Test Cleanup

[DrLynch]

This is primarily a placeholder issue for discussion. At present the Testing code is wholly unorganized. To the extent it exists most of it has been placed inside the AWE_Workbench repository. The code will need to be sorted, assigned to specific subpackages, and extended to cover areas not otherwise evaluated. This will also require developing some bespoke tests for the AWE_LanguageTool which must be wholly unique.

This is a local placeholder for the ETS level issue: ETS-Next-Gen#2

In addressing this you will need to:

1. Survey the tests to see where they should be located and make a catalog of the items to share.
2. Move tests to the appropriate location.
3. Develop novel tests for the uncovered items and set up a single suite.

[duckduckdoof]

First goal is to get tests in Workbench/tests working. As of right now, pytest does not properly load extensions specified by components in custom spacy pipeline:

• lexical features
• syntax discourse features
• viewpoint features
• lexical clusters
• content segmentation

I have verified that pytest (python=3.10.13, pytest-8.0.0, pluggy-1.4.0) does not run these tests properly. I have also been asked to verify if the tests run in a non-pytest setting (i.e. - run the holmes wrapper spacy pipeline interactively): this appears to not work either:

import holmes_extractor.manager as holmes
import unittest
from awe_components.components.utility_functions import print_parse_tree
from awe_workbench.pipeline import pipeline_def

holmes_manager = holmes.Manager(
    'en_core_web_lg', perform_coreference_resolution=False, number_of_workers=2, extra_components=pipeline_def)

holmes_manager.parse_and_register_document(...)

extended_test = doc._.extended_core_sentences
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/duck/anaconda3/envs/ets_test/lib/python3.10/site-packages/spacy/tokens/underscore.py", line 47, in __getattr__
    raise AttributeError(Errors.E046.format(name=name))
AttributeError: [E046] Can't retrieve unregistered extension attribute 'extended_core_sentences'. Did you forget to call the `set_extension` method?

I also note that, with tests which do not have this error, there is a mismatch in the expected dictionary and the returned dictionary from holmes manager:

>>> len(doc._.vwp_perspective_spans)
705
>>> len(persp_actual)
4

[duckduckdoof]

I looked deeper into the called components in the holmes spacy pipeline, to see if I could find the extensions referred to in the tests in Workbench/tests. As of now, there are three components which have custom added extensions:

• syntaxDiscourseFeats.py
• lexicalFeatures.py
• viewpointFeatures.py

After searching the extensions name by name, I compared them to extensions called in the workbench tests. Of these components, we are missing custom extensions (ex: extended_core_sentences, as mentioned in #1 (comment)).

However, there are components called in the tests which are in fact created here; I'm not sure at this point where the other extensions could have disappeared from.

[PDDeane]

It looks like I renamed extended_core_sentences ... see here: (https://github.com/ETS-Next-Gen/AWE_Components/blob/main/awe_components/components/contentSegmentation.py,) line 26 ... I remember doing a restructuring on vwp_perspective_spans that I may not have updated the test for.

For the other attribute issues, there's a check you need to run.

This script is designed to pretty much run ALL of the extended attributes I create to produce a document summary:

https://github.com/ETS-Next-Gen/AWE_Workbench/blob/main/examples/batch_summary.py

It gives you an example of how to run all of the code OUTSIDE the testing environment. If you can run that without error, the attributes exist under normal circumstances, and something weird is happening in the testing environment.

[duckduckdoof]

@PDDeane I ran batch_summary.py, but got an error; appears the dataframe in question is empty:

processing files in ./
Running LanguageTool
Running spellcorrect
Running parser
Empty DataFrame
Columns: []
Index: []
Traceback (most recent call last):
  File "/mnt/c/Users/duckd/Documents/dev/ets_dev/AWE_Workbench/examples/batch_summary.py", line 117, in <module>
    syntactic_profile.set_index('ID', inplace=True)
  File "/home/duck/anaconda3/envs/ets_test_2/lib/python3.10/site-packages/pandas/core/frame.py", line 6106, in set_index
    raise KeyError(f"None of {missing} are in the columns")
KeyError: "None of ['ID'] are in the columns"

Both servers were running before I ran this one. I also went ahead and printed the contents of "syntactic_profile".

[PDDeane]

Clearly something is broken here. But note that I ran this batch script last July and did not encounter this problem.

I also ran this script without problems:

https://github.com/ETS-Next-Gen/AWE_Workbench/blob/main/examples/standalone_parse.py

If this doesn't work, something is broken that was not broken in the main repository in July 2023 ...

[PDDeane]

I will try this weekend to pull down the main branch from ETS Next Gen and see if the standalone script runs. Clearly something is broken. The question is where the break happened and what exactly broke ... :(

[duckduckdoof]

Is there any way I can see your environment that you're running? That would help me tons, to see if I can compare it with mine.

[PDDeane]

Remind me what the command is to display the environment? The machine I normally use is at home, so I'll need to run that command tonight to see what happens.

…

On Wed, Feb 7, 2024 at 4:50 PM Caleb Scott ***@***.***> wrote: Is there any way I can see your environment that you're running? That would help me tons, to see if I can compare it with mine. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHGKSEZ2U5C5SX5EEQD4GBDYSPZJLAVCNFSM6AAAAABCM4SBQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSHE4TMMRQHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[PDDeane]

(and of course I need to verify that the bare parse script runs before that would do you much good) On Wed, Feb 7, 2024 at 4:55 PM Paul Deane ***@***.***> wrote:

…

Remind me what the command is to display the environment? The machine I normally use is at home, so I'll need to run that command tonight to see what happens. On Wed, Feb 7, 2024 at 4:50 PM Caleb Scott ***@***.***> wrote: > Is there any way I can see your environment that you're running? That > would help me tons, to see if I can compare it with mine. > > — > Reply to this email directly, view it on GitHub > <#1 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHGKSEZ2U5C5SX5EEQD4GBDYSPZJLAVCNFSM6AAAAABCM4SBQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSHE4TMMRQHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[duckduckdoof]

With pip, you can run "pip freeze" and pipe that into a file; I'd also need the python version too.

Alternatively, you can install and run "pipdeptree" to see a tree of dependencies.

[duckduckdoof]

After looking at all the "missing" parts of the test, here are the attributes claimed to be missing:

• core_sentences
• extended_core_sentences
• propn_allocentric
• propn_egocentric
• vwp_direct_speech_spans

I found this code blurb from contentSegmentation.py (starting at line 28):

doc._.main_ideas_ = core_sentences
doc._.supporting_ideas_ = extended_core_sentences
doc._.supporting_details_ = elaboration_sentences
doc._.prompt_related_ = pclusters
doc._.prompt_language_ = plemmas

For the first two, maybe the tests should be calling "main_ideas" and "supporting_ideas", respectfully? Not sure about the last three though. These errors are suggestive that maybe this isn't a spacy version error.

[PDDeane]

The first four reflect changes to viewpointFeatures.py. Looks like I renamed core_sentences and extended_core_sentences. The propn_ attributes were probably removed because when I added a new function that allows the same indicator to be summarized in multiple ways, they became redundant. vwp_direct_speech_spans is set in viewpointFeatures.py, It looks like I set up the function that defines it so that it only get called if you first call for vwp_attribution, vwp_cite, vwp_source, nominal_references, tense_changes, concrete_details. I recall wanting to call this function only if I absolutely had to, because it's very expensive. So I ran it when one of the attributes dependent on it was called, and not otherwise. So you should rename the attributes in the core_sentences and extended_core_sentences tests, delete the two propn tests, and (probably best choice here), call nominal_references before you test vwp_direct_speech_spans. Paul

…

On Wed, Feb 7, 2024 at 4:59 PM Caleb Scott ***@***.***> wrote: After looking at all the "missing" parts of the test, here are the attributes claimed to be missing: - core_sentences - extended_core_sentences - propn_allocentric - propn_egocentric - vwp_direct_speech_spans I found this code blurb from contentSegmentation.py: doc._.main_ideas_ = core_sentences doc._.supporting_ideas_ = extended_core_sentences doc._.supporting_details_ = elaboration_sentences doc._.prompt_related_ = pclusters doc._.prompt_language_ = plemmas For the first two, maybe the tests should be calling "main_ideas" and "supporting_ideas", respectfully? Not sure about the last three though. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHGKSE6UV3IKQDKULIW2KODYSP2LHAVCNFSM6AAAAABCM4SBQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZTGAYDOMRTGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[duckduckdoof]

Did you manage to get a description of your python environment dependencies?