Update schema handling and corrected various errors

.gitignore

@@ -62,6 +62,7 @@ var/
 .installed.cfg
 *.egg
 tests/scratch
+tests/test_output
 
 # Installer logs
 pip-log.txt

README.md

@@ -1,102 +1,103 @@
-[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8056010.svg)](https://doi.org/10.5281/zenodo.8056010)
-[![Maintainability](https://api.codeclimate.com/v1/badges/11bf2329590e7b0164ba/maintainability)](https://codeclimate.com/github/hed-standard/hed-python/maintainability)
-[![Test Coverage](https://api.codeclimate.com/v1/badges/11bf2329590e7b0164ba/test_coverage)](https://codeclimate.com/github/hed-standard/hed-python/test_coverage)
-![PyPI - Status](https://img.shields.io/pypi/v/hedtools)
-[![Documentation Status](https://readthedocs.org/projects/hed-python/badge/?version=latest)](https://hed-python.readthedocs.io/en/latest/?badge=latest)
-
-# HEDTools - Python
-HED (Hierarchical Event Descriptors) is a framework for systematically describing
-both laboratory and real-world events as well as other experimental metadata.
-HED tags are comma-separated path strings.
-HED, itself, is platform-independent, extendable, and data-neutral. 
-
-This repository contains the underlying python tools that support validation,
-summarization, and analysis of datasets using HED tags.
-
-Most people will simply annotate their events by creating a spreadsheet
-or a BIDS JSON sidecar that associates HED tags with event codes or the events themselves.
-If you have such a spreadsheet or a JSON, 
-you can use the HED Online Validator currently available at 
-[https://hedtools.org](https://hedtools.org) to validate or transform
-your files without downloading any tools. 
-
-A version of the online tools corresponding to the `develop` branch can be found at:
-[https://hedtools.org/hed_dev](https://hedtools.org/hed_dev).
-
-### Installation
-Use `pip` to install `hedtools` from PyPI:
-
-   ```
-       pip install hedtools
-   ```
-
-To install directly from the 
-[GitHub](https://github.com/hed-standard/hed-python) repository `master` branch:
-
-   ```
-       pip install git+https://github.com/hed-standard/hed-python/@master
-   ```
-
-The HEDTools in this repository require Python 3.8 or greater.
-
-### Relationship to other repositories
-
-The `hed-python` repository contains the Python infrastructure for validating
-and analyzing HED. This repository has several companion repositories:
-- [`hed-web`](https://github.com/hed-standard/hed-web) contains the web interface
-for HED as well as a deployable docker module that supports web services for HED.  
-- [`hed-examples`](https://github.com/hed-standard/hed-examples) contains examples of
-using HED in Python and MATLAB. This repository also houses the HED resources.
-See [https://www.hed-resources.org](https://www.hed-resources.org) for access to these.
-- [`hed-specification`](https://github.com/hed-standard/hed-specification) contains
-the HED specification documents. The `hed-python` validator is keyed to error codes
-in this document.
-- [`hed-schemas`](https://github.com/hed-standard/hed-schemas) contains
-the official HED schemas. The tools access this repository to retrieve and cache schema versions
-during execution. Starting with `hedtools 0.2.0` local copies of the most recent schema versions
-are stored within the code modules for easy access.
-
-#### Develop versus master versus stable branches
-
-The `hed-python` repository
-
-| Branch |  Meaning | Synchronized with |
-| ------ | -------- | ------------------ |
-| stable | Officially released on PyPI as a tagged version. | `stable@hed-web`<br/>`stable@hed-specification`<br/>`stable@hed-examples` |
-| latest | Most recent usable version. | `latest@hed-web`<br/>`latest@hed-specification`<br/>`latest@hed-examples` |
-| develop | Experimental and evolving. | `develop@hed-web`<br/>`develop@hed-specification`<br/>`develop@hed-examples` |
-
-As features are integrated, they first appear in the `develop` branches of the
-repositories.
-The `develop` branches of the repositories will be kept in sync as much as possible
-If an interface change in `hed-python` triggers a change in `hed-web` or `hed-examples`,
-every effort will be made to get the three types of branches
-(`develop`, `latest`, `stable`) of the respective repositories in
-sync.
-
-API documentation is generated on ReadTheDocs when a new version is
-pushed on any of the three branches. For example, the API documentation for the
-`latest` branch can be found on [hed-python.readthedocs.io/en/latest/](hed-python.readthedocs.io/en/latest/).
-
-#### To contribute
-
-Contributions are welcome.
-Please use the [Github issues](https://github.com/hed-standard/hed-python/issues)
-for suggestions or bug reports.
-The [Github pull request](https://github.com/hed-standard/hed-python/pulls)
-may also be used for contributions.
-These PRs should be made to the `develop` branch, not the `master` branch.
-
-#### Local Settings Storage
-Cached Schemas by default are stored in "home/.hedtools/" 
-Location of "home" directory varies by OS.
-
-Use `hed.schema.set_cache_directory` to change the location.
-The HED cache can be shared across processes.
-
-Starting with `hedtools 0.2.0` local copies of the most recent schema versions
-are stored within the code modules for easy access.  
-
-### Other links of interest
-
-Code climate reports: [https://codeclimate.com/github/hed-standard/hed-python](https://codeclimate.com/github/hed-standard/hed-python).
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8056010.svg)](https://doi.org/10.5281/zenodo.8056010)
+[![Maintainability](https://api.codeclimate.com/v1/badges/11bf2329590e7b0164ba/maintainability)](https://codeclimate.com/github/hed-standard/hed-python/maintainability)
+[![Test Coverage](https://api.codeclimate.com/v1/badges/11bf2329590e7b0164ba/test_coverage)](https://codeclimate.com/github/hed-standard/hed-python/test_coverage)
+![Python3](https://img.shields.io/badge/python->=3.8-yellow.svg)
+![PyPI - Status](https://img.shields.io/pypi/v/hedtools)
+[![Documentation Status](https://readthedocs.org/projects/hed-python/badge/?version=latest)](https://hed-python.readthedocs.io/en/latest/?badge=latest)
+
+# HEDTools - Python
+HED (Hierarchical Event Descriptors) is a framework for systematically describing
+both laboratory and real-world events as well as other experimental metadata.
+HED tags are comma-separated path strings.
+HED, itself, is platform-independent, extendable, and data-neutral. 
+
+This repository contains the underlying python tools that support validation,
+summarization, and analysis of datasets using HED tags.
+
+Most people will simply annotate their events by creating a spreadsheet
+or a BIDS JSON sidecar that associates HED tags with event codes or the events themselves.
+If you have such a spreadsheet or a JSON, 
+you can use the HED Online Validator currently available at 
+[https://hedtools.org](https://hedtools.org) to validate or transform
+your files without downloading any tools. 
+
+A version of the online tools corresponding to the `develop` branch can be found at:
+[https://hedtools.org/hed_dev](https://hedtools.org/hed_dev).
+
+### Installation
+Use `pip` to install `hedtools` from PyPI:
+
+   ```
+       pip install hedtools
+   ```
+
+To install directly from the 
+[GitHub](https://github.com/hed-standard/hed-python) repository `master` branch:
+
+   ```
+       pip install git+https://github.com/hed-standard/hed-python/@master
+   ```
+
+The HEDTools in this repository require Python 3.8 or greater.
+
+### Relationship to other repositories
+
+The `hed-python` repository contains the Python infrastructure for validating
+and analyzing HED. This repository has several companion repositories:
+- [`hed-web`](https://github.com/hed-standard/hed-web) contains the web interface
+for HED as well as a deployable docker module that supports web services for HED.  
+- [`hed-examples`](https://github.com/hed-standard/hed-examples) contains examples of
+using HED in Python and MATLAB. This repository also houses the HED resources.
+See [https://www.hed-resources.org](https://www.hed-resources.org) for access to these.
+- [`hed-specification`](https://github.com/hed-standard/hed-specification) contains
+the HED specification documents. The `hed-python` validator is keyed to error codes
+in this document.
+- [`hed-schemas`](https://github.com/hed-standard/hed-schemas) contains
+the official HED schemas. The tools access this repository to retrieve and cache schema versions
+during execution. Starting with `hedtools 0.2.0` local copies of the most recent schema versions
+are stored within the code modules for easy access.
+
+#### Develop versus master versus stable branches
+
+The `hed-python` repository
+
+| Branch |  Meaning | Synchronized with |
+| ------ | -------- | ------------------ |
+| stable | Officially released on PyPI as a tagged version. | `stable@hed-web`<br/>`stable@hed-specification`<br/>`stable@hed-examples` |
+| latest | Most recent usable version. | `latest@hed-web`<br/>`latest@hed-specification`<br/>`latest@hed-examples` |
+| develop | Experimental and evolving. | `develop@hed-web`<br/>`develop@hed-specification`<br/>`develop@hed-examples` |
+
+As features are integrated, they first appear in the `develop` branches of the
+repositories.
+The `develop` branches of the repositories will be kept in sync as much as possible
+If an interface change in `hed-python` triggers a change in `hed-web` or `hed-examples`,
+every effort will be made to get the three types of branches
+(`develop`, `latest`, `stable`) of the respective repositories in
+sync.
+
+API documentation is generated on ReadTheDocs when a new version is
+pushed on any of the three branches. For example, the API documentation for the
+`latest` branch can be found on [hed-python.readthedocs.io/en/latest/](hed-python.readthedocs.io/en/latest/).
+
+#### To contribute
+
+Contributions are welcome.
+Please use the [Github issues](https://github.com/hed-standard/hed-python/issues)
+for suggestions or bug reports.
+The [Github pull request](https://github.com/hed-standard/hed-python/pulls)
+may also be used for contributions.
+These PRs should be made to the `develop` branch, not the `master` branch.
+
+#### Local Settings Storage
+Cached Schemas by default are stored in "home/.hedtools/" 
+Location of "home" directory varies by OS.
+
+Use `hed.schema.set_cache_directory` to change the location.
+The HED cache can be shared across processes.
+
+Starting with `hedtools 0.2.0` local copies of the most recent schema versions
+are stored within the code modules for easy access.  
+
+### Other links of interest
+
+Code climate reports: [https://codeclimate.com/github/hed-standard/hed-python](https://codeclimate.com/github/hed-standard/hed-python).

docs/requirements.txt

@@ -9,4 +9,4 @@ semantic-version>=2.10.0
 myst-parser>=1.0.0
 Sphinx>=5.2.2
 sphinx_rtd_theme>=1.0.0
-wordcloud==1.9.3
+wordcloud==1.9.4

hed/errors/error_messages.py

@@ -52,15 +52,15 @@ def val_error_invalid_tag_character(tag, problem_tag):
     return f"Invalid character '{problem_tag}' in tag '{tag}'"
 
 
-@hed_tag_error(ValidationErrors.INVALID_VALUE_CLASS_CHARACTER, has_sub_tag=True,
+@hed_tag_error(ValidationErrors.INVALID_VALUE_CLASS_CHARACTER, has_sub_tag=False,
                actual_code=ValidationErrors.CHARACTER_INVALID)
-def val_error_INVALID_VALUE_CLASS_CHARACTER(tag, problem_tag, value_class):
+def val_error_val_error_invalid_value_class_character(tag, problem_tag, value_class):
     return f"Invalid character '{problem_tag}' in tag '{tag}' for value class '{value_class}'"
 
 
 @hed_tag_error(ValidationErrors.INVALID_VALUE_CLASS_VALUE, has_sub_tag=True,
                actual_code=ValidationErrors.VALUE_INVALID)
-def val_error_INVALID_VALUE_CLASS_VALUE(tag, problem_tag, value_class):
+def val_error_invalid_value_class_value(tag, problem_tag, value_class):
     return f"'{tag}' has an invalid value portion for value class '{value_class}'"
 
 
@@ -70,15 +70,21 @@ def val_error_tildes_not_supported(source_string, char_index):
     return f"Tildes not supported.  Replace (a ~ b ~ c) with (a, (b, c)).  '{character}' at index {char_index}'"
 
 
-@hed_tag_error(ValidationErrors.CURLY_BRACE_UNSUPPORTED_HERE, has_sub_tag=True,
+@hed_tag_error(ValidationErrors.HED_PLACEHOLDER_OUT_OF_CONTEXT, has_sub_tag=False,
+               actual_code=ValidationErrors.PLACEHOLDER_INVALID)
+def val_error_hed_placeholder_out_of_context(tag):
+    return f"'{tag}' has a '#' placeholder where it is not allowed or where it should have been replaced with a value."
+
+
+@hed_tag_error(ValidationErrors.CURLY_BRACE_UNSUPPORTED_HERE, has_sub_tag=False,
                actual_code=SidecarErrors.SIDECAR_BRACES_INVALID)
-def val_error_CURLY_BRACE_UNSUPPORTED_HERE(tag, problem_tag):
+def val_error_curly_brace_unsupported_here(tag, problem_tag):
     return (f"Curly braces are only permitted in sidecars, fully wrapping text in place of a tag.  "
             f"Invalid character '{problem_tag}' in tag '{tag}'")
 
 
 @hed_error(ValidationErrors.ONSETS_UNORDERED, default_severity=ErrorSeverity.WARNING)
-def val_error_ONSETS_UNORDERED():
+def val_error_onsets_unordered():
     return "Onsets need to be temporally increasing and defined for many downstream tools to work."
 
 
@@ -97,6 +103,17 @@ def val_error_duplicate_group(group):
     return f'Repeated group - "{group}"'
 
 
+@hed_error(ValidationErrors.HED_RESERVED_TAG_REPEATED, actual_code=ValidationErrors.TAG_GROUP_ERROR)
+def val_error_duplicate_reserved_tag(tag, group):
+    return f'Repeated reserved tag "{tag}" or multiple reserved tags in group "{group}"'
+
+
+@hed_error(ValidationErrors.HED_RESERVED_TAG_GROUP_ERROR, actual_code=ValidationErrors.TAG_GROUP_ERROR)
+def val_error_group_for_reserved_tag(group, group_count):
+    return (f'The number of non-def-expand subgroups for group "{group}" is {group_count}, "'
+            f'which does not meet reserved tag requirements."')
+
+
 @hed_error(ValidationErrors.PARENTHESES_MISMATCH)
 def val_error_parentheses(opening_parentheses_count, closing_parentheses_count):
     return f'Number of opening and closing parentheses are unequal. '\
@@ -194,8 +211,14 @@ def val_error_extra_slashes_spaces(tag, problem_tag):
 
 
 @hed_error(ValidationErrors.SIDECAR_KEY_MISSING, default_severity=ErrorSeverity.WARNING)
-def val_error_sidecar_key_missing(invalid_key, category_keys):
-    return f"Category key '{invalid_key}' does not exist in column.  Valid keys are: {category_keys}"
+def val_error_sidecar_key_missing(invalid_keys, category_keys):
+    return f"Category keys '{invalid_keys}' do not exist in column.  Valid keys are: {category_keys}"
+
+
+@hed_error(ValidationErrors.TSV_COLUMN_MISSING, actual_code=ValidationErrors.SIDECAR_KEY_MISSING,
+           default_severity=ErrorSeverity.WARNING)
+def val_error_tsv_column_missing(invalid_keys):
+    return f"{invalid_keys} used as column references in sidecar but are not columns in the tabular file"
 
 
 @hed_tag_error(ValidationErrors.HED_DEF_EXPAND_INVALID, actual_code=ValidationErrors.DEF_EXPAND_INVALID)
@@ -236,12 +259,12 @@ def val_error_def_expand_value_extra(tag):
 
 @hed_tag_error(ValidationErrors.HED_TOP_LEVEL_TAG, actual_code=ValidationErrors.TAG_GROUP_ERROR)
 def val_error_top_level_tag(tag):
-    return f"A tag that must be in a top level group was found in another location.  {str(tag)}"
+    return f'Tag "{tag}" must be in a top level group but was found in another location.'
 
 
 @hed_tag_error(ValidationErrors.HED_TAG_GROUP_TAG, actual_code=ValidationErrors.TAG_GROUP_ERROR)
 def val_error_tag_group_tag(tag):
-    return f"A tag that must be in a group was found in another location.  {str(tag)}"
+    return f'Tag "{tag}" that must be in a group was found in another location.'
 
 
 @hed_tag_error(ValidationErrors.HED_MULTIPLE_TOP_TAGS, actual_code=ValidationErrors.TAG_GROUP_ERROR)
@@ -251,6 +274,11 @@ def val_error_top_level_tags(tag, multiple_tags):
            f"Remainder:{str(tags_as_string)}"
 
 
+@hed_tag_error(ValidationErrors.HED_TAGS_NOT_ALLOWED, actual_code=ValidationErrors.TAG_GROUP_ERROR)
+def val_error_tags_in_group_with_reserved(tag, group):
+    return f'Tag "{tag}" is not allowed with the other tag(s) or Def-expand sub-group in group "{group}"'
+
+
 @hed_error(ValidationErrors.REQUIRED_TAG_MISSING)
 def val_warning_required_prefix_missing(tag_namespace):
     return f"Tag with namespace '{tag_namespace}' is required"
@@ -261,11 +289,6 @@ def val_warning_capitalization(tag):
     return f"First word not capitalized or camel case - '{tag}'"
 
 
-@hed_tag_error(ValidationErrors.UNITS_MISSING, default_severity=ErrorSeverity.WARNING)
-def val_warning_default_units_used(tag, default_unit):
-    return f"Tag '{tag}' expects units, but no units were given."
-
-
 @hed_error(SidecarErrors.BLANK_HED_STRING)
 def sidecar_error_blank_hed_string():
     return "No HED string found for Value or Category column."
@@ -293,15 +316,10 @@ def sidecar_error_unknown_column(column_name):
 
 
 @hed_error(SidecarErrors.SIDECAR_HED_USED, actual_code=ValidationErrors.SIDECAR_INVALID)
-def SIDECAR_HED_USED():
+def sidecar_hed_used():
     return "'HED' is a reserved name and cannot be used as a sidecar except in expected places."
 
 
-@hed_error(SidecarErrors.SIDECAR_HED_USED_COLUMN, actual_code=ValidationErrors.SIDECAR_INVALID)
-def SIDECAR_HED_USED_COLUMN():
-    return "'HED' is a reserved name and cannot be used as a sidecar column name"
-
-
 @hed_error(SidecarErrors.SIDECAR_NA_USED, actual_code=ValidationErrors.SIDECAR_INVALID)
 def sidecar_na_used(column_name):
     return f"Invalid category key 'n/a' found in column {column_name}."
@@ -384,7 +402,7 @@ def onset_error_inset_before_onset(tag):
 
 @hed_tag_error(TemporalErrors.ONSET_NO_DEF_TAG_FOUND, actual_code=ValidationErrors.TEMPORAL_TAG_ERROR)
 def onset_no_def_found(tag):
-    return f"'{tag}' tag has no def or def-expand tag in string."
+    return f"'{tag}' tag has no def tag or def-expand group or too many when 1 is required in string."
 
 
 @hed_tag_error(TemporalErrors.ONSET_TOO_MANY_DEFS, actual_code=ValidationErrors.TEMPORAL_TAG_ERROR)
@@ -401,7 +419,7 @@ def onset_too_many_groups(tag, tag_list):
 
 
 @hed_tag_error(TemporalErrors.DURATION_WRONG_NUMBER_GROUPS, actual_code=ValidationErrors.TEMPORAL_TAG_ERROR)
-def onset_DURATION_WRONG_NUMBER_GROUPS(tag, tag_list):
+def onset_duration_wrong_number_groups(tag, tag_list):
     tag_list_strings = [str(a_tag) for a_tag in tag_list]
     return f"A duration and/or delay tag '{tag}'should have exactly one child group." \
            f"Found {len(tag_list_strings)}: {tag_list_strings}"
@@ -421,7 +439,7 @@ def onset_wrong_placeholder(tag, has_placeholder):
 
 
 @hed_tag_error(TemporalErrors.DURATION_HAS_OTHER_TAGS, actual_code=ValidationErrors.TEMPORAL_TAG_ERROR)
-def onset_DURATION_HAS_OTHER_TAGS(tag):
+def onset_duration_has_other_tags(tag):
     return f"Tag '{tag}' should not be grouped with Duration or Delay.  Context tags should be in a sub-group."