Test suites for SHACL constraints - how to integrate with nonconformant exemplars? #9
Description
As we build out SHACL constraints, it is becoming clear that some partner endpoints make choices that probably don't make sense in the long run. For example, the API endpoint:
returns some null values that confuse the toolchain in rdf/js land, and may not even be legal at all in rdf.
This issue is not about deciding what fields should be required or how to handle error cases; this is just about how our test suite can represent that there are test cases we currently EXPECT to fail. I believe it is important to document these real-world cases that our code doesn't immediately handle, and doing so in a test suite makes the code more of a living documentation to me.
In the immediate term, the issue is that our testing runs on .nt files that are already generated, because we have to manually inject a type:Docmap rule into each dataset for validation to work. So there is no practical way to automagically indicate that our pipeline SHOULD have failed upstream.
Ideally we would see the above URL's content represented at each stage of the pipeline, and see where it fails as a WARN. This also applies to the eLife and embo reference examples, which use nonconformant IDs and rely on the baseIRI in our json-ld decoder, which is not sustainable longer term.