Skip to content

Rationalize and expand testing model #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
RichardBruskiewich merged 7 commits into from
Mar 21, 2024
Merged

Rationalize and expand testing model #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c2a1cc6
Added OneHopTests and StandardsValidation to TestObjectiveEnum; rege…
RichardBruskiewich Mar 4, 2024
d469c1a
Subclass KnowledgeGraphNavigationTestCase from ComplianceTestCase
RichardBruskiewich Mar 5, 2024
180869f
Refactored runner_settings into test_run_settings inside TestEntity; …
RichardBruskiewich Mar 7, 2024
15b6163
Repaired example data to fix unit tests
RichardBruskiewich Mar 7, 2024
4b38ace
ditto
RichardBruskiewich Mar 7, 2024
51964bb
Fixed TestObjectiveEnum values as per feedback from Max
RichardBruskiewich Mar 8, 2024
f1268b3
Merge branch 'main' into add-new-test-object-enum and regenerated all…
RichardBruskiewich Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified project/excel/translator_testing_model.xlsx
View file
Open in desktop
Binary file not shown.
104 changes: 21 additions & 83 deletions project/graphql/translator_testing_model.graphql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ type AcceptanceTestAsset
inV1: Boolean
wellKnown: Boolean
testReference: Uriorcurie
runnerSettings: [String]!
testMetadata: TestMetadata!
id: Uriorcurie!
tags: [String]
testRunnerSettings: [String]
mustPassDate: Date
mustPassEnvironment: TestEnvEnum
scientificQuestion: String
Expand All @@ -39,17 +39,15 @@ type AcceptanceTestCase
id: Uriorcurie!
name: String
description: String
testEnv: TestEnvEnum
testRunnerSettings: [String]
queryType: QueryTypeEnum
preconditions: [Precondition]
trapiTemplate: TrapiTemplateEnum
components: [ComponentEnum]
testCaseObjective: TestObjectiveEnum
testCaseSource: TestSourceEnum
testCasePredicateName: String
testCasePredicateId: String
testCaseInputId: Uriorcurie
testCaseRunnerSettings: [String]!
tags: [String]
testAssets: [AcceptanceTestAsset]!
}
Expand All @@ -60,6 +58,7 @@ type AcceptanceTestSuite
name: String
description: String
tags: [String]
testRunnerSettings: [String]
testMetadata: TestMetadata!
testPersona: TestPersonaEnum
testCases: [TestCase]
Expand All @@ -70,74 +69,13 @@ type BenchmarkTestSuite
{
}

type ComplianceTestCase
{
id: Uriorcurie!
name: String
description: String
testEnv: TestEnvEnum
queryType: QueryTypeEnum
testAssets: [TestAsset]!
preconditions: [Precondition]
trapiTemplate: TrapiTemplateEnum
components: [ComponentEnum]
testCaseObjective: TestObjectiveEnum
testCaseSource: TestSourceEnum
testCasePredicateName: String
testCasePredicateId: String
testCaseInputId: Uriorcurie
testCaseRunnerSettings: [String]!
tags: [String]
trapiVersion: String
biolinkVersion: String
}

type KnowledgeGraphNavigationTestCase
{
id: Uriorcurie!
name: String
description: String
testEnv: TestEnvEnum
queryType: QueryTypeEnum
testAssets: [TestAsset]!
preconditions: [Precondition]
trapiTemplate: TrapiTemplateEnum
components: [ComponentEnum]
testCaseObjective: TestObjectiveEnum
testCaseSource: TestSourceEnum
testCasePredicateName: String
testCasePredicateId: String
testCaseInputId: Uriorcurie
testCaseRunnerSettings: [String]!
tags: [String]
}

type OneHopTestCase
{
id: Uriorcurie!
name: String
description: String
testEnv: TestEnvEnum
queryType: QueryTypeEnum
testAssets: [TestAsset]!
preconditions: [Precondition]
trapiTemplate: TrapiTemplateEnum
components: [ComponentEnum]
testCaseObjective: TestObjectiveEnum
testCaseSource: TestSourceEnum
testCasePredicateName: String
testCasePredicateId: String
testCaseInputId: Uriorcurie
testCaseRunnerSettings: [String]!
tags: [String]
}

type OneHopTestSuite
{
id: Uriorcurie!
name: String
description: String
tags: [String]
testRunnerSettings: [String]
testMetadata: TestMetadata!
testPersona: TestPersonaEnum
testCases: [TestCase]
Expand All @@ -150,6 +88,7 @@ type Precondition
name: String
description: String
tags: [String]
testRunnerSettings: [String]
}

type Qualifier
Expand All @@ -163,18 +102,16 @@ type QuantitativeTestCase
id: Uriorcurie!
name: String
description: String
testEnv: TestEnvEnum
testRunnerSettings: [String]
queryType: QueryTypeEnum
testAssets: [TestAsset]!
preconditions: [Precondition]
trapiTemplate: TrapiTemplateEnum
components: [ComponentEnum]
testCaseObjective: TestObjectiveEnum
testCaseSource: TestSourceEnum
testCasePredicateName: String
testCasePredicateId: String
testCaseInputId: Uriorcurie
testCaseRunnerSettings: [String]!
tags: [String]
}

Expand All @@ -184,6 +121,7 @@ type StandardsComplianceTestSuite
name: String
description: String
tags: [String]
testRunnerSettings: [String]
testMetadata: TestMetadata!
testPersona: TestPersonaEnum
testCases: [TestCase]
Expand All @@ -210,29 +148,27 @@ type TestAsset
inV1: Boolean
wellKnown: Boolean
testReference: Uriorcurie
runnerSettings: [String]!
testMetadata: TestMetadata!
id: Uriorcurie!
tags: [String]
testRunnerSettings: [String]
}

type TestCase
{
id: Uriorcurie!
name: String
description: String
testEnv: TestEnvEnum
testRunnerSettings: [String]
queryType: QueryTypeEnum
testAssets: [TestAsset]!
preconditions: [Precondition]
trapiTemplate: TrapiTemplateEnum
components: [ComponentEnum]
testCaseObjective: TestObjectiveEnum
testCaseSource: TestSourceEnum
testCasePredicateName: String
testCasePredicateId: String
testCaseInputId: Uriorcurie
testCaseRunnerSettings: [String]!
tags: [String]
}

Expand All @@ -242,6 +178,7 @@ type TestCaseResult
name: String
description: String
tags: [String]
testRunnerSettings: [String]
testSuiteId: Uriorcurie
testCase: TestCase
testCaseResult: TestCaseResultEnum
Expand All @@ -267,10 +204,10 @@ type TestEdgeData
inV1: Boolean
wellKnown: Boolean
testReference: Uriorcurie
runnerSettings: [String]!
testMetadata: TestMetadata!
id: Uriorcurie!
tags: [String]
testRunnerSettings: [String]
}

interface TestEntity
Expand All @@ -279,6 +216,7 @@ interface TestEntity
name: String
description: String
tags: [String]
testRunnerSettings: [String]
}

type TestEntityParameter
Expand All @@ -293,6 +231,7 @@ type TestMetadata
name: String
description: String
tags: [String]
testRunnerSettings: [String]
testSource: TestSourceEnum
testReference: Uriorcurie
testObjective: TestObjectiveEnum
Expand All @@ -305,6 +244,7 @@ type TestOutput
name: String
description: String
tags: [String]
testRunnerSettings: [String]
testCaseId: String
pks: [TestResultPKSet]
}
Expand All @@ -315,6 +255,7 @@ type TestResultPKSet
name: String
description: String
tags: [String]
testRunnerSettings: [String]
parentPk: String
mergedPk: String
aragorn: String
Expand All @@ -324,22 +265,17 @@ type TestResultPKSet
improving: String
}

type TestRunnerConfiguration
{
id: Uriorcurie!
name: String
description: String
testRunParameters: [TestEntityParameter]
tags: [String]
}

type TestRunSession
{
id: Uriorcurie!
name: String
description: String
tags: [String]
testRunnerSettings: [String]
components: [ComponentEnum]
Copy link
Collaborator

@maximusunc maximusunc Mar 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think components should be all the way up at the TestRunSession. I think it needs to stay down at TestCase. We could conceivably want to run tests against different components in the same Test Run (Session)

Copy link
Collaborator

@maximusunc maximusunc Mar 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A use case could be TRAPI validation. We would hit all levels, KPs, ARAs, ARS, make sure that none of them are breaking the standard.

Copy link
Collaborator Author

@RichardBruskiewich RichardBruskiewich Mar 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @maximusunc, components are a List[str] of infores identifiers of target components. The implicit assumption that was made (perhaps incorrect, I guess) was that all of the test_entities (test data - TestAssets/Cases/Suites) of a given TestRunSession will be run against all of the specified components. If a TestRunSession is a heterogeneous mix of testing targets for which the data is not mapped one-to-one, then of course, this assumption is violated. I guess it just depends how one decides to cut the cake.

Copy link
Collaborator Author

@RichardBruskiewich RichardBruskiewich Mar 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As promised @maximusunc (cc: @sierra-moxon), I will try to express my sense of the testing landscape. I suspect there is no "right answer" to this... that your current implementation is equally tenable... Just trying to "think out loud" here.

Anyhow, I would see the planning of the test process as an ordered series of test configuration decisions. Here's an order I have sort of followed in earlier testing work:

  1. which component do you wish to test (UI, ARS, ARA, KP, ??)
  2. within which environment are you running the component to be tested?
  3. which category of test ("TestRunner") do you which to run against the component
  4. within what global context do you wish to test (e.g. TRAPI version, Biolink version, ???)
  5. what test data (TestAssets/Cases/Suites) do you wish to use as input?
  6. when do you want to run the test?
  7. how do you wish to assess the test results?

My general sense is that the answers to all of the above questions could be aggregated within a top level TestRunSession declaration.

What I understand, however, is that from your perspective, a TestRunSession is mainly a scheduled (timestamped) container for any ad hoc set of test data (TestAssets/Cases/Suites) to be processed, and that the test data itself will specify where it is consumed (i.e. which component) thus different members of the test_entities array of test data could be routed to different target components, albeit, all processed by the same specified TestRunner.

Nothing wrong with that, I guess... one is simply embedding the answer to question 1 above, inside the test data returned in response to question 5. On that note, would the test_env also belong there as well, or does that parameter tend to be more global to a given test run?

Just an alternate way of organizing the testing... "cutting the cake..." LOL. In terms of granularity, I might tend to suggest that the granularity for this is the TestSuite - in that components might "own" a TestSuite, right (even if in some cases, there is duplication of test data in TestSuite data across components).

I guess that is actually what the legacy SRI_Testing test harness did to some extent, since the test data was resolved from KP's using their Translator SmartAPI 'info.x-trapi.test_data_location' property. that said, the top level test run specification decision of SRI_Testing was still to specify which component(s) were to be tested.... OK... just a different way of cutting the cake.

In the context of the above list of questions, I should probably ask your opinion about question 4. In particular, I suppose that TestRunner specific parameters like TRAPI version and Biolink version (e.g. for a standards validation test runner) could be pushed down into the TestSuite level, if it is the case that specific TestSuite instances are run as checks on standards. Alternately, if one expects to generally run the standards validation as an overlay test using many of the test data sets used in other tests, then where is it best specified? The TestRunSession seemed to be the best location but if so, should these Translator level declarations be specified at the TestHarness level CLI (perhaps defaulting to the 'current' consortium releases, if not specified)?

Come to think of it, the TestRunner also currently seems to be specified at the TestCase level. This seems very granular. Is there a possible rationale for moving this up to the TestSuite level, or even, the TestRunSession level (actually, there is already a test_runner_name slot in TestRunSession which reflects this idea...). More alternative ways of cutting the cake?

Any further thoughts @maximusunc and @sierra-moxon on any or all of the above?

testEnv: TestEnvEnum
testRunnerName: String
testRunParameters: [TestEntityParameter]
testEntities: [TestEntity]
testCaseResults: [TestCaseResult]
timestamp: Datetime
Expand All @@ -351,6 +287,7 @@ type TestSuite
name: String
description: String
tags: [String]
testRunnerSettings: [String]
testMetadata: TestMetadata!
testPersona: TestPersonaEnum
testCases: [TestCase]
Expand All @@ -363,6 +300,7 @@ type TestSuiteSpecification
name: String
description: String
tags: [String]
testRunnerSettings: [String]
testDataFileLocator: Uriorcurie
testDataFileFormat: FileFormatEnum
}
Expand Down
2 changes: 1 addition & 1 deletion project/jsonld/translator_testing_model.context.jsonld
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"comments": {
"description": "Auto generated by LinkML jsonld context generator",
"generation_date": "2024-03-18T12:26:22",
"generation_date": "2024-03-21T14:12:54",
"source": "translator_testing_model.yaml"
},
"@context": {
Expand Down
Loading