Protocol Validator Behavior and OEM URI Support:

[plmanik]

Currently, the protocol validator verifies only standard resource URIs and does not validate OEM-specific URIs.

• Do we have any mandatory requirements for validating OEM URIs?
• How are the test cases currently mapped to the Redfish specification?
  Is there a dedicated section or framework that defines this mapping?
  Additionally, is there a way to extract or identify all relevant test cases directly from the specification?
• Is there any plan to extend the validator's functionality to include OEM URI verification?

Thanks,
Mani

[mraineri]

Do we have any mandatory requirements for validating OEM URIs?

No. The reason for this is regardless of the URI being OEM versus standard, the protocol semantics for Redfish are the same.

How are the test cases currently mapped to the Redfish specification?

This has been a largely manual process to sift through the spec for all of the "shall" statements. For each "shall", a test is constructed to verify the service conforms to the behavior dictated by the spec.

Is there any plan to extend the validator's functionality to include OEM URI verification?

No. As indicated earlier, the protocol semantics for Redfish don't change between standard and OEM URIs. Beyond the protocol, the only difference between OEM and standard URIs is the data model. Data model validation is covered by the Redfish Service Validator.

[plmanik]

Thanks @mraineri for details.Should we ensure that OEM URIs also pass all relevant Redfish Protocol Validator test cases?
Or, if certain test cases fail specifically due to OEM URIs, can those failures be considered non-critical and safely discarded?

[mraineri]

The problem is it becomes a scaling issue that increases test time. When creating the tool, we came in with the assumption that protocol handling is largely common for the entire service. For example, we assume that if the HTTP Allow header is behaving correctly for a few standard resources, that gives us high confidence it'll behave the same for all resources.

So, instead of running the same test cases on all URIs, we'd focus on where it made the most sense to perform testing.

[plmanik]

I understand your point regarding testing time. As you mentioned, the HTTP Allow header is a common implementation and behaves similarly for both standard and OEM URIs. Therefore, if it fails for a standard resource, it can be addressed at that level.

That said, based on the schema, are there any specific test cases or a defined set of tests that need to be executed for OEM URIs?

Additionally, I’d like to understand how standard URIs are selected for each test case. For example, in the case of array element removal, this functionality could apply to multiple URIs, yet the test seems to validate against ManagerNetworkProtocol. Is there a selection criteria behind this?

[mraineri]

That said, based on the schema, are there any specific test cases or a defined set of tests that need to be executed for OEM URIs?

Beyond testing the payload contents against the schema, no, I wouldn't expect the need for additional tests for OEM URIs.

Additionally, I’d like to understand how standard URIs are selected for each test case.

The selection was based on the likelihood of a particular URI (or general function) being supported as well as what types of data modifications we can make without disturbing the overall unit under test. For example, it's a fairly reasonable assumption all services support creating/modifying user accounts, so performing POST and PATCH testing on the ManagerAccount resource seemed fairly universal, and adding new user accounts shouldn't affect the operational state of the system. Likewise with ManagerNetworkProtocol, the NTPServers array was also highly likely to be supported and safely PATCH with accidentally bringing down the test unit.