Method to disable individual tests

[duncangreene]

Raising issue further to e-mail conversation with @jboddey.

Use case
User should be able to disable individual tests from running, either from test pack (preferred), or from module_config.json files.

I tried two methods as instructed.

1. Deleting the highlighted block below from a test pack...

...in this case the tests were still conducted, despite not being present in the test pack.

2. Adding "enabled": false to the module_config.json file...

...yields the below in the report. I would maybe have expected a result of "Disabled" or for the tests to not appear at all, as opposed to "Feature Not Detected" (the detection didn't happen).

These may warrant being two separate issues, but were discussed in the same thread of a need to disable individual tests. I can create a separate issue if wanted.

On a related point, it is noted that entire modules of tests can be disabled via the UI below, and so I'm unsure how the proposed disabling of individual tests may interact/interfere with the below in the UI. As an observation I would have thought test pack choice would override and even avoid the need to specify these options in the UI to some extent.

[jhughesoti]

When you change a module configuration file, you need to rebuild the modules for it to take effect.

sudo cmd/build

This is not the recommended approach as it will disable these tests for all devices for all future tests and will also get overridden during upgrades.

The recommended approach is to modify the specific device configuration file.

For your example, if you navigate to the local/devices/< device folder >/device_config.json file and modify:

  "test_modules": {
    "protocol": {
      "enabled": true,
      "tests": {
        "protocol.valid_bacnet": {
          "enabled": false
        },
        "protocol.bacnet.version": {
          "enabled": false
        },
        "protocol.valid_modbus": {
          "enabled": false
        }
      }
    },

Which will give the following results:

You should be able to expand this approach to any individual test. You do not need to itemize every test within a module, only the specific ones you want to modify.

[duncangreene]

When you change a module configuration file, you need to rebuild the modules for it to take effect.

sudo cmd/build

This is not the recommended approach as it will disable these tests for all devices for all future tests and will also get overridden during upgrades.

@jhughesbiot thanks for the nod on this. Just confirming here for my own notes that changing module_config.json and then rebuild has the below effect.

[duncangreene]

@jhughesbiot Is there scope to get the functionality as discussed in method 1 in the original post working?

I would anticipate that the expected behaviour be that the exclusion of a test from a test pack should cause it not to run at all, nor appear in the test result list. What are your thoughts?

[jboddey]

@duncangreene We will go ahead with the disabling of tests if it is not included in the test pack.

[duncangreene]

@duncangreene We will go ahead with the disabling of tests if it is not included in the test pack.

@jboddey Sounds good, a couple of supplementary questions if possible.

1. If a test is not included in a test pack, will it not appear in the results at all (preferable), or will it appear in the results as 'disabled'?
2. Will this functionality supersede/negate the need for the options in the UI below?
   

[jboddey]

If a test is not included in the test pack, it will not be included at all. It also negates the need to disable modules in the UI (if none of those module's tests are in the test pack). However, the test module must first be enabled if any of the tests are to run.

[jboddey]

This has now been merged into dev and will be released in v2.1