Sporadic results in Services tests

[duncangreene]

Describe the bug
In 2.2 I'm still seeing some sporadic results in Services tests. This issue was initially uncovered in #885 and seems to occur on more resource constrained devices, which are prevalent (and constrained by design) in the IoT/smart building world.

Test A

Test B

Test C

Test D

Expected behavior
The same device under the same conditions should yield the same results every time.

Error logs
testA.zip
testB.zip (Ignore the TLS server test errors, I suspect the device lost power during the TLS module tests)
testC.zip (Ignore the TLS module timeout, this is due to #1178)
testD.zip

Environment (please provide the following information about your setup):

• Version 2.2

Additional context
There was discussion on #885 previously about utilising nmap's inbuilt rate limiting features to mitigate the false negatives/false positives that can/will occur against devices currently.

The comment here states that a second HTTP specific scan is carried out in 2.2 that may well resolve false negatives/false positives with the HTTP related tests, but this naturally doesn't cover all the other possible (good/bad?) service/port permutations out there (TCP 502 in the case of the device above, but that's just one device/service in this example).

[duncangreene]

My two cents on the mention of 'constrained by design' devices above.

Many devices in the smart building/OT/IoT world are going to be constrained/limited in resources by design. Take DALI as an example, a lighting protocol that operates at 1200 baud (~1.2kbs to all intents). In the context of enterprise IT, that would seem archaic, but the reality is, 1200 baud is plenty fast enough for lighting commands. Something operating at that speed simply doesn't require more powerful hardware, nor should it be expected to have it. Devices that need to run an IP stack may have a bit more horsepower, but it may not need to be so much more horsepower that it stands up to unplanned/unexpected traffic (potentially such as that from pentests/Testrun/etc.) if that's simply not the intended design of the device. To that end, I would suggest making/tuning Testrun to match the capabilities of the device (i.e. rate limiting if needed), instead of the expectation being the other way around.

FWIW, any devices running mission critical or life safety systems will (by design) have their own redundancy/resilience protocols/methods etc. all built in. Electrical substation protocols like GOOSE and network redundancy stuff like MRP take care of all this, and you wouldn't expect something like Testrun or a pentest to trip them up, but the key thing is being mindful of 'by design', and that's something Testrun should probably have flexibility for if needed.

I know Testrun has been designed to be 'quick', but IMHO a scan that takes 40 minutes to complete, but provides a true reflection of device behaviour, should be seen as better than one that takes 20 minutes to complete but provides false negatives and/or false positives.

[duncangreene]

I would also add that if there is an expectation that the devices rise to meet the performance of Testrun (which I can see there would be merit in), it's worth noting that there is nothing within the Testrun report currently that actually detects and flags the above false negatives/false positives. I.e. Studying the above reports A, B, C or D in isolation would not yield the opinion that the device is struggling to keep up with Testrun.

[hitnik]

The issue has been reported 419479030.