Strategy on manual test execution.

[pahmann]

Topic originated in #2364.

What is the best way and frequency to execute manual test cases to be as close to the automated test execution as possible.

Challenge statement

"We shall consider how we want to ensure fail/pass doesn't just happen on a single machine. And who will be responsible on checking those manual tests on PR/main/release?"

Starting point to resolve

I would try to find an argument that manual test cases are only executed for specific releases not for every PR. It could be an idea to execute once a month on main, so we do not get surprises during a release, but still keep the effort low. The actual execution may need to be done by someone from the Testing Community. Maybe we take a release candidate (e.g. when we are closer to a release) and execute the tests then. The maintainer of the component/feature/platform test will confirm the proper execution. We can also swap and let the maintainer execute the test and testing community reviews. But at least this should overcome some single person effects. Best would be two people execute, but this would even increase efforts.
The execution would in best case be on the machine where the reference hardware will be to avoid local machine execution. This would also be the place where the automated tests got executed and where the logs are stored. In this way, we guarantee that the right software was deployed and the right version of the test is executed and so on.

One point though is, that manual execution is always effort, so we should limit the number to the least possible, but should no get unexpected surprises when it comes to the formal release testing. For one project we executed the tests once a month (depending on changes in the component) where we had a formal deliverable which had to be fully tested once a quarter.