testutil helpers are hard to use and make too many assumptions on command outcomes

[apostasie]

Description

Specifically, some helpers testing stdout do first assert that exit code == 0.

However, there are legitimate cases where exit code is not 0 and you still want to test the output.

Recently, we also faced issues with commands being executed multiple times as each helper invocation used to call Run again.

In light of these issues, maybe it will soon be time to have a look at type Cmd and possibly revamp it to make testing easier.

Steps to reproduce the issue

na

Describe the results you received and expected

na

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

Besides bugs in our testing tools - hopefully got fixed by now - laying out issues here with the current design, that should inform a proposal to enhance:

• use of defer instead of t.Cleanup - cleaning up should be taken care of automatically
• not calling cleanup routines before running tests - making failing tests successive reruns to fail (locally) - cleaning up should be taken care of automatically
• Cwd for binary calls is by default the current working directory of the test process - this is causing a variety of issues, as it could very well be read-only (lima), and should by default be a temp directory
• manipulating the environment directly has side-effects for other tests - there should be an obvious mechanism for people to set additional env without polluting the parent scope
• tests are usually a blurb of mixed together setup, cleanup, and actual test routines - making them hard to read and figuring out what is actually being tested - abstraction should encourage a clean separation of Setup, Cleanup, Command
• tests are inherently repeating the same boilerplate over and over (tID := testutil.Identifier()) - since all tests are doing it, that should be hidden away
• structuring tests and subtests is left as an exercise to the developer - leading to a wide variety of approaches and complex boilerplate - we should have a cleaner, simpler, ready to go abstraction for that
• debugging is hard: some assert methods will provide no feedback whatsoever on what was the command that actually failed, what was the output of it - all failing commands should provide complete output automatically
• icmd.Expected is too limiting, and is making assumptions that:
  • there is only one error you can test
  • you can only test the output if exitCode == 0
• running commands with other binaries than the current target (eg: not calling base.Cmd) is left as an exercise to the developer, leading to all of the issues above (Chdir, Env manipulation, no debugging output, etc)
• no-parallel is the default - unless specified otherwise - it should be the opposite
• we currently very rarely test stderr in case of error - this is partly because we do not use typed errors, but also because it is cumbersome / not obvious / limited - we should provide primitives that encourage testing the actual error we return to the user