New test tooling by apostasie · Pull Request #3418 · containerd/nerdctl

apostasie · 2024-09-07T04:17:13Z

We are not in a good situation wrt our testing. The build would be constantly red if it wasn't for the fact we are retrying every test multiple times. Some of it may be bugs, but most of it is definitely broken tests.

Despite lengthy efforts to reduce flakiness one by one, it seems unlikely to me that we can make significant progress without better test tooling to use.

This here proposes a new testing approach that should hopefully address our endemic issues and promote better testing in the future, by providing by default better test isolation, simpler, more debuggable helpers, and overall a more expressive testing approaching.

Commit 1 is the tooling itself.

Commit 2 is a handful of tests being rewritten with the new tooling.

Commit 3 is documentation.

Background information:

the commit message has the full details - read it for more details
this fixes testutil helpers are hard to use and make too many assumptions on command outcomes #3130

Obviously, this should keep evolving a lot, and some stuff might have to change as we rewrite more tests and discover more use-cases.

I would suggest reading the testtool.md doc first before engaging with code.

Right now, what matters most IMHO is that:

we agree on the problem statement - hopefully we do :-)
we are overall comfortable with the high-level goals and design approach (let's call it "table driven test++")
we find the proposed DX acceptable enough

Improvements can always follow.

apostasie · 2024-09-18T05:38:29Z

@AkihiroSuda this is ready for a first review.

CI is pending.

AkihiroSuda · 2024-09-18T05:49:07Z

+	WithConfig(key ConfigKey, value ConfigValue) Data
+	ReadConfig(key ConfigKey) ConfigValue
+
+	// Private methods


private methods do not need to be defined in the interface?

I do need getLabels and getConfig here:

nerdctl/pkg/testutil/test/data.go

Lines 131 to 137 in f1434f4

func configureData(t *testing.T, seedData Data, parent Data) Data {

if seedData == nil {

seedData = &data{}

}

dat := &data{

config: seedData.getConfig(),

labels: seedData.getLabels(),

Will remove adopt, which is not necessary.

Done (removed adopt).

Lmk if there is a better way to achieve this without defining the other two on the interface.

AkihiroSuda · 2024-09-18T05:50:18Z

+	"github.com/containerd/nerdctl/v2/pkg/inspecttypes/dockercompat"
+	"github.com/containerd/nerdctl/v2/pkg/inspecttypes/native"
+	"github.com/containerd/nerdctl/v2/pkg/testutil"
+	"github.com/containerd/nerdctl/v2/pkg/testutil/test"


Why separate pkg?

These are nerdctl specific (NerdctlToml, etcetera). Motivation was that there is already a lot inside the test package, and the nerdtest package will likely grow a lot more as it subsumes base.

I can merge them back together though if preferable. No strong opinion here.

AkihiroSuda

Thanks

apostasie · 2024-09-18T23:42:26Z

Thanks

@AkihiroSuda

I forgot to convert some tests that are commented out.
Let me take care of that ASAP before we merge.

apostasie · 2024-09-19T23:02:40Z

@AkihiroSuda last push finish converting and re-enabled the mistakenly commented out tests.
Apologies for missing it.

Also note a small change in the test tooling, enabling Parallel for test.Group.

apostasie · 2024-09-20T00:36:53Z

CI failures are network flukes, which have been endemic with github recently.
Concerning for sure, but unrelated.

AkihiroSuda · 2024-09-20T18:15:05Z

Is this ready to be merged, or do you plan to migrate more tests in this single PR?

apostasie · 2024-09-20T18:30:32Z

Is this ready to be merged, or do you plan to migrate more tests in this single PR?

I think we should merge now if you are ok with it. I would like to use the new tooling in other PRs (like the "rewrite cp" one).

I'll migrate more tests separately. This is going to take time.

apostasie · 2024-09-20T19:23:42Z

Latest push has minor wording fixes in the documentation.

Our testing tools have served us well, and there are plenty of good things with them that we absolutely need to keep. We have grown a large test suite though, and the sheer size of it is putting pressure on it and it is starting to show cracks. Besides bugs per-se (double execution) - that we hopefully fixed - our design is reaching its limits. We do have rather impactful issues with regard to test isolation and test concurrency, leading to situations where it is hard or outright impossible to figure out which test is causing a cascading failure, or why the adding of a new test file working individually breaks everything. Furthermore, as we are not prescriptive on certain things, we do see a lot of negative patterns emerging from test authors: - defer being used instead of t.Cleanup - not calling cleanup routines before running tests - outright forgetting to cleanup resources - Cwd for binary calls is by default the current working directory of the test process - this is causing a variety of issues, as it could very well be read-only (lima), and should by default be a temp directory - manipulating the environment directly - which has side-effects for other tests - tests becoming big blurbs of mixed together setup, cleanup, and actual test routines - making them hard to read and figuring out what is actually being tested - subtests repetitiveness w. shadowing testutil.T leading to confusing code - ... or not dereferencing the current test in a loop - in-test homegrown abstractions being inherently repetitive, with the same boilerplate over and over again - structuring tests and subtests being left as an exercise to the developer - leading to a wide variety of approaches and complex boilerplate - hard to debug: a lot of the assert methods do not provide any feedback whatsoever on what was the command that actually failed, or what was the output, or environment of it - icmd.Expected showing its limits, and making assumptions that there is only one error you can test, or that you can only test the output if exitCode == 0 - running commands with other binaries than the current target (eg: not calling base.Cmd) being left as an exercise to the developer, leading to all of the issues above (Chdir, Env manipulation, no debugging output, etc) - no-parallel being the default - unless specified otherwise - which should be the other way around - very rarely testing stderr in case of error - partly because we do not use typed errors, but also because it is cumbersome / not obvious / limited This new tooling offers a set of abstractions that should address all of these and encourage more expressive, better structured, better isolated, more debuggable tests. Signed-off-by: apostasie <spam_blackhole@farcloser.world>

apostasie · 2024-09-22T17:05:09Z

Effed-up my commit rewrite :/. Hang on a sec.

Signed-off-by: apostasie <spam_blackhole@farcloser.world>

apostasie · 2024-09-22T19:15:40Z

All set.

Apologies for the back and forth.

While I was in the process of rewriting more tests this weekend, a few fixes and changes came up as necessary for the tooling, which is better to have here directly:

there is now a dedicated way to express Requirements, and a set of "requires" have been ported from base (Rootless, OnlyIPv6)
marked a few functions as t.Helper to avoid having them show in back traces
inline documentation fixes
subtests now inherit their parent custom namespace if any
Command now supports WithStdin and Background to accommodate for tests that require async or stdin manipulation
Command methods signature changed to return the command itself to allow for easy chaining (sugar)
fixed identifier hashing method to replace more characters (commas) so that it does not break image names

Thanks for your patience... This task is a large and tricky endeavor, but we are starting to see benefits:

Merging at your convenience as other PRs would (mutually) benefit (from) the new testing tools.

AkihiroSuda

Thanks

apostasie force-pushed the dev-3130-tesutil-nextgen branch 4 times, most recently from 8cb9f25 to 814418a Compare September 7, 2024 22:37

apostasie force-pushed the dev-3130-tesutil-nextgen branch 4 times, most recently from 37bc5e9 to 193a67d Compare September 18, 2024 05:30

apostasie changed the title ~~[WIP] Test tooling refresh~~ New test tooling Sep 18, 2024

apostasie marked this pull request as ready for review September 18, 2024 05:34

AkihiroSuda reviewed Sep 18, 2024

View reviewed changes

Comment thread pkg/testutil/test/test.go Outdated

AkihiroSuda reviewed Sep 18, 2024

View reviewed changes

Comment thread cmd/nerdctl/main_test_test.go

AkihiroSuda reviewed Sep 18, 2024

View reviewed changes

Comment thread docs/dev/testtool.md

apostasie force-pushed the dev-3130-tesutil-nextgen branch 2 times, most recently from 6b60abc to 28263b1 Compare September 18, 2024 06:32

AkihiroSuda approved these changes Sep 18, 2024

View reviewed changes

AkihiroSuda added the area/ci e.g., CI failure label Sep 18, 2024

AkihiroSuda marked this pull request as draft September 18, 2024 23:44

apostasie force-pushed the dev-3130-tesutil-nextgen branch from 28263b1 to 4a27229 Compare September 19, 2024 23:01

apostasie marked this pull request as ready for review September 19, 2024 23:02

apostasie requested a review from AkihiroSuda September 19, 2024 23:03

apostasie force-pushed the dev-3130-tesutil-nextgen branch from 4a27229 to 5d9e243 Compare September 20, 2024 19:22

apostasie force-pushed the dev-3130-tesutil-nextgen branch from 5d9e243 to 0eb301e Compare September 22, 2024 16:40

apostasie marked this pull request as draft September 22, 2024 16:45

apostasie force-pushed the dev-3130-tesutil-nextgen branch 3 times, most recently from 39b55d7 to ac18215 Compare September 22, 2024 18:24

apostasie added 2 commits September 22, 2024 11:47

Move root and volume tests to new test tooling

cc56a09

Signed-off-by: apostasie <spam_blackhole@farcloser.world>

Testtool initial documentation

a74f0f3

Signed-off-by: apostasie <spam_blackhole@farcloser.world>

apostasie force-pushed the dev-3130-tesutil-nextgen branch from ac18215 to a74f0f3 Compare September 22, 2024 18:47

apostasie marked this pull request as ready for review September 22, 2024 19:15

AkihiroSuda approved these changes Sep 23, 2024

View reviewed changes

AkihiroSuda merged commit 772feb4 into containerd:main Sep 23, 2024

AkihiroSuda added this to the v2.0.0 milestone Sep 23, 2024

apostasie mentioned this pull request Sep 23, 2024

Test rework, part 2 #3455

Merged

apostasie mentioned this pull request Oct 5, 2024

Test rework, part 4 #3492

Merged

	func configureData(t *testing.T, seedData Data, parent Data) Data {
	if seedData == nil {
	seedData = &data{}
	}
	dat := &data{
	config: seedData.getConfig(),
	labels: seedData.getLabels(),

Conversation

apostasie commented Sep 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apostasie commented Sep 18, 2024

Uh oh!

Uh oh!

AkihiroSuda Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

apostasie Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

apostasie Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

apostasie Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

apostasie commented Sep 18, 2024

Uh oh!

apostasie commented Sep 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apostasie commented Sep 20, 2024

Uh oh!

AkihiroSuda commented Sep 20, 2024

Uh oh!

apostasie commented Sep 20, 2024

Uh oh!

apostasie commented Sep 20, 2024

Uh oh!

apostasie commented Sep 22, 2024

Uh oh!

apostasie commented Sep 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

apostasie commented Sep 7, 2024 •

edited

Loading

apostasie commented Sep 19, 2024 •

edited

Loading

apostasie commented Sep 22, 2024 •

edited

Loading