Kube on the CI by apostasie · Pull Request #3296 · containerd/nerdctl

apostasie · 2024-08-12T07:33:40Z

This is meant to bring in kube testing (fix #3282):

using kind to spin-up kube
- script usable locally or on the CI indifferently
- most of it comes from what I committed recently to nydus
mimic the ipv6 mechanism: -test.kube will run only "kube-compatible" tests - see testutils
example / placeholder test inside container_commit_test, since this is where we will need the first one

Tagging @lingdie because of recent work on kube issues with #827 and #3268 and facing that question of how to test on kube in here.

@fahedouch @AkihiroSuda and other folks - thoughts/feeback?

apostasie · 2024-08-12T08:38:11Z

 # We specifically want the static ones
 TINI_CHECKSUM=static

-


Sharing that with the new kube script (moved to lib.sh)

apostasie · 2024-08-12T08:40:39Z

Ok, target is green.

This is a bit rough in its current shape, but we can make it better / more integrated in the future as we start seeing usage for it (eg: actual kube tests).

apostasie · 2024-08-12T17:11:30Z

FreeBSD failure is unrelated. Last push just removes two debugging print that are not consequential.

apostasie · 2024-08-16T00:53:06Z

@AkihiroSuda

To run kube tests:

go test ./cmd/nerdctl/ -test.kube

This must be run in the (containerd) context where kube containers have been created (by kubectl for example).
In the case of kind (on the CI) , I am running that command inside the kind control plane container.

There is now one test in https://github.com/containerd/nerdctl/pull/3296/files#diff-cb510ac1ecc8b03a6de395f8bfde3d161432dfbbf3d26814eec378cb350b41b7

It is an "anti" test - it assumes failure (because right now our code is broken), but MUST be transformed to assume success as soon as we fix the problem.

The gist of things is:

create stuff with kubectl
test that nerdctl can perform operations on these containers

BaseForKube puts you in the k8s.io namespace, where kube created containers are expected to live.

apostasie · 2024-08-16T03:23:35Z

@AkihiroSuda from the comments above - most of them addressed, BUT:

kubectl wait will timeout with a busybox image - no idea why, and not particularly interested in debugging kube - I thus switched the image to testutil.CommonImage instead
kubectl namespaces are not containerd namespaces as far as I can tell, so, for now, we will operate on the standard namespace k8s.io (not sure how to change that)
moved the kube test to a linux only test file, as linking to testutil images would fail the tests on windows

Kube-CI is green.

AkihiroSuda · 2024-08-16T16:09:14Z

kubectl namespaces are not containerd namespaces as far as I can tell, so, for now, we will operate on the standard namespace k8s.io (not sure how to change that)

The containerd objects should stay in containerd's k8s.io namespace, but the Kubernetes objects (Pods, etc.) should be in Kubernetes API's nerdctl-test namespace

Signed-off-by: apostasie <spam_blackhole@farcloser.world>

apostasie · 2024-08-16T16:36:25Z

Latest push:

rebased, conflict resolved
uses k8s namespace for kubectl created objects
tentative rename for the flags:
- test.allow-kill-daemon
- test.only-ipv6
- test.only-kube

Let me know if you would like these flag names tweaked to something else.

apostasie · 2024-08-16T18:00:42Z

CI failures are:

github belly-up
the usual TestIPFSComposeUp

If we can re-trigger these two - and eventually we need to figure out #3305

apostasie · 2024-08-17T15:35:10Z

CI is green.

AkihiroSuda

Thanks

fahedouch · 2024-08-18T13:32:16Z

Sorry for late review! thanks for effort I recommend using k8s rather then kube to be more explicit.

fahedouch · 2024-08-17T15:31:18Z

+	setup()
+
+	t.Run("test commit / push on Kube (https://github.com/containerd/nerdctl/issues/827)", func(t *testing.T) {
+		t.Log("This test is meant to verify that we can commit / push an image from a pod." +


Suggested change

t.Log("This test is meant to verify that we can commit / push an image from a pod." +

t.Log("This test is meant to verify that we can commit / push an image that belongs to containerd `k8s` namespace." +

BTW it is a nerdctl issue not only a kube

Do you mean issue #827 can be reproduced in a non-kube environment?

Is there a separate ticket, or do you have a simple reproducer for it that does not involve kube?

apostasie · 2024-08-18T17:11:27Z

Sorry for late review! thanks for effort I recommend using k8s rather then kube to be more explicit.

Thanks @fahedouch !

It is not too late - can definitely send a follow-up PR - and yes, I can rename kube -> k8s.

Just answered your comments above. Let me know what you think when you get a chance.

ouyangningdong · 2024-08-21T02:58:31Z

Has the issue of the save error prompt been resolved? Thank you very much! content digest sha256:a58ecd4f0c864650a4286c3c2d49c7219a3f2fc8d7a0bf478aa9834acfe14ae7: not found

apostasie · 2024-08-21T03:41:30Z

@ouyangningdong #827 (comment)

AkihiroSuda reviewed Aug 12, 2024

View reviewed changes

Comment thread .github/workflows/test-kube.yml Outdated

AkihiroSuda reviewed Aug 12, 2024

View reviewed changes

Comment thread .github/workflows/test-kube.yml Outdated

AkihiroSuda reviewed Aug 12, 2024

View reviewed changes

Comment thread hack/build-integration-kube.sh Outdated

apostasie force-pushed the dev-kube branch 12 times, most recently from 9ab161f to 05b2421 Compare August 12, 2024 08:33

apostasie changed the title ~~[WIP] Kube on the CI~~ Kube on the CI Aug 12, 2024

apostasie commented Aug 12, 2024

View reviewed changes

apostasie marked this pull request as ready for review August 12, 2024 08:40

apostasie requested a review from AkihiroSuda August 12, 2024 08:40

AkihiroSuda reviewed Aug 12, 2024

View reviewed changes

Comment thread pkg/testutil/testutil.go

apostasie force-pushed the dev-kube branch from 05b2421 to c2942ba Compare August 12, 2024 16:35

This was referenced Aug 13, 2024

Cannot commit and push container image from Kubernetes pods #827

Closed

Canary broken: follow-up to #3224 #3306

Closed

apostasie force-pushed the dev-kube branch from c2942ba to bb401e2 Compare August 15, 2024 17:04

AkihiroSuda reviewed Aug 15, 2024

View reviewed changes

Comment thread cmd/nerdctl/container_commit_test.go Outdated

AkihiroSuda added area/ci e.g., CI failure area/kubernetes labels Aug 15, 2024

apostasie force-pushed the dev-kube branch from bb401e2 to a31d043 Compare August 16, 2024 00:46

apostasie force-pushed the dev-kube branch from a31d043 to 4167281 Compare August 16, 2024 00:47

AkihiroSuda reviewed Aug 16, 2024

View reviewed changes

Comment thread cmd/nerdctl/container_commit_test.go Outdated

AkihiroSuda reviewed Aug 16, 2024

View reviewed changes

Comment thread cmd/nerdctl/container_commit_test.go Outdated

AkihiroSuda reviewed Aug 16, 2024

View reviewed changes

Comment thread cmd/nerdctl/container_commit_test.go Outdated

AkihiroSuda reviewed Aug 16, 2024

View reviewed changes

Comment thread .github/workflows/test-kube.yml Outdated

apostasie force-pushed the dev-kube branch 4 times, most recently from 94eaea7 to c7a1032 Compare August 16, 2024 03:18

Kube on the CI

97c5492

Signed-off-by: apostasie <spam_blackhole@farcloser.world>

apostasie force-pushed the dev-kube branch from c7a1032 to 97c5492 Compare August 16, 2024 16:34

AkihiroSuda added this to the v2.0.0 milestone Aug 17, 2024

AkihiroSuda approved these changes Aug 17, 2024

View reviewed changes

AkihiroSuda merged commit a44a1cb into containerd:main Aug 17, 2024

fahedouch reviewed Aug 18, 2024

View reviewed changes

apostasie mentioned this pull request Aug 21, 2024

Kubernetes minor cleanup (rename and -f on delete) #3338

Merged

apostasie mentioned this pull request Oct 15, 2024

Part 5: tests #3550

Merged

	t.Log("This test is meant to verify that we can commit / push an image from a pod." +
	t.Log("This test is meant to verify that we can commit / push an image that belongs to containerd `k8s` namespace." +

Conversation

apostasie commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

apostasie Aug 12, 2024

Choose a reason for hiding this comment

Uh oh!

apostasie commented Aug 12, 2024

Uh oh!

Uh oh!

apostasie commented Aug 12, 2024

Uh oh!

Uh oh!

apostasie commented Aug 16, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

apostasie commented Aug 16, 2024

Uh oh!

AkihiroSuda commented Aug 16, 2024

Uh oh!

apostasie commented Aug 16, 2024

Uh oh!

apostasie commented Aug 16, 2024

Uh oh!

apostasie commented Aug 17, 2024

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

fahedouch commented Aug 18, 2024

Uh oh!

Uh oh!

Uh oh!

fahedouch Aug 17, 2024

Choose a reason for hiding this comment

Uh oh!

fahedouch Aug 17, 2024

Choose a reason for hiding this comment

Uh oh!

apostasie Aug 18, 2024

Choose a reason for hiding this comment

Uh oh!

fahedouch Aug 20, 2024

Choose a reason for hiding this comment

Uh oh!

apostasie Aug 21, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

apostasie commented Aug 18, 2024

Uh oh!

ouyangningdong commented Aug 21, 2024

Uh oh!

apostasie commented Aug 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

apostasie commented Aug 12, 2024 •

edited

Loading