Refactor run command networking options for Windows support. by aznashwan · Pull Request #1953 · containerd/nerdctl

aznashwan · 2023-01-30T14:27:29Z

Refactor the loading and application of the networking-related
arguments for the nerdctl run command in order to facilitate
Windows support.

Fixes/alleviates #559 and #1680.

Signed-off-by: Nashwan Azhari nazhari@cloudbasesolutions.com

AkihiroSuda · 2023-01-30T14:46:45Z

compose: show detailed errors again

This is now merged in the main, please rebase

aznashwan · 2023-02-09T14:12:13Z

Just pushed some updates, and despite still have some issues which will need ironing out, I would like to offer some notes for any pre-emptive reviews due to the initially excessive-looking amount of abstractions which are added:

this initial implementation only offers CNI networking on Windows (host mode networking should be doable as well)
OCI hooks (which is the current mechanism through which nerdctl sets up and tears down CNI networking) are not currently supported on Windows, so the only route forwards is to have the main nerdctl run/rm process set up and tear down the container's networking itself. (this has lead to potential dangling interface definitions and port mappings which I'll need to add some more explicit tests for)
the nerdctl/pkg/netutil package only saves/reads CNI v1.0.0 configurations, but Windows container networking can only handle version up to v0.4.0.

Although that last one isn't as big of a problem as it sounds (it only means networks created by nerdctl cannot be read/used by HCS, so only affects tests which create their own network), I'm still looking into a better way to handle them without bypassing netutil.

Any feedback is appreciated in the meantime.

dcantah · 2023-02-16T11:58:56Z

@aznashwan I don't think there's a technical reason why OCI hooks couldn't work on Windows, I just think we never had a reason to really implement them, but I feel like this is as good of a reason as they come hahah

dcantah · 2023-02-16T11:59:53Z

Maybe worth exploring 😳

aznashwan · 2023-02-16T15:00:57Z

Just pushed some updates, I believe I have managed to completely bypass the CNI versioning issue (at least in nerdctl itself while creating the container), and at this stage nerdctl seems to reliably create and clean up networking for Windows containers.

I still have to port/fix some tests, but this is the point where any reviews would be extra-appreciated!

aznashwan · 2023-02-28T18:19:47Z

LE: The cleanup errors were actually due to a failure to remove containers created with --rm on non-Windows which was introduced in this patch, but should be fixed now.

~~This PR should be review-able, and it looks like the rest of the failing tests are somehow related to this recent BuildKit layer pruning issue.~~

Failing tests are:

TestBuildSourceDateEpoch states that the image it just build cannot be deleted since it is being used by a residual container.
TestImageEncryptJWE fails because an image pull expected to fail actually succeeds.
the following IPFS-build-related tests all fail to remove the testutil.CommonImage image:
TestIPFS, TestIPFSBuild, TestIPFSAddress, TestIPFSCommit, TestIPFSWithLazyPulling, TestIPFSWithLazyPullingCommit, TestIPFSRegistry, TestIPFSRegistryWithLazyPulling

I'm pretty stumped at this stage on what the cause may be because as far as I can tell my patch should not modify/manipulate IPFS settings at all. (it's strictly related to the networking arguments for the containers themselves)

It's probably something obvious I'm missing, but I'm not familiar enough with how IPFS registries should work to know if any of my modifications to container creation even can bear any relevance to those tests?

@AkihiroSuda @fahedouch any pointers you may have for the IPFS failure would be greatly appreciated.

@dcantah @jsturtevant if you could please have a look over the Windows side of the code, that would be greatly appreciated as well!

jsturtevant · 2023-03-01T20:18:30Z

+func (m *containerNetworkManager) VerifyNetworkOptions(_ context.Context) error {
+	// TODO: check host OS, not client-side OS.
+	if runtime.GOOS != "linux" {
+		return errors.New("container networking mode is currently only supported on Linux")


I was surprised to see this. Maybe I don't see how this is used?

The initial PR's scope was abstracting the networking logic for all platforms and only adding minimal CNI networking support for Windows for now as that is the only network setup which is tested on Windows.

aznashwan · 2023-03-16T14:29:26Z

@AkihiroSuda could I please get a recheck whenever you can spare the time?

AkihiroSuda · 2023-03-16T15:26:31Z

 	// https://github.com/microsoft/Windows-Containers/issues/179
 	CommonImage = WindowsNano
+
+	// NOTE(aznashwan): the upstream e2e Nginx test image is actually based on BusyBox.


You can change the constant name to match the reality. Can be another PR.

AkihiroSuda · 2023-03-16T15:28:29Z

+		return "", errors.New("namespace is required")
+	}
+	if strings.Contains(globalOptions.Namespace, "/") {
+		return "", errors.New("namespace with '/' is unsupported")


For Windows you may want to validate \ (and :?) too

jsturtevant · 2023-03-17T16:37:39Z

+		return err
+	}
+
+	// NOTE: only currently supported network type on Windows is nat:


why is only nat supported?

The main goal of the patch was to get minimal CNI networking running on Windows, enable as many tests which use networking (e.g. tests which set up a temporary local registry), and then add other network modes later.

Seeing as though nat was the default network type set up by the CI, and is the only one that gets tested, I added this explicit constraint for now.

jsturtevant · 2023-03-17T16:42:46Z

-			var isErr bool
-			if errE := os.RemoveAll(stateDir); errE != nil {
-				isErr = true
+			if containerErr == nil {


do we need && runtime.GOOS == "windows" here too?

The variable netSetupErr is initially set to nil and will only be non-nil if runtime.GOOS == "windows". Therefore, it is unnecessary IMO.

ah, Thanks! I see that now. Just a thought, It isn't immediately obvious that this code is only executing for Windows when reading through. Adding something here to indicate that would could be helpful.

jsturtevant · 2023-03-20T16:29:00Z

lgtm for Windows
I agree with @dcantah the ideal path would be OCI hooks since the project heavily uses it. It is technically feasible for Windows as far as I know but would take some effort in the hcsshim project.

AkihiroSuda · 2023-03-21T17:31:50Z

CI is failing. Please try rebasing.

Refactor the loading and application of the networking-related arguments for the `nerdctl run` command in order to enable Windows container networking support through CNI. To facilitate this, the following major changes were made: * moved all networking-related container spec definitions to pkg/containerutil/container_network_manager_* * refactored network-related argiment loading and application in cmd/nerdctl/container_run.go * enabled some of the networking-related tests on Windows Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

aznashwan · 2023-03-22T11:08:04Z

@AkihiroSuda looks all green now, thanks!

AkihiroSuda

Thanks

aznashwan force-pushed the hyperv-netns branch 3 times, most recently from dff5c2d to 7751426 Compare January 30, 2023 14:36

AkihiroSuda reviewed Jan 30, 2023

View reviewed changes

Comment thread go.mod Outdated

aznashwan force-pushed the hyperv-netns branch from 7751426 to 5e71481 Compare January 30, 2023 15:38

AkihiroSuda added platform/Windows/Non-WSL2 Microsoft Windows (non-WSL2) kind/refactor labels Jan 30, 2023

jsturtevant mentioned this pull request Feb 6, 2023

[Windows] Networking doesn't work #559

Closed

AkihiroSuda added this to the v1.3.0 (tentative) milestone Feb 8, 2023

aznashwan force-pushed the hyperv-netns branch from 5e71481 to a188461 Compare February 9, 2023 13:13

AkihiroSuda modified the milestones: v1.2.1, v1.3.0 (tentative) Feb 11, 2023

aznashwan force-pushed the hyperv-netns branch from a188461 to 925c06e Compare February 16, 2023 14:52

aznashwan force-pushed the hyperv-netns branch 2 times, most recently from 93c1a7c to 9c1bd98 Compare February 27, 2023 14:22

AkihiroSuda marked this pull request as draft February 28, 2023 05:31

AkihiroSuda requested a review from ktock February 28, 2023 22:33

jsturtevant reviewed Mar 1, 2023

View reviewed changes

ktock reviewed Mar 2, 2023

View reviewed changes

Comment thread cmd/nerdctl/container_create_linux_test.go Outdated

ktock reviewed Mar 2, 2023

View reviewed changes

Comment thread pkg/containerutil/container_network_manager_test.go

aznashwan force-pushed the hyperv-netns branch 4 times, most recently from 7e76e18 to c110fe4 Compare March 6, 2023 16:44

aznashwan force-pushed the hyperv-netns branch from 3be79db to 932b698 Compare March 14, 2023 11:31

AkihiroSuda reviewed Mar 16, 2023

View reviewed changes

Comment thread pkg/containerutil/containerutil.go Outdated

AkihiroSuda reviewed Mar 16, 2023

View reviewed changes

Comment thread go.mod Outdated

AkihiroSuda requested a review from yuchanns March 16, 2023 15:30

aznashwan force-pushed the hyperv-netns branch from 932b698 to 67ff631 Compare March 16, 2023 17:43

jsturtevant reviewed Mar 17, 2023

View reviewed changes

yuchanns reviewed Mar 18, 2023

View reviewed changes

Comment thread cmd/nerdctl/container_create_linux_test.go Outdated

Comment thread cmd/nerdctl/container_run_log_driver_syslog_test.go Outdated

Comment thread cmd/nerdctl/container_run_log_driver_syslog_test.go Outdated

Comment thread cmd/nerdctl/container_run_log_driver_syslog_test.go Outdated

aznashwan force-pushed the hyperv-netns branch from 67ff631 to 8e24dfa Compare March 20, 2023 09:57

aznashwan requested review from AkihiroSuda, jsturtevant and yuchanns and removed request for AkihiroSuda, jsturtevant and yuchanns March 20, 2023 10:25

aznashwan force-pushed the hyperv-netns branch from 8e24dfa to 89a78b4 Compare March 22, 2023 09:52

AkihiroSuda approved these changes Mar 22, 2023

View reviewed changes

AkihiroSuda merged commit 41ea1c7 into containerd:main Mar 22, 2023

Conversation

aznashwan commented Jan 30, 2023

Uh oh!

AkihiroSuda commented Jan 30, 2023

Uh oh!

Uh oh!

aznashwan commented Feb 9, 2023

Uh oh!

dcantah commented Feb 16, 2023

Uh oh!

dcantah commented Feb 16, 2023

Uh oh!

aznashwan commented Feb 16, 2023

Uh oh!

aznashwan commented Feb 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsturtevant Mar 1, 2023

Choose a reason for hiding this comment

Uh oh!

aznashwan Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aznashwan commented Mar 16, 2023

Uh oh!

AkihiroSuda Mar 16, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AkihiroSuda Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsturtevant Mar 17, 2023

Choose a reason for hiding this comment

Uh oh!

aznashwan Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

jsturtevant Mar 17, 2023

Choose a reason for hiding this comment

Uh oh!

yuchanns Mar 18, 2023

Choose a reason for hiding this comment

Uh oh!

jsturtevant Mar 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsturtevant commented Mar 20, 2023

Uh oh!

AkihiroSuda commented Mar 21, 2023

Uh oh!

aznashwan commented Mar 22, 2023

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

aznashwan commented Feb 28, 2023 •

edited

Loading

AkihiroSuda Mar 16, 2023 •

edited

Loading

jsturtevant Mar 20, 2023 •

edited

Loading