[Windows] Networking doesn't work

[lippertmarkus]

When using nerdctl run to start a Windows container, networking doesn't work as there is no ethernet adapter attached:

.\nerdctl.exe pull mcr.microsoft.com/windows/nanoserver:1809
.\nerdctl.exe run --rm mcr.microsoft.com/windows/nanoserver:1809 cmd /c ipconfig

# Windows IP Configuration

Same behaviour like with ctr:

./ctr.exe i pull mcr.microsoft.com/windows/nanoserver:1809
./ctr.exe run -rm  mcr.microsoft.com/windows/nanoserver:1809 test cmd /c ipconfig

# Windows IP Configuration

With crictl it works however:

.\crictl.exe pull mcr.microsoft.com/windows/nanoserver:1809
$POD_ID=(./crictl runp .\pod.json)
$CONTAINER_ID=(./crictl create $POD_ID .\container.json .\pod.json)
./crictl start $CONTAINER_ID
.\crictl.exe exec $CONTAINER_ID cmd /c ipconfig

# Windows IP Configuration
#
# Ethernet adapter vEthernet (3cceb90e9342cc0a23db648c9c4da002a329d285ba3733e8feb666a374e9f842_nat):
#
#   Connection-specific DNS Suffix  . : 
#   Link-local IPv6 Address . . . . . : fe80::1d7f:3f4c:efb5:457b%15
#   IPv4 Address. . . . . . . . . . . : 172.30.197.170
#   Subnet Mask . . . . . . . . . . . : 255.255.240.0
#   Default Gateway . . . . . . . . . : 172.30.192.1

with pod.json:

{
  "metadata": {
    "name": "mycont-sandbox",
    "namespace": "default",
    "attempt": 1,
    "uid": "hdishd83djaidwnduwk28basb"
  },
  "log_directory": "/tmp",
  "linux": {
  }
}

and container.json:

{
  "metadata": {
      "name": "mycont"
  },
  "image":{
      "image": "mcr.microsoft.com/windows/nanoserver:1809"
  },
  "command": [
      "cmd",
      "/c",
      "ping -t 127.0.0.1"
  ],
  "log_path":"mycont.0.log",
  "linux": {
  }
}

Without having any knowledge about the internals, I would assume it's because crictl uses a pod sandbox where networking is set up for and ctr/nerdctl doesn't?

cc @jsturtevant

[jsturtevant]

Looks like CTR has a flag for CNI that needs to be enabled:

• https://github.com/containerd/containerd/blob/af1a0908d06b4f7db4a11bfc67ecc2f34baeb04b/cmd/ctr/commands/run/run.go#L210-L214
• https://github.com/containerd/containerd/blob/af1a0908d06b4f7db4a11bfc67ecc2f34baeb04b/cmd/ctr/commands/run/run.go#L137

Could you try that and see if CTR is working with that flag?

For nerdctl, it look like the network is set up via a OCI hook. Something doesn't seem to be hooked up correctly for Windows. The hook is here:

nerdctl/cmd/nerdctl/internal.go

Lines 24 to 34 in fae1696

	var internalCommand = &cobra.Command{
	Use: "internal",
	Short: "DO NOT EXECUTE MANUALLY",
	Hidden: true,
	SilenceUsage: true,
	SilenceErrors: true,
	}
	internalCommand.AddCommand(
	newInternalOCIHookCommandCommand(),
	)

and

Here is where the network created with that hook:

nerdctl/pkg/ocihook/ocihook.go

Line 175 in 288848d

o.cni, err = gocni.New(cniOpts...)

[jsturtevant]

I will see if I can get this wired up properly.

[lippertmarkus]

The cni flag is part of the platformRunFlags and only available on Linux:
https://github.com/containerd/containerd/blob/af1a0908d06b4f7db4a11bfc67ecc2f34baeb04b/cmd/ctr/commands/run/run_unix.go#L82-L85

but not on Windows:
https://github.com/containerd/containerd/blob/af1a0908d06b4f7db4a11bfc67ecc2f34baeb04b/cmd/ctr/commands/run/run_windows.go#L33

[lippertmarkus]

I tried adding the cni flag to the platformRunFlags for Windows as well in containerd/containerd#6304 to try it out. It seems that it tries to do the right thing but fails in the end, see containerd/containerd#6305

[jsturtevant]

thanks for trying it out and logging the issue on containerd 💯

For nerdctl implementation, it looks like the current use of OCI hooks will not work as they are POSIX only: https://github.com/opencontainers/runtime-spec/blob/v1.0.2/config.md#posix-platform-hooks. A quick look through the hcshim code showed they seem to be implemented for LCOW.

I am guessing we will need to create the network during the run flow (similar to cri flow works) and then pass that information to the OCI spec as defined here: https://github.com/opencontainers/runtime-spec/blob/v1.0.2/config-windows.md#network

@AkihiroSuda do you have any input before I take look at implementing this?

[AkihiroSuda]

Why can't Windows support OCI hooks?

[AkihiroSuda]

I am guessing we will need to create the network during the run flow (similar to cri flow works) and then pass that information to the OCI spec as defined here:

Does this work with the restart manager (--restart=always)?
On Linux it doesn't work, so it's implemented in the hooks.

[jsturtevant]

I am not sure why it doesn't have support. Just inferring from the OCI docs and hcsshim code. spoke briefly with @dcantah and it sounds like it should be technically possible but would need implementation.

Does this work with the restart manager (--restart=always)?
On Linux it doesn't work, so it's implemented in the hooks.

I will have to experiment to figure this out.

[lippertmarkus]

Thanks to @kevpar I made networking work via ctr::

./ctr.exe run --cni -rm -t mcr.microsoft.com/windows/nanoserver:1809 test cmd /c ipconfig

Windows IP Configuration

Ethernet adapter vEthernet (default-test_nat):

  Connection-specific DNS Suffix  . :
  Link-local IPv6 Address . . . . . : fe80::a527:27d2:6f6a:6d53%23
  IPv4 Address. . . . . . . . . . . : 172.27.19.133
  Subnet Mask . . . . . . . . . . . : 255.255.240.0
  Default Gateway . . . . . . . . . : 172.27.16.1

Maybe that implementation in containerd/containerd#6304 helps you for nerdctl

[jsturtevant]

Maybe that implementation in containerd/containerd#6304 helps you for nerdctl

That is one way to do it but would be different form the Linux implementation here. I also think it might have issues as pointed out above with the flag --restart=always.

I've looked into the OCI hooks and there doesn't seem to be any reason it can't be supported in hcsshim it just hasn't been implemented for Windows. I plan to get a prototype of that working to gauge effort and feasibility and will update here.

[kkbruce]

In the beginning, I installed it through install-containerd-runtime.ps1, because the specified SYMLINKD directory was missing, so the --cni of ctr could not be used.

After solving the problems of ctr and --cni, I then tested nerdctl and found that there are similar problems. Provide my environment settings, and whether it can provide a useful solution.

---

if run nerdctl network ls, command auto-generate nerdctl-nat.conflist file and see two nat network:

PS C:\> nerdctl network ls
NETWORK ID    NAME    FILE
              nat     C:\Program Files\containerd\cni\conf\0-containerd-nat.conf
0             nat     C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist
              host
              none

nerdctl-nat.conflist content:

{
  "cniVersion": "1.0.0",
  "name": "nat",
  "nerdctlID": 0,
  "nerdctlLabels": {},
  "plugins": [
    {
      "type": "nat",
      "master": "Ethernet",
      "ipam": {
        "routes": [
          {
            "gateway": "10.4.0.1"
          }
        ],
        "subnet": "10.4.0.0/24",
        "type": ""
      }
    }
  ]
}

PS C:\> nerdctl -v
nerdctl version 0.21.0

PS C:\> nerdctl info
Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: v1.6.6
 Storage Driver: windows
 Logging Driver: json-file
 Cgroup Driver:
 Cgroup Version:
 Plugins:
  Log: fluentd journald json-file
  Storage: windows-lcow windows
 Security Options:
 Kernel Version:
 Operating System:
 OSType: windows
 Architecture:
 CPUs: 0
 Total Memory: 0B
 Name: labtestcontaine
 ID: 2755fc58-0723-4ec1-91b9-c652d11765de

Running with --net nat:

PS C:\> nerdctl run --rm --net nat mcr.microsoft.com/windows/nanoserver:ltsc2022 ipconfig

Windows IP Configuration

The run result of ctr with --cni:

PS C:\> ctr run --cni --rm mcr.microsoft.com/windows/nanoserver:ltsc2022 test ipconfig

Windows IP Configuration

Ethernet adapter vEthernet (default-test_nat):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::fdd7:f487:e1fc:be8f%17
   IPv4 Address. . . . . . . . . . . : 172.20.86.86
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 172.20.80.1

[lippertmarkus]

@kkbruce Networking support for Windows is not implemented yet in nerdctl. Currently networking only works with crictl and ctr command line tools.

[kkbruce]

@lippertmarkus Thanks for your info. Let's take a look at crictl first, and we can only look forward to its implementation.