e2e: add environment for kind

[squat]

This commit adds a new implementation of the Environment interface
implemented using kind, ie Kubernetes in Docker. The motivation is that
you might want to manually run some complex configurations in the e2e
tests using Kubernetes resources and still manage the tests simply using
the e2e package.

Users who need the escape hatch of deploying things to a Kubernetes
cluster manually, e.g. workloads that are not a simple deployment, can
use the kubeconfig file located in the environment's shared directory.

Signed-off-by: Lucas Servén Marín lserven@gmail.com

[squat]

Despite what CI says, the tests on macOS are actually failing: https://github.com/efficientgo/e2e/actions/runs/4491794039/jobs/7900760164#step:6:194

Mounting the volumes is working seamlessly across all of the layers! However, there seems to be an issue with connecting to the NodePort service for pods. I am not familiar with how Colima/Lima forward ports and don't have a mac to test this. This works out-of-the-box on Linux, but it is also jumping one layer less:
on Linux:
host->"node"->iptables->container
on macOS:
host->Lima->"node"->iptables->container

I'm guessing, since I don't have a mac to test, but I image the issue is with getting packets from the host to the Lima VM, since the discovered IP address is the address of the "node" and not of the Lima VM, so the packets destined to the "node" can't be routed. Does anybody have a mac to help debug this? Maybe we can pair?

[matej-g]

@squat let me try to run the tests on my Mac and report back 👍

[squat]

Thanks @matej-g ⭐

We should also make sure that CI actually fails when the tests fail

[matej-g]

So I got the same results initially on my local (could not reach container).

I roughly followed the guide from here (Step 5 - https://baptistout.net/posts/kubernetes-clusters-on-macos-with-loadbalancer-without-docker-desktop/, slightly adjusted for Colima), to setup the whole end-to-end networking. After which the test passed fine.

So presumably (unless there's an easier way to set it up), we'd need to automate this and add necessary config on both the runner host and inside of Lima VM.

Happy to pair on this!

[squat]

Oh cool that you got that to work! I was reading that exact same blog last nicht at 3am hahaha

I have a suspicion that for a typical Lima/Colima setup we might be able to get away with only one (max two) step(s):

1. Add routes to the macOS host so that all packets destined for and IP in the CIDR of the Docker bridge in the Lima guest go through the IP of the Lima guest
2. Not certain if it will be required, but if necessary, configure the Lima guest to forward packets (sysctl and/or iptables). This would mean issuing one single lima iptables ... command after colima start

It would be great to pair on this to see if it could be this simple.

[squat]

I got the tests to pass on macOS! However, I was wrong, not only were those two steps necessary, but a third one was necessary as well!

PS: I was finally able to test this by running a macOS VM on my Linux machine, so the entire stack is:
Linux host -> macOS guest -> Lima guest -> "node" -> container 😈

[matej-g]

I have tested the updated CI setup locally, tests are working 🌟.

Nice addition, this looks good to me overall. I think it would be good to document the KinD environment and especially document the usage (i.e. runnables can be only Deployment, for other objects users need to do it by hand). But this can be a separate PR.

[squat]

oki everything should be ready for now! I'll make a follow-up with some more documentation, if that works

[matej-g]

Thank you @squat 🙇