Add pipeline to run integration test on k3d

[mszostok]

Description

Recently, we added support for k3d (#466). To ensure no regression, we need to create a pipeline that will run on each PR and main branch. It will be exactly the same as we have with the kind.

capact/.github/workflows/pr-build.yaml

Lines 390 to 443 in fa6712d

	integration-tests:
	name: Integration tests
	runs-on: ubuntu-latest
	if: github.event.pull_request.draft == false
	needs: [ push-apps, push-tests, build-cli ]
	permissions:
	contents: read
	env:
	CAPACT_BINARY: "${{ github.workspace}}/capact-linux-amd64"
	steps:
	- name: Checkout code
	uses: actions/checkout@v2
	with:
	ref: ${{ github.event.pull_request.head.sha }}
	- name: Setup environment
	env:
	PR_NUMBER: ${{ github.event.pull_request.number }}
	run: |
	. ./hack/ci/setup-env.sh
	- name: Download Capact CLI
	uses: actions/download-artifact@v2
	with:
	name: cli_${{github.sha}}
	path: ${{ github.workspace }}
	- name: Setup Capact CLI
	run: |
	chmod +x $CAPACT_BINARY
	$CAPACT_BINARY version
	- name: Setup Go
	uses: actions/setup-go@v2
	with:
	go-version: ${{env.GO_VERSION}}
	- name: Run K8s Controller integration tests
	run: |
	make test-k8s-controller
	- name: Run cross-functional integration tests
	env:
	BUILD_IMAGES: "false"
	ARTIFACTS: "output/"
	DISABLE_MONITORING_INSTALLATION: "true"
	run: |
	make test-integration
	- name: Upload artifacts
	uses: actions/upload-artifact@v2
	if: ${{ always() }}
	with:
	name: cluster_dump_${{github.sha}}
	path: "output"
	retention-days: 5 # Default 90 days
	- name: Delete Capact CLI artifact
	uses: geekyeggo/delete-artifact@v1
	with:
	name: cli_${{github.sha}}

We need to add it as a parallel step to the kind one.

AC

• Integration test are executed on k3d cluster, if failed, pipeline should be red

• Integration test on kind and k3d are run in parallel

• Stretch:
  Use preloaded images for our dependencies. We should build image tarball only if something was changed in deploy/kubernetes/charts. Later, this tarball can be used by k3d.

  # Cache images
  capact alpha archive-images helm -v --version @latest --helm-repo @latest --output ./archive.tar --save-component=neo4j,ingress-nginx,argo,cert-manager,kubed
  
  # Preload images
  DISABLE_MONITORING_INSTALLATION=true K3D_IMAGES_ARCHIVE=archive.tar CLUSTER_TYPE=k3d make test-integration
  

  Not sure where this tarball should be stored, as we may run to GitHub cache limits: https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy. Maybe it will be better to push it to our GCS.

Reason

Ensure no regression. Additionally, test which environment is more stable and faster.

[thisisommore]

Can I give it a try?

[mszostok]

Hi @thisisommore!

Feel free to take it :)

In the first phase, just focus on executing the integration test on k3d:

DISABLE_MONITORING_INSTALLATION=true K3D_IMAGES_ARCHIVE=archive.tar CLUSTER_TYPE=k3d make test-integration

If you need help, please join #capact channel on CNCF workspace: https://capact.io/community/slack 👍

The problematic part is that the change needs to be tested on your fork. On our repo we use the pull_request_target event which takes the workflow definition from the main branch. As a result, changes to any capact/.github/workflows/*.yaml files on pull-request are ignored on our pipelines.

[thisisommore]

I usually code in different branch even on fork, I guess in this case I have to code in main branch

[thisisommore]

Hey @mszostok I was working on it and got an error

Can you please check if there is something wrong with my changes https://github.com/thisisommore/capact

[mszostok]

Hi @thisisommore, sorry for late response but I was on vacation during the long weekend ;)

First, I like your approach with CLUSTER_TYPE: ${{ matrix.cluster }} 👍

Regarding your issues I found:

1. Unfortunately, you deleted the function call: thisisommore@c164908#diff-079edea40bca6863258884390b26184956a3f24fecaa122eb2c5803b1e14847cL99-L100. This is the reason that you don't see anything in the logs.
2. You commented out the code responsible for pushing images. In general that's ok but in normal workflow we use them later in integration test step to make it faster and reuse already built images. For your tests, you can simply change the BUILD_IMAGES env to true on the integration step.
3. Don't define the K3D_IMAGES_ARCHIVE as you don't produce this archive yet.

Here are the fixes that I added: mszostok@1e1eb59

Unfortunately, I got:

Node 'k3d-k3d-ci-capact-serverlb' is restarting for more than a minute now. Possibly it will recover soon (e.g. when it's waiting to join). Consider using a creation timeout to avoid waiting forever in a Restart Loop. [started 22:04 UTC]

I will try to debug that tomorrow :)

[mszostok]

Hi @thisisommore I figured out the problem, here is the PR with fix: #566

after merging the above PR you can retry your pipelines with the changes that I mentioned here: #492 (comment)

I checked that and it worked on my fork :)