[question]: Support of pod spec fields

[alexandrst88]

Hi! Thanks, for this cool tool. I have few questions, do you have a plan to add pod annotations and servicaccount to runners?

spec:
  replicas: 3
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: role-arn```

I'd like to build docker images and push them to ecr or have ability to access k8s cluster api for example as use case.

If nope, I could work on this, thanks

[mumoshu]

@alexandrst88 Hey! I believe this being a valid feature request. Probably adding annotations next to https://github.com/summerwind/actions-runner-controller/blob/0edf0d59f74bcbbc7c9bd899c6547c2a49a9b1b5/api/v1alpha1/runner_types.go#L34 would make sense.

cc/ @summerwind

[summerwind]

Thank you for the feedback!
That's nice idea! I'm also considering a similar feature. In my use case, I'd like to set Resource Limits.

However, managing each field of a pod individually can be complicated. How about adding a podTemplate field to Runner?

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: example
spec:
  repository: summerwind/actions-runner-controller
  podTemplate:
    metadata:
      annotations:
        iam.amazonaws.com/role: role-arn
    spec:
      serviceAccountName: example
      ...

[summerwind]

I'm currently working on StatefulSet based runner management in #4. After that, it might be a good time to implement podTemplate.

[alexandrst88]

I might me wrong, but in runner_controller.go

pod := corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{
			Name:      runner.Name,
			Namespace: runner.Namespace,
		},
		Spec: corev1.PodSpec{
			RestartPolicy: "OnFailure",
			Containers: []corev1.Container{
				{
					Name:            containerName,
					Image:           runnerImage,
					ImagePullPolicy: "Always",
					Env:             env,
					VolumeMounts: []corev1.VolumeMount{
						{
							Name:      "docker",
							MountPath: "/var/run",
						},
					},
					SecurityContext: &corev1.SecurityContext{
						RunAsGroup: &group,
					},
				},
				{
					Name:  "docker",
					Image: r.DockerImage,
					VolumeMounts: []corev1.VolumeMount{
						{
							Name:      "docker",
							MountPath: "/var/run",
						},
					},
					SecurityContext: &corev1.SecurityContext{
						Privileged: &privileged,
					},
				},
			},
			Volumes: []corev1.Volume{
				corev1.Volume{
					Name: "docker",
					VolumeSource: corev1.VolumeSource{
						EmptyDir: &corev1.EmptyDirVolumeSource{},
					},
				},
			},
		},
	}

there is one way to create the pod, where pod spec is just kind of hardcoded some fields like ImagePullPolicy, or RestartPolicy, if you defined,

  repository: summerwind/actions-runner-controller
  podTemplate:
    metadata:
      annotations:
        iam.amazonaws.com/role: role-arn
    spec:
      serviceAccountName: example
      ...

you need also specify all fields here all fields from the previous code snippet, but, it would be create to deepcopy or merge fields from pod template into pod creation above?

like podTemplate.Metadata.DeepCopy(pod.Metadata) and
podTemplate.Spec.DeepCopy(pod.Spec)

[mumoshu]

you need also specify all fields here all fields from the previous code snippet, but, it would be create to deepcopy or merge fields from pod template into pod creation above?

like podTemplate.Metadata.DeepCopy(pod.Metadata) and
podTemplate.Spec.DeepCopy(pod.Spec)

Yeah, that would work!

We'd ideally want "strategic-merge-patch" on containers, volumes, and volumeMounts. But that seems like a YAGNI and we can defer it to another github issue/feature request.

So, in the meantime, I believe you can just deepcopy the pod template into a new pod spec, and set default values to the missing fields.

For instance, when the pod template in the runner spec misses RestartPolicy, you'd set it to OnFailure as we already do. Similarly, if the pod template misses Containers[], you'd set it to the two containers as already do.

[alexandrst88]

Funny thing, DeepCopy it's actually removed existing ones, fields like it wiped out containers fields if I choose it in annotations.

Second, one Kubernetes also checking embedded fields spec, like if you define podTemplate.

  podTemplate:
    metadata:
      annotations:
        iam.amazonaws.com/role: role-arn
    template:
      spec:
        serviceAccountName: testing

Kuberenetes api validation force you also specify container fileds :)

lol, it seems like here elastic/cloud-on-k8s#1822, they removed validation from CRD.

what do you think about just pod creation into CRD within runner?

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:

or just skip validation like in https://github.com/elastic/cloud-on-k8s, because creation of pod will be control by api validation too.

it will hard to maintain of fields of PodTemplate, because k8s will update API and add or remove f fields.

[missedone]

wow, looking forward for this feature as well, i'd like to set iam role to the runner pod with podTemplate.

what's the status of PR #7? i can contribute as well.

thanks again for the nice operator.

[missedone]

re-comment #5 (comment)

I think it depends on what's the principle we want to follow, 1) either minimize the custom podTemplate for user, or 2) always enforce the user to explicitly specify the full runner podTemplate.
i'd prefer option 2, since it makes people very clear on what's containers spec they are using. Ex. what's the version of runner docker image, whether need extra docker container, what's the runner container resource spec, etc.
just my 2 cents.

Thanks,
Nick

[missedone]

BTW, @alexandrst88 , regards to the pod spec merge logic, yes, if you copy the custom podTemplate into the controller pod instance, the containers spec gets lost.
so I think we can implement the merge logic in other way, saying merge controller managed pod instance to the custom pod.

something like this (there is still space to improve the code below):

	// copy custom podTemplate
	pod := corev1.Pod{}
	if runner.Spec.PodTemplate.ObjectMeta.Size() > 0 {
		runner.Spec.PodTemplate.ObjectMeta.DeepCopyInto(&pod.ObjectMeta)
	}
	if runner.Spec.PodTemplate.Template.Spec.Size() > 0 {
		runner.Spec.PodTemplate.Template.Spec.DeepCopyInto(&pod.Spec)
	}

	// merge runner specific pod spec
	pod.ObjectMeta.Name = runner.Name
	if pod.ObjectMeta.Namespace == "" {
		pod.ObjectMeta.Namespace = runner.Namespace
	}
	if len(pod.Spec.Containers) == 0 {
		pod.Spec.Containers = []corev1.Container{
			{
				Name:            containerName,
				Image:           runnerImage,
				ImagePullPolicy: "Always",
				Env:             env,
				VolumeMounts: []corev1.VolumeMount{
					{
						Name:      "docker",
						MountPath: "/var/run",
					},
				},
				SecurityContext: &corev1.SecurityContext{
					RunAsGroup: &group,
				},
			},
			{
				Name:  "docker",
				Image: r.DockerImage,
				VolumeMounts: []corev1.VolumeMount{
					{
						Name:      "docker",
						MountPath: "/var/run",
					},
				},
				SecurityContext: &corev1.SecurityContext{
					Privileged: &privileged,
				},
			},
		}
		pod.Spec.Volumes = append(pod.Spec.Volumes, []corev1.Volume{
			corev1.Volume{
				Name: "docker",
				VolumeSource: corev1.VolumeSource{
					EmptyDir: &corev1.EmptyDirVolumeSource{},
				},
			},
		}...)
		pod.Spec.RestartPolicy = "OnFailure"
	} else {
		// append runner specific env vars
		for i := 0; i < len(pod.Spec.Containers); i++ {
			pod.Spec.Containers[i].Env = append(pod.Spec.Containers[i].Env, env...)
		}
	}

[alexandrst88]

yeap, @missedone thanks, it makes sense. about container field, I think, kustomize could remove this field as required in crd.

[summerwind]

Does anyone have a real use case that needs adding a sidecar to the runner pod or replacing the runner container? If not, I'd like to choose a way to just ignore the containers field in the pod template to keep controller simple.

[missedone]

Yes, for example, I’d like to build golang app, but turns out the default runner image doesn’t have make installed.
Also we have serval teams build different type of tech stack, and different requirements of tool dependency. That says I’d like to give the team flexibility to custom the runner docker image.

[summerwind]

Thank you for sharing your use case!
If you want to use custom runner image, you can use the image field ;-)

Currently the controller watches the runner's pod state and restarts the pod after actions' job is completed. This is depends on the container name. If we introduce a Pod Template, the restart process could be difficult because the container name will be dynamic value.

To solve this issue, I propose the following solution:

• Make 'Runner' resource's annotations and labels are inherited to their 'Pod' resource.
• Add serviceAccountName field to allow specifying Pod service account

The manifest based on the proposal would be:

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: example
  annotations:
    iam.amazonaws.com/role: role-arn
spec:
  repository: summerwind/actions-runner-controller
  serviceAccountName: example
  image: your/runner:latest