Usage of `spec.version: latest` is not clear

[nak3]

Describe the bug

• Add latest as a special version supported by the operator #443 introduces spec.version: latest support. But it is not clear how we can use it.

Expected behavior

• The goal is that we want to use manifest with serving.knative.dev/release: devel label.

To Reproduce

1. Add nightly manifests into cmd/operator/kodata/knative-serving/latest/

$ cd <OPERATOR_SOURCE_DIRECTORY>
$ mkdir cmd/operator/kodata/knative-serving/latest/

$ wget -O cmd/operator/kodata/knative-serving/latest/crd.yaml  https://storage.googleapis.com/knative-nightly/serving/latest/serving-crds.yaml
$ wget -O cmd/operator/kodata/knative-serving/latest/core.yaml https://storage.googleapis.com/knative-nightly/serving/latest/serving-core.yaml

2. Install operator

$ ko apply -f config

3. Set version to latest

cat <<-EOF | kubectl apply -f -
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: ks
spec:
  version: latest
EOF

4. Operator fails to start

$ kubectl logs knative-operator-7b86d56f8c-9r9mm  -p
 ....
{"severity":"INFO","timestamp":"2021-04-13T07:46:06.779626756Z","logger":"knative-operator","caller":"knativeserving/knativeserving.go:99","message":"Reconciling KnativeServing","knative.dev/pod":"knative-operator-7b86d56f8c-9r9mm","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"5e702b0c-884e-442f-a959-ab8895e20fc3","knative.dev/key":"default/ks","status":{"observedGeneration":2,"conditions":[{"type":"DependenciesInstalled","status":"Unknown","lastTransitionTime":"2021-04-13T04:25:16Z"},{"type":"DeploymentsAvailable","status":"Unknown","lastTransitionTime":"2021-04-13T04:25:16Z"},{"type":"InstallSucceeded","status":"False","lastTransitionTime":"2021-04-13T04:27:10Z","reason":"Error","message":"Install failed with message: The manifests of the target version latest are not available to this release."},{"type":"Ready","status":"False","lastTransitionTime":"2021-04-13T04:27:10Z","reason":"Error","message":"Install failed with message: The manifests of the target version latest are not available to this release."},{"type":"VersionMigrationEligible","status":"True","lastTransitionTime":"2021-04-13T04:25:16Z"}]}}
panic: runtime error: slice bounds out of range [1:0]

$ kubectl describe knativeserving 
  ...
    Last Transition Time:  2021-04-13T04:27:10Z
    Message:               Install failed with message: The manifests of the target version latest are not available to this release.

Knative release version

HEAD (a2913e3)

[houshengbo]

@nak3 This feature only facilitates the prow for now. I do not think we expect end-users to use it.
In prow, it downloads the source code of eventing and serving, builds the yamls and puts them under a dir named latest in kodata.
Then, set spec.version to latest, so it will install the yamls under latest.

[houshengbo]

but one option we can offer is to use the latest available version if latest dir is not available.

[nak3]

Thank you @houshengbo

@nak3 This feature only facilitates the prow for now. I do not think we expect end-users to use it.
In prow, it downloads the source code of eventing and serving, builds the yamls and puts them under a dir named latest in kodata.
Then, set spec.version to latest, so it will install the yamls under latest.

Th prow is https://testgrid.k8s.io/knative-operator#knative-operator-continuous, correct?
As per the log and script test/e2e-common.sh, it surely does "downloads the source code of eventing and serving, builds the yamls and puts them under a dir named latest in kodata" but it never does "set spec.version to latest". So the issue I reported does not happen. Could you please confirm it?

but one option we can offer is to use the latest available version if latest dir is not available.

Umm... Sorry could you elaborate on it?

[houshengbo]

@nak3 Check on this line: https://github.com/knative/operator/blob/main/test/e2e-common.sh#L26
This is where to set the value of the target version. It can be set to "latest".

https://github.com/knative/operator/blob/main/test/e2e-common.sh#L282
This line is how we create the CR.

create_latest_custom_resource is called by the golang code: https://github.com/knative/operator/blob/main/test/upgrade/installation/shell.go#L30

[nak3]

I see. It is in upgrade test. And yeah, the bug I reported definitely happens on CI as well.

You can find the log by going to:

• the log of pull/563 -> Artifcacts -> artifacts -> f875cca3-a75c-11eb-bb19-6e5dccb14317 -> k8s.log.txt

You can see the following panic in the operator pod logs:

default/knative-operator-75c95694f7-8zxv9[knative-operator]: panic: runtime error: slice bounds out of range [1:0]
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 
default/knative-operator-75c95694f7-8zxv9[knative-operator]: goroutine 516 [running]:
default/knative-operator-75c95694f7-8zxv9[knative-operator]: knative.dev/operator/pkg/reconciler/knativeserving/ingress.getIngress(0xc0005cbb9a, 0x6, 0xc000b8f800, 0x6, 0x2c74eb8)
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 	knative.dev/operator/pkg/reconciler/knativeserving/ingress/ingress.go:86 +0x419
default/knative-operator-75c95694f7-8zxv9[knative-operator]: knative.dev/operator/pkg/reconciler/knativeserving/ingress.AppendTargetIngresses(0x201e820, 0xc000958c30, 0xc000b8f800, 0x204aa40, 0xc000e29400, 0x0, 0x0)
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 	knative.dev/operator/pkg/reconciler/knativeserving/ingress/ingress.go:98 +0x57
default/knative-operator-75c95694f7-8zxv9[knative-operator]: knative.dev/operator/pkg/reconciler/common.Stages.Execute(0xc0007198a8, 0x9, 0x9, 0x201e820, 0xc000958c30, 0xc000b8f800, 0x204aa40, 0xc000e29400, 0x0, 0x0)
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 	knative.dev/operator/pkg/reconciler/common/stages.go:37 +0x83
default/knative-operator-75c95694f7-8zxv9[knative-operator]: knative.dev/operator/pkg/reconciler/knativeserving.(*Reconciler).ReconcileKind(0xc0001fc070, 0x201e820, 0xc000958c30, 0xc000e29400, 0x0, 0x0)
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 	knative.dev/operator/pkg/reconciler/knativeserving/knativeserving.go:122 +0x878
default/knative-operator-75c95694f7-8zxv9[knative-operator]: knative.dev/operator/pkg/client/injection/reconciler/operator/v1alpha1/knativeserving.(*reconcilerImpl).Reconcile(0xc0002148c0, 0x201e820, 0xc000958c00, 0xc000e663a0, 0x20, 0xc000b1f408, 0x201e820)
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 	knative.dev/operator/pkg/client/injection/reconciler/operator/v1alpha1/knativeserving/reconciler.go:246 +0x1248
default/knative-operator-75c95694f7-8zxv9[knative-operator]: knative.dev/pkg/controller.(*Impl).processNextWorkItem(0xc00067a040, 0x0)
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 	knative.dev/pkg@v0.0.0-20210426180040-cfc1eed82870/controller/controller.go:530 +0x63e
default/knative-operator-75c95694f7-8zxv9[knative-operator]: knative.dev/pkg/controller.(*Impl).RunContext.func3(0xc000432ea0, 0xc00067a040)
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 	knative.dev/pkg@v0.0.0-20210426180040-cfc1eed82870/controller/controller.go:468 +0x53
default/knative-operator-75c95694f7-8zxv9[knative-operator]: created by knative.dev/pkg/controller.(*Impl).RunContext
default/knative-operator-75c95694f7-8zxv9[knative-operator]: 	knative.dev/pkg@v0.0.0-20210426180040-cfc1eed82870/controller/controller.go:466 +0x1a5