Skip to content

Enable webhook leader election. #8386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
knative-prow-robot merged 2 commits into from
Jun 22, 2020
Merged

Enable webhook leader election. #8386

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
41267d2
Enable webhook leader election.
mattmoor Jun 19, 2020
3ebd764
Fix admission v1 merge conflict
mattmoor Jun 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions config/core/configmaps/leader-election.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ metadata:
labels:
serving.knative.dev/release: devel
annotations:
knative.dev/example-checksum: ecc94014
knative.dev/example-checksum: 2a1bda27
data:
_example: |
################################
Expand Down Expand Up @@ -63,4 +63,4 @@ data:
# - certcontroller
# - istiocontroller
# - nscontroller
enabledComponents: "controller,hpaautoscaler,certcontroller,istiocontroller,nscontroller"
enabledComponents: "controller,hpaautoscaler,certcontroller,istiocontroller,nscontroller,webhook"
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ require (
k8s.io/kube-openapi v0.0.0-20200410145947-bcb3869e6f29
knative.dev/caching v0.0.0-20200606210318-787aec80f71c
knative.dev/networking v0.0.0-20200619041525-1faac2ec5d38
knative.dev/pkg v0.0.0-20200619020725-7df8fc5d7743
knative.dev/test-infra v0.0.0-20200618184825-a7b2980a8884
knative.dev/pkg v0.0.0-20200619182625-b6a13e2894ee
knative.dev/test-infra v0.0.0-20200619200026-0b0587234302
)

replace (
Expand Down
4 changes: 4 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1398,6 +1398,8 @@ knative.dev/pkg v0.0.0-20200611204322-2ddcfef739a2/go.mod h1:rA+FklsrVahwF4a+D63
knative.dev/pkg v0.0.0-20200618002824-96c250871fac/go.mod h1:4ty6MSlNjZk5qBaGb3Gt4gopjMD4gRknfTABblcFpQ8=
knative.dev/pkg v0.0.0-20200619020725-7df8fc5d7743 h1:W1NKMizoXYYX5e2mkFXnn21T7X6ROKKwL8YetGu7xCQ=
knative.dev/pkg v0.0.0-20200619020725-7df8fc5d7743/go.mod h1:DquzK0hsLDcg2q63Sn+CngAyRwv4cKMpt5F19YzBfb0=
knative.dev/pkg v0.0.0-20200619182625-b6a13e2894ee h1:nudgQQo72NJuzaB4gBHE7Vm2BJVBTmpRt7M2psgvKxY=
knative.dev/pkg v0.0.0-20200619182625-b6a13e2894ee/go.mod h1:DquzK0hsLDcg2q63Sn+CngAyRwv4cKMpt5F19YzBfb0=
knative.dev/sample-controller v0.0.0-20200510050845-bf7c19498b7e/go.mod h1:D2ZDLrR9Dq9LiiVN7TatzI7WMcEPgk1MHbbhgBKE6W8=
knative.dev/test-infra v0.0.0-20200407185800-1b88cb3b45a5/go.mod h1:xcdUkMJrLlBswIZqL5zCuBFOC22WIPMQoVX1L35i0vQ=
knative.dev/test-infra v0.0.0-20200505052144-5ea2f705bb55/go.mod h1:WqF1Azka+FxPZ20keR2zCNtiQA1MP9ZB4BH4HuI+SIU=
Expand All @@ -1412,6 +1414,8 @@ knative.dev/test-infra v0.0.0-20200615231324-3a016f44102c/go.mod h1:+BfrTJpc++rH
knative.dev/test-infra v0.0.0-20200617235125-6382dba95484/go.mod h1:+BfrTJpc++rH30gX/C0QY6NT2eYVzycll52uw6CrQnc=
knative.dev/test-infra v0.0.0-20200618184825-a7b2980a8884 h1:qGxu/U/8VxhAuyFedrrne4s0vfY+YfoRwJJCY0AKpbw=
knative.dev/test-infra v0.0.0-20200618184825-a7b2980a8884/go.mod h1:qeiTuhDKO/HHheqVfepbxy5/q+O9toSJW6CO/DgjxFY=
knative.dev/test-infra v0.0.0-20200619200026-0b0587234302 h1:nGw173QprRCSZab6KT4uSj0GTp3WNRo0nfk9Lpo3F1I=
knative.dev/test-infra v0.0.0-20200619200026-0b0587234302/go.mod h1:H8QEB2Y35+vAuVtDbn7QBD+NQr9zQbbxNiovCLNH7F4=
modernc.org/cc v1.0.0/go.mod h1:1Sk4//wdnYJiUIxnW8ddKpaOJCF37yAdqYnkxUpaYxw=
modernc.org/golex v1.0.0/go.mod h1:b/QX9oBD/LhixY6NDh+IdGv17hgB+51fET1i2kPSmvk=
modernc.org/mathutil v1.0.0/go.mod h1:wU0vUrJsVWBZ4P6e7xtFJEhFSNsfRLJ8H458uRjg03k=
Expand Down
5 changes: 3 additions & 2 deletions pkg/client/injection/reconciler/autoscaling/v1alpha1/metric/reconciler.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions pkg/client/injection/reconciler/autoscaling/v1alpha1/podautoscaler/reconciler.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions pkg/client/injection/reconciler/serving/v1/configuration/reconciler.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions pkg/client/injection/reconciler/serving/v1/revision/reconciler.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions pkg/client/injection/reconciler/serving/v1/route/reconciler.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions pkg/client/injection/reconciler/serving/v1/service/reconciler.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion pkg/leaderelection/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ var (
"hpaautoscaler",
"certcontroller",
"istiocontroller",
"nscontroller")
"nscontroller",
"webhook",
)
)

// ValidateConfig enriches the leader election config validation
Expand Down
2 changes: 1 addition & 1 deletion pkg/leaderelection/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ func TestValidateConfig(t *testing.T) {
data["enabledComponents"] = "controller,frobulator"
return data
}(),
err: errors.New(`invalid enabledComponent "frobulator": valid values are ["certcontroller" "controller" "hpaautoscaler" "istiocontroller" "nscontroller"]`),
err: errors.New(`invalid enabledComponent "frobulator": valid values are ["certcontroller" "controller" "hpaautoscaler" "istiocontroller" "nscontroller" "webhook"]`),
}}

for _, tc := range cases {
Expand Down
73 changes: 39 additions & 34 deletions test/e2e-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,47 @@ if (( HTTPS )); then
fi

# Enable allow-zero-initial-scale before running e2e tests (for test/e2e/initial_scale_test.go)
kubectl -n ${SYSTEM_NAMESPACE} patch configmap/config-autoscaler --type=merge --patch='{"data":{"allow-zero-initial-scale":"true"}}'
kubectl -n ${SYSTEM_NAMESPACE} patch configmap/config-autoscaler --type=merge --patch='{"data":{"allow-zero-initial-scale":"true"}}' || failed=1
add_trap "kubectl -n ${SYSTEM_NAMESPACE} patch configmap/config-autoscaler --type=merge --patch='{\"data\":{\"allow-zero-initial-scale\":\"false\"}}'" SIGKILL SIGTERM SIGQUIT

kubectl -n "${SYSTEM_NAMESPACE}" patch configmap/config-leader-election --type=merge \
--patch='{"data":{"enabledComponents":"controller,hpaautoscaler,webhook", "buckets": "10"}}' || failed=1
add_trap "kubectl get cm config-leader-election -n ${SYSTEM_NAMESPACE} -oyaml | sed '/.*enabledComponents.*/d' | kubectl replace -f -" SIGKILL SIGTERM SIGQUIT

# Save activator HPA original values for later use.
min_replicas=$(kubectl get hpa activator -n "${SYSTEM_NAMESPACE}" -ojsonpath='{.spec.minReplicas}')
max_replicas=$(kubectl get hpa activator -n "${SYSTEM_NAMESPACE}" -ojsonpath='{.spec.maxReplicas}')
Comment on lines +94 to +95
Copy link
Copy Markdown
Contributor

@vagababov vagababov Jun 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

kubectl get hpa activator -n "${SYSTEM_NAMESPACE}" -ojsonpath='"minReplicas": {.spec.minReplicas} "maxReplicas": {.spec.maxReplicas}'

will give you the innards directly

Copy link
Copy Markdown
Member Author

@mattmoor mattmoor Jun 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am just moving this 🤷

Copy link
Copy Markdown
Contributor

@vagababov vagababov Jun 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah but why not improve things.

hpa_template='{"spec": {"maxReplicas": %s, "minReplicas": %s}}'

kubectl patch hpa activator -n "${SYSTEM_NAMESPACE}" \
--type "merge" \
--patch "$(printf "$hpa_template" "2" "2")" || failed=1
add_trap "kubectl patch hpa activator -n ${SYSTEM_NAMESPACE} \
--type 'merge' \
--patch $(printf "$hpa_template" "$max_replicas" "$min_replicas")" SIGKILL SIGTERM SIGQUIT

for deployment in controller autoscaler-hpa webhook; do
# Make sure all pods run in leader-elected mode.
kubectl -n "${SYSTEM_NAMESPACE}" scale deployment "$deployment" --replicas=0 || failed=1
# Give it time to kill the pods.
sleep 5
# Scale up components for HA tests
kubectl -n "${SYSTEM_NAMESPACE}" scale deployment "$deployment" --replicas=2 || failed=1
done
add_trap "for deployment in controller autoscaler-hpa webhook; do \
kubectl -n ${SYSTEM_NAMESPACE} scale deployment $deployment --replicas=0; \
kubectl -n ${SYSTEM_NAMESPACE} scale deployment $deployment --replicas=1; done" SIGKILL SIGTERM SIGQUIT

# Wait for a new leader Controller to prevent race conditions during service reconciliation
wait_for_leader_controller || failed=1

# Dump the leases post-setup.
header "Leaders"
kubectl get lease -n "${SYSTEM_NAMESPACE}"

# Give the controller time to sync with the rest of the system components.
sleep 30

# Run conformance and e2e tests.

go_test_e2e -timeout=30m \
Expand All @@ -105,7 +143,6 @@ if (( HTTPS )); then
turn_off_auto_tls
fi


# Certificate conformance tests must be run separately
# because they need cert-manager specific configurations.
kubectl apply -f ${TMP_DIR}/test/config/autotls/certmanager/selfsigned/
Expand All @@ -130,38 +167,6 @@ if [[ -n "${ISTIO_VERSION}" ]]; then
fi

# Run HA tests separately as they're stopping core Knative Serving pods
kubectl -n "${SYSTEM_NAMESPACE}" patch configmap/config-leader-election --type=merge \
--patch='{"data":{"enabledComponents":"controller,hpaautoscaler"}}'
add_trap "kubectl get cm config-leader-election -n ${SYSTEM_NAMESPACE} -oyaml | sed '/.*enabledComponents.*/d' | kubectl replace -f -" SIGKILL SIGTERM SIGQUIT

# Save activator HPA original values for later use.
min_replicas=$(kubectl get hpa activator -n "${SYSTEM_NAMESPACE}" -ojsonpath='{.spec.minReplicas}')
max_replicas=$(kubectl get hpa activator -n "${SYSTEM_NAMESPACE}" -ojsonpath='{.spec.maxReplicas}')
hpa_template='{"spec": {"maxReplicas": %s, "minReplicas": %s}}'

kubectl patch hpa activator -n "${SYSTEM_NAMESPACE}" \
--type "merge" \
--patch "$(printf "$hpa_template" "2" "2")"
add_trap "kubectl patch hpa activator -n ${SYSTEM_NAMESPACE} \
--type 'merge' \
--patch $(printf "$hpa_template" "$max_replicas" "$min_replicas")" SIGKILL SIGTERM SIGQUIT

for deployment in controller autoscaler-hpa; do
# Make sure all pods run in leader-elected mode.
kubectl -n "${SYSTEM_NAMESPACE}" scale deployment "$deployment" --replicas=0
# Scale up components for HA tests
kubectl -n "${SYSTEM_NAMESPACE}" scale deployment "$deployment" --replicas=2
done
add_trap "for deployment in controller autoscaler-hpa; do \
kubectl -n ${SYSTEM_NAMESPACE} scale deployment $deployment --replicas=0; \
kubectl -n ${SYSTEM_NAMESPACE} scale deployment $deployment --replicas=1; done" SIGKILL SIGTERM SIGQUIT

# Wait for a new leader Controller to prevent race conditions during service reconciliation
wait_for_leader_controller || failed=1

# Give the controller time to sync with the rest of the system components.
sleep 30

# Define short -spoofinterval to ensure frequent probing while stopping pods
go_test_e2e -timeout=15m -failfast -parallel=1 ./test/ha -spoofinterval="10ms" || failed=1

Expand Down
4 changes: 2 additions & 2 deletions vendor/k8s.io/api/admission/v1beta1/doc.go → vendor/k8s.io/api/admission/v1/doc.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading