OCPBUGS-6861: Allow TopoLVM initial grace period while service-ca attempts to start

[copejon]

TopoLVM on MicroShift suffers flakey behavior that stems in part from a race window with the service-ca. This PR pads TopoLVM's startup time to give the service-ca a chance to issue certs. This PR sets the initial startupProbe values to a 5 minute timeout (30 retries at 10 second intervals).

The race window was exposed by flakey behavior with service-ca startup that is being addressed separately. The work there will reduce the scale of the race window, however it will still be present. The window appears during (re)boot because topolvm depends on the service-ca to issue cert. In this scenario, topolvm and service-ca are started concurrently; their boot order is not controlled by microshift. This means that both services' retry loops are initiated at the same time, so the diff between the service-ca issuing a cert and topolvm reaching it's retry max creates this race window. By providing topolvm a startup grace period, we allow the service ca to stabilize and improve the user experience since topolvm will not enter a crashloopbackoff as a result of timing out.

Note that this grace period does not delay topolvm's startup, it only delays the point at which topolvm startup failures are considered true failures.

Which issue(s) this PR addresses:

[openshift-ci-robot]

@copejon: This pull request references Jira Issue OCPBUGS-6861, which is invalid:

• expected the bug to target the "4.13.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

TopoLVM on MicroShift suffers flakey behavior that stems in part from a race window with the service-ca. This PR pads TopoLVM's startup time to give the service-ca a chance to issue certs. This PR sets the initial startupProbe values to a 5 minute timeout (30 retries at 10 second intervals).

Which issue(s) this PR addresses:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[copejon]

/pj-rehearse ack

[copejon]

/pj-rehearse

[pacevedom]

Do you have some more details on the race condition? In the logs from the bug I see there is some trouble here and there in both pods, but not clear to me.

[copejon]

/pj-rehearse

[copejon]

/pj-rehearse

[copejon]

/pj-rehearse

[ggiguash]

/jira refresh

[openshift-ci-robot]

@ggiguash: This pull request references Jira Issue OCPBUGS-6861, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.13.0) matches configured target version for branch (4.13.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@copejon: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

TopoLVM on MicroShift suffers flakey behavior that stems in part from a race window with the service-ca. This PR pads TopoLVM's startup time to give the service-ca a chance to issue certs. This PR sets the initial startupProbe values to a 5 minute timeout (30 retries at 10 second intervals).

Which issue(s) this PR addresses:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[copejon]

Hey @pacevedom I've updated the description to provide a bit more context.

I've also remove the jira reference as it's determined that the cause of that bug is not related to this patch.

[copejon]

/jira refresh

[openshift-ci-robot]

@copejon: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dhellmann]

This looks OK, except for the messy history.

@pmtk and @pacevedom , we could set the merge method to squash to take the changes. Thoughts?

[openshift-ci-robot]

@copejon: This pull request references Jira Issue OCPBUGS-6861, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.13.0) matches configured target version for branch (4.13.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

TopoLVM on MicroShift suffers flakey behavior that stems in part from a race window with the service-ca. This PR pads TopoLVM's startup time to give the service-ca a chance to issue certs. This PR sets the initial startupProbe values to a 5 minute timeout (30 retries at 10 second intervals).

The race window was exposed by flakey behavior with service-ca startup that is being addressed separately. The work there will reduce the scale of the race window, however it will still be present. The window appears during (re)boot because topolvm depends on the service-ca to issue cert. In this scenario, topolvm and service-ca are started concurrently; their boot order is not controlled by microshift. This means that both services' retry loops are initiated at the same time, so the diff between the service-ca issuing a cert and topolvm reaching it's retry max creates this race window. By providing topolvm a startup grace period, we allow the service ca to stabilize and improve the user experience since topolvm will not enter a crashloopbackoff as a result of timing out.

Note that this grace period does not delay topolvm's startup, it only delays the point at which topolvm startup failures are considered true failures.

Which issue(s) this PR addresses:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dhellmann]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: copejon, dhellmann

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [copejon,dhellmann]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 3278f55 and 2 for PR HEAD 6d4a374 in total

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 0b6dc3d and 1 for PR HEAD 6d4a374 in total

[openshift-ci]

@copejon: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-ci-robot]

@copejon: Jira Issue OCPBUGS-6861: All pull requests linked via external trackers have merged:

• openshift/microshift#1378

Jira Issue OCPBUGS-6861 has been moved to the MODIFIED state.

Details

In response to this:

TopoLVM on MicroShift suffers flakey behavior that stems in part from a race window with the service-ca. This PR pads TopoLVM's startup time to give the service-ca a chance to issue certs. This PR sets the initial startupProbe values to a 5 minute timeout (30 retries at 10 second intervals).

The race window was exposed by flakey behavior with service-ca startup that is being addressed separately. The work there will reduce the scale of the race window, however it will still be present. The window appears during (re)boot because topolvm depends on the service-ca to issue cert. In this scenario, topolvm and service-ca are started concurrently; their boot order is not controlled by microshift. This means that both services' retry loops are initiated at the same time, so the diff between the service-ca issuing a cert and topolvm reaching it's retry max creates this race window. By providing topolvm a startup grace period, we allow the service ca to stabilize and improve the user experience since topolvm will not enter a crashloopbackoff as a result of timing out.

Note that this grace period does not delay topolvm's startup, it only delays the point at which topolvm startup failures are considered true failures.

Which issue(s) this PR addresses:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[copejon]

/cherry-pick release-4.13

[openshift-cherrypick-robot]

@copejon: new pull request created: #1537

Details

In response to this:

/cherry-pick release-4.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.