Skip to content

Conversation

@locriandev
Copy link

Fix GOTOOLCHAIN invalid value when GO_VERSION has "v" prefix

Problem

OKD builds for ose-installer-etcd-artifacts started failing on 2026-01-16 with:

go: invalid GOTOOLCHAIN "gov1.24.11"
Go compliance shim [2329] [okd-4.21][ose-installer-etcd-artifacts]: Exited with: 1
subprocess exited with status 1
Error: building at STEP "RUN export GOFLAGS='-mod=readonly' && export GO_BUILD_FLAGS='-v' && CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 ./build.sh": exit status 1

Root Cause

The golang builder base image sets GO_VERSION=v1.24.11 as an environment variable (with "v" prefix):

ENV GO_VERSION=v1.24.11

When build.sh sources scripts/test_lib.sh, the determine_go_version() function runs:

function determine_go_version {
  GO_VERSION="${GO_VERSION:-"$(cat "${ETCD_ROOT_DIR}/.go-version")"}"
  if [ "${GOTOOLCHAIN:-auto}" != 'auto' ]; then
    :
  elif [ -n "${FORCE_HOST_GO:-}" ]; then
    export GOTOOLCHAIN='local'
  else
    GOTOOLCHAIN="go${GO_VERSION}"     # BUG: Creates "gov1.24.11"
    export GOTOOLCHAIN
  fi
}

Since GO_VERSION is already set in the environment to v1.24.11, the default substitution never triggers, and line 476 creates an invalid GOTOOLCHAIN=gov1.24.11 (with extra "v"), which Go rejects.

Why This Only Affected OKD Builds Recently

The bug existed all along, but was masked by the Go compliance shim in OpenShift builds:

Build Type __doozer_group Go Compliance Result
OpenShift (RHEL) openshift-4.21 EXEMPT: 0 → Forces GOTOOLCHAIN=local Build succeeds ✅
OKD (CentOS Stream) okd-4.21 EXEMPT: 1 → Does NOT override GOTOOLCHAIN Build fails ❌

The Go compliance shim enforces FIPS compliance for OpenShift builds by forcing GOTOOLCHAIN=local, which overrode the invalid value. OKD builds are exempt from this enforcement, exposing the bug.

When OKD builds were correctly labeled as __doozer_group=okd-4.21 (instead of the previous mislabeled openshift-4.21), the compliance shim exemption activated and the bug was exposed.

Solution

Normalize GO_VERSION by stripping any leading "v" prefix before using it in GOTOOLCHAIN:

function determine_go_version {
  GO_VERSION="${GO_VERSION:-"$(cat "${ETCD_ROOT_DIR}/.go-version")"}"
  if [ "${GOTOOLCHAIN:-auto}" != 'auto' ]; then
    :
  elif [ -n "${FORCE_HOST_GO:-}" ]; then
    export GOTOOLCHAIN='local'
  else
    # Strip "v" prefix from GO_VERSION for GOTOOLCHAIN compatibility
    GOTOOLCHAIN="go${GO_VERSION#v}"
    export GOTOOLCHAIN
  fi
}

This ensures GOTOOLCHAIN=go1.24.11 regardless of whether GO_VERSION is set to v1.24.11 or 1.24.11.

Testing

Before Fix

export GO_VERSION=v1.24.11
source scripts/test_lib.sh
echo $GOTOOLCHAIN  # Shows: gov1.24.11 ❌

After Fix

export GO_VERSION=v1.24.11
source scripts/test_lib.sh
echo $GOTOOLCHAIN  # Shows: go1.24.11 ✅

Verified by test build ose-etcd-container-v4.22.0-202601201705.p2.g5baac2e.assembly.stream.scos9

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jan 20, 2026
@openshift-ci-robot
Copy link

@locriandev: This pull request references Jira Issue OCPBUGS-74168, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Fix GOTOOLCHAIN invalid value when GO_VERSION has "v" prefix

Problem

OKD builds for ose-installer-etcd-artifacts started failing on 2026-01-16 with:

go: invalid GOTOOLCHAIN "gov1.24.11"
Go compliance shim [2329] [okd-4.21][ose-installer-etcd-artifacts]: Exited with: 1
subprocess exited with status 1
Error: building at STEP "RUN export GOFLAGS='-mod=readonly' && export GO_BUILD_FLAGS='-v' && CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 ./build.sh": exit status 1

Root Cause

The golang builder base image sets GO_VERSION=v1.24.11 as an environment variable (with "v" prefix):

ENV GO_VERSION=v1.24.11

When build.sh sources scripts/test_lib.sh, the determine_go_version() function runs:

function determine_go_version {
 GO_VERSION="${GO_VERSION:-"$(cat "${ETCD_ROOT_DIR}/.go-version")"}"
 if [ "${GOTOOLCHAIN:-auto}" != 'auto' ]; then
   :
 elif [ -n "${FORCE_HOST_GO:-}" ]; then
   export GOTOOLCHAIN='local'
 else
   GOTOOLCHAIN="go${GO_VERSION}"     # BUG: Creates "gov1.24.11"
   export GOTOOLCHAIN
 fi
}

Since GO_VERSION is already set in the environment to v1.24.11, the default substitution never triggers, and line 476 creates an invalid GOTOOLCHAIN=gov1.24.11 (with extra "v"), which Go rejects.

Why This Only Affected OKD Builds Recently

The bug existed all along, but was masked by the Go compliance shim in OpenShift builds:

Build Type __doozer_group Go Compliance Result
OpenShift (RHEL) openshift-4.21 EXEMPT: 0 → Forces GOTOOLCHAIN=local Build succeeds ✅
OKD (CentOS Stream) okd-4.21 EXEMPT: 1 → Does NOT override GOTOOLCHAIN Build fails ❌

The Go compliance shim enforces FIPS compliance for OpenShift builds by forcing GOTOOLCHAIN=local, which overrode the invalid value. OKD builds are exempt from this enforcement, exposing the bug.

When OKD builds were correctly labeled as __doozer_group=okd-4.21 (instead of the previous mislabeled openshift-4.21), the compliance shim exemption activated and the bug was exposed.

Solution

Normalize GO_VERSION by stripping any leading "v" prefix before using it in GOTOOLCHAIN:

function determine_go_version {
 GO_VERSION="${GO_VERSION:-"$(cat "${ETCD_ROOT_DIR}/.go-version")"}"
 if [ "${GOTOOLCHAIN:-auto}" != 'auto' ]; then
   :
 elif [ -n "${FORCE_HOST_GO:-}" ]; then
   export GOTOOLCHAIN='local'
 else
   # Strip "v" prefix from GO_VERSION for GOTOOLCHAIN compatibility
   GOTOOLCHAIN="go${GO_VERSION#v}"
   export GOTOOLCHAIN
 fi
}

This ensures GOTOOLCHAIN=go1.24.11 regardless of whether GO_VERSION is set to v1.24.11 or 1.24.11.

Testing

Before Fix

export GO_VERSION=v1.24.11
source scripts/test_lib.sh
echo $GOTOOLCHAIN  # Shows: gov1.24.11 ❌

After Fix

export GO_VERSION=v1.24.11
source scripts/test_lib.sh
echo $GOTOOLCHAIN  # Shows: go1.24.11 ✅

Verified by test build ose-etcd-container-v4.22.0-202601201705.p2.g5baac2e.assembly.stream.scos9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from deads2k and hasbro17 January 20, 2026 18:26
@locriandev
Copy link
Author

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jan 20, 2026
@openshift-ci-robot
Copy link

@locriandev: This PR has been marked as verified by test build [ose-etcd-container-v4.22.0-202601201705.p2.g5baac2e.assembly.stream.scos9](https://okd-build-history-okd-build-history.apps.artc2023.pc3z.p1.openshiftapps.com/build?nvr=ose-etcd-container-v4.22.0-202601201705.p2.g5baac2e.assembly.stream.scos9&outcome=success&type=image).

Details

In response to this:

/verified by test build ose-etcd-container-v4.22.0-202601201705.p2.g5baac2e.assembly.stream.scos9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@locriandev
Copy link
Author

/cherry-pick openshift-4.21

@openshift-cherrypick-robot

@locriandev: once the present PR merges, I will cherry-pick it on top of openshift-4.21 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick openshift-4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Jan 22, 2026
@locriandev
Copy link
Author

ART managed to fix OKD builds by patching their config. Changing this PR content to remove unneeded ART dockerfiles

@tjungblu
Copy link

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 22, 2026
@openshift-ci
Copy link

openshift-ci bot commented Jan 22, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: locriandev, tjungblu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants