Fix install_libstdcc(): replace invalid getDownloadUrl op, add curl -f, improve error diagnostics

[Copilot]

install_libstdcc() was failing in CI with jq: parse error: Invalid numeric literal due to two bugs: getDownloadUrl is not a public op on BinaryPackagePublishingHistory (returning an error body instead of a URL), and missing -f on curl caused HTTP 5xx responses to be silently written to files, fooling retry into thinking the call succeeded.

Changes

• Bug 1 — replace getDownloadUrl with +files/<filename> redirect:
  Use curl -fsSIL -w '%{url_effective}' against the archive's +files/<pkg>_<ver>_<arch>.deb endpoint, which performs a 303 → launchpadlibrarian.net. Sanity-check the resolved URL starts with https://launchpadlibrarian.net/ before proceeding.

  download_url=$(retry 5 10 curl -fsSIL \
      -w '%{url_effective}' \
      -o /dev/null \
      "${ppa_web_url}/+files/${deb_filename}")

• Bug 2 — add -f (+ -S) to every curl in the function:
  API GETs (-fsSG) and the .deb download (-fsSL) now fail on HTTP errors, so retry actually retries instead of forwarding error HTML to jq.

• Error diagnostics:
  JSON parsing is guarded with if ! jq ...; on failure (e.g. HTML error page), dumps the first 2048 bytes to stderr before exiting. On success, logs total_size and display_name from the API response so zero-entry regressions are immediately visible in CI logs.

• Extract ppa_web_url as a local variable alongside the existing ppa_url to avoid the hardcoded web URL inside the loop.

Original prompt

Context

The current install_libstdcc() on branch ppa-workaround (file bazel/sysroot/build_sysroot.sh) fails in CI with:

jq: parse error: Invalid numeric literal at line 1, column 3

There are two distinct bugs causing this. Fix both.

Bug 1: getDownloadUrl is not a public op on BinaryPackagePublishingHistory

The current code does:

retry 5 10 curl -sG "${self_link}" \
    --data-urlencode "ws.op=getDownloadUrl" \
    -o "$url_file"
download_url=$(jq -r '.' "$url_file")

getDownloadUrl exists on LibraryFileAlias objects, not on BinaryPackagePublishingHistory. The call returns an error response (JSON error object, sometimes an HTML page on infra failures), jq -r '.' then prints something that isn't a URL, and the subsequent curl either fails or succeeds writing nonsense to the .deb path. The "Invalid numeric literal" error fires when jq later tries to parse it.

The correct, documented way to get the librarian URL for a published binary is the +files/<filename> HTTP redirect. The Launchpad webservice serves a 303 to the launchpadlibrarian.net URL.

Replace the getDownloadUrl block with:

local download_url
download_url=$(retry 5 10 curl -fsSIL \
    -o /dev/null \
    -w '%{url_effective}' \
    "${self_link}/+files/${pkg}_${pkg_version}_${ARCH}.deb")

Notes:

• -I does HEAD (no body, just follow the redirect chain).
• -L follows the redirect.
• %{url_effective} prints the final URL after redirects — this is the launchpadlibrarian.net URL we want.
• -f makes curl fail on HTTP error (so retry actually retries).
• -s silences progress.
• -S shows errors on stderr.
• The whole pipeline is one curl invocation, so retry wraps it cleanly.

Then sanity-check the result starts with https://launchpadlibrarian.net/:

if [[ "$download_url" != https://launchpadlibrarian.net/* ]]; then
    echo "Error: unexpected download URL for ${pkg}: ${download_url}" >&2
    exit 1
fi

Bug 2: curl -s without -f silently writes error bodies to the output file

The current code uses:

retry 5 10 curl -sG "$ppa_url" ... -o "$meta_file"

Without -f, an HTTP 5xx response from api.launchpad.net makes curl exit 0 and dump the error body (often HTML, sometimes plain text) into $meta_file. retry sees exit 0, does NOT retry, and jq then parse-errors on the HTML.

Fix every curl invocation in install_libstdcc() to add -f (and -S so genuine errors surface). For the API GET calls:

retry 5 10 curl -fsSG "$ppa_url" \
    --data-urlencode "ws.op=getPublishedBinaries" \
    --data-urlencode "binary_name=\"${pkg}\"" \
    --data-urlencode "distro_arch_series=${distro_arch_series}" \
    --data-urlencode "exact_match=true" \
    --data-urlencode "status=\"Published\"" \
    --data-urlencode "order_by_date=true" \
    -o "$meta_file"

For the .deb download:

retry 5 10 curl -fsSL -o "$deb_file" "$download_url"

Add a debug log so future failures are diagnosable in one CI run

Right after the API response is captured, print the first ~300 bytes if the parse looks suspicious. The cleanest way: always log total_size and the first entry's display_name, and dump the whole file on parse failure:

local total_size
total_size=$(jq -r '.total_size // "missing"' "$meta_file" 2>/dev/null || echo "PARSE_FAILED")
if [[ "$total_size" == "PARSE_FAILED" ]] || [[ "$total_size" == "0" ]] || [[ "$total_size" == "missing" ]]; then
    echo "Error: API response for ${pkg} (${PPA_TOOLCHAIN}/${ARCH}) was empty or unparseable:" >&2
    echo "--- begin response ---" >&2
    head -c 2000 "$meta_file" >&2 || true
    echo >&2
    echo "--- end response ---" >&2
    exit 1
fi
echo "  API returned ${total_size} entries; first: $(jq -r '.entries[0].display_name' "$meta_file")"

This means if the API ever returns 0 entries again (e.g. the binary_name quoting regression), the CI log will show exactly what we got, instead of an opaque "Invalid numeric literal".

Don't pipeline inside retry (general hygiene)

The existing retry helper runs "$@" — so a shell pipeline inside an argument list won't work the way you'd expect, and set -e -o pipefail plus partial pipeline output is what produced the interleaved log mess. The fix above already avoids this (every retry call wraps a single curl with -o file), so just keep it that way: never put | jq ... directly after retry curl …. Always curl -o file then jq … file as separate steps.

Acceptance criteria

• install_libstdcc() no longer uses the non-existent getDownloadUrl op.
• Librarian URL is resolved via +files/<filename> HEAD-redirect (-fsSIL -w '%{url_effective}').
• Sanity check: resolved URL starts with https://launchpadlibrarian.net/.
• All curl invocations in install_libstdcc() use -f (and -S) so HTTP errors fail the retry attempt.
• If the API returns 0 entries or an unparseable body, the...

This pull request was created from Copilot chat.