Skip to content

Support downloading preview SRPMs #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
schmittjoseph merged 3 commits into from
Sep 30, 2020
Merged

Support downloading preview SRPMs #160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ad971db
Replace SRPM_URL* with SRPM_URL_LIST
schmittjoseph Sep 29, 2020
a268af2
Split urls onto new lines
schmittjoseph Sep 30, 2020
a81ff8a
Fix srpm url priorities
schmittjoseph Sep 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions toolkit/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,9 @@ endif
SOURCE_URL ?=
PACKAGE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/$(build_arch)/rpms
PACKAGE_UPDATE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/$(build_arch)/rpms
SRPM_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/srpms
SRPM_UPDATE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/srpms
SRPM_URL_LIST ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/srpms \
https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/srpms \
https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/preview/srpms
REPO_LIST ?=
CA_CERT ?=
TLS_CERT ?=
Expand Down
24 changes: 8 additions & 16 deletions toolkit/docs/building/building.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,7 @@
- [`SOURCE_URL=...`](#source_url)
- [`PACKAGE_URL=...`](#package_url)
- [`PACKAGE_UPDATE_URL=...`](#package_update_url)
- [`SRPM_URL=...`](#srpm_url)
- [`SRPM_UPDATE_URL=...`](#srpm_update_url)
- [`SRPM_URL_LIST=...`](#srpm_url_list)
- [`REPO_LIST=...`](#repo_list)
- [Build Enable/Disable Flags](#build-enabledisable-flags)
- [`REBUILD_TOOLCHAIN=...`](#rebuild_toolchain)
Expand Down Expand Up @@ -292,8 +291,7 @@ Direct file downloads are by default pulled from:
SOURCE_URL ?=
PACKAGE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/$(build_arch)/rpms
PACKAGE_UPDATE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/$(build_arch)/rpms
SRPM_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/srpms
SRPM_UPDATE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/srpms
SRPM_URL_LIST ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/srpms https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/srpms https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/preview/srpms
```

While `tdnf` uses a list of repo files:
Expand Down Expand Up @@ -323,8 +321,7 @@ The build system can operate without using pre-built components if desired. Ther
SOURCE_URL ?=
PACKAGE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/$(build_arch)/rpms
PACKAGE_UPDATE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/$(build_arch)/rpms
SRPM_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/srpms
SRPM_UPDATE_URL ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/srpms
SRPM_URL_LIST ?= https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/srpms https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/srpms https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/preview/srpms
REPO_LIST ?=
```

Expand Down Expand Up @@ -355,7 +352,7 @@ sudo make go-tools REBUILD_TOOLS=y
# then rebuild the toolchain properly using the provided sources
# NOTE: Source files must made available via one of:
# - `SOURCE_URL=<YOUR_SOURCE_SERVER>`
# - DOWNLOAD_SRPMS=y (will download pre-packages sources from SRPM_URL=... and SRPM_UPDATE_URL=...)
# - DOWNLOAD_SRPMS=y (will download pre-packages sources from SRPM_URL_LIST=...)
# - manually placing the correct sources in each /SPECS/* package folder
# (SRPM_FILE_SIGNATURE_HANDLING=update must be used if the new sources files to not match the existing hashes)
sudo make toolchain PACKAGE_URL="" PACKAGE_UPDATE_URL="" REPO_LIST="" DISABLE_UPSTREAM_REPOS=y REBUILD_TOOLCHAIN=y REBUILD_TOOLS=y
Expand All @@ -365,7 +362,7 @@ sudo make toolchain PACKAGE_URL="" PACKAGE_UPDATE_URL="" REPO_LIST="" DISABLE_UP
# Complete rebuild of all tool, package, and image files from source.
# NOTE: Source files must made available via one of:
# - `SOURCE_URL=<YOUR_SOURCE_SERVER>`
# - DOWNLOAD_SRPMS=y (will download pre-packages sources from SRPM_URL=... and SRPM_UPDATE_URL=...)
# - DOWNLOAD_SRPMS=y (will download pre-packages sources from SRPM_URL_LIST=...)
# - manually placing the correct sources in each /SPECS/* package folder
# (SRPM_FILE_SIGNATURE_HANDLING=update must be used if the new sources files to not match the existing hashes)
sudo make image PACKAGE_URL="" PACKAGE_UPDATE_URL="" REPO_LIST="" DISABLE_UPSTREAM_REPOS=y REBUILD_TOOLCHAIN=y REBUILD_PACKAGES=y REBUILD_TOOLS=y
Expand All @@ -391,13 +388,9 @@ If that is not desired all remote sources can be disabled by clearing the follow

> URL to download RPM packages from if not found under `$(PACKAGE_URL)` and `$(USE_UPDATE_REPO)` is set to `y`, used to populate the toolchain packages if they are missing.

#### `SRPM_URL=...`
#### `SRPM_URL_LIST=...`

> URL to download packed SRPM packages from prior to build if `$(DOWNLOAD_SRPMS)` is set to `y`.

#### `SRPM_UPDATE_URL=...`

> URL to download updated versions of packed SRPM packages from prior to build if `$(DOWNLOAD_SRPMS)` is set to `y`.
> Space seperated list of URLs to download packed SRPM packages from prior to build if `$(DOWNLOAD_SRPMS)` is set to `y`.

#### `REPO_LIST=...`

Expand Down Expand Up @@ -616,8 +609,7 @@ To reproduce an ISO build, run the same make invocation as before, but set:
| Variable | Default | Description
|:------------------------------|:---------------------------------------------------------------------------------------------------------|:---
| SOURCE_URL | | URL to request package sources from
| SRPM_URL | `https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/srpms` | URL to request packed SRPMs from if `$(DOWNLOAD_SRPMS)` is set to `y`
| SRPM_UPDATE_URL | `https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/srpms` | URL to request updated versions of packed SRPMs from if `$(DOWNLOAD_SRPMS)` is set to `y`
| SRPM_URL_LIST | `https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/srpms https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/srpms https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/preview/srpms` | Space seperated list of URLs to request packed SRPMs from if `$(DOWNLOAD_SRPMS)` is set to `y`
| PACKAGE_URL | `https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/base/$(build_arch)/rpms` | URL to request full toolchain packages from
| PACKAGE_UPDATE_URL | `https://packages.microsoft.com/cbl-mariner/$(RELEASE_MAJOR_ID)/prod/update/$(build_arch)/rpms` | URL to request full toolchain packages from if not found under `$(PACKAGE_URL)` and `$(USE_UPDATE_REPO)` is set to `y`
| REPO_LIST | | Space separated list of repo files for tdnf to pull packages form
Expand Down
4 changes: 1 addition & 3 deletions toolkit/scripts/srpm_pack.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,11 @@ $(BUILD_SRPMS_DIR): $(STATUS_FLAGS_DIR)/build_srpms.flag
@echo Finished updating $@

ifeq ($(DOWNLOAD_SRPMS),y)
srpm_download_urls := $(SRPM_URL) $(SRPM_UPDATE_URL)

$(STATUS_FLAGS_DIR)/build_srpms.flag: $(local_specs) $(local_spec_dirs) $(SPECS_DIR)
for spec in $(local_specs); do \
spec_file=$${spec} && \
srpm_file=$$(rpmspec -q $${spec_file} --srpm --define='with_check 1' --define='dist $(DIST_TAG)' --queryformat %{NAME}-%{VERSION}-%{RELEASE}.src.rpm) && \
for url in $(srpm_download_urls); do \
for url in $(SRPM_URL_LIST); do \
Comment thread
schmittjoseph marked this conversation as resolved.
wget $${url}/$${srpm_file} \
-O $(BUILD_SRPMS_DIR)/$${srpm_file} \
--no-verbose \
Expand Down