Skip to content

Conversation

@thaJeztah
Copy link
Member


add local fork of github.com/docker/docker/builder/remotecontext

Adds a local fork of this package for use in the classic builder.

Code was taken at commit d33d46d01656e1d9ee26743f0c0d7779f685dd4e.

Migration was done using the following steps:

# install filter-repo (https://github.com/newren/git-filter-repo/blob/main/INSTALL.md)
brew install git-filter-repo

# create a temporary clone of docker
cd ~/Projects
git clone https://github.com/docker/docker.git build_context_temp
cd build_context_temp

# commit taken from
git rev-parse --verify HEAD
d33d46d01656e1d9ee26743f0c0d7779f685dd4e

git filter-repo --analyze

# remove all code, except for the remotecontext packages, and move to build/internal docs and previous locations of it
git filter-repo \
  --path 'builder/remotecontext/git' \
  --path 'builder/remotecontext/urlutil' \
  --path-rename builder/remotecontext:cli/command/image/build/internal


# go to the target repository
cd ~/go/src/github.com/docker/cli

# create a branch to work with
git checkout -b fork_remotecontext

# add the temporary repository as an upstream and make sure it's up-to-date
git remote add build_context_temp ~/Projects/build_context_temp
git fetch build_context_temp

# merge the upstream code
git merge --allow-unrelated-histories --signoff -S build_context_temp/master

- Human readable description for the release notes

- A picture of a cute animal (not mandatory but encouraged)

dnephin and others added 30 commits June 2, 2017 16:54
Signed-off-by: Daniel Nephin <dnephin@docker.com>
`docker build` accepts remote repositories
using either the `git://` notation, or `git@`.

Docker attempted to parse both as an URL, however,
`git@` is not an URL, but an argument to `git clone`.

Go 1.7 silently ignored this, and managed to
extract the needed information from these
remotes, however, Go 1.8 does a more strict
validation, and invalidated these.

This patch adds a different path for `git@` remotes,
to prevent them from being handled as URL (and
invalidated).

A test is also added, because there were no
tests for handling of `git@` remotes.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This function was only used inside gitutils,
and is written specifically for the requirements
there.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If the HEAD request fails, use a GET request to properly test if git
server is smart-http.

Signed-off-by: Andrew He <he.andrew.mail@gmail.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Daniel Nephin <dnephin@docker.com>
gty-migrate-from-testify --ignore-build-tags

Signed-off-by: Daniel Nephin <dnephin@docker.com>
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 723b107ca4fba14580a6cd971e63d8af2e7d2bbe)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
gitutils: add validation for ref
```
builder/remotecontext/remote.go:48:        G107: Potential HTTP request made with variable url (gosec)
builder/remotecontext/git/gitutils.go:145: G107: Potential HTTP request made with variable url (gosec)
builder/remotecontext/git/gitutils.go:147: G107: Potential HTTP request made with variable url (gosec)
pkg/fileutils/fileutils_test.go:185:       G303: File creation in shared tmp directory without using ioutil.Tempfile (gosec)
pkg/tarsum/tarsum_test.go:7:               G501: Blacklisted import `crypto/md5`: weak cryptographic primitive (gosec)
pkg/tarsum/tarsum_test.go:9:               G505: Blacklisted import `crypto/sha1`: weak cryptographic primitive (gosec)
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Replace gometalinter with golangci-lint
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
urlutil.IsUrl() was merely checking if the url had a http(s)://
prefix, which is just as well handled through using url.Parse()

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: gotestyourself/gotest.tools@v2.3.0...v3.0.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
replace pkg/symlink with github.com/moby/sys/symlink
Signed-off-by: Tibor Vass <tibor@docker.com>
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
pkg/urlutil (despite its poorly chosen name) is not really intended as a generic
utility to handle URLs, and should only be used by the builder to handle (remote)
build contexts.

- IsURL() only does a very rudimentary check for http(s):// prefixes, without any
  other validation, but due to its name may give incorrect expectations.
- IsGitURL() is written specifically with docker build remote git contexts in
  mind, and has handling for backward-compatibility, where strings that are
  not URLs, but start with "github.com/" are accepted.

Because of the above, this patch:

- moves the package inside builder/remotecontext, close to where it's intended
  to be used (ideally this would be part of build/remotecontext itself, but this
  package imports many other dependencies, which would introduce those as extra
  dependencies in the CLI).
- deprecates pkg/urlutil, but adds aliases as there are some external consumers.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Simplify some of the logic, and add documentation about the package,
as well as warnings that this package should not be used as a general-
purpose utility.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
pkg/urlutil: deprecate, and move to builder/remotecontext/urlutil
Older versions of Go don't format comments, so committing this as
a separate commit, so that we can already make these changes before
we upgrade to Go 1.19.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Make the test more debuggable by logging all git command output and
running each table-driven test case as a subtest.

Signed-off-by: Cory Snider <csnider@mirantis.com>
Keep It Simple! Set the working directory for git commands by...setting
the git process's working directory. Git commands can be run in the
parent process's working directory by passing the empty string.

Signed-off-by: Cory Snider <csnider@mirantis.com>
Prevent git commands we run from reading the user or system
configuration, or cloning submodules from the local filesystem.

Signed-off-by: Cory Snider <csnider@mirantis.com>
@thaJeztah
Copy link
Member Author

Fixed; had that done in my previous attempt, but should be good now

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

📢 Thoughts on this report? Let us know!

@thaJeztah thaJeztah force-pushed the fork_remotecontext branch from 1d1d0b2 to ed9ea2c Compare July 16, 2025 14:10
Adds a local fork of this package for use in the classic builder.

Code was taken at commit [d33d46d01656e1d9ee26743f0c0d7779f685dd4e][1].

Migration was done using the following steps:

    # install filter-repo (https://github.com/newren/git-filter-repo/blob/main/INSTALL.md)
    brew install git-filter-repo

    # create a temporary clone of docker
    cd ~/Projects
    git clone https://github.com/docker/docker.git build_context_temp
    cd build_context_temp

    # commit taken from
    git rev-parse --verify HEAD
    d33d46d01656e1d9ee26743f0c0d7779f685dd4e

    git filter-repo --analyze

    # remove all code, except for the remotecontext packages, and move to build/internal docs and previous locations of it
    git filter-repo \
      --path 'builder/remotecontext/git' \
      --path 'builder/remotecontext/urlutil' \
      --path-rename builder/remotecontext:cli/command/image/build/internal

    # go to the target repository
    cd ~/go/src/github.com/docker/cli

    # create a branch to work with
    git checkout -b fork_remotecontext

    # add the temporary repository as an upstream and make sure it's up-to-date
    git remote add build_context_temp ~/Projects/build_context_temp
    git fetch build_context_temp

    # merge the upstream code
    git merge --allow-unrelated-histories --signoff -S build_context_temp/master

[1]: https://github.com/docker/docker/d33d46d01656e1d9ee26743f0c0d7779f685dd4e

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah force-pushed the fork_remotecontext branch from ed9ea2c to b8c794b Compare July 16, 2025 14:20
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Copy link
Member Author

Looks like we may need to add some extra things in the dev container;

#18 66.08 === FAIL: cli/command/image/build/internal/git TestCheckoutGit (0.14s)
#18 66.08     gitutils_test.go:179: git version 2.49.1
#18 66.08     gitutils_test.go:239: Initialized empty Git repository in /tmp/TestCheckoutGit3938753936/001/repo/.git/
#18 66.08     gitutils_test.go:254: [master (root-commit) 2a3382f] First commit
#18 66.08          4 files changed, 5 insertions(+)
#18 66.08          create mode 100644 Dockerfile
#18 66.08          create mode 120000 absolutelink
#18 66.08          create mode 120000 parentlink
#18 66.08          create mode 100644 subdir/Dockerfile
#18 66.08     gitutils_test.go:255: Switched to a new branch 'test'
#18 66.08     gitutils_test.go:261: [test 5094963] Branch commit
#18 66.08          2 files changed, 3 insertions(+), 2 deletions(-)
#18 66.08     gitutils_test.go:262: Switched to branch 'master'
#18 66.08     gitutils_test.go:266: Initialized empty Git repository in /tmp/TestCheckoutGit3938753936/001/subrepo/.git/
#18 66.08     gitutils_test.go:273: [master (root-commit) 0922701] Subrepo initial
#18 66.08          1 file changed, 1 insertion(+)
#18 66.08          create mode 100644 subfile
#18 66.08 2025/07/16 14:30:17 cgi: no headers
#18 66.08     gitutils_test.go:210: git-http-backend: git: 'http-backend' is not a git command. See 'git --help'.
#18 66.08         
#18 66.08     gitutils_test.go:210: git-http-backend: 
#18 66.08     gitutils_test.go:275: Cloning into '/tmp/TestCheckoutGit3938753936/001/repo/sub'...
#18 66.08         fatal: unable to access 'http://127.0.0.1:44973/subrepo/': The requested URL returned error: 500
#18 66.08         fatal: clone of 'http://127.0.0.1:44973/subrepo' into submodule path '/tmp/TestCheckoutGit3938753936/001/repo/sub' failed
#18 66.08     gitutils_test.go:275: assertion failed: error is not nil: exit status 128

@thaJeztah thaJeztah force-pushed the fork_remotecontext branch from 0ccc247 to 016137d Compare July 16, 2025 14:44
    gitutils_test.go:210: git-http-backend: git: 'http-backend' is not a git command. See 'git --help'.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah force-pushed the fork_remotecontext branch from 016137d to b05aa46 Compare July 16, 2025 14:49
@thaJeztah
Copy link
Member Author

Another flaky test; looks like we may not be matching errors correctly (or just race condition) 🤔

65.72 === Failed
65.72 === FAIL: cli/command/plugin TestUpgradePromptTermination (0.00s)
65.72     upgrade_test.go:37: assertion failed: error is "plugin upgrade has been cancelled" (plugin.cancelledErr), not "prompt terminated" (prompt.ErrTerminated prompt.cancelledErr)

@thaJeztah thaJeztah added this to the 29.0.0 milestone Jul 16, 2025
@thaJeztah thaJeztah marked this pull request as ready for review July 16, 2025 15:02
@thaJeztah
Copy link
Member Author

@Benehiko @vvoland this should be ready for review

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah merged commit 8c317ad into docker:master Jul 16, 2025
87 checks passed
@thaJeztah thaJeztah deleted the fork_remotecontext branch July 16, 2025 23:34
thaJeztah added a commit to thaJeztah/cli that referenced this pull request Aug 14, 2025
add local fork of github.com/docker/docker/builder/remotecontext

(cherry picked from commit 8c317ad)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah added a commit to thaJeztah/cli that referenced this pull request Aug 14, 2025
add local fork of github.com/docker/docker/builder/remotecontext

(cherry picked from commit 8c317ad)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.