Skip to content

batches: Remote Mount #5 - src-cli #816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Piszmog merged 83 commits into from
Oct 6, 2022
Merged

batches: Remote Mount #5 - src-cli #816

Piszmog merged 83 commits into from
Oct 6, 2022

Conversation

@Piszmog
Copy link
Contributor

@Piszmog Piszmog commented Aug 15, 2022 •
edited
Loading

Part of #31792.

Dependent on sourcegraph#40932.

  • The remote command uploads files specified in the mount section of a step. The files are upload via the files API.
    • The files are uploaded as chunks - thus saving on memory
  • Updated the exec entry command to accept the flag workspaceFiles and to mount this directory when running Docker
  • Various other small changes to support running locally and remote. They are not super compatible.

Test plan

Go Unit tests and functional testing. Running mounts locally also confirmed to still work.

Example

Screen.Recording.2022-09-21.at.19.39.11.mov

Piszmog added 30 commits August 9, 2022 13:58
@Piszmog
Upload mounted files
573cc58
@Piszmog
Temporarily point to local sourcegraph code
2a326ce
@Piszmog
Add mount upload progress
3a214d9
@Piszmog
Specify the directory where the files are located
9f63867
@Piszmog
Remove unneeded tests
1a1b4f3
@Piszmog
Do not update the path of the mount
5d8f987
@Piszmog
Allow mount files to be ran in executor
d402823
@Piszmog
Add functions to upload files
1794e7f
@Piszmog
Save the directory of the batch spec
9d2554d
@Piszmog
Use mount directory for caching
bb9f6f4
@Piszmog
Specify the mount directory
e836e7a
@Piszmog
Remove unused parameter
0552186
@Piszmog
Remove unused parameter
ded8b16
@Piszmog
Use the batch spec directory for caching
5596cf8
@Piszmog
Get the mount directory
dc21e08
@Piszmog
Merge remote-tracking branch 'origin/main' into rc/remote-mounting
e2411fc
@Piszmog
Set the mount directory
b56d8d1
@Piszmog
Be able to upload multiple files
bd2e771
@Piszmog
Add test for uploading files
7326479
@Piszmog
Add mock implementation for api client
f6bc5ed
@Piszmog
Run go mod tidy
ff1b30d
@Piszmog
Fixes based on unit tests
7803bea
@Piszmog
Remove TODO
cab0852
@Piszmog
Remove special logic for caching mounts
c8b8608
@Piszmog
Pass the file mod time
948809b
@Piszmog
Update caching to support local and remote
dcc6607
@Piszmog
Remove mount directory
90c9892
@Piszmog
Rename from batchSpecDir to workingDirectory to fit with remote and l…
fb1903d 
…ocal usage
@Piszmog
Update dependencies
af61c2d
@Piszmog
Fix issue with mix pointer and receiver functions
69119aa
@Piszmog Piszmog changed the title batches: Remote Mount #7 - src-cli batches: Remote Mount #5 - src-cli Sep 27, 2022
@Piszmog Piszmog mentioned this pull request Sep 27, 2022
Merged
eseliger
eseliger approved these changes Oct 4, 2022
View reviewed changes
Copy link
Member

@eseliger eseliger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go!!!11

internal/batches/service/remote_windows_test.go
"github.com/sourcegraph/sourcegraph/lib/batches"
)

func TestService_UploadBatchSpecWorkspaceFiles_Windows_Path(t *testing.T) {
Copy link
Member

@eseliger eseliger Oct 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the fact that this test has to exist makes me sad 🙃

Copy link
Contributor Author

@Piszmog Piszmog Oct 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol, I had a dream about windows paths. This is the result

@Piszmog Piszmog merged commit ad1b36a into main Oct 6, 2022
@Piszmog Piszmog deleted the rc/remote-mounting branch October 6, 2022 14:24
scjohns pushed a commit that referenced this pull request Apr 24, 2023
@Piszmog
batches: Remote Mount #5 - src-cli (#816)
ee8a2b9
@chrisdruta chrisdruta mentioned this pull request Jan 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eseliger eseliger eseliger approved these changes

@LawnGnome LawnGnome Awaiting requested review from LawnGnome

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Piszmog @eseliger @LawnGnome