-
Notifications
You must be signed in to change notification settings - Fork 20
Add S3Handle #33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Add S3Handle #33
Changes from all commits
Commits
Show all changes
55 commits
Select commit
Hold shift + click to select a range
eb08c3f
Add S3Handle using minio 5.0.0
joshmoore 3289c57
Location: wider s3 support including parent/child logic
joshmoore 384e397
S3Handle: parse more complex URLs
joshmoore f464dc3
Drop JDK7 for the moment
joshmoore bfc6e88
Location: partially deal with null child arguments
joshmoore 38166c0
S3 bucket must be the first path component
manics edb69c9
Location.length return handle.length
manics 0f4abc5
S3 server and protocol are always required
manics 91bcbe1
Add in-depth traces to S3Handle
manics 028e0a0
Make default s3 protocol a var
manics abcec31
Fix S3HandleTest protocol
manics dd61bb8
S3Handle default to https, add mock object for testing
manics a7479b4
Add S3Handle.toString()
manics 817e88d
Add caching to S3Handle
manics edd1d94
Cache S3 files when BF_REMOTE_CACHE_ROOTDIR set
manics ef32294
S3Handle: only create cache dir if s3 object exists
manics c23ed87
Throw HandleException if downloading null path
manics 8920edb
Custom protocol is handled as s3+protocol:// instead of env-var
manics 4756a08
More testing/fixes to ensure consistent s3 url parsing
manics bc02711
S3Handle determines whether it can handle URLs
manics 97bef00
Use java.net.URI for parsing s3 URL
manics 4a6bdfe
S3 isDirectory equivalent to bucket-exists
manics bba2fa1
S3Handle.canHandleScheme can test scheme only
manics c102bb9
Be explicit about S3 instead of inferring from isUrl/null
manics 2532d25
Top parent of URLs is the server not the scheme
manics b237bbb
Replace Location.isDirectory IOException with UncheckedIOException
manics 0e31186
URL detection needs to handle unregistered schemes separately
manics 19b5c95
Use static patterns for all Location regex
manics 625ac5c
If Location is an invalid URL ignore and treat as non-URL
manics b24fa30
Use a single MinioClient. Use getObject offset for large seeks.
manics f44ed18
Update LocationTest including real minio/s3 server
manics e56a65b
S3 throw if object not found or stat is null
manics 4dd0eef
exists() for URLs is handle by the handles not Location
manics b5c0c47
Add docker minio for s3 tests
manics ce68184
s3 test private bucket
manics 0255fa9
Fix order of assertEquals parameters
manics 6f95f63
close handle in Location.exists()
manics 3f679ad
Replace docker minio server with local
manics 47b84a4
Make url and s3 tests mandatory
manics d72bd1f
Revert "Make url and s3 tests mandatory"
manics e7d951c
Disable HTTP and S3 tests by default
manics 34866f5
Remove skip(-1) in S3Handle.resetStream
manics 045b5d8
Remove unused imports, avoid wildcard imports
manics 8f7771e
Don't catch IOException in S3Handle constructor
manics 4b5a082
Add exists() to IRandomAccess
manics 76bea96
Bump to minio 5.0.2
manics 7d7c480
skip more LocationTests if S3
manics cb23387
Add more info when object not found exception throw
manics fe7b76c
Don't show found status in connect trace message
manics a506212
Location.exists don't cast handle
manics 740a88b
S3Handle set user-agent to `Bio-Formats/dev`
manics 6811bac
Create 2MB file for testing
manics 3720981
Test S3Handle read and seek
manics 795cc2b
Add note about handle caching (doesn't work)
manics 056b1cb
Cache URL exists and length
manics File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| bioformats.test.private | ||
| bioformats.test.public | ||
| mc | ||
| minio | ||
| .minio.sys |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/bin/sh | ||
| set -eux | ||
|
|
||
| PORT=31836 | ||
| PLATFORM=`uname | tr '[:upper:]' '[:lower:]'` | ||
|
|
||
| [ -f minio ] || \ | ||
| curl -sfSo minio "https://dl.minio.io/server/minio/release/$PLATFORM-amd64/minio" | ||
| [ -f mc ] || \ | ||
| curl -sfSo mc "https://dl.minio.io/client/mc/release/$PLATFORM-amd64/mc" | ||
| chmod +x minio mc | ||
| ./minio version | ||
| ./mc version | ||
|
|
||
| export MINIO_ACCESS_KEY=accesskey MINIO_SECRET_KEY=secretkey | ||
| ./minio server --address localhost:$PORT . & | ||
| sleep 2; | ||
|
|
||
| ./mc config host add ome-common-java-minio-test http://localhost:$PORT ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY} | ||
|
|
||
| set +x | ||
|
|
||
| for SUFFIX in public private; do | ||
| ./mc ls ome-common-java-minio-test/bioformats.test.$SUFFIX || \ | ||
| ./mc mb ome-common-java-minio-test/bioformats.test.$SUFFIX | ||
| ./mc ls ome-common-java-minio-test/bioformats.test.$SUFFIX/single-channel.ome.tiff || \ | ||
| curl -sfS https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/bioformats-artificial/single-channel.ome.tiff | \ | ||
| ./mc pipe ome-common-java-minio-test/bioformats.test.$SUFFIX/single-channel.ome.tiff | ||
| # 2MB file for testing seek | ||
| for n in `seq 65536`; do printf '.% 30d\n' $n; done | \ | ||
| ./mc pipe ome-common-java-minio-test/bioformats.test.$SUFFIX/2MBfile.txt | ||
| done | ||
|
|
||
| ./mc policy public ome-common-java-minio-test/bioformats.test.public |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.