Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
eb08c3f
Add S3Handle using minio 5.0.0
joshmoore Sep 20, 2018
3289c57
Location: wider s3 support including parent/child logic
joshmoore Sep 26, 2018
384e397
S3Handle: parse more complex URLs
joshmoore Sep 26, 2018
f464dc3
Drop JDK7 for the moment
joshmoore Sep 28, 2018
bfc6e88
Location: partially deal with null child arguments
joshmoore Oct 1, 2018
38166c0
S3 bucket must be the first path component
manics Oct 4, 2018
edb69c9
Location.length return handle.length
manics Oct 4, 2018
0f4abc5
S3 server and protocol are always required
manics Oct 5, 2018
91bcbe1
Add in-depth traces to S3Handle
manics Oct 5, 2018
028e0a0
Make default s3 protocol a var
manics Oct 5, 2018
abcec31
Fix S3HandleTest protocol
manics Oct 5, 2018
dd61bb8
S3Handle default to https, add mock object for testing
manics Oct 5, 2018
a7479b4
Add S3Handle.toString()
manics Oct 5, 2018
817e88d
Add caching to S3Handle
manics Oct 5, 2018
edd1d94
Cache S3 files when BF_REMOTE_CACHE_ROOTDIR set
manics Oct 5, 2018
ef32294
S3Handle: only create cache dir if s3 object exists
manics Oct 5, 2018
c23ed87
Throw HandleException if downloading null path
manics Oct 8, 2018
8920edb
Custom protocol is handled as s3+protocol:// instead of env-var
manics Oct 10, 2018
4756a08
More testing/fixes to ensure consistent s3 url parsing
manics Oct 10, 2018
bc02711
S3Handle determines whether it can handle URLs
manics Oct 10, 2018
97bef00
Use java.net.URI for parsing s3 URL
manics Oct 10, 2018
4a6bdfe
S3 isDirectory equivalent to bucket-exists
manics Oct 11, 2018
bba2fa1
S3Handle.canHandleScheme can test scheme only
manics Oct 11, 2018
c102bb9
Be explicit about S3 instead of inferring from isUrl/null
manics Oct 12, 2018
2532d25
Top parent of URLs is the server not the scheme
manics Oct 12, 2018
b237bbb
Replace Location.isDirectory IOException with UncheckedIOException
manics Oct 19, 2018
0e31186
URL detection needs to handle unregistered schemes separately
manics Oct 19, 2018
19b5c95
Use static patterns for all Location regex
manics Oct 19, 2018
625ac5c
If Location is an invalid URL ignore and treat as non-URL
manics Oct 19, 2018
b24fa30
Use a single MinioClient. Use getObject offset for large seeks.
manics Oct 23, 2018
f44ed18
Update LocationTest including real minio/s3 server
manics Oct 23, 2018
e56a65b
S3 throw if object not found or stat is null
manics Oct 24, 2018
4dd0eef
exists() for URLs is handle by the handles not Location
manics Oct 24, 2018
b5c0c47
Add docker minio for s3 tests
manics Oct 24, 2018
ce68184
s3 test private bucket
manics Oct 24, 2018
0255fa9
Fix order of assertEquals parameters
manics Oct 24, 2018
6f95f63
close handle in Location.exists()
manics Oct 25, 2018
3f679ad
Replace docker minio server with local
manics Oct 30, 2018
47b84a4
Make url and s3 tests mandatory
manics Oct 30, 2018
d72bd1f
Revert "Make url and s3 tests mandatory"
manics Oct 31, 2018
e7d951c
Disable HTTP and S3 tests by default
manics Oct 31, 2018
34866f5
Remove skip(-1) in S3Handle.resetStream
manics Nov 1, 2018
045b5d8
Remove unused imports, avoid wildcard imports
manics Nov 2, 2018
8f7771e
Don't catch IOException in S3Handle constructor
manics Nov 2, 2018
4b5a082
Add exists() to IRandomAccess
manics Nov 2, 2018
76bea96
Bump to minio 5.0.2
manics Nov 2, 2018
7d7c480
skip more LocationTests if S3
manics Nov 5, 2018
cb23387
Add more info when object not found exception throw
manics Nov 6, 2018
fe7b76c
Don't show found status in connect trace message
manics Nov 6, 2018
a506212
Location.exists don't cast handle
manics Nov 7, 2018
740a88b
S3Handle set user-agent to `Bio-Formats/dev`
manics Nov 8, 2018
6811bac
Create 2MB file for testing
manics Nov 13, 2018
3720981
Test S3Handle read and seek
manics Nov 13, 2018
795cc2b
Add note about handle caching (doesn't work)
manics Nov 15, 2018
056b1cb
Cache URL exists and length
manics Nov 15, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,12 @@ cache:
jdk:
- oraclejdk11
- oraclejdk8
- openjdk7

before_script:
- cd location-server && ./start-location.sh && cd ..

script:
- mvn test -B -Dtestng.runHttpRemoteTests=1 -Dtestng.runS3RemoteTests=1

matrix:
fast_finish: true
5 changes: 5 additions & 0 deletions location-server/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
bioformats.test.private
bioformats.test.public
mc
minio
.minio.sys
34 changes: 34 additions & 0 deletions location-server/start-location.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/sh
set -eux

PORT=31836
PLATFORM=`uname | tr '[:upper:]' '[:lower:]'`

[ -f minio ] || \
curl -sfSo minio "https://dl.minio.io/server/minio/release/$PLATFORM-amd64/minio"
[ -f mc ] || \
curl -sfSo mc "https://dl.minio.io/client/mc/release/$PLATFORM-amd64/mc"
chmod +x minio mc
./minio version
./mc version

export MINIO_ACCESS_KEY=accesskey MINIO_SECRET_KEY=secretkey
./minio server --address localhost:$PORT . &
sleep 2;

./mc config host add ome-common-java-minio-test http://localhost:$PORT ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}

set +x

for SUFFIX in public private; do
./mc ls ome-common-java-minio-test/bioformats.test.$SUFFIX || \
./mc mb ome-common-java-minio-test/bioformats.test.$SUFFIX
./mc ls ome-common-java-minio-test/bioformats.test.$SUFFIX/single-channel.ome.tiff || \
curl -sfS https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/bioformats-artificial/single-channel.ome.tiff | \
./mc pipe ome-common-java-minio-test/bioformats.test.$SUFFIX/single-channel.ome.tiff
# 2MB file for testing seek
for n in `seq 65536`; do printf '.% 30d\n' $n; done | \
./mc pipe ome-common-java-minio-test/bioformats.test.$SUFFIX/2MBfile.txt
done

./mc policy public ome-common-java-minio-test/bioformats.test.public
9 changes: 9 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,11 @@
</distributionManagement>

<dependencies>
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>5.0.2</version>
</dependency>
Comment thread
manics marked this conversation as resolved.
<dependency>
<groupId>com.esotericsoftware.kryo</groupId>
<artifactId>kryo</artifactId>
Expand Down Expand Up @@ -233,6 +238,10 @@
<suiteXmlFiles>
<suiteXmlFile>src/test/java/loci/common/utests/testng-template.xml</suiteXmlFile>
</suiteXmlFiles>
<systemPropertyVariables>
<testng.runHttpRemoteTests>0</testng.runHttpRemoteTests>
<testng.runS3RemoteTests>0</testng.runS3RemoteTests>
</systemPropertyVariables>
</configuration>
</plugin>

Expand Down
6 changes: 6 additions & 0 deletions src/main/java/loci/common/AbstractNIOHandle.java
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@ public abstract class AbstractNIOHandle implements IRandomAccess {

// -- AbstractNIOHandle methods --

/* @see IRandomAccess#exists() */
@Override
public boolean exists() throws IOException {
return length() >= 0;
}

/**
* Ensures that the file mode is either "r" or "rw".
* @param mode Mode to validate.
Expand Down
6 changes: 6 additions & 0 deletions src/main/java/loci/common/FileHandle.java
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,12 @@ public long getFilePointer() throws IOException {
return raf.getFilePointer();
}

/* @see IRandomAccess#exists() */
@Override
public boolean exists() throws IOException {
return length() >= 0;
}

/* @see IRandomAccess.length() */
@Override
public long length() throws IOException {
Expand Down
8 changes: 8 additions & 0 deletions src/main/java/loci/common/IRandomAccess.java
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,14 @@ public interface IRandomAccess extends DataInput, DataOutput {
*/
long getFilePointer() throws IOException;

/**
* Returns whether this refers to a valid object
*
* @return true if this refers to a valid object
* @throws IOException if unable to determine whether the object is valid
*/
boolean exists() throws IOException;

/**
* Returns the length of this stream.
*
Expand Down
Loading