druid.storage.type=s3 now honors druid.storage.zip to allow storing segments in s3 without zip compression#18544
Conversation
…egments in s3 without zip compression
clintropolis
force-pushed
the
s3_zip_no_zip
branch
from
93a05a5 to
981707c
Compare
| ); | ||
| } | ||
| catch (AmazonServiceException e) { | ||
| if (S3Utils.ERROR_ENTITY_TOO_LARGE.equals(S3Utils.getS3ErrorCode(e))) { |
There was a problem hiding this comment.
This whole block is duplicated, would be good to dedupe it.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| final URI uri = objectLocation.toUri(S3StorageDruidModule.SCHEME); | ||
| final ByteSource byteSource = getByteSource(uri); | ||
| final File outFile = new File(outDir, Paths.get(objectLocation.getPath()).getFileName().toString()); | ||
| outFile.createNewFile(); |
Check notice
Code scanning / CodeQL
Ignored error status of call Note
| private static final DataSegment DATA_SEGMENT_1_NO_ZIP = new DataSegment( | ||
| "test", | ||
| Intervals.of("2015-04-12/2015-04-13"), | ||
| "1", | ||
| ImmutableMap.of("bucket", TEST_BUCKET, "key", KEY_1 + "/"), | ||
| null, | ||
| null, | ||
| NoneShardSpec.instance(), | ||
| 0, | ||
| 1 | ||
| ); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note test
| private static final DataSegment DATA_SEGMENT_2_NO_ZIP = new DataSegment( | ||
| "test", | ||
| Intervals.of("2015-04-13/2015-04-14"), | ||
| "1", | ||
| ImmutableMap.of("bucket", TEST_BUCKET, "key", KEY_2 + "/"), | ||
| null, | ||
| null, | ||
| NoneShardSpec.instance(), | ||
| 0, | ||
| 1 | ||
| ); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note test
| object1.setBucketName(bucket); | ||
| object1.setKey(keyPrefix + "meta.smoosh"); | ||
| object1.getObjectMetadata().setLastModified(new Date(0)); | ||
| object1.setObjectContent(new FileInputStream(tmpFile)); |
Check warning
Code scanning / CodeQL
Potential input resource leak Warning test
| object2.setBucketName(bucket); | ||
| object2.setKey(keyPrefix + "00000.smoosh"); | ||
| object2.getObjectMetadata().setLastModified(new Date(0)); | ||
| object2.setObjectContent(new FileInputStream(tmpFile)); |
Check warning
Code scanning / CodeQL
Potential input resource leak Warning test
4 participants
Description
This PR adds the capability to store segments in s3 without compressing with zip, similar to the 'local' deep storage option. This is mainly for experimentation at this point, but went ahead and documented it just in case anyone else wants to experiment 🤷
When
druid.storage.zipis false, the load spec stores the prefix to use instead of the exact location, and the pullers/killers/movers when for a path that ends with/and then do a list operation and iterate over and apply the operation to the results.