HADOOP-16085: S3Guard versioning: get ITestS3ARemoteFileChanged to work consistently #824
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
commit ae876ab2df46c68ddd923edf8dd1d314191fcc94 Merge: 2e0254e 6a42745 Author: Ben Roling <ben.roling@cerner.com> Date: Thu May 2 10:14:10 2019 -0500 Merge branch 'trunk' into HADOOP-16085-squashed-2 commit 2e0254e Author: Ben Roling <ben.roling@cerner.com> Date: Thu Apr 18 12:13:40 2019 -0500 Remove unused import commit d1275e4 Merge: 450ba66 df76cdc Author: Ben Roling <ben.roling@cerner.com> Date: Thu Apr 18 12:10:01 2019 -0500 Merge branch 'trunk' into HADOOP-16085-squashed commit 450ba66 Author: Ben Roling <ben.roling@cerner.com> Date: Thu Apr 18 11:45:41 2019 -0500 Improvements to TestObjectChangeDetectionAttributes, AbstractS3AMockTest commit 408af6c Author: Ben Roling <ben.roling@cerner.com> Date: Thu Apr 18 10:29:05 2019 -0500 Use HttpStatus code constant instead of magic number commit 5f0532b Author: Ben Roling <ben.roling@cerner.com> Date: Thu Apr 18 10:02:50 2019 -0500 Update core-default.xml commit 3488b20 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Apr 17 16:14:43 2019 -0500 Fix runaround of creating FileStatus and then calling fromFileStatus() commit 90d5c9c Author: Ben Roling <ben.roling@cerner.com> Date: Wed Apr 17 15:45:51 2019 -0500 Fix minor nits commit 3ff59e4 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Apr 17 15:07:02 2019 -0500 Mutate S3AFileStatus instead of creating new instance commit 13fab97 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Apr 17 14:30:55 2019 -0500 Rename S3LocatedFileStatus to S3ALocatedFileStatus commit bee4e52 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Apr 17 14:25:17 2019 -0500 Stop pretending to support group and permission attributes on S3AFileStatus commit 807e13b Author: Ben Roling <ben.roling@cerner.com> Date: Wed Apr 17 14:20:14 2019 -0500 Add serialVersionUID to S3LocatedFileStatus commit 9974cec Author: Ben Roling <ben.roling@cerner.com> Date: Mon Apr 8 13:34:38 2019 -0500 Fix missed group or owner tweak commit 708c001 Author: Ben Roling <ben.roling@cerner.com> Date: Mon Apr 8 12:58:14 2019 -0500 Fix S3AFileStatus group handling ITestS3AConfiguration.testUsernameFromUGI was failing, expecting the user to be copied into the group. Strict copying of user into group causes TestLocalMetadataStore.testPutNew() to fail since it expects the group to be preserved from the original FileStatus. This change copies user into group when group is null/empty. With this change, all existing tests pass. commit 5239a9f Author: Ben Roling <ben.roling@cerner.com> Date: Thu Apr 4 16:38:31 2019 -0500 Skip tests that require versionId when bucket doesn't have versioning enabled commit 4c6331e Author: Ben Roling <ben.roling@cerner.com> Date: Mon Apr 1 13:58:24 2019 -0500 Fix broken TestObjectChangeDetectionAttributes commit 8a19c42 Author: Ben Roling <ben.roling@cerner.com> Date: Mon Apr 1 10:08:33 2019 -0500 Squashed commit of the following: commit 9f4ad88 Author: Ben Roling <ben.roling@cerner.com> Date: Mon Apr 1 09:29:35 2019 -0500 Add test for 412 response commit dc0a3fb Author: Ben Roling <ben.roling@cerner.com> Date: Thu Mar 28 16:53:46 2019 -0500 Update tests that started failing due to HADOOP-15999 commit 5e1f3e3 Author: Ben Roling <ben.roling@cerner.com> Date: Thu Mar 28 15:49:26 2019 -0500 Speed up ITestS3ARemoteFileChanged commit 1b6be40 Author: Ben Roling <ben.roling@cerner.com> Date: Thu Mar 28 14:23:53 2019 -0500 Skip invalid test when object versioning enabled commit 8597d2e Merge: 2d235f8 b5db238 Author: Ben Roling <ben.roling@cerner.com> Date: Thu Mar 28 11:54:50 2019 -0500 Merge remote-tracking branch 'apache/trunk' into HADOOP-16085 commit 2d235f8 Author: Ben Roling <ben.roling@cerner.com> Date: Thu Mar 28 11:51:26 2019 -0500 Fix typo commit dc83cef Author: Ben Roling <ben.roling@cerner.com> Date: Thu Mar 28 10:28:09 2019 -0500 Generalize TestObjectETag to cover versionId and test overwrite commit 0d71f32 Author: Ben Roling <ben.roling@cerner.com> Date: Thu Mar 28 08:45:42 2019 -0500 Fix trailing whitespace commit 324be6d Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 22:00:57 2019 -0500 S3GuardTool updates to correct ETag or versionId metadata commit 2a2bba7 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 21:27:27 2019 -0500 Clarify log message commit 6e62a3a Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 21:17:48 2019 -0500 Documentation updates per PR feedback commit 1ff8bef Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 16:05:59 2019 -0500 check version.required on CopyResult commit e296275 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 16:04:50 2019 -0500 Minor javadoc improvements from PR review commit 3e9ea19 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 13:15:58 2019 -0500 Skip tests that aren't applicable with change.detection.source=versionId commit ddbf68b Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 11:56:38 2019 -0500 Add tests of case where no version metadata is present commit 21d37dd Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 09:25:46 2019 -0500 Fix compiler deprecation warning commit b8e1569 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 09:19:46 2019 -0500 Fix license issue commit 33bb5f9 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 09:19:32 2019 -0500 Fix findbugs issue commit 5b7fadb Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 09:00:39 2019 -0500 Fix checkstyle issues commit 6110a11 Author: Ben Roling <ben.roling@cerner.com> Date: Wed Mar 27 08:28:37 2019 -0500 Remove trailing whitespace commit d82069b Author: Ben Roling <ben.roling@cerner.com> Date: Tue Mar 26 16:05:01 2019 -0500 Improve S3Guard doc commit ca2f0e9 Author: Ben Roling <ben.roling@cerner.com> Date: Tue Mar 26 14:29:03 2019 -0500 Fix ITestS3ARemoteFileChanged commit 1e4fa85 Author: Ben Roling <ben.roling@cerner.com> Date: Tue Mar 26 11:37:48 2019 -0500 Increase local metastore cache timeout commit 34b0c80 Author: Ben Roling <ben.roling@cerner.com> Date: Tue Mar 26 11:35:34 2019 -0500 Fix isEmptyDir inconsistency commit bbf8365 Author: Ben Roling <ben.roling@cerner.com> Date: Mon Mar 25 16:55:24 2019 -0500 TestPathMetadataDynamoDBTranslation tests null etag, versonId commit 2ae7d16 Author: Ben Roling <ben.roling@cerner.com> Date: Mon Mar 25 16:54:49 2019 -0500 Add constants in TestDirListingMetadata commit 068a55d Author: Ben Roling <ben.roling@cerner.com> Date: Mon Mar 25 15:43:45 2019 -0500 Add copy exception handling commit 0eca6f3 Author: Ben Roling <ben.roling@cerner.com> Date: Mon Mar 25 12:43:51 2019 -0500 Don't process response from copy commit ad9e152 Author: Ben Roling <ben.roling@cerner.com> Date: Mon Feb 25 16:41:54 2019 -0600 HADOOP-16085-003.patch Rebase of previous work after merge of HADOOP-15625.
Includes retries for regular reads, select(), and rename()
+add stevel review (primarily of tests) Change-Id: I75a3b70917eefc0a0ec3190ca1de527e2081551e
-move CopyOutcome -add new (failing) test for directory renaming where one of the files underneath is eventually consistent. -pull out commonanity from test methods (spyOnFilesystem(), expectReadFailure()) -intercept() calls provide details on failure, primarily just by returning the result of the operation expected to fail. -ChangeDetectionPolicy-raised exceptions on copy failures don't include position any more -cache/restore metastore so even on test failures mockito is involved on teardown Change-Id: I1fbdc38e9083881e4d72f13f29845e34539ac4ed
eTag and versionid weren't coming through on directory listings.
Use ChangeTracker in getObjectMetadata() and avoid calling getObjectMetadata() in a retry loop since it retries internally.
AWS response stubbing was incorrect for inconsistency during copyObject(). AWS throws AmazonClientException caused by NPE if constraints aren't met per aws/aws-sdk-java#1644. Documentation and tests updated to reflect this.
…rk consistently * clear auth mode flag from FS config, so there is a metastore check before rename(file), ensuring that the values passed to the mock are consistent everywhere * change error in ChangeDetectionPolicy to make clear when an error is coming on the client for a getObjectMetadata call even when the copy policy is "server" * and fix up ITestS3ARemoteFileChanged to use the constant string defined in ChangeDetectionPolicy precisely to stop tests being brittle against changes in the text. * new tests in ITestS3ARemoteFileChanged to not lose stack traces when examining causes/inner causes of exceptions, just to throw new assertions with nested causes. * deleted a couple of unused imports * moved from two spaces after a "." to one. Sorry. Change-Id: I30e28fdc9431668058ce1eae8517cc25f31fa051
|
🎊 +1 overall
This message was automatically generated. |
* added auth mode as a param on tests; switch half the existing tests over to it (so improve coverage without adding new runs) * javadocs for the tests to help maintenance and debugging in future * updated test docs to cover versioned buckets * s3guard bucket info prints change mode/policy (and signing policy, while I am there) * and updated examples of bucket-info to be consistent with the current output. Change-Id: Ib2577da02f5d91df427a0359e6ccca2add4894da
…rivial >80 char line width Change-Id: I4782ae8d7e4da74251f30b81589f67954869127d
Contributor
Author
|
tested: s3 london, versioned store, s3guard auth mode. Also tested versioning tests against us-west + unguarded; outcome there was those tests which require versioning were skipped (i.e. probes for versioning work) |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
merged into #794 |
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
- To support long classpath with deterministic jar loading we use a concept called pathing jar - A new archive called pathing.jar is created with a custom manifest - This custom manifest file contains the deterministic classpath that we construct Readings: - [Adding Classes to the JAR File's Classpath ](https://docs.oracle.com/javase/tutorial/deployment/jar/downman.html) - [How to avoid argument line too long with manifest files ](https://stackoverflow.com/questions/3057841/too-long-line-in-manifest-file-while-trying-to-create-jar) - [Pathing Jar information: (from tools) ](https://stackoverflow.com/questions/201816/how-to-set-a-long-java-classpath-in-windows) Author: Sanil15 <sanil.jain15@gmail.com> Reviewers: Boris Shkolnik <boryas@apache.org>, svenkataraman@linkedin.com Closes apache#824 from Sanil15/SAMZA-1817-pathing
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
HADOOP-16085: S3Guard versioning: get ITestS3ARemoteFileChanged to work consistently
this is #794 with 1+ extra patch on to be merged in