AWS S3 support for Atum by dk1844 · Pull Request #38 · AbsaOSS/atum

dk1844 · 2020-08-31T11:26:53Z

AWS support for ATUM

new Atum entry point for S3 impl: sparkSession.enableControlMeasuresTrackingForS3 - no breaking changes for existing usage
original persistence implementation is moved under za.co.absa.atum.persistence.hdfs, while the s3 one is under za.co.absa.atum.persistence.s3 for the structures to make sense.
README updated
Unit tests both for HDFS and S3 added - ControlMeasures(Hdfs|S3)(Loader|Storer)JsonSpec
Spark-bound listener (SparkQueryExecutionListener) and output info file inference for the S3 case support added.

Test-run support

Sample measurement object (runnable locally, but with disabled test) available for S3 Impl as well - SampleMeasurementsS3RunnerSpec
- to run the loader, one needs to provide s3 location with existing info file and an existing saml profile
- to run the storer, one needs to provide writable s3 location, KMS key Id string, and an existing saml profile

Test-run

I have successfully run SampleS3Measurements1 and SampleS3Measurements2 with a real S3 location, the data were read from and written to. ✔️

… added.

…stilled

…nd by explictly providing the kmsKeyId

…sStorerJsonSpec

- org.mockito.plugins.MockMaker with "mock-maker-inline" allows mocking final classes. (hint: https://www.baeldung.com/mockito-final)

…Request that is used to query object on S3.

…ded from environment property: TOOLING_KMS_KEY_ID cleanup

dk1844 · 2020-08-31T11:38:30Z

atum/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker

@@ -0,0 +1 @@
+mock-maker-inline


This allows mocking final classes for Mockito, which would not be otherwise possible.

README.md update with S3 specific example

atum/src/main/scala/za/co/absa/atum/persistence/ControlMeasuresParser.scala

AdrianOlosutean · 2020-09-01T07:50:02Z

atum/src/main/scala/za/co/absa/atum/utils/FileUtils.scala

@@ -0,0 +1,12 @@
+package za.co.absa.atum.utils


Would it be an idea to have FileUtils, HdfsFileUtils, S3Utils together in one object?

Well, they could in theory. I thought the separation clearly communicates the different concerns they solve. 🤷 I will think about it.

The more I think about this, the less I think it would improve things. So unless there is an obvious upside in this (that I am not seeing at the moment), I'd rather keep it separate.

"implicit def StringToPath" changed to an implicit class wrapper. version update to 0.3.0-SNAPSHOT

lokm01 · 2020-09-03T12:21:37Z

Very cool PR, please include @yruslan to review when ready ;)

…torer (s3 based or not). may expose the kmsKeyId this way, though!

todo = s3Location work, remove extensive logging

…ined storer, no storing otherwise. toS3Location test added

dk1844 · 2020-09-07T14:07:36Z

atum/src/test/scala/za/co/absa/atum/BigDecimalToJsonSerializationSpec.scala

 import za.co.absa.atum.model.Measurement

-class BigDecimalToJsonSerializationSpec extends FlatSpec with Matchers {
+class BigDecimalToJsonSerializationSpec extends AnyFlatSpec with Matchers {


This change (org.scalatest.{FlatSpec, Matchers} -> org.scalatest.{flatspec.AnyFlatSpec, matchers.should.Matchers}) is due to ScalaTest version upgrade.

dk1844 · 2020-09-07T14:32:55Z

atum/src/main/scala/za/co/absa/atum/core/Accumulator.scala

+   * Ability to view the storer if set.
+   * @return
+   */
+  private[atum] def getStorer: Option[ControlMeasuresStorer] = if (isStorerLoaded) Some(storer) else None


This way, the accumulator insides are opened to the rest of the Atum to behave differently based on what kind of storer (HDFS vs S3) is used.

dk1844 added 12 commits August 21, 2020 10:24

testrun fix (spark.testing.memory=1g)

1648c9b

spark-version update to v2.4.5, json4s update, aws sdk latest version…

b1f5e17

… added.

tests enabled in examples, order for the sample measurements tests in…

d0145ca

…stilled

Loader from s3 crudely works on local host by reading my SAML profile

ac00d10

Storer to s3 crudely works on local host by reading my SAML profile a…

aa50232

…nd by explictly providing the kmsKeyId

Unit tests for ControlMeasuresHdfsLoaderJsonFile | ControlMeasuresHdf…

6ecf7f2

…sStorerJsonSpec

Unit tests for ControlMeasuresS3LoaderJsonSpec

11b751b

- org.mockito.plugins.MockMaker with "mock-maker-inline" allows mocking final classes. (hint: https://www.baeldung.com/mockito-final)

ScalaTest version update to 3.2.2, Mockito -> ScalaMockito

89def5c

ControlMeasuresS3LoaderJsonSpec enhanced - now checking the GetObject…

003bb24

…Request that is used to query object on S3.

ControlMeasuresS3StorerJsonSpec added

3e2125d

SampleS3Measurements1|2 now work with SAML profile and KMS Key ID loa…

98f27b9

…ded from environment property: TOOLING_KMS_KEY_ID cleanup

scaladoc fix (reformat revert)

e993f78

dk1844 linked an issue Aug 31, 2020 that may be closed by this pull request

PoC _INFO files generation in S3 #37

Closed

dk1844 commented Aug 31, 2020

View reviewed changes

scala doc touchups

6c51d92

README.md update with S3 specific example

dk1844 marked this pull request as ready for review September 1, 2020 07:02

AdrianOlosutean approved these changes Sep 1, 2020

View reviewed changes

storeCurrentInfoFile divided into HDFS and S3 version

def1d73

"implicit def StringToPath" changed to an implicit class wrapper. version update to 0.3.0-SNAPSHOT

dk1844 added the work in progress label Sep 3, 2020

dk1844 added 3 commits September 3, 2020 17:09

writeInfoFileForQueryForS3 vs writeInfoFileForQuery - based on used s…

f609686

…torer (s3 based or not). may expose the kmsKeyId this way, though!

writeInfoFileForQueryForS3 vs writeInfoFileForQuery - fix

60c1be4

todo = s3Location work, remove extensive logging

SparkQueryExecutionListerner.onSuccess - only writing info with a def…

5d051b9

…ined storer, no storing otherwise. toS3Location test added

dk1844 commented Sep 7, 2020

View reviewed changes

isValidS3Path added to S3Utils, test updates

3fa2198

dk1844 commented Sep 7, 2020

View reviewed changes

dk1844 requested a review from AdrianOlosutean September 7, 2020 14:33

dk1844 removed the work in progress label Sep 7, 2020

dk1844 requested a review from yruslan September 7, 2020 14:34

dk1844 merged commit 16d3806 into master Sep 21, 2020

dk1844 deleted the feature/37-aws-sdk-poc branch September 21, 2020 07:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS S3 support for Atum#38

AWS S3 support for Atum#38
dk1844 merged 18 commits intomasterfrom
feature/37-aws-sdk-poc

dk1844 commented Aug 31, 2020 •

edited

Loading

Uh oh!

dk1844 Aug 31, 2020

Uh oh!

Uh oh!

AdrianOlosutean Sep 1, 2020

Uh oh!

dk1844 Sep 1, 2020

Uh oh!

dk1844 Sep 7, 2020

Uh oh!

lokm01 commented Sep 3, 2020

Uh oh!

dk1844 Sep 7, 2020

Uh oh!

dk1844 Sep 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1 @@
		mock-maker-inline

Conversation

dk1844 commented Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AWS support for ATUM

Test-run support

Test-run

Uh oh!

dk1844 Aug 31, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AdrianOlosutean Sep 1, 2020

Choose a reason for hiding this comment

Uh oh!

dk1844 Sep 1, 2020

Choose a reason for hiding this comment

Uh oh!

dk1844 Sep 7, 2020

Choose a reason for hiding this comment

Uh oh!

lokm01 commented Sep 3, 2020

Uh oh!

dk1844 Sep 7, 2020

Choose a reason for hiding this comment

Uh oh!

dk1844 Sep 7, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dk1844 commented Aug 31, 2020 •

edited

Loading