[SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack #45619
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
Could you set |
Author
|
On it now, will paste shortly. |
Member
|
Thank you, @murrelljenna ! |
|
|
||
| val withOptionalConfiguration = dynamoDBEndpointUrl match { | ||
| case Some(endpoint) => baseClientLibConfiguration | ||
| .withDynamoDBEndpoint(endpoint).withMetricsLevel(metricsLevel) |
Member
There was a problem hiding this comment.
nit. Shall we remove .withMetricsLevel(metricsLevel) like the following because we already have it in the above?
- .withDynamoDBEndpoint(endpoint).withMetricsLevel(metricsLevel)
+ .withDynamoDBEndpoint(endpoint)
dongjoon-hyun
changed the title
dongjoon-hyun
changed the title
Author
|
So I said I'd paste shortly - turns out it'd take around 3 hours haha. It seems I have no failed tests but some suites didn't run. I can post the whole output but that would be enormous. This is with kinesis tests enabled. |
Author
|
Just as a heads up, I'll be unavailable tomorrow and monday, but will be back on this pr come tuesday. |
Member
|
Ya, let's continue next week because the above result looks insufficient to me. I expected the following with |
Author
|
Weird, is there a way to just run the kinesis tests? Even with ENABLE_KINESIS_TESTS=1, dev/run-tests still spends 3 hours running every single suite of tests |
Member
|
@murrelljenna Could you try the following command? (#45619 (comment))
|
Author
|
Thanks! That was way faster. This is likely some local laptop issue I'm having, but I'm getting class not found errors. Also, I have not set up to talk to any particular aws resources on this laptop for spark, so I'm not sure how these tests will work. |
Member
|
Ya, unfortunately, that could be one of the root cause why Apache Spark community was unable to approve and accept this domain like the previous PR. |
Member
|
BTW, could you give me some more data points, please, @murrelljenna ? What is the result when you run the test on the vanilla |
Author
|
Ah, I see, that may be a complicating factor. Here is my output on master. It looks pretty much like the same output |
Author
|
Spark is using java 17, correct? |
Author
|
This error occurs here: Am I expected to have credentials to my own aws in order to run this? |
Author
|
Do I need to set up my own aws environment to run these tests? |
Author
|
@dongjoon-hyun , do these tests generally require one to configure their own aws kinesis environment? Have you run these tests before? This change is pretty simple, it'd be a shame if the pr were to die again. |
Member
|
Author
|
I see, well in theory the kinesis tests could use localstack kinesis (ironically its this PR that would make that possible), but that would be a bit of an effort. I feel like this pr is at an impasse then, no? Are there any community members that have the infrastructure to run these tests? |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a rebased and updated version of a previous pr by @etspaceman from 2019. This PR allows the spark kinesis library to work with localstack instances of kinesis.
I have not added any changes to @etspaceman 's original code, except to remove the
cloudwatchMetricsLevelvalue in the original PR, as subsequent PRs have added that same value in since.What changes were proposed in this pull request?
Currently the spark kinesis library allows you to override the kinesis endpoint, but not the dynamodb endpoint the kinesis consumer will use to manage leases. This PR adds that configurable url into the kinesis input stream builder.
Why are the changes needed?
Localstack has only become more popular for integration testing since the original PR. Currently, one cannot use spark streaming and localstack kinesis together, as the kinesis consumer will only attempt to manage leases through a remote aws dynamo instance.
Does this PR introduce any user-facing change?
This PR will allow the user to specify their own dynamo endpoint, much like how they can currently specify their own kinesis endpoint.
How was this patch tested?
This PR adds unit tests to cover this new value.