[BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery 1 #17406

mosche · 2022-04-20T10:01:28Z

Usage of forkEvery 1 is typically a strong sign of poor quality / bad code and should be avoided:

It often hides resource leaks, either in code or worse, potentially in the runner itself.
It significantly impacts performance when running tests.

This PR introduces new JUnit rules to better manage the lifecyle of SparkContext or SparkContextOptions and allows to safely reuse those for all tests of a suite where applicable. The previous ReuseSparkContextRule leaks the SparkContext and was removed. Besides, with forkEvery 1 trying to reuse the SparkContext is absolutely pointless ...

There's might be more issues hidden I didn't discover locally ... I'll keep a look.

This also reduce the runtime of tests to ~30 secs from 1:30+ mins.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

mosche · 2022-04-21T06:36:22Z

Hmm, I noticed a related issue here. SparkContextOptions doesn't work with TestPipeline because providedSparkContext is ignored during the serde roundtrip to test that everything can be serialized before actually running the pipeline :/

IMHO providedSparkContext really doesn't belong into PipelineOptions, it can't be serialized and the resulting behavior is very inconsistent... tough that would be a breaking change. I suggest adding methods to set the provided Spark context to SparkContextFactory. If a context is provided using SparkContextOptions, it will be stored in the factory using setProvidedSparkContext as well.

This also allows to clear the provided Spark context as well, allowing for much cleaner code.

  /**
   * Set an externally managed {@link JavaSparkContext} that will be used if {@link
   * SparkContextOptions#getUsesProvidedSparkContext()} is set to {@code true}.
   *
   * <p>A Spark context can also be provided using {@link
   * SparkContextOptions#setProvidedSparkContext(JavaSparkContext)}. However, it will be dropped
   * during serialization potentially leading to confusing behavior. This is particularly the case
   * when used in tests with {@link org.apache.beam.sdk.testing.TestPipeline}.
   */
  public static synchronized void setProvidedSparkContext(JavaSparkContext providedSparkContext) 

  public static synchronized void clearProvidedSparkContext()

mosche · 2022-04-21T08:14:47Z

@echauchot I pushed one more commit with the change suggested above

mosche · 2022-04-21T08:15:28Z

I'll follow up with another PR to remove forkEvery 1 from the validates runner tasks

echauchot · 2022-04-21T08:35:24Z

@mosche need to finish something and I take a look

…ove forkEvery 1.

…during a serde roundtrip in TestPipenline

mosche · 2022-04-28T19:49:38Z

@echauchot ping :)

echauchot · 2022-04-29T07:49:26Z

@mosche sorry for the delay. I should be able to start your review by the end of the afternoon

echauchot · 2022-05-02T07:09:37Z

@mosche reviewing ...

echauchot · 2022-05-02T07:41:18Z

Usage of forkEvery 1 is typically a strong sign of poor quality / bad code and should be avoided:

It often hides resource leaks, either in code or worse, potentially in the runner itself.

It significantly impacts performance when running tests.

This PR introduces new JUnit rules to better manage the lifecyle of SparkContext or SparkContextOptions and allows to safely reuse those for all tests of a suite where applicable. The previous ReuseSparkContextRule leaks the SparkContext and was removed. Besides, with forkEvery 1 trying to reuse the SparkContext is absolutely pointless ...

There's might be more issues hidden I didn't discover locally ... I'll keep a look.

This also reduce the runtime of tests to ~30 secs from 1:30+ mins.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).

Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.

Update CHANGES.md with noteworthy changes.

If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Bad smell indeed ! Thanks for catching this @mosche and reducing the test time

echauchot

Thanks Moritz for your work. And sorry for the delay I was drawn on other things and this PR had a lot a changes to review.

runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java

...c/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/SparkMetricsSinkTest.java

...park/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java

...rg/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpersTest.java

runners/spark/src/test/java/org/apache/beam/runners/spark/metrics/SparkMetricsPusherTest.java

runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java

echauchot · 2022-05-10T16:16:42Z

...spark/src/test/java/org/apache/beam/runners/spark/coders/SparkRunnerKryoRegistratorTest.java

-    public WrapperKryoRegistrator() {
-      wasInitiated = true;
+    @Test
+    public void testDefaultSerializerNotCallingKryo() {


the assertion here is just that no exception is thrown upon serde ?

Not my code, but that seems to be the approach... it would throw if it attempted to initialize the Kryo registrator

Yes I know it is not your code but I took a look anyway as you changed the test structure.
I don't like the approach that the original developer took in several tests that the test assertion is the absence of exception. It is not explicit enough IMHO. But changing the test would we out of the scope of this PR. Can you just add a comment in the code that "the fact that exception is not thrown means that the Kryo serializer is not called by default"

fixed the test to be clearer

mosche · 2022-05-11T09:08:11Z

Thanks for the thorough review @echauchot 🙏 I should have addressed all your questions / comments. Let me know if anything is unclear

runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java

echauchot

Only one comment and a javadoc precision to add to the code and we're done. Thanks

...spark/src/test/java/org/apache/beam/runners/spark/coders/SparkRunnerKryoRegistratorTest.java

echauchot

Thanks for your work Moritz ! And thanks also for the little extra fixes. LGTM !

commit 5064cc2 Author: Robert Burke <lostluck@users.noreply.github.com> Date: Sat May 14 15:49:04 2022 -0700 [BEAM-14470] Use Generic Registrations in loadtests. (apache#17673) commit 780ad62 Author: Danny McCormick <dannymccormick@google.com> Date: Fri May 13 19:04:58 2022 -0400 [BEAM-14371] (and BEAM-14372) - enable a couple staticchecks (apache#17670) commit 66e85da Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri May 13 15:00:42 2022 -0700 Add some auto-starting runners to the typescript SDK. (apache#17580) Adds out-of-the-box support for FlinkRunner, DataflowRunner, and the Python Universal Local runner. Also adds a DefaultRunner which chooses between the DirectRunner and the ULR depending on the properties of the pipeline. commit c110365 Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Fri May 13 17:20:49 2022 -0400 [BEAM-14469] Allow nil primary returns from TrySplit in a single-window context (apache#17667) commit b774133 Author: Ryan Thompson <ryanthompson591@gmail.com> Date: Fri May 13 15:28:22 2022 -0400 [BEAM-14014] Support impersonation credentials in dataflow runner (apache#17244) commit 04f4984 Merge: 0a2aed7 26b16d0 Author: Heejong Lee <heejong@gmail.com> Date: Fri May 13 11:37:31 2022 -0700 Merge pull request apache#17605 from ihji/BEAM-14455 [BEAM-14455] Add UUID to sub-schemas for PythonExternalTransform commit 0a2aed7 Author: Marco Robles <marco.robles@wizeline.com> Date: Fri May 13 12:50:53 2022 -0500 Merge pull request apache#17365 from [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob * Added a zero row job to bigquery writeTable * added unit test * add sdf for update schema destination * add loadjob with inputstreamcontent parameter * add updateschemadest with zeroloadjob before copying to dest * add test for write temp tables * add nullness supress warning * remove unnecesary variable and use global variable for updateschemadestination-dofn Co-authored-by: Miguel Anzo <miguel.anzo@wizeline.com> commit 2d57753 Merge: 9085345 2d36feb Author: Heejong Lee <heejong@gmail.com> Date: Fri May 13 10:29:55 2022 -0700 Merge pull request apache#17608 from ihji/BEAM-14430 [BEAM-14430] Adding a logical type support for Python callables to Row schema commit 9085345 Merge: 2b7aab0 b8d78b7 Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri May 13 09:23:25 2022 -0700 Merge pull request apache#17653 Revert "Better test assertion." commit 2b7aab0 Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Fri May 13 12:16:31 2022 -0400 [BEAM-14465] Reduce DefaultS3ClientBuilderFactory logging to debug level (apache#17645) commit 4ce6056 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri May 13 09:14:26 2022 -0700 Bump github.com/spf13/cobra from 1.3.0 to 1.4.0 in /sdks (apache#17647) Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/spf13/cobra/releases) - [Changelog](https://github.com/spf13/cobra/blob/v1.4.0/CHANGELOG.md) - [Commits](spf13/cobra@v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 787479f Author: Brian Hulette <bhulette@google.com> Date: Thu May 12 23:56:21 2022 -0700 Drop dataclasses requirement, we only support python 3.7+ (apache#17640) commit b8d78b7 Author: tvalentyn <tvalentyn@users.noreply.github.com> Date: Fri May 13 08:54:11 2022 +0200 Revert "Better test assertion. (apache#17551)" This reverts commit dc20185. commit 969f5b8 Author: Danny McCormick <dannymccormick@google.com> Date: Thu May 12 22:12:53 2022 -0400 [BEAM-14347] Add function for simple function registration (apache#17650) commit 2d36feb Author: Heejong Lee <heejong@gmail.com> Date: Thu May 12 18:02:09 2022 -0700 add comment commit 4ac6820 Merge: 1c476e7 6c18bcf Author: Ahmet Altay <aaltay@gmail.com> Date: Thu May 12 16:34:49 2022 -0700 Merge pull request apache#17631 from chamikaramj/update_release_md commit 2fce769 Author: Heejong Lee <heejong@gmail.com> Date: Thu May 12 16:12:46 2022 -0700 put a default type hint for PythonCallableSource commit 1c476e7 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu May 12 15:25:59 2022 -0700 Bump cloud.google.com/go/bigquery from 1.28.0 to 1.32.0 in /sdks (apache#17625) Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.28.0 to 1.32.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.28.0...spanner/v1.32.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigquery dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 514e325 Author: Heejong Lee <heejong@gmail.com> Date: Thu May 12 15:10:50 2022 -0700 add micros_instant urn commit 792baa2 Author: Heejong Lee <heejong@gmail.com> Date: Thu May 12 14:46:24 2022 -0700 move logical types def commit 704ecc6 Author: bullet03 <bulat.safiullin@akvelon.com> Date: Fri May 13 02:48:41 2022 +0600 Merge pull request apache#17408 from [BEAM-14312] [Website] change section order, move socials to footer * [BEAM-14312] [Website] change section order, move socials to footer * [BEAM-14312] [Website] change margin * [BEAM-14312] [Website] change link * [BEAM-14312] [Website] delete section, change title and links of the pillars * [BEAM-14312] [Website] add pipelines section * [BEAM-14312] [Website] change image in graphic section * [BEAM-14312] [Website] delete width of 95%, change flink link * [BEAM-14312] [Website] move css style, delete unused html, optimized logos size * [BEAM-14312] [Website] substitute images, fix css styles for pipelines and logos sections * [BEAM-14312] [Website] add kinesis runner * [BEAM-14312] [Website] add kinesis runner, add margin-right to pipelines-logos * [BEAM-14312] [Website] change amazon kinesis logo * [BEAM-14312] change structure of logos in the footer * [BEAM-14312] add license header to linkedin.svg * [BEAM-14312] change margin in quote section commit c36f544 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu May 12 13:44:23 2022 -0700 Bump cloud.google.com/go/pubsub from 1.18.0 to 1.21.1 in /sdks (apache#17646) Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.18.0 to 1.21.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@pubsub/v1.18.0...pubsub/v1.21.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/pubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 6680261 Author: Pablo <pabloem@users.noreply.github.com> Date: Thu May 12 16:38:55 2022 -0400 Merge pull request apache#17584 from [BEAM-14415] Exception handling tests and logging for partial failure BQIO * [BEAM-14415] Exception handling tests and logging for partial failures in BQ IO * fix DLQ integration test * fix lint * fix postcommit * fix formatter * Fixing tests and adding test info * fix skipping tests commit 3cca763 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu May 12 13:06:09 2022 -0700 Bump github.com/lib/pq from 1.10.4 to 1.10.5 in /sdks (apache#17626) Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.10.4 to 1.10.5. - [Release notes](https://github.com/lib/pq/releases) - [Commits](lib/pq@v1.10.4...v1.10.5) --- updated-dependencies: - dependency-name: github.com/lib/pq dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit cc571a3 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu May 12 13:05:11 2022 -0700 Bump github.com/testcontainers/testcontainers-go in /sdks (apache#17627) Bumps [github.com/testcontainers/testcontainers-go](https://github.com/testcontainers/testcontainers-go) from 0.12.0 to 0.13.0. - [Release notes](https://github.com/testcontainers/testcontainers-go/releases) - [Commits](testcontainers/testcontainers-go@v0.12.0...v0.13.0) --- updated-dependencies: - dependency-name: github.com/testcontainers/testcontainers-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 59212f3 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu May 12 13:04:42 2022 -0700 Bump github.com/google/go-cmp from 0.5.7 to 0.5.8 in /sdks (apache#17628) Bumps [github.com/google/go-cmp](https://github.com/google/go-cmp) from 0.5.7 to 0.5.8. - [Release notes](https://github.com/google/go-cmp/releases) - [Commits](google/go-cmp@v0.5.7...v0.5.8) --- updated-dependencies: - dependency-name: github.com/google/go-cmp dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit dc20185 Author: Robert Bradshaw <robertwb@gmail.com> Date: Thu May 12 13:02:14 2022 -0700 Better test assertion. (apache#17551) commit 5bb50d1 Author: Danny McCormick <dannymccormick@google.com> Date: Thu May 12 16:00:55 2022 -0400 [BEAM-14347] Add generic registration feature to CHANGES (apache#17643) commit d5df315 Merge: ec85d28 7c23d52 Author: Aizhamal Nurmamat kyzy <aizhamal.nk@gmail.com> Date: Thu May 12 10:13:12 2022 -0700 Merge pull request apache#17588 from damccorm/users/damccorm/issueTemplates [BEAM-14442] Add GitHub issue templates commit ec85d28 Author: Yichi Zhang <zyichi@google.com> Date: Thu May 12 10:06:16 2022 -0700 Revert "[BEAM-14429] Force java load test on dataflow runner v2 forceNumInitialBundles to 1 (apache#17576)" (apache#17609) This reverts commit 178441d. commit fd61a90 Author: Danny McCormick <dannymccormick@google.com> Date: Thu May 12 10:37:27 2022 -0400 Trigger go precommits on go mod/sum changes (apache#17636) commit 7c23d52 Author: Danny McCormick <dannymccormick@google.com> Date: Thu May 12 09:30:51 2022 -0400 Pare down to fewer templates commit 30f1a0c Author: Robert Burke <lostluck@users.noreply.github.com> Date: Thu May 12 06:17:35 2022 -0700 Typo & link update (apache#17633) commit a6ee885 Author: Moritz Mack <mmack@talend.com> Date: Thu May 12 15:08:19 2022 +0200 [BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery 1 (apache#17406) * [BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery 1 and set provided SparkContext via SparkContextFactory to avoid losing it during a serde roundtrip in TestPipenline. commit e2b2966 Author: Heejong Lee <heejong@gmail.com> Date: Wed May 11 22:29:12 2022 -0700 fix lint errors commit 6797787 Author: Heejong Lee <heejong@gmail.com> Date: Wed May 11 20:34:20 2022 -0700 add urn, type inference for PythonCallableSource commit a167424 Author: Kiley Sok <kileysok@google.com> Date: Wed May 11 14:48:57 2022 -0700 [BEAM-13695] Add jamm jvm options to Java 11 (apache#17178) * Add jamm jvm options to Java 11 * fix test comments commit 6ee09b6 Author: Danny McCormick <dannymccormick@google.com> Date: Wed May 11 17:16:50 2022 -0400 Correctly route go dependency changes to go label (apache#17632) commit 6c18bcf Author: Chamikara Jayalath <chamikaramj@gmail.com> Date: Wed May 11 13:33:08 2022 -0700 Updates CHANGES.md to include some recently discovered known issues commit 59cdb58 Author: Danny McCormick <dannymccormick@google.com> Date: Wed May 11 15:59:21 2022 -0400 [BEAM-14347] Add some benchmarks for generic registration (apache#17613) commit 559877a Author: Danny McCormick <dannymccormick@google.com> Date: Wed May 11 15:57:56 2022 -0400 Cut p4 commit b215351 Author: Danny McCormick <dannymccormick@google.com> Date: Wed May 11 15:48:56 2022 -0400 We don't need outage commit 51f8dc8 Author: Danny McCormick <dannymccormick@google.com> Date: Wed May 11 15:45:03 2022 -0400 Ask for beam version + other dependencies commit f81119a Author: Yichi Zhang <zyichi@google.com> Date: Wed May 11 12:08:45 2022 -0700 Remove python 3.6 postcommit from mass_comment.py (apache#17630) commit d6df306 Merge: fbc495c 86b68aa Author: Kenn Knowles <kenn@apache.org> Date: Wed May 11 11:46:04 2022 -0700 Merge pull request apache#17519: [BEAM-14096] bump junit-quickcheck to 1.0 commit fbc495c Author: Danny McCormick <dannymccormick@google.com> Date: Wed May 11 14:09:39 2022 -0400 [BEAM-12526] Add Dependabot (apache#17563) * Create dependabot.yml * Apache header * See if dependabot will parse now * Explanatory comment * Reduce duplication * Bug + style from feedback * Change format to make dependabot happy commit f30e4e5 Author: Igor Krasavin <krasavin.is@yandex.ru> Date: Wed May 11 19:23:23 2022 +0300 [BEAM-14081] [CdapIO] Add context classes for CDAP plugins (apache#17104) * [BEAM-14048] Add ConfigWrapper for building CDAP PluginConfigs * [BEAM-14048] Fix checkstyle * [BEAM-14048] Fix warnings * [BEAM-14048] Fix warnings * [BEAM-14048] Fix warning * [BEAM-14048] Fix warning * [BEAM-14048] Remove unused dependencies * [BEAM-14048] Add needed dependencies * [BEAM-14048] Fix spotless * [BEAM-14048] Fix typo * [BEAM-14048] Use fori instead of stream * [BEAM-14048] Suppress warning * [BEAM-14048] Add used undeclared artifacts * [BEAM-14048] Change dependencies to test * Add context. * Fix dependencies issue * Add null annotation * [BEAM-14048] Refactoring * Add SuppressWarning. * Fix style. * Determine dependencies. * [BEAM-14048] Use CDAP InstantiatorFactory for creating config objects * [BEAM-14048] Suppress warning * [BEAM-14081] Refactoring * Update maven repo * Update build.gradle * [BEAM-14081] Refactoring * [BEAM-14048] Use ServiceNow CDAP dependency from Maven central * [BEAM-14048] Set macroFields * [BEAM-14081] Fix javadoc * [BEAM-14081] Make BatchContextImpl class abstract Co-authored-by: vitaly.terentyev <vitaly.terentyev@akvelon.com> Co-authored-by: Alex Kosolapov <alex.kosolapov@gmail.com> Co-authored-by: Elizaveta Lomteva <elizaveta.lomteva@akvelon.com> Co-authored-by: Elizaveta Lomteva <57974525+Lizzfox@users.noreply.github.com> commit 43cc865 Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Wed May 11 11:58:53 2022 -0400 [BEAM-11104] Add self-checkpointing to CHANGES.md (apache#17612) commit 0f38c82 Author: tvalentyn <tvalentyn@users.noreply.github.com> Date: Wed May 11 15:58:59 2022 +0200 [BEAM-14396] Bump httplib2 upper bound. (apache#17602) commit a181053 Author: andoni-guzman <89603073+andoni-guzman@users.noreply.github.com> Date: Wed May 11 08:29:32 2022 -0500 [BEAM-5492] Python Dataflow integration tests should export the pipeline console output to Jenkins Test Result section (apache#17530) commit 5a5e51e Author: Heejong Lee <heejong@gmail.com> Date: Thu May 5 20:26:31 2022 -0700 [BEAM-14430] Adding a logical type support for Python callables to Row schema commit 26b16d0 Author: Heejong Lee <heejong@gmail.com> Date: Tue May 10 14:05:50 2022 -0700 [BEAM-14455] Add UUID to sub-schemas for PythonExternalTransform commit d181445 Author: Danny McCormick <dannymccormick@google.com> Date: Mon May 9 11:19:30 2022 -0400 [BEAM-14441] Add GitHub issue templates commit 86b68aa Author: masahitojp <randomstep@gmail.com> Date: Sun May 1 18:52:09 2022 +0900 [BEAM-14096] bump junit-quickcheck to 1.0

github-actions bot added runners spark labels Apr 20, 2022

echauchot self-requested a review April 20, 2022 10:34

mosche force-pushed the BEAM-14334-SparkTests branch from 817cc3d to 744c360 Compare April 21, 2022 08:14

Moritz Mack added 2 commits April 21, 2022 16:13

[BEAM-14334] Fix leakage of SparkContext in Spark runner tests to rem…

bd24635

…ove forkEvery 1.

Set provided SparkContext via SparkContextFactory to avoid losing it …

e2a3095

…during a serde roundtrip in TestPipenline

mosche force-pushed the BEAM-14334-SparkTests branch from 744c360 to e2a3095 Compare April 21, 2022 14:14

echauchot requested changes May 10, 2022

View reviewed changes

PR feedback

ae0a524

Add more javadocs

b042e16

echauchot reviewed May 11, 2022

View reviewed changes

runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java Outdated Show resolved Hide resolved

echauchot requested changes May 11, 2022

View reviewed changes

Addressing final review comments

ca71ea7

mosche commented May 12, 2022

View reviewed changes

...spark/src/test/java/org/apache/beam/runners/spark/coders/SparkRunnerKryoRegistratorTest.java Show resolved Hide resolved

echauchot approved these changes May 12, 2022

View reviewed changes

echauchot merged commit a6ee885 into apache:master May 12, 2022

mosche deleted the BEAM-14334-SparkTests branch May 12, 2022 16:08

mosche mentioned this pull request May 13, 2022

[BEAM-14334] Remove remaining forkEvery 1 from all Spark tests and stop mixing unit tests with runner validations. #17662

Merged

4 tasks

[BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery 1 #17406

[BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery 1 #17406

Uh oh!

Conversation

mosche commented Apr 20, 2022

GitHub Actions Tests Status (on master branch)

Uh oh!

mosche commented Apr 21, 2022

Uh oh!

mosche commented Apr 21, 2022

Uh oh!

mosche commented Apr 21, 2022

Uh oh!

echauchot commented Apr 21, 2022

Uh oh!

mosche commented Apr 28, 2022

Uh oh!

echauchot commented Apr 29, 2022

Uh oh!

echauchot commented May 2, 2022

Uh oh!

echauchot commented May 2, 2022

GitHub Actions Tests Status (on master branch)

Uh oh!

echauchot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

echauchot May 10, 2022

Choose a reason for hiding this comment

Uh oh!

mosche May 11, 2022

Choose a reason for hiding this comment

Uh oh!

echauchot May 11, 2022

Choose a reason for hiding this comment

Uh oh!

mosche May 12, 2022

Choose a reason for hiding this comment

Uh oh!

echauchot May 12, 2022

Choose a reason for hiding this comment

Uh oh!

mosche commented May 11, 2022

Uh oh!

Uh oh!

echauchot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

echauchot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

echauchot left a comment •

edited

Loading