[BEAM-10529] nullable xlang coder #16923

johnjcasey · 2022-02-22T21:02:59Z

Support KafkaIO with null keys and values. The majority of this change is to enable nullable coders for xlang, and then to make minor changes to the KafkaIO itself to use them

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…nullable-Xlang-Coder # Conflicts: # runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java # runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/wire/CommonCoderTest.java

…sidentify coders as nullable

johnjcasey · 2022-02-23T15:44:39Z

R: @chamikaramj
R: @TheNeuralBit
R: @charlesccychen

codecov · 2022-02-23T16:59:58Z

Codecov Report

Merging #16923 (dbe1bb3) into master (ca47cd8) will increase coverage by 0.12%.
The diff coverage is 58.82%.

@@            Coverage Diff             @@
##           master   #16923      +/-   ##
==========================================
+ Coverage   73.98%   74.10%   +0.12%     
==========================================
  Files         674      675       +1     
  Lines       88558    88715     +157     
==========================================
+ Hits        65521    65746     +225     
+ Misses      21916    21822      -94     
- Partials     1121     1147      +26

Flag	Coverage Δ
go	`50.11% <53.04%> (+0.54%)`	⬆️
python	`83.67% <90.47%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/go/pkg/beam/core/graph/coder/map.go	`65.00% <ø> (+1.27%)`	⬆️
sdks/go/pkg/beam/core/typex/fulltype.go	`41.83% <0.00%> (-0.34%)`	⬇️
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go	`53.93% <36.84%> (-1.39%)`	⬇️
sdks/go/pkg/beam/core/runtime/graphx/coder.go	`53.72% <43.75%> (-0.39%)`	⬇️
sdks/go/pkg/beam/core/runtime/exec/coder.go	`56.46% <48.83%> (-0.47%)`	⬇️
sdks/go/pkg/beam/core/graph/coder/nil.go	`59.09% <59.09%> (ø)`
sdks/python/apache_beam/coders/coders.py	`88.24% <87.50%> (+0.25%)`	⬆️
sdks/python/apache_beam/typehints/typehints.py	`93.37% <90.90%> (-0.07%)`	⬇️
sdks/go/pkg/beam/core/graph/coder/coder.go	`85.98% <100.00%> (+0.68%)`	⬆️
sdks/go/pkg/beam/core/graph/coder/row_decoder.go	`63.73% <100.00%> (ø)`
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca47cd8...dbe1bb3. Read the comment docs.

…g-Coder' into feature/BEAM-10529-nullable-Xlang-Coder

TheNeuralBit

Thanks @johnjcasey! I have a few suggestions

TheNeuralBit · 2022-02-23T19:14:52Z

model/pipeline/src/main/proto/beam_runner_api.proto

+    // A Nullable coder encodes nullable values of wrapped coder value that does
+    // not tolerate null values. A Nullable coder uses exactly 1 byte per entry
+    // to indicate whether the value is null, then adds the encoding of the
+    // inner coder for non-null values.


Could you have this clarify that 1 implies non-null? It would also be nice to update the row coder doc (under "Nullable types in container types (ArrayType, MapType)") to refer to this

You mean 1 byte (\u0000) implies null ? I might be misunderstanding something.

TheNeuralBit · 2022-02-23T19:19:58Z

sdks/python/apache_beam/coders/typecoders_test.py

+  def test_nullable_coder(self):
+    expected_coder = coders.NullableCoder(coders.BytesCoder())
+    real_coder = typecoders.registry.get_coder(
+        typehints.UnionConstraint([type(None), type(bytes)]))


nit: could you construct this with Optional instead of using Union directly?

Also I think bytes is already a type:

>>> bytes <class 'bytes'> >>> type(bytes) <class 'type'>

Shouldn't this be Optional[bytes]?

Yep, I'll make the change in the test case

TheNeuralBit · 2022-02-23T19:22:53Z

sdks/python/apache_beam/coders/typecoders.py

        return coders.ListCoder.from_type_hint(typehint, self)
+      elif (isinstance(typehint, typehints.UnionConstraint) and
+            typehint.contains_type(type(None) and
+                                   len(list(typehint._inner_types())) == 2)):


Could you extract this logic into an is_optional helper in typehints.py?

It also looks like there may be a paren mistake, the len is inside the contains_type call.

Let's make sure this is covered by a unit test.

TheNeuralBit · 2022-02-23T19:25:14Z

sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java

+              return NullableCoder.of(VarIntCoder.of());
            } else if (returnType.equals(Long.class)) {
-              return VarLongCoder.of();
+              return NullableCoder.of(VarLongCoder.of());


Is this going to break the Go Kafka connector?

CC: @youngoli

It will. I checked with @lostluck, and Go doesn't have a nullable coder yet, so that will need to be built before this can be merged

:( sorry, Go didn't have a kafkaio when BEAM-10529 was filed.

Probably we can introduce a new transform that is only exposed through the Python API ?

Another option is to update URN_WITH_METADATA transforms to support nulls:

beam/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java

Line 803 in 90dd44e

public static final String URN_WITH_METADATA =

Seems like Go SDK uses "URN_WITHOUT_METADATA" transform:

beam/sdks/go/pkg/beam/io/xlang/kafkaio/kafka.go

Line 84 in 90dd44e

readURN = "beam:transform:org.apache.beam:kafka_read_without_metadata:v1"

I think exposing the Go Nullable coder is a good option. That way we have it in the future, and we don't need to introduce work arounds in our code

Agreed that would be the best solution, but to be clear, are you planning on doing that for the next step on this PR?

Yep. I'm working on that now. I'm unfamiliar with Go, but making good progress so far

Awesome! Thank you for taking this on

Sounds good. Thanks.

chamikaramj

Thanks!

chamikaramj · 2022-03-08T01:48:20Z

model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml

+examples:
+  "\u0001\u0003\u0061\u0062\u0063" : "abc"
+  "\u0001\u000a\u006d\u006f\u0072\u0065\u0020\u0062\u0079\u0074\u0065\u0073" : "more bytes"
+  "\u0000" : null


To clarify, only the last byte here will be in the an encoded element that has value null ?

yep. 0000 means null, 0001 means "the rest of this is non-null"

chamikaramj · 2022-03-08T01:51:09Z

model/pipeline/src/main/proto/beam_runner_api.proto

+    // A Nullable coder encodes nullable values of wrapped coder value that does
+    // not tolerate null values. A Nullable coder uses exactly 1 byte per entry
+    // to indicate whether the value is null, then adds the encoding of the
+    // inner coder for non-null values.


You mean 1 byte (\u0000) implies null ? I might be misunderstanding something.

chamikaramj · 2022-03-08T01:52:35Z

sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java

+              return NullableCoder.of(VarIntCoder.of());
            } else if (returnType.equals(Long.class)) {
-              return VarLongCoder.of();
+              return NullableCoder.of(VarLongCoder.of());


Sounds good. Thanks.

chamikaramj · 2022-03-08T01:56:40Z

sdks/python/apache_beam/coders/typecoders.py

        return coders.ListCoder.from_type_hint(typehint, self)
+      elif (isinstance(typehint, typehints.UnionConstraint) and
+            typehint.contains_type(type(None) and
+                                   len(list(typehint._inner_types())) == 2)):


Let's make sure this is covered by a unit test.

chamikaramj · 2022-03-08T02:12:50Z

sdks/python/apache_beam/coders/typecoders.py

+      elif (isinstance(typehint, typehints.UnionConstraint) and
+            typehint.contains_type(type(None) and
+                                   len(list(typehint._inner_types())) == 2)):
+        return coders.NullableCoder.from_type_hint(typehint, self)


Did you mean to use the inner type in this call ?

No, I thought it made more sense to use the full typehint, to try and match the existing coder.from_type_hint pattern

chamikaramj · 2022-03-08T02:13:42Z

sdks/python/apache_beam/typehints/typehints.py

      for t in self.union_types:
        yield t

+    def contains_type(self, maybe_type):


Does the order matter (for nullable and other cases) ?

Optional doesn't care about order, and contains type doesn't care about order either

…g-Coder' into feature/BEAM-10529-nullable-Xlang-Coder

…nullable-Xlang-Coder

chamikaramj

LGTM. Thanks.

chamikaramj · 2022-03-29T20:17:48Z

Lemme know if this is ready to be merged.

johnjcasey · 2022-03-29T20:21:04Z

Waiting on @lostluck to review the Go changes, but then I am good to go

lostluck

A few quick fixes to the Go changes, but otherwise, I like it. Thank you!

model/pipeline/src/main/proto/beam_runner_api.proto

lostluck · 2022-03-30T15:45:34Z

sdks/go/pkg/beam/core/graph/coder/coder.go

+	coders := make([]*Coder, 1)
+	coders[0] = component


Suggested change

coders := make([]*Coder, 1)

coders[0] = component

coders := []*Coder{ component }

lostluck · 2022-03-30T15:45:51Z

sdks/go/pkg/beam/core/graph/coder/coder.go

+	return &Coder{
+		Kind:       Nullable,
+		T:          typex.New(typex.NullableType, component.T),
+		Components: []*Coder{component},


Suggested change

Components: []*Coder{component},

Components: coders,

lostluck · 2022-03-30T15:52:59Z

sdks/go/pkg/beam/core/graph/coder/coder_test.go

+		component   *Coder
+		shouldpanic bool
+		want        *Coder
+	}{{


+1
In this case, you should split the }} with a new line, and then run gofmt on this file.

I would strongly recommend having your IDE do so automatically on save anyway.

sdks/go/pkg/beam/core/runtime/exec/coder.go

sdks/go/test/regression/coders/fromyaml/fromyaml.go

lostluck

LGTM for Go. Thanks!

lostluck · 2022-03-31T16:08:34Z

The GoPortable error is xlang related but not due to this PR.

johnjcasey · 2022-04-01T14:04:32Z

run java precommit

…nullable-Xlang-Coder # Conflicts: # sdks/go/pkg/beam/core/typex/special.go

johnjcasey · 2022-04-01T14:46:18Z

Run Portable_Python PreCommit

…g-Coder' into feature/BEAM-10529-nullable-Xlang-Coder

johnjcasey · 2022-04-04T16:26:32Z

Failures are due to https://issues.apache.org/jira/browse/BEAM-14231, and this can be merged

TheNeuralBit · 2022-04-04T16:33:44Z

Re-running the Python failure, looks like it should be resolved by rollback #17255. I will merge when we get it green

@lostluck is there a jira for the Go xlang failure?

TheNeuralBit · 2022-04-04T17:01:13Z

Re-running the Python failure, looks like it should be resolved by rollback #17255. I will merge when we get it green

Oh I forgot, GHA checks do not run with changes from HEAD, so it's still running code with this flaky behavior.

jrmccluskey · 2022-04-04T18:01:16Z

@johnjcasey The Actions build breakage was due to bad formatting in core/typex/special.go. I've got a PR out to fix it (linked above) but please don't ignore the build status moving forward!

TheNeuralBit · 2022-04-04T18:03:44Z

@jrmccluskey This was my fault, I assumed the Go failure was the one @lostluck was referencing here.

Abacn · 2022-04-04T21:06:13Z

Hi,

I got an error on ./gradlew :sdks:python:wordCount checking out latest master:

Traceback (most recent call last):
  File "/Users/<USER>/.pyenv/versions/3.8.7/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/<USER>/.pyenv/versions/3.8.7/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/Users/<USER>/beam/sdks/python/apache_beam/__init__.py", line 92, in <module>
    from apache_beam import coders
  File "/Users/<USER>/beam/sdks/python/apache_beam/coders/__init__.py", line 17, in <module>
    from apache_beam.coders.coders import *
  File "/Users/<USER>/beam/sdks/python/apache_beam/coders/coders.py", line 638, in <module>
    Coder.register_structured_urn(common_urns.coders.NULLABLE.urn, NullableCoder)
AttributeError: type object 'Enum' has no attribute 'NULLABLE'

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/<USER>/beam/sdks/python/build.gradle' line: 119

* What went wrong:
Execution failed for task ':sdks:python:wordCount'.
> Process 'command 'sh'' finished with non-zero exit value 1

Seems related to this change. Could anyone please have a look, thanks!

Abacn · 2022-04-05T00:42:31Z

Hi,

I got an error on ./gradlew :sdks:python:wordCount checking out latest master:

Traceback (most recent call last):
  File "/Users/<USER>/.pyenv/versions/3.8.7/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/<USER>/.pyenv/versions/3.8.7/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/Users/<USER>/beam/sdks/python/apache_beam/__init__.py", line 92, in <module>
    from apache_beam import coders
  File "/Users/<USER>/beam/sdks/python/apache_beam/coders/__init__.py", line 17, in <module>
    from apache_beam.coders.coders import *
  File "/Users/<USER>/beam/sdks/python/apache_beam/coders/coders.py", line 638, in <module>
    Coder.register_structured_urn(common_urns.coders.NULLABLE.urn, NullableCoder)
AttributeError: type object 'Enum' has no attribute 'NULLABLE'

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/<USER>/beam/sdks/python/build.gradle' line: 119

* What went wrong:
Execution failed for task ':sdks:python:wordCount'.
> Process 'command 'sh'' finished with non-zero exit value 1

Seems related to this change. Could anyone please have a look, thanks!

It turns out that this was a problem of my protobuf dependency, which causes sdks/python/gen_protos.py not running to generate new versions of RegularCoders.py. Force install protobuf with c++ implementation resolves this problem.

* [BEAM-10529] add java and generic components of nullable xlang tests * [BEAM-10529] fix test case * [BEAM-10529] add coders and typehints to support nullable xlang coders * [BEAM-10529] update external builder to support nullable coder * [BEAM-10529] clean up coders.py * [BEAM-10529] add coder translation test * [BEAM-10529] add additional check to typecoder to not accidentally misidentify coders as nullable * [BEAM-10529] add test to retrieve nullable coder from typehint * [BEAM-10529] run spotless * [BEAM-10529] add go nullable coder * [BEAM-10529] cleanup extra println * [BEAM-10529] improve comments, clean up python * [BEAM-10529] remove changes to kafkaIO to simplify pr * [BEAM-10529] add coders to go exec, add asf license text * [BEAM-10529] clean up error handlign * [BEAM-10529] update go fromyaml to handle nullable cases * [BEAM-10529] add unit test, register nullable coder in dataflow.go * [BEAM-10529] remove mistaken commit * [BEAM-10529] add argument check to CoderTranslators * [BEAM-10529] Address python comments & cleanup * [BEAM-10529] address go comments * [BEAM-10529] remove extra check that was added in error * [BEAM-10529] fix typo * [BEAM-10529] re-order check for nonetype to prevent attribute errors * [BEAM-10529] change isinstance to ==

* added initial commit * removed modified file * removed params that dont exist * added clock, removed generics that were causing pickle error, fixed metrics name * fixed class names removed class that goes in apis * added base test file * Added unit tests * reordered imports * replied to comments * apis to api * added license * added mock clock test for metrics, realized our metric wouldn't work right with a generator * Minor changes from Andys comments. Push metric namespace decision to modleLoader class * Update sdks/python/apache_beam/ml/inference/base.py typo fix valentyn's suggestion Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com> * updated with changes from valentyns comments * merged from tfx version * added comment * linted * changed import order for jenkins linter * added a bug to track a todo * fixed for Roberts comments * make clock and metrics collector private * make shared second parameter * mark RunInferenceDoFn private * moved initialization of shared.Shared into constructor * added todo * Update sdks/python/apache_beam/ml/inference/base.py Co-authored-by: Brian Hulette <hulettbh@gmail.com> * Update sdks/python/apache_beam/ml/inference/base.py Co-authored-by: Brian Hulette <hulettbh@gmail.com> * Update sdks/python/apache_beam/ml/inference/base.py Co-authored-by: Brian Hulette <hulettbh@gmail.com> * updated to correct variable names * udpated variable names * added typevar * remove unbatch * added note that users should expect changes * Update python container version * Add --dataflowServiceOptions=enable_prime to useUnifiedWorker conditions (#17213) * Add self-descriptive message for expected errors. Ideally we would not log these in the first place, but this is an easy hack. * [BEAM-10529] nullable xlang coder (#16923) * [BEAM-10529] add java and generic components of nullable xlang tests * [BEAM-10529] fix test case * [BEAM-10529] add coders and typehints to support nullable xlang coders * [BEAM-10529] update external builder to support nullable coder * [BEAM-10529] clean up coders.py * [BEAM-10529] add coder translation test * [BEAM-10529] add additional check to typecoder to not accidentally misidentify coders as nullable * [BEAM-10529] add test to retrieve nullable coder from typehint * [BEAM-10529] run spotless * [BEAM-10529] add go nullable coder * [BEAM-10529] cleanup extra println * [BEAM-10529] improve comments, clean up python * [BEAM-10529] remove changes to kafkaIO to simplify pr * [BEAM-10529] add coders to go exec, add asf license text * [BEAM-10529] clean up error handlign * [BEAM-10529] update go fromyaml to handle nullable cases * [BEAM-10529] add unit test, register nullable coder in dataflow.go * [BEAM-10529] remove mistaken commit * [BEAM-10529] add argument check to CoderTranslators * [BEAM-10529] Address python comments & cleanup * [BEAM-10529] address go comments * [BEAM-10529] remove extra check that was added in error * [BEAM-10529] fix typo * [BEAM-10529] re-order check for nonetype to prevent attribute errors * [BEAM-10529] change isinstance to == * Fix go fmt break in core/typex/special.go (#17266) * [BEAM-8970] Add docs to run wordcount example on portable Spark Runner * [BEAM-8970] Add period to end of sentence * [BEAM-5436] Add doc page on Go cross compilation. (#17256) * Pr-bot Don't count all reviews as approvals (#17269) * Fix postcommits (#17263) * [BEAM-14241] Address staticcheck warnings in boot.go (#17264) * [BEAM-14157] GrpcWindmillServer: Use stream specific boolean to do client closed check (#17191) * [BEAM-14157] GrpcWindmillServer: Use stream specific boolean to do client closed check This is a follow up to #17162. An AbstractWindmillStream can have more than one grpc stream during its lifetime, new streams can be created after client closed for sending pending requests. So it is not correct to check `if(clientClosed)` in `send()`, this PR adds a new grpc stream level boolean to do the closed check in `send()`. * [BEAM-14157] Add unit test testing CommitWorkStream retries around stream closing * [BEAM-14157] review comments * [BEAM-14157] review comments * [BEAM-14157] review comments * [BEAM-14157] fix test * [BEAM-14157] fix test Co-authored-by: Arun Pandian <pandiana@google.com> * [BEAM-10582] Allow (and test) pyarrow 7 (#17229) * [BEAM-13519] Solve race issues when the server responds with an error before the GrpcStateClient finishes being constructed. (#17240) * [BEAM-13519] Solve race issues when the server responds with an error before the GrpcStateClient finishes. The issue was that the InboundObserver can be invoked before outboundObserverFactory#outboundObserverFor returns meaning that the server is waiting for a response for cache.remove but cache.computeIfAbsent is being invoked at the same time. Another issue was that the outstandingRequests map could be updated with another request within GrpcStateClient during closeAndCleanup meaning that the CompleteableFuture would never be completed exceptionally. Passes 1000 times locally now without getting stuck or failing. * [BEAM-14256] update SpEL dependency to 5.3.18.RELEASE * [BEAM-14256] remove .RELEASE * [BEAM-13015] Disable retries for fnapi grpc channels which otherwise defaults on. (#17243) * [BEAM-13015] Disable retries for grpc channels which otherwise default to true. Since the channel is to the local runner process, retries are not expected to help. This simplifies the grpc stream stack to not involve a RetryStream object. * fixup comment * Update sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java * Update sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java Co-authored-by: Lukasz Cwik <lcwik@google.com> * [BEAM-9649] Add region option to Mongo Dataflow test. * Fix dependency issue causing failures * [BEAM-13952] Sickbay testAfterProcessingTimeContinuationTriggerUsingState (#17214) * BEAM-14235 parquetio module does not parse PEP-440 compliant Pyarrow version (#17275) * Update parquetio.py * Update CHANGES.md * Fix import order * [BEAM-14250] Fix coder registration for types defined in __main__. Until all runners are portable and we can get rid of all round trips between Pipeline and proto representatons, register types in __main__ according to their string representations as pickling does not preserve identity. * Allow get_coder(None). Co-authored-by: Andy Ye <andyye333@gmail.com> * [Website] Contribution guide page indent bug fix (#17287) * Fix markdown indent issue in Development Setup section * update query * [BEAM-10976] Document go sdk bundle finalization (#17048) * [BEAM-13829] Expose status API from Go SDK Harness (#16957) * Avoid pr-bot state desync (#17299) * [BEAM-14259] Clean up staticcheck warnings in the exec package (#17285) * Minor: Prefer registered schema in SQL docs (#17298) * Prefer registered schema in SQL docs * address review comments * [Playground] add meta tags (#17207) * playground add meta tags * playground fix meta tags * fixes golint and deprecated issues in recent Go SDK import (#17304) * [BEAM-14262] Update plugins for Dockerized Jenkins. I copied the list from the cwiki and removed all of the ones that failed to install. https://cwiki.apache.org/confluence/display/INFRA/ci-beam.apache.org * Add ansicolor and ws-cleanup plugins. Without them, the seed job prints warnings: Warning: (CommonJobProperties.groovy, line 107) plugin 'ansicolor' needs to be installed Warning: (CommonJobProperties.groovy, line 113) plugin 'ws-cleanup' needs to be installed * [BEAM-14266] Replace deprecated ptypes package uses (#17302) * [BEAM-11936] Fix rawtypes warnings in SnowflakeIO (#17257) * [BEAM-10556] Fix rawtypes warnings in SnowflakeIO * fixup! [BEAM-10556] Fix rawtypes warnings in SnowflakeIO * Merge pull request #17262: [BEAM-14244] Use the supplied output timestamp for processing time timers rather than the input watermark * removed unused typing * added list typing * linted Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com> Co-authored-by: Brian Hulette <hulettbh@gmail.com> Co-authored-by: kileys <kileysok@google.com> Co-authored-by: Yichi Zhang <zyichi@google.com> Co-authored-by: Kyle Weaver <kcweaver@google.com> Co-authored-by: johnjcasey <95318300+johnjcasey@users.noreply.github.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Benjamin Gonzalez <benjamin.gonzalez@wizeline.com> Co-authored-by: Robert Burke <lostluck@users.noreply.github.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Arun Pandian <arunpandianp@gmail.com> Co-authored-by: Arun Pandian <pandiana@google.com> Co-authored-by: Brian Hulette <bhulette@google.com> Co-authored-by: Lukasz Cwik <lcwik@google.com> Co-authored-by: johnjcasey <johnjcasey@google.com> Co-authored-by: scwhittle <scwhittle@users.noreply.github.com> Co-authored-by: Arwin Tio <arwin.tio@adroll.com> Co-authored-by: Robert Bradshaw <robertwb@gmail.com> Co-authored-by: Andy Ye <andyye333@gmail.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Michael Li <bingyeli@google.com> Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com> Co-authored-by: Kamil Breguła <kamil.bregula@snowflake.com> Co-authored-by: Steven Niemitz <steveniemitz@gmail.com>

johnjcasey added 6 commits February 14, 2022 14:16

[BEAM-10529] add java and generic components of nullable xlang tests

b4b0025

[BEAM-10529] fix test case

798fd53

[BEAM-10529] add coders and typehints to support nullable xlang coders

8743549

[BEAM-10529] update external builder to support nullable coder

1ce6605

[BEAM-10529] clean up coders.py

a6a38d0

github-actions bot added core fn-execution io java kafka model python runners labels Feb 22, 2022

johnjcasey added 2 commits February 22, 2022 16:57

[BEAM-10529] add coder translation test

e3e0223

[BEAM-10529] add additional check to typecoder to not accidentally mi…

30f06d6

…sidentify coders as nullable

johnjcasey marked this pull request as ready for review February 23, 2022 15:44

[BEAM-10529] add test to retrieve nullable coder from typehint

24dd876

chamikaramj requested a review from TheNeuralBit February 23, 2022 18:32

johnjcasey added 2 commits February 23, 2022 13:35

[BEAM-10529] run spotless

e642e7c

Merge remote-tracking branch 'origin/feature/BEAM-10529-nullable-Xlan…

f55074c

…g-Coder' into feature/BEAM-10529-nullable-Xlang-Coder

TheNeuralBit changed the title ~~Feature/beam 10529 nullable xlang coder~~ [BEAM-10529] nullable xlang coder Feb 23, 2022

TheNeuralBit reviewed Feb 23, 2022

View reviewed changes

chamikaramj reviewed Mar 8, 2022

View reviewed changes

johnjcasey added 2 commits March 8, 2022 15:28

[BEAM-10529] add go nullable coder

4dd1c54

[BEAM-10529] cleanup extra println

43c84e6

github-actions bot added the go label Mar 8, 2022

[BEAM-10529] improve comments, clean up python

3e93b56

Merge remote-tracking branch 'origin/feature/BEAM-10529-nullable-Xlan…

40556b7

…g-Coder' into feature/BEAM-10529-nullable-Xlang-Coder

johnjcasey requested a review from chamikaramj March 29, 2022 18:25

Merge remote-tracking branch 'origin/master' into feature/BEAM-10529-…

488e1d1

…nullable-Xlang-Coder

chamikaramj approved these changes Mar 29, 2022

View reviewed changes

lostluck requested changes Mar 30, 2022

View reviewed changes

[BEAM-10529] address go comments

cc243f1

lostluck approved these changes Mar 31, 2022

View reviewed changes

johnjcasey added 3 commits April 1, 2022 10:06

Merge remote-tracking branch 'origin/master' into feature/BEAM-10529-…

a343a57

…nullable-Xlang-Coder # Conflicts: # sdks/go/pkg/beam/core/typex/special.go

[BEAM-10529] remove extra check that was added in error

c9761f5

[BEAM-10529] fix typo

c5ae489

johnjcasey added 3 commits April 1, 2022 11:52

[BEAM-10529] re-order check for nonetype to prevent attribute errors

d0a5979

Merge remote-tracking branch 'origin/feature/BEAM-10529-nullable-Xlan…

962ec4b

…g-Coder' into feature/BEAM-10529-nullable-Xlang-Coder

[BEAM-10529] change isinstance to ==

dbe1bb3

TheNeuralBit merged commit 11cccb4 into apache:master Apr 4, 2022

jrmccluskey mentioned this pull request Apr 4, 2022

Fix go fmt break in core/typex/special.go #17266

Merged

4 tasks

johnjcasey deleted the feature/BEAM-10529-nullable-Xlang-Coder branch April 5, 2022 16:48

	coders := make([]*Coder, 1)
	coders[0] = component
	coders := []*Coder{ component }

[BEAM-10529] nullable xlang coder #16923

[BEAM-10529] nullable xlang coder #16923

Uh oh!

Conversation

johnjcasey commented Feb 22, 2022

GitHub Actions Tests Status (on master branch)

Uh oh!

johnjcasey commented Feb 23, 2022

Uh oh!

codecov bot commented Feb 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TheNeuralBit left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

chamikaramj commented Mar 29, 2022

Uh oh!

johnjcasey commented Mar 29, 2022

Uh oh!

lostluck left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 23, 2022 •

edited

Loading