Skip to content

[Bug]: textio and parquetio are not -beam_strict compatible #26468

@jeremyje

Description

@jeremyje

What happened?

If you run a pipeline with -beam_strict in Go you'll get the following errors from the v2 SDK.

There are possibly more issues like this within the SDK as well.

Full error:
strictness check failed
        caused by:
        validating pipeline with vet runner
pipeline is not performant, see diagnostic summary:
/**
PTransform Audit:
pardo github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio.writeFileFn struct[[*textio.writeFileFn]] FOUND textio.writeFileFn
        - ProcessElementFn function[[func(context.Context, int, func(*string) bool) error]] NEED_SHIM OK
pardo github.com/apache/beam/sdks/v2/go/pkg/beam/io/parquetio.parquetWriteFn struct[[*parquetio.parquetWriteFn]] FOUND parquetio.parquetWriteFn
        - ProcessElementFn function[[func(context.Context, int, func(*typex.X) bool) error]] NEED_SHIM OK

Summary
All exported?: true
0        Imports
0        Functions
0        Types
2        Shims
0        Emits
0        Inputs
Pipeline is not performant:
        requires additional shim generation
        Good News! All identifiers are exported; the pipeline's package can be patched with generated output.
Diagnostic output pre-amble for the code generator
Functions
Types
Shims
func(context.Context, int, func(*string) bool) error, func(context.Context, int, func(*string) bool) error
func(context.Context, int, func(*typex.X) bool) error, func(context.Context, int, func(*typex.X) bool) error
Emitters
Iterators 
*/

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions