[Prism]: Support Docker in MacOS with transforms for Cross Language

[nguymin4]

What would you like to happen?

I tried to run the example of Go SDK - SQL locally with Prism Runner

• Linux: ✅ - Succeeded
• MacOS: ❌ - Failed

Issue

localhost is set and used in many places in Prism runner (Go), and thus this doesn't work on Mac as we should use host.docker.internal instead

• Python SDK

  beam/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py

  Line 747 in 2b4056b

  return 'host.docker.internal'

• Java SDK

  beam/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java

  Line 209 in 2b4056b

  private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";

Proposed changes

I now have a working solution for this issue locally.
Based on my testing, there are at least 2-3 places needs to be modified (depends on implementation)

• func GetProvisionInfo

  beam/sdks/go/pkg/beam/runners/prism/internal/worker/worker.go

  Line 128 in 2b4056b

  func (wk *W) GetProvisionInfo(_ context.Context, _ *fnpb.GetProvisionInfoRequest) (*fnpb.GetProvisionInfoResponse, error) {

• func NewMultipleW

  beam/sdks/go/pkg/beam/runners/prism/internal/worker/worker.go

  Line 683 in 2b4056b

  endpoint: "localhost:" + p,

• func dockerEnvironment

  beam/sdks/go/pkg/beam/runners/prism/internal/environments.go

  Line 210 in 2b4056b

  ccr, err := cli.ContainerCreate(ctx, &container.Config{

Challenges

1. We need to modify the GetProvisionInfo and dockerEnvironment at the same time. I found this ticket https://issues.apache.org/jira/browse/BEAM-8201
   => If I understand correctly, it still relies on the combination of the CLI args and the provisioning API. We probably want to consolidate this?

2. In NewMultipleW actually I can only make this work if I replace localhost with the actual ipv4 address of host e.g. 192.168.x.x
   => And if we use the IP address, we don't need to modify func dockerEnvironment. But I'm open for advice and suggestion as I don't like using IP like this either.

3. I saw a TODO in pkg/beam/runners/prism/internal/environments.go // TODO move environment handling to the worker package. but I don't know our current plan with this.

Summary

• I have a working version in my local machine, tested on both Linux and MacOS.
  • However, as mentioned in the challenges section, before creating the PR, I hope we can discuss my implementation plan above.
• I can contribute more to this beside my issue with Docker on MacOS

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[shunping]

Thanks for reporting. I run into this problem too and I am looking into that.

[nguymin4]

Thank you. I thought about working on a PR last weekkend, but also happy to see your PR, as you may have better solution for this.

[shunping]

Do you mind trying my PR and see if it works for you for both linux and macOS?

I found this ticket https://issues.apache.org/jira/browse/BEAM-8201
=> If I understand correctly, it still relies on the combination of the CLI args and the provisioning API. We probably want to consolidate this?If I understand correctly, it still relies on the combination of the CLI args and the provisioning API. We probably want to consolidate this?

Also thanks for bringing this up. I have the same doubt when debugging this issue. Consolidating this may introduce breaking changes, so we have to be cautious about that. Maybe it is worthwhile to send out a discussion in the dev list to get some feedback from others.

I can contribute more to this beside my issue with Docker on MacOS

You are absolutely welcome to contribute to Beam! Feel free to pick up issues, propose new features or work on any aspects you like. If you need help with that, you can tag me.

[nguymin4]

Absolutely, I can help with testing this out. I think this is the PR #35964

[shunping]

Closing this issue since #35964 has been merged.