Adds helper functions for computing and using code object identifiers to facilitate more efficient pickling of lambdas. (#35656) #35656

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

tvalentyn merged 142 commits into apache:master from kristynsmith:master

Aug 15, 2025

Contributor

kristynsmith commented Jul 22, 2025

Creates consistent pickling for lambdas by creating a unique name for lambdas that can be used for stable referencing.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

kristynsmith added 20 commits

July 22, 2025 12:05


          Update code_object_pickler.py to include consistent pickling for lambdas

74f690a


          Create code_object_pickler_test.py

23fa4d8


          Update code_object_pickler_test.py

86e12b2


          Create after_module_add_function.py

d17d08a


          Create after_module_add_lambda_variable.py

6b66de9


          Create after_module_add_variable.py

a1106e6


          Create after_module_remove_lambda_variable.py

bbe555c


          Create after_module_remove_variable.py

4a8d350


          Create after_module_with_classes.py


          Create after_module_with_global_variable.py

c2ab713


          Create after_module_with_nested_function.py

ca853c2


          Create after_module_with_nested_function_2.py

8c26a6c


          Create after_module_with_single_class.py

40b767c


          Create before_module_with_classes.py

251b1d7


          Create before_module_with_functions.py


          Create before_module_with_lambdas.py

6169dbe


          Create module_with_default_argument.py

03ba258


          Create pickle_code_path_test.py

f1ce3ca


          Update code_object_pickler.py

2e9ff8b


          Update pickle_code_path_test.py

d940cfa

github-actions bot added the python label

kristynsmith added 9 commits

July 22, 2025 14:06


          Update code_object_pickler_test.py

0438ba4


          Update pickle_code_path_test.py

30ece4b


          Update after_module_add_function.py

3dc6856


          Update after_module_add_lambda_variable.py

74cb523


          Update after_module_add_variable.py

82e5656


          Update after_module_remove_lambda_variable.py

0c0cf87


          Update after_module_remove_variable.py

50011e4


          Update after_module_with_classes.py

35f2dfe


          Update after_module_with_global_variable.py

256a010

tvalentyn reviewed

View reviewed changes

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

tvalentyn reviewed

View reviewed changes

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

kristynsmith added 6 commits

August 7, 2025 11:12


          change type hints

cd83d96


          fix typo that caused error

9660d41


          fix formatting and lint errors


          fix typehints

d49f850


          fix formatting and lint

8f4344b


          Update code_object_pickler.py

a4ebbd5

tvalentyn reviewed

View reviewed changes

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

kristynsmith closed this

kristynsmith reopened this

kristynsmith added 3 commits

August 12, 2025 15:31


          Update code_object_pickler.py

bac70eb


          fix union error

7be70e6


          import union

eca023d

tvalentyn reviewed

View reviewed changes

sdks/python/apache_beam/internal/code_object_pickler.py Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler.py Show resolved Hide resolved

tvalentyn reviewed

View reviewed changes

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler_test.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler_test.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/pickle_code_path_test.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/pickle_code_path_test.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/test_cases/__init__.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/test_cases/__init__.py Show resolved Hide resolved

sdks/python/apache_beam/internal/test_cases/after_module_add_function.py Show resolved Hide resolved

sdks/python/apache_beam/internal/pickle_code_path_test.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

kristynsmith added 2 commits

August 14, 2025 11:47


          update code_object_pickler.py docstring, move pickle_code_path_test.p…

ff59041

…y into code_object_pickler_test.py, rename files in test_data, rename tests in code_object_pickler_test.py


          fix error

2a14db5

tvalentyn approved these changes

View reviewed changes

Contributor

tvalentyn left a comment

Thanks!

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler.py Outdated

    
                return path

              def get_code_path(callable: types.FunctionType):

Contributor

tvalentyn Aug 14, 2025

(totally optional) wdyt about naming this method:

get_code_object_identifier

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler_test.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/internal/code_object_pickler_test.py Show resolved Hide resolved

kristynsmith added 3 commits

August 14, 2025 14:07


          rename functions in code_object_pickler.py

81f5a33


          fix errors

a17c69d


          fix formatting and lint errors

7ee69f1

tvalentyn approved these changes

View reviewed changes

sdks/python/apache_beam/internal/code_object_pickler.py Outdated Show resolved Hide resolved


          Update sdks/python/apache_beam/internal/code_object_pickler.py

a9838a1

tvalentyn merged commit afe94ad into apache:master

81 of 86 checks passed

tvalentyn changed the title ~~Add support for lambda name pickling.~~ Adds helper functions for computing and using code object identifiers to facilitate more efficient pickling of lambdas. (#35656)

DKER2 pushed a commit to DKER2/beam that referenced this pull request


          Add helper functions for computing and using code object identifiers …

0ba5323

…to facilitate more efficient pickling of lambdas. (apache#35656)

* Update code_object_pickler.py to include consistent pickling for lambdas

* Create code_object_pickler_test.py

* Update code_object_pickler_test.py

* Create after_module_add_function.py

* Create after_module_add_lambda_variable.py

* Create after_module_add_variable.py

* Create after_module_remove_lambda_variable.py

* Create after_module_remove_variable.py

* Create after_module_with_classes.py

* Create after_module_with_global_variable.py

* Create after_module_with_nested_function.py

* Create after_module_with_nested_function_2.py

* Create after_module_with_single_class.py

* Create before_module_with_classes.py

* Create before_module_with_functions.py

* Create before_module_with_lambdas.py

* Create module_with_default_argument.py

* Create pickle_code_path_test.py

* Update code_object_pickler.py

* Update pickle_code_path_test.py

* Update code_object_pickler_test.py

* Update pickle_code_path_test.py

* Update after_module_add_function.py

* Update after_module_add_lambda_variable.py

* Update after_module_add_variable.py

* Update after_module_remove_lambda_variable.py

* Update after_module_remove_variable.py

* Update after_module_with_classes.py

* Update after_module_with_global_variable.py

* Update after_module_with_nested_function.py

* Update after_module_with_nested_function_2.py

* Update after_module_with_single_class.py

* Update before_module_with_classes.py

* Update before_module_with_functions.py

* Update before_module_with_lambdas.py

* Update module_with_default_argument.py

* Update after_module_add_lambda_variable.py

* Update after_module_with_nested_function.py

* Update code_object_pickler.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler.py

* Update pickle_code_path_test.py

* Update after_module_with_nested_function.py

* Update after_module_with_classes.py

* Update after_module_with_nested_function_2.py

* Update code_object_pickler_test.py

* Update code_object_pickler.py

* Update pickle_code_path_test.py

* Update pickle_code_path_test.py

* Update code_object_pickler.py

* Update after_module_with_classes.py

* Update after_module_with_nested_function.py

* Update after_module_with_nested_function_2.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update pickle_code_path_test.py

* Update pickle_code_path_test.py

* Create __init__.py

* Update before_module_with_classes.py

* Update after_module_with_single_class.py

* Update after_module_with_classes.py

* Update pickle_code_path_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update pickle_code_path_test.py

* Update code_object_pickler_test.py

* Update pickle_code_path_test.py

* Update code_object_pickler_test.py

* Update pickle_code_path_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler_test.py

* Update pickle_code_path_test.py

* Update after_module_with_nested_function.py

* Update after_module_with_classes.py

* Update after_module_with_single_class.py

* Update before_module_with_functions.py

* Update after_module_add_lambda_variable.py

* Update after_module_with_global_variable.py

* Update after_module_with_nested_function_2.py

* Update after_module_add_function.py

* Update before_module_with_lambdas.py

* Update before_module_with_classes.py

* Update after_module_add_variable.py

* Update code_object_pickler.py

* Update pickle_code_path_test.py

* Update after_module_add_function.py

* Update after_module_add_lambda_variable.py

* Update after_module_add_function.py

* Update after_module_add_function.py

* Update after_module_add_variable.py

* Update after_module_remove_lambda_variable.py

* Update after_module_remove_variable.py

* Update after_module_with_classes.py

* Update after_module_with_global_variable.py

* Update after_module_with_nested_function.py

* Update after_module_with_nested_function_2.py

* Update after_module_with_single_class.py

* Update before_module_with_classes.py

* Update before_module_with_functions.py

* Update before_module_with_lambdas.py

* Update module_with_default_argument.py

* Update after_module_with_classes.py

* Update before_module_with_classes.py

* Update code_object_pickler_test.py

* Update after_module_add_function.py

* Update pickle_code_path_test.py

* Update code_object_pickler_test.py

* Update code_object_pickler.py

* Update code_object_pickler_test.py

* Update pickle_code_path_test.py

* fix formatting and lint

* formatting and lint changes

* Update code_object_pickler.py

* change type hints

* fix typo that caused error

* fix formatting and lint errors

* fix typehints

* fix formatting and lint

* Update code_object_pickler.py

* Update code_object_pickler.py

* fix union error

* import union

* update code_object_pickler.py docstring, move pickle_code_path_test.py into code_object_pickler_test.py, rename files in test_data, rename tests in code_object_pickler_test.py

* fix error

* rename functions in code_object_pickler.py

* fix errors

* fix formatting and lint errors

* Update sdks/python/apache_beam/internal/code_object_pickler.py

---------

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

kristynsmith mentioned this pull request

Fix typos and update test implementation from #35656 #35958

Merged

3 tasks

tvalentyn pushed a commit that referenced this pull request


          Fix typos and update test implementation from #35656 (#35958)

81e4db7

* implement lambda name pickling in cloudpickle

* add enable_lambda_name to __init__

* fix formatting and lint

* fix typo

* fix code paths in test

* fix tests

* fix lint

* fix formatting and failing test

* fix formatting again

* remove cloudpickle implementation to leave only typo fixes and fixing test structure.

* fix _make_function typo

* revert regex

* fix failing tests

* fix formatting

* update prefix to not hardcode

damccorm added a commit that referenced this pull request


          Properly add milvus as extra dependency (#35986)

c082ab9

* sdks/python: properly make milvus as extra dependency

* sdks/python: update image requirements

* .github: trigger postcommit python

* sdks/python: fix linting issues

* sdks/python: fix formatting issues

* .github: trigger beam postcommit python

* sdks/python: revert milvus version in itests

* sdks/python: update image requirements

* trigger_files: trigger postcommit python

* Bump github.com/docker/go-connections from 0.5.0 to 0.6.0 in /sdks (#35906)

Bumps [github.com/docker/go-connections](https://github.com/docker/go-connections) from 0.5.0 to 0.6.0.
- [Commits](docker/go-connections@v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: github.com/docker/go-connections
  dependency-version: 0.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add the readme link to new YAML examples (#35941)

* Bump google.golang.org/api from 0.247.0 to 0.248.0 in /sdks (#35969)

* Remove mysql-connector-python dependency (#35932)

* Fix typos and update test implementation from #35656 (#35958)

* implement lambda name pickling in cloudpickle

* add enable_lambda_name to __init__

* fix formatting and lint

* fix typo

* fix code paths in test

* fix tests

* fix lint

* fix formatting and failing test

* fix formatting again

* remove cloudpickle implementation to leave only typo fixes and fixing test structure.

* fix _make_function typo

* revert regex

* fix failing tests

* fix formatting

* update prefix to not hardcode

* feat(mongodb): upgrade MongoDB Java driver to version 5.5.0 (#35946)

* feat(mongodb): upgrade MongoDB Java driver to version 5.5.0

Update MongoDB Java driver from 3.12.11 to 5.5.0 and refactor code to use new API
Add mongo-bson dependency required by new driver version
Replace deprecated MongoClient with MongoClients and update GridFS implementation

* refactor(mongodb): update MongoDB client usage to modern API

Replace deprecated MongoClient with MongoClients.create() and update database drop method

* build(dependencies): add mongodb driver core dependency

Add mongodb-driver-core to support MongoDB Java driver functionality.
Also mark mongo_java_driver as permitUnusedDeclared and add testImplementation.

* fix(mongodb): update embedded mongo version and fix split key filtering

Update embedded MongoDB test dependency to version 3.5.4 and simplify split key filtering logic by using BsonObjectId for range queries. This ensures proper type handling when filtering MongoDB documents by _id field.

* build: add mongodb-driver-core dependency

Add mongodb-driver-core version 5.5.0 to support MongoDB Java driver functionality

* use version

* refactor: simplify mongo client creation logic

Remove redundant null check and consolidate uri handling in MongoDbGridFSIO

* Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks (#35974)

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.18.6 to 1.18.7.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.18.7/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@config/v1.18.6...config/v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.18.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump google.golang.org/grpc from 1.74.2 to 1.75.0 in /sdks (#35971)

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.74.2 to 1.75.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.74.2...v1.75.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.75.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Override localhost endpoint when a worker is running in docker on mac (#35964)

* fix(parquetio): handle missing nullable fields in row conversion (#35948)

* fix(parquetio): handle missing nullable fields in row conversion

Add null value handling when converting rows to Arrow tables for nullable fields that are missing from input data. This fixes KeyError when writing to Parquet with missing nullable fields, addressing issue #35791.

* fix lint

* Bump cloud.google.com/go/storage from 1.56.0 to 1.56.1 in /sdks (#35980)

Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.56.0 to 1.56.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@spanner/v1.56.0...storage/v1.56.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-version: 1.56.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [Prism] Fix segv when docker container self-terminated. (#35977)

* Fix segv when docker container is self-terminated

* Add some debug logging for docker and process env.

* add a jinja % include/import pipeline example to docs (#35931)

* add a jinja include pipeline example

* update yaml doc with import example

* address gemini and other comments

* fix table of contents for readme

* add link to jinja pipeline examples

* Bump github.com/aws/aws-sdk-go-v2/config from 1.31.2 to 1.31.3 in /sdks (#35983)

* Add a security GCP log analyzer (#35922)

* Add the base log_analyzer

* Add github action for security logging

* Enhance LogAnalyzer to filter logs by time range and include file names in event summary

* Add dry-run option for weekly email report generation in LogAnalyzer

* Better error handling for timezones and missing details

* Refactor LogAnalyzer to use SinkCls for type consistency and enhance bucket permission management for log sinks

* update py containers (#35982)

* [YAML]: add import jinja pipeline example (#35945)

* add import jinja pipeline example

* revert name change

* update overall examples readme

* fix lint issue

* fix gemini small issue

* Update sdks/python/apache_beam/yaml/examples/transforms/jinja/import/README.md

---------

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

* workflows: capture DinD tests in PreCommit Py Coverage workflow

* workflows: temporarily removing `ubuntu-latest` till resolving deps

* workflows: add `matrix.os` label to `beam_PreCommit_Python_Coverage`

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Mohamed Awnallah <mohamedmohey2352@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chamikara Jayalath <chamikaramj@gmail.com>
Co-authored-by: Yi Hu <yathu@google.com>
Co-authored-by: kristynsmith <kristynsmith@google.com>
Co-authored-by: liferoad <huxiangqian@gmail.com>
Co-authored-by: Shunping Huang <shunping@google.com>
Co-authored-by: Derrick Williams <derrickaw@google.com>
Co-authored-by: Enrique Calderon <71863693+ksobrenat32@users.noreply.github.com>
Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com>
Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Next Action: Reviewers python