Skip to content

Conversation

@Lunderberg
Copy link
Contributor

@Lunderberg Lunderberg commented Mar 11, 2024

Summary

Prior to this commit, the CallsiteParameterAdder object could not be pickled. As a result, structlog configurations could not be propagated to subprocesses using pickle.dumps(structlog.get_config()). This commit updates the handlers in CallsiteParameterAdder._handlers to be free functions, rather than lambda functions, so that all class members can be pickled.

Closes #600

Pull Request Check List

  • Do not open pull requests from your main branch – use a separate branch!
    • There's a ton of footguns waiting if you don't heed this warning. You can still go back to your project, create a branch from your main branch, push it, and open the pull request from the new branch.
    • This is not a pre-requisite for your your pull request to be accepted, but you have been warned.
  • Added tests for changed code.
    • The CI fails with less than 100% coverage.
  • [n/a] New APIs are added to our typing tests in api.py.
  • [n/a] Updated documentation for changed code.
    • [n/a] New functions/classes have to be added to docs/api.rst by hand.
    • [n/a] Changed/added classes/methods/functions have appropriate versionadded, versionchanged, or deprecated directives.
      • The next version is the second number in the current release + 1. The first number represents the current year. So if the current version on PyPI is 23.1.0, the next version is gonna be 23.2.0. If the next version is the first in the new year, it'll be 24.1.0.
  • Documentation in .rst and .md files is written using semantic newlines.
  • Changes (and possible deprecations) are documented in the changelog.
  • Consider granting push permissions to the PR branch, so maintainers can fix minor issues themselves without pestering you.

Lunderberg and others added 2 commits March 11, 2024 18:15
Prior to this commit, the `CallsiteParameterAdder` object could not be
pickled.  As a result, structlog configurations could not be propagated
to subprocesses using `pickle.dumps(structlog.get_config())`.  This
commit updates the handlers in `CallsiteParameterAdder._handlers` to be
free functions, rather than lambda functions, so that all class members
can be pickled.

Closes hynek#600
Lunderberg added a commit to octoml/mlc-llm that referenced this pull request Mar 11, 2024
Required for compatibility with TVM PR
apache/tvm#16618.  Can be removed after
upstream `structlog` hynek/structlog#603
lands.

This is a backport of the OLLM PR
https://github.com/octoml/ollm/pull/409, and is not needed after the
MLC-serve migration.
masahi pushed a commit to octoml/mlc-llm that referenced this pull request Mar 11, 2024
Required for compatibility with TVM PR
apache/tvm#16618.  Can be removed after
upstream `structlog` hynek/structlog#603
lands.

This is a backport of the OLLM PR
https://github.com/octoml/ollm/pull/409, and is not needed after the
MLC-serve migration.
Copy link
Owner

@hynek hynek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make the function names verbs please? (iow: _pathname_get_pathname).

Changelog entry would be great too ("structlog.processors.CallsiteParameterAdder is now pickleable." + link here).

(oh and get the CI green too, pls 😇 – I think it's just missing type hints for the functions?)

@Lunderberg
Copy link
Contributor Author

Can you make the function names verbs please?

Can do. I've updated each function name from FOO to _get_callsite_FOO.

Changelog entry would be great too

And updated. (Made the commit for the changelog yesterday, but forgot to push it.)

(oh and get the CI green too, pls 😇 – I think it's just missing type hints for the functions?)

Thank you on pointing me in the right direction. Updated the signatures from (module, frame_info) to (module: str, frame_info: inspect.Traceback) to match the existing type annotation for CallsiteParameterAdder._handlers.

Copy link
Owner

@hynek hynek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

@hynek hynek enabled auto-merge (squash) March 14, 2024 05:35
@hynek hynek merged commit a62bbf6 into hynek:main Mar 14, 2024
@Lunderberg Lunderberg deleted the picklable_callsite_parameter_adder branch March 14, 2024 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CallsiteParameterAdder not pickleable

2 participants