Skip to content

Conversation

@mobuchowski
Copy link
Contributor

We've moved to a model where DagRun level events utilize ProcessPoolExecutor to send data, to not do this on the direct scheduler process. This PR adds more guards to that model:

  • do not send/pickle whole DagRun model to process, since serializing only selected attributes is cheaper and less error prone
  • add notification on ProcessPoolExecutor complete - this allows listener to log potential serialization failures.
  • guard whole listener with BaseException checks - we don't want errors to bubble to scheduler.

@mobuchowski mobuchowski force-pushed the notify-serialization-exception branch 12 times, most recently from 6593ede to 8e7ccf7 Compare August 29, 2024 11:22
@mobuchowski mobuchowski force-pushed the notify-serialization-exception branch from 8e7ccf7 to c8bca89 Compare August 31, 2024 13:59
…un, don't serialize unnecessary params, guard listener for exceptions

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants