Skip to content

Conversation

@jscheffl
Copy link
Contributor

@jscheffl jscheffl commented Jul 20, 2024

In order to leverage the AIP-44 interface for AIP-69 / Remote Executor I need some small adjustments to "hook into" the RPC API. This PR extracts from PoC implementation of AIP-69 (#40900) the needed internal API adjustments:

  • The private cached method map _initialize_map() needs to be extended, not to replicate the function list I'd like to remove the private tag to extend it with a few methods in remote executor. Would not add it to be a "public" interface though.
  • Allowing to use a different API path from the standard internal API URL. For the remote case we would expose the URL to another (plugin hosted or separately operated) path, such a URL with path should be configurable as API endpoint
  • For the setup of the CLI for Remote worker a way to force usage of internal API as in the remote case we would configure other config.
  • Allow the (Remote Provider Package) to hook-in additional classes for serialization to Pydantic mapping for extended API communication via RPC

@jscheffl jscheffl added this to the Airflow 2.10.0 milestone Jul 20, 2024
@jscheffl jscheffl marked this pull request as ready for review July 20, 2024 12:57
@potiuk potiuk force-pushed the feature/have-integration-points-for-aip-69-in-internal-api branch from 39ff6f6 to e8291d3 Compare July 20, 2024 14:50
@potiuk
Copy link
Member

potiuk commented Jul 20, 2024

I rebased it after I fixed selective checks in #40904

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one question.

@jscheffl jscheffl merged commit 6745cb8 into apache:main Jul 21, 2024
@ephraimbuddy ephraimbuddy added AIP-44 Airflow Internal API changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) labels Jul 22, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
* Have integration points for AIP-69 in Internal API

* Allow adding custom classes into serialization mapping

* Have integration points for AIP-69 in Internal API

* Fix pytest, reverse validation check for scheme

* Have integration points for AIP-69 in Internal API

* Review feedback
@jscheffl jscheffl deleted the feature/have-integration-points-for-aip-69-in-internal-api branch October 5, 2025 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIP-44 Airflow Internal API changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..)

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants