AIP-81 - API Communication Mechanism

[bugraoz93]

closes: #42561 and #43656

Summary
This PR introduces a central API communication mechanism for the CLI and adopts one of the broadest endpoint ranges in the API, focusing on connection commands. After several iterations to automate API call generation and handle edge cases, I couldn't achieve a fully dynamic request generation process. Instead, I landed on a structure similar to TaskSDK <-> ExecutionAPI. This approach simplifies reviews, maintains consistency across clients, and opens the door for better automation in future iterations, possibly in 3.1 or 3.2.

Implementation Details

• Added a central API communication mechanism for the CLI.
• Implemented most of the necessary CLI operations.
• Data model generation from RestAPI to CLI is handled using uv.
• Added decorators to cascade generic processes to operations.
• Created a testing framework for CLI testing, with an example implementation for connection command.
• Pending Work
• Some minor features related to the connection command are still in progress and will be added in follow-up updates before 3.0.

Note: There are two missing functionalities in this PR for the connection command. Those will be included with follow-up tasks.

• AIP-81 Implement Create Default Connections in REST API (FastAPI) #45302 (Create Default Connections)
• AIP-81 Include Overwrite Functionality to Bulk Insert API Endpoints for Connection&Pool #45303 (Overwrite the Bulk Insert Connections when Flag is passed)

Notes on task_command
While working on this, I realized that task_command behaves more like a local_command than a remote_command. This needs further discussion, especially regarding its integration with TaskSDK. From the CLI perspective, depending on TaskSDK seems cleaner than directly calling ExecutionAPI, it avoids duplication and keeps the structure consistent. Even if the CLI calls TaskSDK, the two-hop process to reach the ExecutionAPI feels a bit redundant. What do you think, @ashb (to all, please jump into discussion, I tagged Ash to follow up the previous discussion started on this in Slack)?

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[jscheffl]

Note: As PR #45312 has been merged, the code formatting rules have changed for new UI. Please rebase and re-run pre-commit checks to ensure that formatting in folder airflow/ui is adjusted.

[bugraoz93]

Note Update: The missing parts are included in the PR.

• Create default connections in connection_command
• Include overwrite functionality for file import operations for connections and pool endpoint
• Include overwrite functionality to connection_command

Thanks @jason810496 for your contributions!

[jscheffl]

I saw this PR and was "afraid" from the size but after the dev call today I looked again. Agree to comments from @ashb - after this I assume it can be merged.

Next time would be better to cut PRs in smaller chunks not to scare reviewers away by number of LoC :-D

[bugraoz93]

Thanks a lot for your time and your comments! I will address them and adjust the code accordingly soon.

Next time would be better to cut PRs in smaller chunks not to scare reviewers away by number of LoC :-D

Indeed, that's on me making it big. I thought most parts were repeated on the operations while making the integration easier for contributers and went for it. I agree, that's on me, I will definitely split next time :D

[jedcunningham]

Failure is unrelated. Merging!

[jedcunningham]

#protm

[jason810496]

Hi @bugraoz93, since this PR has been merged, I think we can start working on the #45661 meta issue, right? Or are there any other dependencies that need to be completed first?

[bugraoz93]

Hi @bugraoz93, since this PR has been merged, I think we can start working on the #45661 meta issue, right? Or are there any other dependencies that need to be completed first?

Hi @jason810496 , thanks for flagging! Normally, yes, they are unblocked. Since the direction has been changed and we don't have backward compatibility with the local commands running in core, we can provide outputs and data in a different and in a standard way than what the current commands do. I am trying something that could generate all the commands on the fly from operations created/implemented. It can save us from lots of effort. That's why I holded pinging all of you from the issue for a bit. I will share more details soon, either today or tomorrow, so the direction of the implementations can also be clear for transitions.