Add create XCom endpoint in RestAPI

[shubhamraj-git]

related: #45966

Update: Modifying the description only for create API and simplifying use case.

The XCom creation API fills a critical gap by enabling dynamic updates to workflows.
Current XComs are tied to task execution, limiting flexibility for long-running workflows.
The API allows dynamic creation of XCom values during execution, This is needed mostly for the use case of fixing manually recover from problems with workflows.
Avoid placeholder tasks just to generate XComs.

We will also have the edit XCom API as part of #45966 coming soon. This PR is only for creation.

---

Steps to play around the feature.

1. Add the following Dag.

This Airflow DAG demonstrates how to use XComs for passing data between tasks. The first task (wait_and_not_push) waits for 1 minute but does not push any XCom. The second task (pull_and_print) attempts to pull an XCom value with the key outbound_key1 from the first task, logs it if found, or warns if absent.

from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
import time
import logging
import pendulum

# Default arguments for the DAG
default_args = {
    'owner': 'airflow',
    'retries': 1
}

# Initialize the DAG
with DAG(
    dag_id="xcom_pull_example_dag",
    default_args=default_args,
    description="A DAG demonstrating XCom pull with key 'outbound_key1'",
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    catchup=False,
    tags=["xcom", "example"],
) as dag:

    def wait_and_not_push_xcom(**kwargs):
        """
        Task to wait for 1 minute.
        """
        logging.info("Waiting for 1 minute...")
        time.sleep(60)  # Wait for 1 minute

    def pull_and_print_xcom(**kwargs):
        """
        Task to pull an XCom entry with key 'outbound_key1' and print its value.
        """
        ti = kwargs["ti"]
        xcom_value = ti.xcom_pull(task_ids="wait_and_not_push", key="outbound_key1")

        if xcom_value:
            logging.info(f"Retrieved XCom value for key 'outbound_key1': {xcom_value}")
        else:
            logging.warning("No XCom value found for key 'outbound_key1'!")

    # First task: Wait for 1 minute
    wait_and_not_push = PythonOperator(
        task_id="wait_and_not_push",
        python_callable=wait_and_not_push_xcom
    )

    # Second task: Pull the XCom entry and print its value
    pull_and_print = PythonOperator(
        task_id="pull_and_print",
        python_callable=pull_and_print_xcom
    )

    # Define the task dependencies
    wait_and_not_push >> pull_and_print

3. Trigger the run.
4. As soon as the run starts, trigger an XCom creation through API (Can do it from swagger: http://localhost:29091/docs#/XCom/create_xcom_entry)

Now, check the logs, You can see the XCom pull was successful.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[jscheffl]

Just adding comments as copy of the 1:1 conversation:
I doubt that pushing an XCom from an external system is a good way to signal data to an existing workflow. Technically it would be working but this opens the door to "hacky integrations" in my view. XCom (as far as I have seen it) was made for inter-task communication and not as an interface to external applications.
I don't know if there is a specific design pattern that need to be considered. Just my thought.

I would understand if the API can be used to "patch" wrong/incorrect XCom values administratively for specific szenarios where upstream tasks can not be re-executed but values are incorrect. But the PR of this API is only adding XCom, will raise an error if the value is existing.

So before merging I'd propose to have a bit of discussion between the other maintainers if we want to open XCom up to external applications to signal a processing status.

If so then this integration pattern would require a bit of documentation as well.

[eladkal]

Just adding comments as copy of the 1:1 conversation:
I doubt that pushing an XCom from an external system is a good way to signal data to an existing workflow.

This is not the use case. The use case is incident recovery. When on call fixes problems with data pipeline manual interventions sometimes required.

[jscheffl]

Just adding comments as copy of the 1:1 conversation:
I doubt that pushing an XCom from an external system is a good way to signal data to an existing workflow.

This is not the use case. The use case is incident recovery. When on call fixes problems with data pipeline manual interventions sometimes required.

If I read the PR description incident recovery is not the use case.

For incident recovery also patching of XCom would be needed. It is called as:

enabling dynamic, external updates to workflows

[eladkal]

@jscheffl The PR is to solve the pain I raised in #45966
I don't think this endpoint adds more risk than creating variables endpoint. Do you agree?

@shubhamraj-git I suggest to strike trough the extra use cases mentioned in the description.

[jscheffl]

Tests are missing

[jscheffl]

@jscheffl The PR is to solve the pain I raised in #45966 I don't think this endpoint adds more risk than creating variables endpoint. Do you agree?

I just fear a undesired integration pattern coming in. But if you say this is a pain the need to be fixed, I am not blocking.

[shubhamraj-git]

Tests are missing

Yes, I was waiting for discussions to be done. I will include the tests before merging. @jscheffl

[pierrejeambrun]

Code looks good to me.

Indeed we need tests before being able to merge it.

Same as Jens on this one, indeed it feels a little weird from a functional point of view, but if this is required for some use case, then I'm fine with it.

[eladkal]

ready to merge?

[shubhamraj-git]

Yes, added the tests and good to merge.

[eladkal]

@pierrejeambrun @jscheffl any further comments?

[jscheffl]

@pierrejeambrun @jscheffl any further comments?

No. No objections.

[pierrejeambrun]

I think we need one last rebase.