Skip to content

Conversation

@kaxil
Copy link
Member

@kaxil kaxil commented Jan 11, 2025

This PR adds a Context class that is used for Type hinting of the Context dictionary. It replaces Context in the Scheduler.

part of #44481

Next PRs:

  • Port remaining keys to Task SDK's Context dict
  • Change Typehint in providers while keeping support for 2.x
  • Remove context dict duplication in the following code:
    # NOTE: If you add to this dict, make sure to also update the following:
    # * Context in airflow/utils/context.pyi
    # * KNOWN_CONTEXT_KEYS in airflow/utils/context.py
    # * Table in docs/apache-airflow/templates-ref.rst
    context: dict[str, Any] = {
    "dag": dag,
    "dag_run": dag_run,
    "data_interval_end": timezone.coerce_datetime(data_interval.end),
    "data_interval_start": timezone.coerce_datetime(data_interval.start),
    "outlet_events": OutletEventAccessors(),
    "ds": ds,
    "ds_nodash": ds_nodash,
    "expanded_ti_count": expanded_ti_count,
    "inlets": task.inlets,
    "inlet_events": InletEventsAccessors(task.inlets, session=session),
    "logical_date": logical_date,
    "macros": macros,
    "map_index_template": task.map_index_template,
    "outlets": task.outlets,
    "params": validated_params,
    "prev_data_interval_start_success": get_prev_data_interval_start_success(),
    "prev_data_interval_end_success": get_prev_data_interval_end_success(),
    "prev_start_date_success": get_prev_start_date_success(),
    "prev_end_date_success": get_prev_end_date_success(),
    "run_id": task_instance.run_id,
    "task": task,
    "task_instance": task_instance,
    "task_instance_key_str": f"{task.dag_id}__{task.task_id}__{ds_nodash}",
    "test_mode": task_instance.test_mode,
    "ti": task_instance,
    "triggering_asset_events": lazy_object_proxy.Proxy(get_triggering_events),
    "ts": ts,
    "ts_nodash": ts_nodash,
    "ts_nodash_with_tz": ts_nodash_with_tz,
    "var": {
    "json": VariableAccessor(deserialize_json=True),
    "value": VariableAccessor(deserialize_json=False),
    },
    "conn": ConnectionAccessor(),
    }
    # Mypy doesn't like turning existing dicts in to a TypeDict -- and we "lie" in the type stub to say it
    # is one, but in practice it isn't. See https://github.com/python/mypy/issues/8890
    return Context(context) # type: ignore

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@kaxil kaxil force-pushed the add-context-in-sdk branch from 614ec7a to 521d92c Compare January 11, 2025 19:13
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in #43902 so we no longer need this code.

In preparation of #45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
@kaxil kaxil force-pushed the add-context-in-sdk branch 8 times, most recently from d881d06 to 372f449 Compare January 13, 2025 23:08
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jan 13, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
@kaxil kaxil force-pushed the add-context-in-sdk branch from e806c84 to 4f2e2a8 Compare January 14, 2025 08:43
@kaxil kaxil marked this pull request as ready for review January 14, 2025 08:43
@kaxil kaxil merged commit f616c62 into apache:main Jan 14, 2025
86 checks passed
@kaxil kaxil deleted the add-context-in-sdk branch January 14, 2025 19:58
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 14, 2025
This is now an empty directory following apache#45583 and can be removed.
kaxil added a commit that referenced this pull request Jan 14, 2025
This is now an empty directory following #45583 and can be removed.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 14, 2025
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 15, 2025
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
…5583)

This PR adds a `Context` class that is used for Type hinting of the Context dictionary. It replaces Context in the Scheduler.

part of apache#44481
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
This is now an empty directory following apache#45583 and can be removed.
dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
…5583)

This PR adds a `Context` class that is used for Type hinting of the Context dictionary. It replaces Context in the Scheduler.

part of apache#44481
dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
This is now an empty directory following apache#45583 and can be removed.
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
…5583)

This PR adds a `Context` class that is used for Type hinting of the Context dictionary. It replaces Context in the Scheduler.

part of apache#44481
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
This is now an empty directory following apache#45583 and can be removed.
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request May 28, 2025
We removed all the deprecated keys in apache/airflow#43902 so we no longer need this code.

In preparation of apache/airflow#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.

GitOrigin-RevId: 5597bb296106648fda48c768814f5ae6f3eb7a53
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request May 28, 2025
This is now an empty directory following apache/airflow#45583 and can be removed.

GitOrigin-RevId: b5b2383ae5fdfc4a200b686227192dec491ace7a
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 23, 2025
We removed all the deprecated keys in apache/airflow#43902 so we no longer need this code.

In preparation of apache/airflow#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.

GitOrigin-RevId: 5597bb296106648fda48c768814f5ae6f3eb7a53
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 23, 2025
This is now an empty directory following apache/airflow#45583 and can be removed.

GitOrigin-RevId: b5b2383ae5fdfc4a200b686227192dec491ace7a
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 21, 2025
We removed all the deprecated keys in apache/airflow#43902 so we no longer need this code.

In preparation of apache/airflow#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.

GitOrigin-RevId: 5597bb296106648fda48c768814f5ae6f3eb7a53
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 21, 2025
This is now an empty directory following apache/airflow#45583 and can be removed.

GitOrigin-RevId: b5b2383ae5fdfc4a200b686227192dec491ace7a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants