Sync v3-1-stable with 3.1.2rc1 changes
#57640
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…cess Listener (apache#54568) (apache#57031) Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
) (apache#57062) * [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (apache#54568) (apache#57031) Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> * [v3-1-test] Fix Extra Links translation key. (apache#57030) (apache#57040) (cherry picked from commit 8d71c07) Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> * [v3-1-test] Fix dags list not rendering (apache#57037) (apache#57046) (cherry picked from commit 8668fe5) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> * [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (apache#57010) (apache#57045) Co-authored-by: Wei Lee <weilee.rx@gmail.com> fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943) * [v3-1-test] Fix dag list cache invalidation on favouriting a dag (apache#57039) (apache#57047) Fix dag list cache invalidation on favouriting a dag (cherry picked from commit 4e46212) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> * [v3-1-test] Add new PL translations. (apache#56359) (apache#57055) (cherry picked from commit 6583b6d) * Add last missing translation for 3.1.1 (apache#57057) * [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (apache#56980) (apache#57048) * [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (apache#56570) * fix: adjust PanelButtons spacing and alignment * Fix PanelButtons layout and button styling * Fix PanelButtons: use bg.subtle and remove unnecessary _hover (cherry picked from commit 4c35cd2) Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
…che#57097) * Fix grid links for tasks with retries * Update following code review (cherry picked from commit a8f2d97) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
…#57107) Changes from https://github.com/apache/airflow/tree/python-client/3.1.0rc1 / apache/airflow-client-python#138 (cherry picked from commit 6fb4e26) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
… of sort memory" error (apache#55589) (apache#57042) * [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (apache#54568) (apache#57031) Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> * [v3-1-test] fix get latest serialized_dag model query to prevent "Out of sort memory" error (apache#55589) * fix get latest serialized_dag model query * fix get latest serialized_dag model query * add db type check logic (cherry picked from commit 757db27) Co-authored-by: Jeongwoo Do <48639483+wjddn279@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> Co-authored-by: Jeongwoo Do <48639483+wjddn279@users.noreply.github.com>
…ion logging (apache#57172) (apache#57179) closes: apache#57167 PR apache#52562 changed `_handle_heartbeat_failures()` to accept an exception parameter and log it as a structured field. Now due to this change, during the first failed heartbeat attempt, the _handle_heartbeat_failures function logs a message by calling log.warning(), which accepts an exception parameter that expects a string type object. However, in the source code, [an exception type object is passed](https://github.com/apache/airflow/blob/54bd5d8cd9f6f477cc83445737614dec81c4323c/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1126) instead of a string type object. This results in a TypeError (like below) which causes task supervision to fail. The error looked like this: ```python 2025-10-23T17:58:22.900129Z [error ] Task execute_workload[aac34f36-54e1-46e4-ba47-15dba8ba7149] raised unexpected: TypeError('can only concatenate str (not "ConnectError") to str') [celery.app.trace] loc=trace.py:267 Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions yield File "/usr/python/lib/python3.10/site-packages/httpx/_transports/default.py", line 250, in handle_request resp = self._pool.handle_request(req) File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request raise exc from None File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request response = connection.handle_request( File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request raise exc File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 78, in handle_request stream = self._connect(request) File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 124, in _connect stream = self._network_backend.connect_tcp(**kwargs) File "/usr/python/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp with map_exceptions(exc_map): File "/usr/python/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/python/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ConnectError: [Errno 111] Connection refused ``` The change in apache#52562 was mainly made due to ruff upgrade reasons, so I am going back to using the standard Python logging pattern: pass exception to `exc_info` parameter. After changes, error looks like this: ```python 2025-10-23T18:24:39.303939Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client] loc=before.py:42 2025-10-23T18:24:40.307404Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client] loc=before.py:42 2025-10-23T18:24:41.417523Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client] loc=before.py:42 2025-10-23T18:24:43.801145Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client] loc=before.py:42 2025-10-23T18:24:46.690339Z [warning ] Failed to send heartbeat. Will be retried [supervisor] failed_heartbeats=3 loc=supervisor.py:1135 max_retries=3 ti_id=UUID('019a1250-48a0-756c-ab0b-9687289ef580') Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions yield File "/usr/python/lib/python3.10/site-packages/httpx/_transports/default.py", line 250, in handle_request resp = self._pool.handle_request(req) File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request raise exc from None File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request response = connection.handle_request( File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request raise exc File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 78, in handle_request stream = self._connect(request) File "/usr/python/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 124, in _connect stream = self._network_backend.connect_tcp(**kwargs) File "/usr/python/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp with map_exceptions(exc_map): File "/usr/python/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/python/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ConnectError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 1105, in _send_heartbeat_if_needed self.client.task_instances.heartbeat(self.id, pid=self._process.pid) File "/opt/airflow/task-sdk/src/airflow/sdk/api/client.py", line 259, in heartbeat self.client.put(f"task-instances/{id}/heartbeat", content=body.model_dump_json()) File "/usr/python/lib/python3.10/site-packages/httpx/_client.py", line 1181, in put return self.request( File "/usr/python/lib/python3.10/site-packages/tenacity/__init__.py", line 338, in wrapped_f return copy(f, *args, **kw) File "/usr/python/lib/python3.10/site-packages/tenacity/__init__.py", line 477, in __call__ do = self.iter(retry_state=retry_state) File "/usr/python/lib/python3.10/site-packages/tenacity/__init__.py", line 378, in iter result = action(retry_state) File "/usr/python/lib/python3.10/site-packages/tenacity/__init__.py", line 420, in exc_check raise retry_exc.reraise() File "/usr/python/lib/python3.10/site-packages/tenacity/__init__.py", line 187, in reraise raise self.last_attempt.result() File "/usr/python/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/usr/python/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/python/lib/python3.10/site-packages/tenacity/__init__.py", line 480, in __call__ result = fn(*args, **kwargs) File "/opt/airflow/task-sdk/src/airflow/sdk/api/client.py", line 894, in request return super().request(*args, **kwargs) File "/usr/python/lib/python3.10/site-packages/httpx/_client.py", line 825, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/python/lib/python3.10/site-packages/httpx/_client.py", line 914, in send response = self._send_handling_auth( File "/usr/python/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth response = self._send_handling_redirects( File "/usr/python/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects response = self._send_single_request(request) File "/usr/python/lib/python3.10/site-packages/httpx/_client.py", line 1014, in _send_single_request response = transport.handle_request(request) File "/usr/python/lib/python3.10/site-packages/httpx/_transports/default.py", line 249, in handle_request with map_httpcore_exceptions(): File "/usr/python/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/python/lib/python3.10/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ConnectError: [Errno 111] Connection refused ``` (cherry picked from commit 970d7da) Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
…e#56124) (apache#56986) * [v3-1-test] Implement integration test for airflowctl with 3.1 (apache#56124) * Implement initial integration test for airflowctl with 3.1 * password can be passed without interaction, update integration tests * Add AIRFLOW_CLI_DEBUG_MODE for enhanced CLI debugging and update integration tests to skip keyring * Warn user while running each command if debug mode enabled and explicitly state it shouldn't be used unless debugging or integration tests * Move python-on-whales to devel-common, use shared docker-composer file, update documentation mistakes * remove shared python-on-whales from airflow-ctl-tests/ Co-authored-by: Jarek Potiuk <jarek@potiuk.com> * Decouple docker compose logic from test method to pytest_sessionstart in conftest * Move python_on_whale import to file level * Reorder dependencies in pyproject.toml for consistency * Add workspace to main pyproject.toml, remove unused variable, move console to singleton __init__.py * Add workspace to main pyproject.toml, remove unused variable, move console to singleton __init__.py --------- (cherry picked from commit 80911e8) Co-authored-by: Bugra Ozturk <bugraoz93@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> * Remove e2e tests came from merge and rerun breeze images for all commands to fix accepting changes while merging * Add env file to compose and update error --------- Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
… en (apache#57060) (apache#57064) * [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (apache#54568) (apache#57031) Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> * [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (apache#56570) (apache#57062) * [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (apache#54568) (apache#57031) Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> * [v3-1-test] Fix Extra Links translation key. (apache#57030) (apache#57040) (cherry picked from commit 8d71c07) Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> * [v3-1-test] Fix dags list not rendering (apache#57037) (apache#57046) (cherry picked from commit 8668fe5) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> * [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (apache#57010) (apache#57045) Co-authored-by: Wei Lee <weilee.rx@gmail.com> fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943) * [v3-1-test] Fix dag list cache invalidation on favouriting a dag (apache#57039) (apache#57047) Fix dag list cache invalidation on favouriting a dag (cherry picked from commit 4e46212) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> * [v3-1-test] Add new PL translations. (apache#56359) (apache#57055) (cherry picked from commit 6583b6d) * Add last missing translation for 3.1.1 (apache#57057) * [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (apache#56980) (apache#57048) * [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (apache#56570) * fix: adjust PanelButtons spacing and alignment * Fix PanelButtons layout and button styling * Fix PanelButtons: use bg.subtle and remove unnecessary _hover (cherry picked from commit 4c35cd2) Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com> * [v3-1-test] Add possibility of removing extra translations missing in en (apache#57060) Sometimes we rename/move translation - we did not have a tool to remove such dangling translations. Now we have. (cherry picked from commit 2314099) Co-authored-by: Jarek Potiuk <jarek@potiuk.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
…pache#57195) (cherry picked from commit f52c0f6) Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
…7231) This commit addresses multiple bugs in systemd service files reported in issue apache#53706: 1. Fix incorrect command in airflow-api.service - Changed 'airflow api' to 'airflow api-server' (correct CLI command) 2. Standardize environment configuration approach - Changed airflow-api.service and airflow-triggerer.service to use EnvironmentFile=/etc/sysconfig/airflow instead of inline Environment - This provides consistency with other services and allows admins flexibility to manage custom environment variables centrally 3. Fix service dependencies - Added proper After= and Wants= clauses for database and message queue services (postgresql, mysql, redis, rabbitmq) to airflow-api.service and airflow-triggerer.service - Services now wait for required dependencies before starting 4. Additional bug fixes discovered during review - airflow-flower.service: Removed duplicate ExecStart line - airflow-kerberos.service: Fixed command from 'kerberos' to 'airflow kerberos' 5. Consistency improvements - Added Type=simple and RestartSec=5s to airflow-api.service and airflow-triggerer.service for consistency with other services All systemd service files now follow the same configuration pattern, making them easier to maintain and more reliable during system startup. Fixes apache#53706 (cherry picked from commit b370160) Co-authored-by: Ashir Alam <alamashir@gmail.com>
…he#56887) (apache#57232) * Make single workflow to run both AMD and ARM builds * Add condition to exclude mysql tests for arm * Fix mypy issues * delete arm and amd workflows * Fix artifact name (cherry picked from commit cf82ae4) Co-authored-by: GPK <gopidesupavan@gmail.com>
…ache#57229) (apache#57244) When any provider is published, we should also automatically build and publish "apache-airflow-providers" documentation which contains summary and "general" providers documentation. (cherry picked from commit c5b7fbb) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* CI: Fix workflow name * Update status badges (cherry picked from commit 0ce2f5c) Co-authored-by: GPK <gopidesupavan@gmail.com>
…pache#57203) (apache#57252) * include task-sdk * switch to RAT 0.17 * update .rat-excludes to be compatible with RAT 0.17 * Add .gitignore to task-sdk to not build release with accidental files (cherry picked from commit 90e3926)
…pache#57219) (apache#57253) * Compile assets when installing airflow from Github Branch in Breeze * Only install pnpm * Fix main UI compliation * Generalize compile_ui_assets for simple_auth_manager ui * Remove "no assets compiled" warning * Fix test due to compile_ui_assets not found (cherry picked from commit 505d9e4) Co-authored-by: LIU ZHE YOU <68415893+jason810496@users.noreply.github.com>
…-run (apache#57255) (apache#57257) (cherry picked from commit d750022) Fixes: apache#50569 Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…#57258) * Fix runner type assignment in selective checks * Update notification workflow
…7260) * Extract prek hooks for Providers * Ups in path * Fix breeze script to consider running also in non repo root * Fix prek script to consider running also in non repo root * Fix prek script to consider running also in non repo root (cherry picked from commit 045fbc5) Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
…ache#57261) Some of the integration tests should not be run for ARM because they do not have multi-platform images. This PR adds the list of those tests that should be removed from ARM runs. (cherry picked from commit d75e913) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…ocs (apache#57243) (apache#57262) This does not remove the actual versions from S3 only limits the drop-down. All versions are available still to see and they are indexed, also links from PyPI continue to work, you can also manually update version in the URL. (cherry picked from commit a0b4d29) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…gration tests (apache#57171) * Sync airflowctl from main to v3.1 to include all changes in integration tests * Remove changed help images, update backfill in integration * Regenerate datamodels * Fix mypy checks
…used (apache#57284) (apache#57285) We now have same workflow running for both ARM and AMD and we need a bit better diagnostics printed to distinguish those different run types. * Tha name of the workflow is just changed to "Tests" * There is a job added that should immediately show the platform in the left-sidebar of GitHub Actions * The title containing platform is printed at the top of summary - before the constraints summary. (cherry picked from commit 4219c57) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Not everything is cherry-picked so the tests need to be fixed.
… (apache#57291) (cherry picked from commit bedd81c) Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
) (cherry picked from commit 91a3552) Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
The dagbag.py file is still not refactored in v3-1-test so in the v3-1-test branch it should still be excluded from task-sdk import checks.
kaxil
force-pushed
the
changes-3.1.2rc1
branch
from
7ad986e to
0725199
Compare
kaxil
requested review from
BasPH,
Lee-W,
XD-DENG,
amoghrajesh,
ashb,
bbovenzi,
bolkedebruin,
bugraoz93,
dstandish,
eladkal,
gopidesupavan,
hussein-awala,
jason810496,
jedcunningham,
jscheffl,
mobuchowski,
o-nikolas,
pierrejeambrun,
potiuk,
rawwar,
romsharon98,
ryanahamilton,
shahar1,
shubhamraj-git,
vatsrahul1001 and
vincbeck
as code owners
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR Syncs
v3-1-stablewithv3-1-testto release 3.1.2.Release notes and version bumps added in
Changes between Task SDK 1.1.1 and 1.1.2
ObjectStoragePath#57156)exc_infofor task instance heartbeat failure exception logging (#57172) #57179)