allows users to write dag_id and task_id in their national characters, added display name for dag / task #32520

xgao1023 · 2023-07-11T10:28:21Z

This is picking up the work from #28183. But do not silently slugify the IDs

boring-cyborg · 2023-07-11T10:28:26Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

airflow/models/baseoperator.py

uranusjr · 2023-07-12T10:13:04Z

Please set up pre-commit locally and fix the static check errors. See guide here: https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#pre-commit-hooks

airflow/migrations/versions/0123_2_5_0_add_display_name_for_dag_and_task_.py

airflow/models/dag.py

airflow/www/templates/airflow/task_instance.html

airflow/models/dag.py

airflow/models/baseoperator.py

airflow/models/dag.py

airflow/models/taskinstance.py

airflow/models/dag.py

bbovenzi · 2023-07-22T03:56:57Z

Could you add an example dag so that we can test it out?

xgao1023 · 2023-07-28T03:35:05Z

Could you add an example dag so that we can test it out?

Added

uranusjr · 2023-07-28T04:39:35Z

Do we also want a unit test? (Open question; there’s not much to be tested tbh, most things are just declarative.)

xgao1023 · 2023-07-28T07:16:27Z

Seems that have display_name a read-only property breaking some ORM code

https://github.com/apache/airflow/actions/runs/5687937364/job/15417962420?pr=32520

uranusjr · 2023-07-28T07:38:40Z

Looks like display_name is already taken by multiple providers. We should probably come up with something else… Maybe task_display_name?

xgao1023 · 2023-07-28T08:38:19Z

Looks like display_name is already taken by multiple providers. We should probably come up with something else… Maybe task_display_name?

Yeah probably. And dag_display_name in DAG?

uranusjr · 2023-07-28T20:09:08Z

Sounds right to me

airflow/models/mappedoperator.py

airflow/models/taskinstance.py

airflow/models/dag.py

jscheffl

I really like this approach, LGTM!

jscheffl

I did a deployment and faced a couple of "glitches". Can you check and rework on these? Otherwise as expressed in the comments I really like the feature and would like to have this merged ASAP!

I see many screens, w.g. Grid view not picking up the cool display names. I asuspect a bit of rework in the UI is needed. Low hanging fruit would be the DAG page header - if you could add using display name there would be great.
Starting the system with breeze raised an import error with two examples. Must be fixed, otherwise I assume also a lot of DAGs "in the wild" will fail in there
Two example DAGs are showing import errors when starting the system, probably the same root cause: "error DAG Import Errors (2)"

Error trace for 2)

ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/airflow/airflow/example_dags/example_dynamic_task_mapping.py
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/dagbag.py", line 345, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/airflow/example_dags/example_dynamic_task_mapping.py", line 37, in <module>
    added_values = add_one.expand(x=[1, 2, 3])
  File "/opt/airflow/airflow/decorators/base.py", line 410, in expand
    return self._expand(DictOfListsExpandInput(map_kwargs), strict=False)
  File "/opt/airflow/airflow/decorators/base.py", line 478, in _expand
    operator = _MappedOperator(
TypeError: __init__() missing 1 required keyword-only argument: 'task_display_name'
ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/airflow/airflow/example_dags/example_params_trigger_ui.py
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/dagbag.py", line 345, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/airflow/example_dags/example_params_trigger_ui.py", line 103, in <module>
    english_greetings = generate_english_greeting.expand(name=names)
  File "/opt/airflow/airflow/decorators/base.py", line 410, in expand
    return self._expand(DictOfListsExpandInput(map_kwargs), strict=False)
  File "/opt/airflow/airflow/decorators/base.py", line 478, in _expand
    operator = _MappedOperator(
TypeError: __init__() missing 1 required keyword-only argument: 'task_display_name'
Database migrating done!

Error trace for 3)

error DAG Import Errors (2) expand_less
expand_moreBroken DAG: [/opt/airflow/airflow/example_dags/example_dynamic_task_mapping.py] Traceback (most recent call last):
  File "/opt/airflow/airflow/decorators/base.py", line 410, in expand
    return self._expand(DictOfListsExpandInput(map_kwargs), strict=False)
  File "/opt/airflow/airflow/decorators/base.py", line 478, in _expand
    operator = _MappedOperator(
TypeError: __init__() missing 1 required keyword-only argument: 'task_display_name'
expand_moreBroken DAG: [/opt/airflow/airflow/example_dags/example_params_trigger_ui.py] Traceback (most recent call last):
  File "/opt/airflow/airflow/decorators/base.py", line 410, in expand
    return self._expand(DictOfListsExpandInput(map_kwargs), strict=False)
  File "/opt/airflow/airflow/decorators/base.py", line 478, in _expand
    operator = _MappedOperator(
TypeError: __init__() missing 1 required keyword-only argument: 'task_display_name'
DAGs

uranusjr · 2023-10-03T09:40:41Z

I tried to add a commit to address the error mentioned above, but there are multiple linting errors here and a merge conflict that must be fixed first.

pierrejeambrun

I think you also want to update the react part (Grid, Cluster Activity, Dataset) to use this new display_name instead of the current task_id, dag_id

xgao1023 · 2023-10-30T06:30:42Z

I'm resuming working on this - fix the issues and also change UI part.

xgao1023 · 2023-11-07T03:50:52Z

I tried to add a commit to address the error mentioned above, but there are multiple linting errors here and a merge conflict that must be fixed first.

I resolved the conflicts and rebased the branch

xgao1023 · 2023-11-07T03:51:37Z

I think you also want to update the react part (Grid, Cluster Activity, Dataset) to use this new display_name instead of the current task_id, dag_id

I will try on this but I’m surely not a frontend expert.

pierrejeambrun · 2023-12-17T18:12:08Z

I will try on this but I’m surely not a frontend expert.

No worries, try your best. If you feel like this is too much, I think we can do that in follow up PRs, step by step.

cc: @bbovenzi

jscheffl

I really like this PR and am looking forward getting this merged. I feel like there are probably another 100 places where the DAG would benefit from displaying the "display name" and not the ID. Might be able to have this updated in the UI incrementally as well to get this merged.

Can you fix-up the DB migration (I hope I understood the alembic tooling right, with the commented fixes it was working for a local test - I hope you don't need to fully re-generate

airflow/migrations/versions/0132_2_8_0_add_display_name_for_dag_and_task_.py

xgao1023 · 2024-02-01T10:25:08Z

I tried to add a commit to address the error mentioned above, but there are multiple linting errors here and a merge conflict that must be fixed first.

@uranusjr
I was trying to fix all errors but I stuck on one thing:

In general the fields on BaseOperator cannot be mapped. But it would be very useful if the display name of each mapped tasks instance can be different according to the expand argument.

Can you give me some hints on how to achieve this?

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Aleksandr Shelukheev <shelukheev@gmail.com>

jscheffl · 2024-03-12T22:55:02Z

I'd really love to have this in - 2.9.0 release cut is across the corner - possible to get the conflicts resolved, the final bug fixed and have this merged?

uranusjr · 2024-03-14T12:15:40Z

Hopefully I can spend some time on this maybe next week.

jscheffl · 2024-03-25T01:29:05Z

As linked in #38446 - I tried to spend time on the weekend to continue working on this PR... let's see if we can get this reviewed and completed...

ephraimbuddy

Nit

ephraimbuddy · 2024-03-25T09:13:18Z

airflow/models/taskmixin.py

+        else:
+            return self.node_id


Suggested change

else:

return self.node_id

return self.node_id

jscheffl · 2024-03-27T20:06:23Z

Closing this PR as completed with #38446 - thanks for the contribution on this branch which was the base for the efforts to make it in (hopefully, if all runs fine) Airflow 2.9.0!

xgao1023 requested review from XD-DENG, ashb, bbovenzi, bolkedebruin, kaxil, pierrejeambrun, ryanahamilton and uranusjr as code owners July 11, 2023 10:28

boring-cyborg bot added area:serialization area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues labels Jul 11, 2023

xgao1023 mentioned this pull request Jul 11, 2023

Separate dag/task UI display name from dag / task id #22073

Closed

2 tasks

uranusjr reviewed Jul 11, 2023

View reviewed changes

airflow/models/baseoperator.py Outdated Show resolved Hide resolved

uranusjr reviewed Jul 11, 2023

View reviewed changes

airflow/models/baseoperator.py Outdated Show resolved Hide resolved

uranusjr reviewed Jul 12, 2023

View reviewed changes

airflow/migrations/versions/0123_2_5_0_add_display_name_for_dag_and_task_.py Outdated Show resolved Hide resolved

uranusjr reviewed Jul 14, 2023

View reviewed changes

airflow/models/dag.py Outdated Show resolved Hide resolved

uranusjr reviewed Jul 14, 2023

View reviewed changes

airflow/www/templates/airflow/task_instance.html Outdated Show resolved Hide resolved

uranusjr reviewed Jul 18, 2023

View reviewed changes

xgao1023 closed this Jul 28, 2023

xgao1023 reopened this Jul 28, 2023

uranusjr approved these changes Jul 28, 2023

View reviewed changes

uranusjr reviewed Sep 6, 2023

View reviewed changes

airflow/models/mappedoperator.py Outdated Show resolved Hide resolved

uranusjr reviewed Sep 6, 2023

View reviewed changes

airflow/models/taskinstance.py Outdated Show resolved Hide resolved

uranusjr reviewed Sep 6, 2023

View reviewed changes

airflow/models/dag.py Outdated Show resolved Hide resolved

jscheffl reviewed Sep 10, 2023

View reviewed changes

vincbeck approved these changes Sep 29, 2023

View reviewed changes

jscheffl reviewed Sep 30, 2023

View reviewed changes

pierrejeambrun reviewed Oct 29, 2023

View reviewed changes

eladkal added this to the Airflow 2.8.0 milestone Oct 30, 2023

eladkal added the type:new-feature Changelog: New Features label Oct 30, 2023

xgao1023 requested a review from potiuk as a code owner November 8, 2023 03:02

ephraimbuddy modified the milestones: Airflow 2.8.0, Airflow 2.9.0 Dec 6, 2023

jscheffl reviewed Jan 30, 2024

View reviewed changes

xgao1023 and others added 2 commits February 5, 2024 16:31

Add display name in DAGs and tasks

b70c86a

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Aleksandr Shelukheev <shelukheev@gmail.com>

Fix frontend issues

2235815

jscheffl mentioned this pull request Mar 25, 2024

Allow users to write dag_id and task_id in their national characters, added display name for dag / task (v2) #38446

Merged

2 tasks

ephraimbuddy approved these changes Mar 25, 2024

View reviewed changes

ephraimbuddy removed this from the Airflow 2.9.0 milestone Mar 25, 2024

kaxil added this to the Airflow 2.10.0 milestone Mar 27, 2024

jscheffl closed this Mar 27, 2024

allows users to write dag_id and task_id in their national characters, added display name for dag / task #32520

allows users to write dag_id and task_id in their national characters, added display name for dag / task #32520

Uh oh!

Conversation

xgao1023 commented Jul 11, 2023

Uh oh!

boring-cyborg bot commented Jul 11, 2023

Uh oh!

Uh oh!

Uh oh!

uranusjr commented Jul 12, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bbovenzi commented Jul 22, 2023

Uh oh!

xgao1023 commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uranusjr commented Jul 28, 2023

Uh oh!

xgao1023 commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uranusjr commented Jul 28, 2023

Uh oh!

xgao1023 commented Jul 28, 2023

Uh oh!

uranusjr commented Jul 28, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

uranusjr commented Oct 3, 2023

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xgao1023 commented Oct 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xgao1023 commented Nov 7, 2023

Uh oh!

xgao1023 commented Nov 7, 2023

Uh oh!

pierrejeambrun commented Dec 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xgao1023 commented Feb 1, 2024

Uh oh!

jscheffl commented Mar 12, 2024

Uh oh!

uranusjr commented Mar 14, 2024

Uh oh!

jscheffl commented Mar 25, 2024

Uh oh!

ephraimbuddy left a comment

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy Mar 25, 2024

xgao1023 commented Jul 28, 2023 •

edited

Loading

xgao1023 commented Jul 28, 2023 •

edited

Loading

pierrejeambrun left a comment •

edited

Loading

xgao1023 commented Oct 30, 2023 •

edited

Loading

pierrejeambrun commented Dec 17, 2023 •

edited

Loading