Added a task try table for better log file retrieval.

[pavel-lexyr]

This PR aims to fix a long-standing logging bug present in Airflow. In particular, any clusters of more than one worker do not handle their logs appropriately - always fetching files from the host used in the last run.

This change adds a lookup table for per-try information for TaskInstances. That allows storing a separate hostname for each try of a task.

Questions to consider before merging - comments would be appreciated:

• What kinds of tests should be added?
• What is the appropriate way to add a database migration to the codebase?

This is a redesign of a previous PR #23135.

closes: #16472

[uranusjr]

The primary key should be TaskInstance’s primary key + try_number.

[tanelk]

Should there also be a map_index in this table?

[uranusjr]

Yes

[pavel-lexyr]

Thank you. Added map_index to the PK, that should make the full TaskInstance key.

[ashb]

Idle thought: Do we need this table and TaskFail table, or could a single table serve both purposes?

[ashb]

We shouldn't have this duplicated here.

[pavel-lexyr]

Fixed. Good catch!

[uranusjr]

Do we need this table and TaskFail table, or could a single table serve both purposes?

We can probably merge them, but it’ll need some work, I have difficulties keeping track of try_number increments and TaskFail insertions in my head…

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.