AIP-38 | Migrate to new UI

[bbovenzi]

Meta epic to track all legacy UI pages we need to migrate over to the new UI

See here for designs: #42371

Feature complete with Legacy UI

General

• Dags List
  • Dag Card to show last and next runs and a chart of recent runs
  • Pause/unpause Dag
  • Trigger dag run AIP-38 | Connect Trigger form to rest API #44857
  • Filter dags by last dag run state, paused, and tags
  • Sort dag list by display name, latest run, next run,
  • Search for a dag by dag_id or dag_display_name
  • Dag import errors
• Dag Details
  • Click on a row in the Dags list to open up Dag Details
  • Show header that has the same information as the Dag Card and includes actions to:
    • Trigger basic Dag AIP-38 | Connect Trigger form to rest API #44857
    • Use dag params to generate forms for triggering a dag AIP-38 | Use Dag params to build an advanced Trigger Dag Run form #44858
  • List all dag runs and link to a specific Dag Run
  • Dag Code
  • Audit Log events filtered by DagId
  • Reparse Dag
  • Delete Dag
  • Dag Run duration bar chart
  • Show Asset conditions and which events have been met for next triggered run
  • Details tab to list all details not in the header card
• Dag Visualizations
  • Graph
    • Show the nodes and edges of a graph
    • Filter by upstream/downstream of a task id
    • Toggle Dag Dependencies (sensors, triggers, data assets with conditions)
    • Show a specific dag run in the graph with each node filled with task instance or asset event info
    • Click on a task to select it as a task or task instance
  • Grid
    • Filter by run type, or state
    • Explore use of icons for each state to be more accessible. And how to balance that with run_type, try_number and has a note functionality
  • Gantt chart
  • Dropdown selector on each viz to change between dag runs
  • Side drawer on the full page modal to show a condensed Task Instance details view with all TI actions
  • Calendar view (do we keep it?)
• Auto refresh AIP-38 | Add auto-refreshing to new UI #46129
• Task Details
  • List task instances
  • Task instance duration chart
  • Audit Log events filtered by TaskId
• Dag Run Details
  • List task instances
  • Manually mark dag run as failed or success
  • Clear a dag run
  • Audit Log events filtered by dag run id
• Task Instance Details
  • Manually mark a task failed or success
  • Clear a task instance AIP-38 | Clear Task Instance #44860
  • See task instance logs across tries AIP-38 | Add Task Instance Logs #44663
  • See task instance details across tries
  • Audit Log events filtered by TI
  • Mapped Task Summary page with list of all mapped instances AIP-38 | Add Mapped Task Instance Summary page #45990
  • Allow selecting a single mapped instance with all task instance functionality
  • Xcoms AIP-38 | Add Task Instance XCom #44667
  • Rendered templates
  • Logs
    • Filter & parse by level
    • Filter & parse by source
    • Auto scroll to bottom of logs
• Dag Runs list
  • List runs across all dags
  • Search, filter and sort
  • Clear and mark state
• Task Instances list
  • List task instances across all dags
  • Search, filter and sort
  • Clear and mark state
• Dashboard (formerly known as Cluster Activity)
  • Import Errors AIP-38 | List Import Errors #43711
  • Pool Summary AIP-38 | Add Pool Summary stats to Dashboard #43328
  • Show historic metrics for dag run and task instances states
  • Component health badges with heartbeat timestamps
• Assets (formerly known as Datasets)
  • List assets
  • List all asset events
  • Manually create asset event
  • Asset Details
    • Show dependency graph of assets
    • List events
• User Profile
  • Toggle dark/light mode
  • Change default timezone
• Docs
  • Link to API, docs, and github
• Admin
  • List Plugins
  • Connections
    • List Connections AIP-38 | List Connections #43703
    • Edit Connection
    • Delete Connection
    • Test Connection
  • Show config
  • List Providers
  • Variables
    • List Variables AIP-38 | List Variables #43709
    • Import variables
    • Add Variable
    • Edit Variable
    • Delete Variable
  • Pools
    • List all pools
    • Edit Pool
    • Delete Pool
    • Create Pool
  • Browse
  • Events
    • List all Audit Log Events
    • Sort list
    • Filter list
    • Search list
  • XComs
    • List all xcoms
• Plugins
  • Show legacy plugins in an iFrame
• Authentication & Permissions
  • Show user management and security in a plugin or natively
  • Show or hide UI elements based on user permissions
• UI Customizations
• #46874

New Features

• General
  • Improve color blind accessibility by not relying just on colors for state AIP-38 | Add colorblind friendly color schemes #43054
• Dashboard
  • Scope out other helpful summaries
• Dags List
  • Advanced search
• Dag Details
  • Overview
    • Quick stats on recent failed runs and task instances
    • Total dag run summary chart
    • Dag run success rate chart
    • Add graph minimap quick link
    • Links to most recent failed task logs
    • Figure out if we can do failed task log previews
    • Scope out other helpful summaries
  • List all tasks
    • Nest list rows based on task groups that expand and collapse like the grid view
  • Task Details
    • Quick stats on failed and retried task instances
    • Total task instance summary chart
    • TI success rate chart
    • Scope out other helpful summaries
• Assets
  • Show summaries of asset events with their upstream/downstreams
  • Dependency graph
    • Select an asset event and see all other related events on the graph
  • Auto refresh Asset pages
  • Show Asset Queue
  • Materialize asset event
• Backfills
  • Create backfill for a dag with a dry run
  • Active backfills banner AIP-38 | Active backfills banner #43968
  • List backfills AIP-38 | List Backfills for a Dag #43967
  • Pause/cancel backfill
• Versioning
  • Show version on dag/dagrun/task/task instance
  • When viewing an older dag run, fetch the dag code for that version
  • Fetch the dag structure for an older version or give an alert that the structure has changed
• Plugins
  • React-based to replace menu items
  • React-based for specific sub views (inside dashboard, dag overview, etc)
  • Dev Tools to aid plugin development
    • Give an iframe template
    • CLI to generate boilerplate plugin

[hterik]

Filter by tags #43715

This appears to link to something else.

[bbovenzi]

Filter by tags #43715

This appears to link to something else.

Good catch. Fixed.

[pierrejeambrun]

I feel like we have everything from the backend perspective. (or planned such as asset dependencies).

The only points I'm not so sure are:

• Include task group information (I'm not sure we have this)
• Show dag import errors on dag list level (I am not sure how we want to display them, as in the legacy we can put them at the top of the dag list, but I don't think we can group them by dag_id there's not really a relation there in the db model)

[bbovenzi]

I feel like we have everything from the backend perspective. (or planned such as asset dependencies).

The only points I'm not so sure are:

• Include task group information (I'm not sure we have this)
• Show dag import errors on dag list level (I am not sure how we want to display them, as in the legacy we can put them at the top of the dag list, but I don't think we can group them by dag_id there's not really a relation there in the db model)

• I think almost all Task Group information is handled by grid_data and structure endpoints
• We literally just need to copy the component from the home page and also show it at the top of the page on the dags list

[pierrejeambrun]

I'll work on the mark as.... (failed/success) etc. For dagruns and tasks.