AIP-84 Add external dependencies to GET Structure Data Endpoint

[pierrejeambrun]

Related to: #42367

Adds the external_dependencies query parameter.

Supports trigger, sensor, asset, asset_aliases.

Sensor is back because ExternalTaskSensor is creating a sensor dependency type.

For now all that is external to the dag, and not specific to the task_level. Meaning that they are attached to the dag (first task of the dag) and not to the specific tasks. (i.e task outlets for assets for instance)

[shubhamraj-git]

Nice!!

[bbovenzi]

Testing locally, I noticed two issues:

example_external_task_marker_parent: did not have any downstream deps

example_trigger_controller_dag`: the dependency comes back as upstream when it should be downstream

[pierrejeambrun]

example_external_task_marker_parent: did not have any downstream deps

Indeed. That's a limitation we have at the moment in the backend in detect_task_dependencies. only TriggerDagRunOperator and ExternalTaskSensor are supported. ExternalTaskMarker is not. We might want to add it but this will impact the serialized dags and I'm not sure yet how free we are of updating that. (and how this dependencies are then used by the scheduler I assume). We will need to discuss that with other to confirm the feasibility of that approach.

I will open a dedicated issue to track that because its not straight forward.

edit: Dedicated issue here #44879

[pierrejeambrun]

example_trigger_controller_dag`: the dependency comes back as upstream when it should be downstream

Good catch, fixed, also added a test for that.

[bbovenzi]

lgtm!