missing conf as a filter field in dag_run browsing

[raphaelauv]

Apache Airflow version

2.9.0

What happened?

I can't use conf as a filter field to search a dag_run in the webserver

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jscheffl]

I also was missing this option many many times. But you need to know that confis not a scalar database field but a serialized object. You can not "easily" set a filter criteria in the database engine and leverage an index to filter.

Filtering on a conf would be very heavy as actually besides filtering on other fields, all rows must be loaded and de-serialized for filtering. On a large DAG run table this could be a performance breaker and multiple such queries can be a DoS on the system.

[raphaelauv]

Okay , but we can already order by conf and on less than 10 000 dag_runs it render almost instantly. So maybe a disclaimer on the front in the case of using a filter on this field would be a good tradeoff

[uranusjr]

Maybe another thing we can consider doing in Airflow 3? I imagine the field would need a dedicated field so we can drill deep into it for filtering. Not sure how.

[jscheffl]

Maybe another thing we can consider doing in Airflow 3? I imagine the field would need a dedicated field so we can drill deep into it for filtering. Not sure how.

Very much depends on what a DB can offer - e.g. if the field is persited as native JSON object/dict, then using JSON query functions. But as not full ystandardized this has a danger of vendor lock-in. Besides performance as probably not indexed