Make the DAG Run UI view more reliable by removing the column config

[tatiana]

The Airflow Scheduler can create DAG Runs with config values that the Airflow Webserver cannot parse. Users cannot list any items in the Airflow DAG Run UI when this happens. The workaround to allow customers to list DAG Runs again was manually remove from the SQL database the DAG Run rows with unparsable config.

An example of an error message:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.9/site-packages/airflow/www/auth.py", line 51, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/decorators.py", line 109, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/decorators.py", line 72, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/views.py", line 2291, in tree
session.query(DagRun)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3373, in all
return list(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 100, in instances
cursor.close()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
compat.raise_(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 80, in instances
rows = [proc(row) for row in fetch]
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 80, in <listcomp>
rows = [proc(row) for row in fetch]
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 579, in _instance
_populate_full(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 725, in _populate_full
dict_[key] = getter(row)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/sqltypes.py", line 1723, in process
return loads(value)
ModuleNotFoundError: No module named 'unusual_prefix_000a97efabd6e981b062d199b3a7ce82d192aeca_scenario-flow'

This change does not deal with the root cause problem (the fact that the Scheduler is creating DAG Runs with config, which cannot be parsed by the Webserver). However, it improves the experience from the perspective of users who want to list DAGs in the UI regardless of the config value.

closes: #32287

[utkarsharma2]

@tatiana What are the use cases we would need to show config in UI? Probably @bbovenzi might have a better idea about the use cases here.

Also, I think your suggestion to validate the input makes more sense to me.

[bbovenzi]

Do we want to leave out the config entirely? I think we'd just want to convert it to a json object or a string. Because a user would want to see the config a run was triggered with.

[xuganyu96]

PR #28777 addressed the JSON serialization of DagRun conf with a custom JSON Encoder class that encodes non-serializable objects with its __repr__ output. I think it is a more elegant solution than simply leaving the conf column out of the DagRun list view completely.

[bbovenzi]

Closing due to #28777