Update conf column in dag_run table type from bytes to JSON

[vatsrahul1001]

closes: #43933

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[vatsrahul1001]

While Testing this I noticed an issue in the downgrade case. We are not removing the data from downgrade and just perform below conversion.

          ALTER TABLE dag_run
          ALTER COLUMN conf TYPE BYTEA
          USING CASE
              WHEN conf IS NOT NULL THEN CONVERT_TO(conf::TEXT, 'UTF8')
              ELSE NULL
          END
 

I am getting UnpicklingError error with this when I try to open DAG page.

Should we also move data to the archive table in case of a downgrade as well?
cc: @kaxil

[kaxil]

While Testing this I noticed an issue in the downgrade case. We are not removing the data from downgrade and just perform below conversion.

          ALTER TABLE dag_run
          ALTER COLUMN conf TYPE BYTEA
          USING CASE
              WHEN conf IS NOT NULL THEN CONVERT_TO(conf::TEXT, 'UTF8')
              ELSE NULL
          END
 

I am getting UnpicklingError error with this when I try to open DAG page.

Should we also move data to the archive table in case of a downgrade as well? cc: @kaxil

Yeah, it is different than XCom, because XCom code had handling of both JSON & pickle type -- for dagrun conf it would be different. For downgrade you might just want to insert all records from the archive table back here.

To allow offline migration to work would require creating an extension in postgres that converst pickled data to json which I think would be a lot to do in this PR. I would probably look at this another time but it shouldn't block this PR