Fix deferrable HttpSensor: HttpAsyncHook falls back to https when port == 443

[nailo2c]

Related: #52319

Why

HttpSensor in deferrable mode fails after the task is resumed by the Triggerer: the schema field is missing, so HttpAsyncHook silently downgrades https://host to http://host.

airflow/providers/http/src/airflow/providers/http/hooks/http.py

Line 465 in eac9a92

schema = conn.schema if conn.schema else "http"

The root cause seems to be here. I guess Triggerer fetches the connection from the metadata DB on the first run and then serves subsequent requests from its cache. However, cache no longer contains schema_ in 3.0.0 somehow. In 2.11.0 the same DAG still works.

(The explanation above is my current understanding based on quick code-reading and logs. If I've misinterpreted how the Triggerer caches connections, please feel free to correct me, feedback is very welcome 😄)

airflow/task-sdk/src/airflow/sdk/execution_time/context.py

Line 155 in d3a33da

msg = SUPERVISOR_COMMS.send(GetConnection(conn_id=conn_id))

# first run
SUPERVISOR_COMMS msg: conn_id='conn_id' conn_type='http' host='dummyjson.com' schema_='https' login=None password=None port=443 extra=None type='ConnectionResult'

# second ... Nth run
SUPERVISOR_COMMS msg: conn_id='conn_id' conn_type='http' host='dummyjson.com' schema_=None login=None password=None port=443 extra=None type='ConnectionResult

The example DAG in the issue works as expected on 2.11.0 but not on 3.0.0. Here is how the DAG runs on 2.11.0:

How

• Short Term Solution

conn.port is still present in every run, so fall back to https when port == 443.

• Long Term Solution

Investigate why schema_ disappears in the Triggerer cache, maybe open a follow-up issue if needed.

What

With this patch the OP’s example DAG now uses https.

[albertwangnz]

Thank you @nailo2c I tried but did not find the root cause of the missing schema value. You are so good!

[gopidesupavan]

can you please add some tests

[nailo2c]

sure, give me a moment

[gopidesupavan]

Hey both the fix has been raised, its issue in triggerer connection result.

BTW thanks for reporting and raising alternatives.

[nailo2c]

Nice!