Disable HTTP2 by default

[trahloff]

Example

Running Prefect at scale with a configuration that utilizes HTTP2 leads to known issues that fail the entire flow run.
The currently recommended mitigation is to disable HTTP2 by setting PREFECT_API_ENABLE_HTTP2=false.

To improve the user experience for new and existing Prefect setups, this PR disables HTTP2 by default without changing any other code.
As soon as httpx finally solves the underlying root cause or we find a different solution, this PR can easily be reverted.

Closes:

1. Client sometimes fails with httpx.LocalProtocolError #7442

Relates to:

1. set http2 to default false prefect-helm#161
2. http2 connection is not renewed after server closes? encode/httpx#2112

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• This pull request includes tests or only affects documentation. (should be covered by existing tests from my perspective)
• This pull request includes a label categorizing the change e.g. fix, feature, enhancement, docs. (no permissions)

[netlify]

👷 Deploy request for prefect-docs-preview pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	447729c

[zanieb]

Are people encountering this during flow runs? My experience thusfar has been that people are encountering this on the agent only. HTTP2 gives a significant performance boost to orchestration of task runs, I believe.

[zanieb]

I'm going to run our benchmarks against Cloud with and without this change and see what the difference is.

[zanieb]

------------------------------------------ benchmark 'bench_async_flow_with_async_tasks': 8 tests -----------------------------------------
Name (time in s)                                             Mean            StdDev                Min                Max            Rounds
-------------------------------------------------------------------------------------------------------------------------------------------
bench_async_flow_with_async_tasks[10] (HTTP2)              2.5541 (1.0)      0.0000 (1.0)       2.5541 (1.0)       2.5541 (1.0)           1
bench_async_flow_with_async_tasks[50] (HTTP2)              3.5271 (1.38)     0.0000 (1.0)       3.5271 (1.38)      3.5271 (1.38)          1
bench_async_flow_with_async_tasks[100] (HTTP2)             4.7060 (1.84)     0.0000 (1.0)       4.7060 (1.84)      4.7060 (1.84)          1
bench_async_flow_with_async_tasks[10] (HTTP1)               4.8862 (1.91)     0.0000 (1.0)       4.8862 (1.91)      4.8862 (1.91)          1
bench_async_flow_with_async_tasks[50] (HTTP1)               8.0075 (3.14)     0.0000 (1.0)       8.0075 (3.14)      8.0075 (3.14)          1
bench_async_flow_with_async_tasks[250] (HTTP2)             9.5130 (3.72)     0.0000 (1.0)       9.5130 (3.72)      9.5130 (3.72)          1
bench_async_flow_with_async_tasks[100] (HTTP1)             10.3310 (4.04)     0.0000 (1.0)      10.3310 (4.04)     10.3310 (4.04)          1
bench_async_flow_with_async_tasks[250] (HTTP1)             21.6182 (8.46)     0.0000 (1.0)      21.6182 (8.46)     21.6182 (8.46)          1
-------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------- benchmark 'bench_async_flow_with_concurrent_subflows': 6 tests ---------------------------------------
Name (time in s)                                                   Mean            StdDev               Min               Max            Rounds
-----------------------------------------------------------------------------------------------------------------------------------------------
bench_async_flow_with_concurrent_subflows[5] (HTTP1)              2.4635 (1.0)      0.0000 (1.0)      2.4635 (1.0)      2.4635 (1.0)           1
bench_async_flow_with_concurrent_subflows[20] (HTTP1)             2.7944 (1.13)     0.0000 (1.0)      2.7944 (1.13)     2.7944 (1.13)          1
bench_async_flow_with_concurrent_subflows[10] (HTTP1)             2.8902 (1.17)     0.0000 (1.0)      2.8902 (1.17)     2.8902 (1.17)          1
bench_async_flow_with_concurrent_subflows[5] (HTTP2)             3.0578 (1.24)     0.0000 (1.0)      3.0578 (1.24)     3.0578 (1.24)          1
bench_async_flow_with_concurrent_subflows[20] (HTTP2)            3.2472 (1.32)     0.0000 (1.0)      3.2472 (1.32)     3.2472 (1.32)          1
bench_async_flow_with_concurrent_subflows[10] (HTTP2)            3.3602 (1.36)     0.0000 (1.0)      3.3602 (1.36)     3.3602 (1.36)          1
-----------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------- benchmark 'bench_async_flow_with_sequential_subflows': 6 tests -----------------------------------------
Name (time in s)                                                    Mean            StdDev                Min                Max            Rounds
--------------------------------------------------------------------------------------------------------------------------------------------------
bench_async_flow_with_sequential_subflows[5] (HTTP1)               7.3431 (1.0)      0.0000 (1.0)       7.3431 (1.0)       7.3431 (1.0)           1
bench_async_flow_with_sequential_subflows[5] (HTTP2)              8.7136 (1.19)     0.0000 (1.0)       8.7136 (1.19)      8.7136 (1.19)          1
bench_async_flow_with_sequential_subflows[10] (HTTP1)             11.2649 (1.53)     0.0000 (1.0)      11.2649 (1.53)     11.2649 (1.53)          1
bench_async_flow_with_sequential_subflows[10] (HTTP2)            12.5701 (1.71)     0.0000 (1.0)      12.5701 (1.71)     12.5701 (1.71)          1
bench_async_flow_with_sequential_subflows[20] (HTTP1)             19.4517 (2.65)     0.0000 (1.0)      19.4517 (2.65)     19.4517 (2.65)          1
bench_async_flow_with_sequential_subflows[20] (HTTP2)            22.3197 (3.04)     0.0000 (1.0)      22.3197 (3.04)     22.3197 (3.04)          1
--------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------- benchmark 'bench_async_flow_with_submitted_sync_tasks': 6 tests ------------------------------------------
Name (time in s)                                                      Mean            StdDev                Min                Max            Rounds
----------------------------------------------------------------------------------------------------------------------------------------------------
bench_async_flow_with_submitted_sync_tasks[10] (HTTP2)              3.2221 (1.0)      0.0000 (1.0)       3.2221 (1.0)       3.2221 (1.0)           1
bench_async_flow_with_submitted_sync_tasks[10] (HTTP1)               3.5167 (1.09)     0.0000 (1.0)       3.5167 (1.09)      3.5167 (1.09)          1
bench_async_flow_with_submitted_sync_tasks[50] (HTTP2)              3.6721 (1.14)     0.0000 (1.0)       3.6721 (1.14)      3.6721 (1.14)          1
bench_async_flow_with_submitted_sync_tasks[100] (HTTP2)             4.2024 (1.30)     0.0000 (1.0)       4.2024 (1.30)      4.2024 (1.30)          1
bench_async_flow_with_submitted_sync_tasks[50] (HTTP1)               5.3244 (1.65)     0.0000 (1.0)       5.3244 (1.65)      5.3244 (1.65)          1
bench_async_flow_with_submitted_sync_tasks[100] (HTTP1)             10.3460 (3.21)     0.0000 (1.0)      10.3460 (3.21)     10.3460 (3.21)          1
----------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'bench_flow_call': 4 tests ------------------------------------------
Name (time in s)                               Mean            StdDev               Min               Max            Rounds
---------------------------------------------------------------------------------------------------------------------------
bench_flow_call[options0] (HTTP1)             1.0465 (1.0)      0.0000 (1.0)      1.0465 (1.0)      1.0465 (1.0)           1
bench_flow_call[options1] (HTTP1)             1.3385 (1.28)     0.0000 (1.0)      1.3385 (1.28)     1.3385 (1.28)          1
bench_flow_call[options0] (HTTP2)            1.4729 (1.41)     0.0000 (1.0)      1.4729 (1.41)     1.4729 (1.41)          1
bench_flow_call[options1] (HTTP2)            1.6737 (1.60)     0.0000 (1.0)      1.6737 (1.60)     1.6737 (1.60)          1
---------------------------------------------------------------------------------------------------------------------------

-------------------------------------------- benchmark 'bench_flow_decorator': 2 tests --------------------------------------------
Name (time in us)                           Mean                StdDev                 Min                    Max            Rounds
-----------------------------------------------------------------------------------------------------------------------------------
bench_flow_decorator (HTTP2)            566.9148 (1.0)      2,063.5867 (1.0)      409.6670 (1.0)      47,107.6250 (1.0)         511
bench_flow_decorator (HTTP1)             638.2730 (1.13)     2,412.4954 (1.17)     415.2080 (1.01)     49,581.0420 (1.05)        414
-----------------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'bench_flow_with_called_tasks': 8 tests -------------------------------------------
Name (time in s)                                         Mean            StdDev                 Min                 Max            Rounds
-----------------------------------------------------------------------------------------------------------------------------------------
bench_flow_with_called_tasks[10] (HTTP2)               9.3720 (1.0)      0.0000 (1.0)        9.3720 (1.0)        9.3720 (1.0)           1
bench_flow_with_called_tasks[10] (HTTP1)               11.4235 (1.22)     0.0000 (1.0)       11.4235 (1.22)      11.4235 (1.22)          1
bench_flow_with_called_tasks[50] (HTTP1)               43.0533 (4.59)     0.0000 (1.0)       43.0533 (4.59)      43.0533 (4.59)          1
bench_flow_with_called_tasks[50] (HTTP2)              49.2414 (5.25)     0.0000 (1.0)       49.2414 (5.25)      49.2414 (5.25)          1
bench_flow_with_called_tasks[100] (HTTP1)              81.1445 (8.66)     0.0000 (1.0)       81.1445 (8.66)      81.1445 (8.66)          1
bench_flow_with_called_tasks[100] (HTTP2)             89.2842 (9.53)     0.0000 (1.0)       89.2842 (9.53)      89.2842 (9.53)          1
bench_flow_with_called_tasks[250] (HTTP1)             211.9108 (22.61)    0.0000 (1.0)      211.9108 (22.61)    211.9108 (22.61)         1
bench_flow_with_called_tasks[250] (HTTP2)            218.6078 (23.33)    0.0000 (1.0)      218.6078 (23.33)    218.6078 (23.33)         1
-----------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------- benchmark 'bench_flow_with_subflows': 6 tests -----------------------------------------
Name (time in s)                                   Mean            StdDev                Min                Max            Rounds
---------------------------------------------------------------------------------------------------------------------------------
bench_flow_with_subflows[5] (HTTP2)              7.1558 (1.0)      0.0000 (1.0)       7.1558 (1.0)       7.1558 (1.0)           1
bench_flow_with_subflows[5] (HTTP1)               8.6027 (1.20)     0.0000 (1.0)       8.6027 (1.20)      8.6027 (1.20)          1
bench_flow_with_subflows[10] (HTTP2)            12.4156 (1.74)     0.0000 (1.0)      12.4156 (1.74)     12.4156 (1.74)          1
bench_flow_with_subflows[10] (HTTP1)             13.2255 (1.85)     0.0000 (1.0)      13.2255 (1.85)     13.2255 (1.85)          1
bench_flow_with_subflows[20] (HTTP1)             20.5468 (2.87)     0.0000 (1.0)      20.5468 (2.87)     20.5468 (2.87)          1
bench_flow_with_subflows[20] (HTTP2)            25.4306 (3.55)     0.0000 (1.0)      25.4306 (3.55)     25.4306 (3.55)          1
---------------------------------------------------------------------------------------------------------------------------------

---------------------------------------- benchmark 'bench_flow_with_submitted_tasks': 6 tests ----------------------------------------
Name (time in s)                                          Mean            StdDev               Min               Max            Rounds
--------------------------------------------------------------------------------------------------------------------------------------
bench_flow_with_submitted_tasks[10] (HTTP1)              2.3953 (1.0)      0.0000 (1.0)      2.3953 (1.0)      2.3953 (1.0)           1
bench_flow_with_submitted_tasks[10] (HTTP2)             3.0064 (1.26)     0.0000 (1.0)      3.0064 (1.26)     3.0064 (1.26)          1
bench_flow_with_submitted_tasks[50] (HTTP2)             3.5601 (1.49)     0.0000 (1.0)      3.5601 (1.49)     3.5601 (1.49)          1
bench_flow_with_submitted_tasks[50] (HTTP1)              4.4364 (1.85)     0.0000 (1.0)      4.4364 (1.85)     4.4364 (1.85)          1
bench_flow_with_submitted_tasks[100] (HTTP2)            4.5209 (1.89)     0.0000 (1.0)      4.5209 (1.89)     4.5209 (1.89)          1
bench_flow_with_submitted_tasks[100] (HTTP1)             7.7314 (3.23)     0.0000 (1.0)      7.7314 (3.23)     7.7314 (3.23)          1
--------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------ benchmark 'bench_task_call': 2 tests -----------------------------------------
Name (time in ms)                      Mean              StdDev                 Min                 Max            Rounds
-------------------------------------------------------------------------------------------------------------------------
bench_task_call (HTTP1)             558.3137 (1.0)        5.2421 (1.0)      554.6070 (1.0)      562.0205 (1.0)           2
bench_task_call (HTTP2)            662.9092 (1.19)     150.8223 (28.77)    556.2617 (1.00)     769.5566 (1.37)          2
-------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'bench_task_decorator': 2 tests --------------------------------------------
Name (time in us)                             Mean              StdDev                Min                    Max            Rounds
----------------------------------------------------------------------------------------------------------------------------------
bench_task_decorator (HTTP1)               706.9701 (1.0)      400.7992 (1.0)      38.3340 (1.93)      4,603.5000 (1.0)        9643
bench_task_decorator (HTTP2)            1,172.6512 (1.66)     710.1507 (1.77)     19.8330 (1.0)      25,349.8330 (5.51)      16383
----------------------------------------------------------------------------------------------------------------------------------

----------------------------------------- benchmark 'bench_task_submit': 4 tests ----------------------------------------
Name (time in ms)                           Mean            StdDev               Min                Max            Rounds
-------------------------------------------------------------------------------------------------------------------------
bench_task_submit[250] (HTTP2)            1.0377 (1.0)      5.4353 (6.10)     0.4363 (1.64)     86.4737 (19.59)       250
bench_task_submit[100] (HTTP1)             1.0542 (1.02)     0.8904 (1.0)      0.3742 (1.41)      4.4147 (1.0)         100
bench_task_submit[250] (HTTP1)             1.1458 (1.10)     5.0043 (5.62)     0.2659 (1.0)      79.2165 (17.94)       250
bench_task_submit[100] (HTTP2)            1.7647 (1.70)     8.1563 (9.16)     0.5783 (2.17)     82.2867 (18.64)       100
-------------------------------------------------------------------------------------------------------------------------

[zanieb]

It looks like this has a significant impact for asynchronous tasks

------------------------------------------ benchmark 'bench_async_flow_with_async_tasks': 8 tests -----------------------------------------
Name (time in s)                                             Mean            StdDev                Min                Max            Rounds
-------------------------------------------------------------------------------------------------------------------------------------------
bench_async_flow_with_async_tasks[10] (HTTP2)              2.1605 (1.0)      0.0000 (1.0)       2.1605 (1.0)       2.1605 (1.0)           1
bench_async_flow_with_async_tasks[50] (HTTP2)              2.7559 (1.28)     0.0000 (1.0)       2.7559 (1.28)      2.7559 (1.28)          1
bench_async_flow_with_async_tasks[10] (HTTP1)              3.2264 (1.49)     0.0000 (1.0)       3.2264 (1.49)      3.2264 (1.49)          1
bench_async_flow_with_async_tasks[100] (HTTP2)             3.4282 (1.59)     0.0000 (1.0)       3.4282 (1.59)      3.4282 (1.59)          1
bench_async_flow_with_async_tasks[50] (HTTP1)              5.0009 (2.31)     0.0000 (1.0)       5.0009 (2.31)      5.0009 (2.31)          1
bench_async_flow_with_async_tasks[250] (HTTP2)             7.2230 (3.34)     0.0000 (1.0)       7.2230 (3.34)      7.2230 (3.34)          1
bench_async_flow_with_async_tasks[100] (HTTP1)             8.0873 (3.74)     0.0000 (1.0)       8.0873 (3.74)      8.0873 (3.74)          1
bench_async_flow_with_async_tasks[250] (HTTP1)            19.5644 (9.06)     0.0000 (1.0)      19.5644 (9.06)     19.5644 (9.06)          1
-------------------------------------------------------------------------------------------------------------------------------------------

[zanieb]

With additional rounds

------------------------------------------- benchmark 'bench_async_flow_with_async_tasks': 8 tests ------------------------------------------
Name (time in s)                                             Mean             StdDev                Min                 Max            Rounds
---------------------------------------------------------------------------------------------------------------------------------------------
bench_async_flow_with_async_tasks[50] (HTTP2)              3.1179 (1.0)       0.2928 (1.0)       2.7073 (1.01)       3.3928 (1.0)           5
bench_async_flow_with_async_tasks[10] (HTTP2)              3.4617 (1.11)      0.7839 (2.68)      2.6750 (1.0)        4.5103 (1.33)          5
bench_async_flow_with_async_tasks[100] (HTTP2)             4.1889 (1.34)      0.4690 (1.60)      3.4745 (1.30)       4.7507 (1.40)          5
bench_async_flow_with_async_tasks[10] (HTTP1)              4.2388 (1.36)      0.8556 (2.92)      2.7388 (1.02)       4.7587 (1.40)          5
bench_async_flow_with_async_tasks[250] (HTTP2)            11.8303 (3.79)      4.6374 (15.84)     8.9392 (3.34)      20.0217 (5.90)          5
bench_async_flow_with_async_tasks[50] (HTTP1)             19.7552 (6.34)     10.9807 (37.50)     8.9543 (3.35)      32.9183 (9.70)          5
bench_async_flow_with_async_tasks[100] (HTTP1)            31.5430 (10.12)    13.5778 (46.37)    17.3778 (6.50)      46.4065 (13.68)         5
bench_async_flow_with_async_tasks[250] (HTTP1)            69.3430 (22.24)    47.7086 (162.92)   18.0398 (6.74)     142.9819 (42.14)         5
---------------------------------------------------------------------------------------------------------------------------------------------

[trahloff]

Oh wow, that is quite a drastic performance delta. I would love to use HTTP2 for our production workload now after seeing this 😁

Your PR over at httpcore (encode/httpcore#679) would also solve the underlying root cause, correct?
If yes, I think it makes much more sense to close this PR and hope for a quick resolution of the httpcore issue.

[zanieb]

I really hoped it would just be a small regression — interesting that the stddev is so high and that it only applies to asynchronous tasks, probably something to dig into further there.

Your PR over at httpcore (encode/httpcore#679) would also solve the underlying root cause, correct?

Yeah it would. I'm a maintainer over there so hopefully we can get it released soon.

[zanieb]

@trahloff this is resolved upstream. Thanks!