Add pagination to HttpOperator and make it more modular #34669 #45872
Description
Description
In the event that I must know the page count beforehand. Two inefficient calls to the API are required for this, and neither one guarantees that every page will be viewed on the second call. HttpOperator by enabling it to handle custom pagination logic, providing users with more flexibility in managing API pagination. The goal is to replace the deprecated SimpleHttpOperator, offering a more efficient and adaptable way to process paginated API data, which improves how workflows interact with APIs within Airflow.
Use case/motivation
For APIs where the pagination logic is not fixed (e.g., using a cursor or other complex structures), this update allows users to implement their custom logic for paging through results. Support for dynamic pagination in APIs is critical for many use cases. By allowing users to pass their own pagination functions, the operator can handle different API structures and pagination methods without modifications.
Related issues
Included in Airflow version 2.8.0
Reference PR: [#34606]
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct