Add support for influx3 in influxdb provider #58929
Conversation
This commit adds a new provider package for InfluxDB 3.x support, separate from the existing influxdb provider which supports InfluxDB 2.x. InfluxDB 3.x uses SQL queries and a different API compared to InfluxDB 2.x which uses Flux queries. This separation allows for: - Clean API design specific to InfluxDB 3.x - Independent evolution of both providers - No breaking changes to existing InfluxDB 2.x users The provider includes: - InfluxDB3Hook for connecting to InfluxDB 3.x databases - InfluxDB3Operator for executing SQL queries - Comprehensive unit and system tests - Complete documentation Closes apache#58610
- Update test_query to properly mock and verify DataFrame return type - Update test_execute in operator tests to use DataFrame - Add test for ImportError when influxdb3-python is not available - Ensure all tests properly validate return types
…w to 2.11.0 - Simplify InfluxDB3Hook.query() to use client's native pandas DataFrame return - Simplify InfluxDB3Operator to handle DataFrame to JSON conversion - Update tests to match simplified implementation - Update minimum Airflow version requirement to 2.11.0 per provider policy - Update documentation to reflect Airflow 2.11.0+ requirement
- Add docstring explaining how connection parameters are read from form widgets - Improve error messages to reference connection form fields explicitly - Add explicit handling for optional org parameter - Clarify that form widgets automatically populate extras JSON field
…essage - Add return type annotation to InfluxDB3Operator.execute() method - Add docstring to execute() method explaining return value - Fix test error message to match actual error message from hook - Improve code formatting consistency
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
- Fix CodeQL security issue: use endswith() instead of 'in' for hostname check - Add proper type annotations for MyPy compliance: - self.connection: Connection | None - self.extras: dict[str, Any] - self.uri: str | None - get_client kwargs: dict[str, Any] - Add None safety check for conn.host in get_uri() - Add Point availability check in write() method
|
Hi All, This is my first PR to Apache Airflow and I am still learning the contribution process. If anything needs to be corrected or reorganized, please let me know and I will be happy to update it. Tagging a few maintainers who have recently worked on the InfluxDB provider for visibility. @bbovenzi @jscheffl @potiuk @kaxil Thank you for reviewing. I appreciate your help. |
|
We are currently in the process of discussing/voting the process of accepting new providers [1][2] - there were potential changes discussed there - including incubation period, having a team that is committed (and accountable) to make sure that the provider I suggest you respond to the governance thread and comment on those proposals and explain how your proposal fits the proposed governance process, what you see as possible changes or where you see difficulties - that would certainly help in finalizing the discussion and helping with unblocking new provider submissions. Actually I think it's a good idea to have anyone who wants to submit a new provider to actively participate in that discussion - because there are certain things we thing about [1] https://lists.apache.org/thread/qrv0j4dxp2yg09gds40vh49dhkbrj5q9 |
…variable Remove Connection type annotation from self.connection to match pattern used in other providers. Connection is only imported under TYPE_CHECKING and MyPy has issues with instance variable type annotations in this case. Function parameter type annotations work fine with TYPE_CHECKING imports.
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
|
not stale |
potiuk
removed
the
stale
|
Rebasing this PR and fixing the failures is a good start to follow the process of adding it |
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
First view on code ad structure looks good. I assume some wiring into dev tooling and GitHub pipelines would be needed to make it complete.
Plus as we are now starting to mature the new policy for providers we need two stewards at least and some committer who is backing the continued work to inter incubation. Is there any sponsor around?
|
Thanks for taking a look @jscheffl, appreciate it. Glad to hear the code and structure look good. Agreed on the remaining wiring needed for dev tooling and GitHub/CI pipelines. I can work on that next. On the provider policy side, I can commit to being one of the stewards for this provider and maintaining it through incubation and beyond. I do not have a second steward or a committer sponsor confirmed yet. Do you have a recommendation for a committer who could sponsor this, or should I reach out on the dev mailing list/Slack to find a sponsor and a second steward? Once that is in place, I can proceed with the remaining integration work and align with the incubation requirements. |
|
We are currently finalizing the policy from AIP-95 (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-95+Provider+lifecycle+update+proposal) in PR #60896 - can you check against here? I think whilst the PR is open a few days and gets some small wording adjusted, feel free to make this a beta test for the process. As of a discussion should be dropped to devlist. Ipropose to request in the devlist also for support (commiter/stwardship). |
3 participants
closes: #58610
Summary
This PR adds a new provider package
apache-airflow-providers-influxdb3to support InfluxDB 3.x (Core/Enterprise/Cloud Dedicated).InfluxDB 3.x uses SQL queries and a different API compared to InfluxDB 2.x, which is why a separate provider is needed.
What's Changed
apache-airflow-providers-influxdb3(version 1.0.0)influxdb3connection type with UI form widgetsRequirements
influxdb3-python>=0.7.0Testing
InfluxDB3HookandInfluxDB3Operator^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.