Document working with "sqlalchemy>0.2.0" in virtualenvs and the induced side-effects #47914
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is the next stage of refactoring of airflow packages, after moving providers to standalone dstribution and separating devel-common as a common distribution. The `task_sdk` has been renamed to `task-sdk` - this way we will never import anything in task_sdk accidentally starting from content root. Some changes have been needed to make it works: * autouse fixture was added to pytest plugin to add `task-sdk/tests` to PYTHONPATH to make it root import * all tests were moved to `task_sdk` package inside the tests folder * all imports for tests are now `from task_sdk` * common tools for task_sdk has been moved to `devel-common/src/test_utils/task_sdk.py` in order to allow importing them before `task-sdk/tests` is added to pythonpath
…der.yaml (apache#47448) * chore: Update description of `execution_timeout` in OpenLineage provider.yaml * chore: Update description of execution_timeout in OpenLineage provider.yaml
* AIP-84 | Add Auth for Config * Fix requires_access_configuration schema * Refactor requires_access_configuration with lambda
* AIP-84 | Add Auth for Extra Link * build: generate airflow/api_fastapi/core_api/openapi/v1-generated.yaml --------- Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Asset evaluation is only done in the scheduler, and requires the database in various cases. It is better to split it out into a dedicated class in Airflow core. Dependency resolution requires database calls for non-direct asset references, but we don't want that to happen in the SDK. This removes the eager resolution code so all asset aliases and refs are no longer resolved, but one only keeps a marker for itself. An additional PR will be submitted later to do the resolution. This is a part of the asset UI project that involves changes in the API anyway.
…he#47360) * Remove regexp from dag.partial_subset method and update callers * Remove casting and convert list checking to old method * Remove not needed casting
Following discusison in https://lists.apache.org/thread/nnttnblj3tcv2wqj79gwovtfhgq8lvmm and PR merged apache#47360 and opened issue apache#47364 ideal that could prevent new additions of regexp to UI. I think it's time to remove google-re2.
…r`connection MacOs/Linux compatible (apache#47491) * Update command for replacing newlines with literals `\n` in `SSHOperator` docs Old command only worked for Windows users but not for MAC nor Linux users. This new command works for all users. * remove raw string prefix from regex pattern to avoid more changes than needed
…hart (apache#47416) * Upgrade pgbouncer_exporter to 0.18.0 - Upgrades pgbouncer_exporter to 0.18.0 - Bumps Airflow pgbouncer_exporter version to current date (2025-03-05) - Bumps expected Go version to 1.23.7 per pgbouncer_exporter release notes * Update helm chart default pgbouncer_exporter image to 0.18.0 * Add newline to 47416.misc.rst * Update chart/newsfragments/47416.misc.rst * fixup! Update chart/newsfragments/47416.misc.rst * Bump pgbouncer to version 1.23.1 * Change to significant news fragment * Update newsfragment content --------- Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
) The AwsGenericHook provides the `@property` decorated function `async_conn` for building and accessing the async boto3 client. Unfortunately, this is results in blocking calls to the Airflow Db or secrets backends within async contexts. This PR provides an async method `get_async_conn` as an alternative that can be awaited in async contexts. This method will call the underlying sync code in a `sync_to_async` wrapper. The `async_conn` property is now deprecated and will be removed in future versions.
* Cleanup leftovers frmo API Connexion * Cleanup leftovers frmo API Connexion, wipe all leftovers * Fix selective checks * Add missing dependecy of connexion to FAB provider * Clean another CODEOWNER leftover * Revert exceptions about project tests missing
Bumps [axios](https://github.com/axios/axios) from 1.7.7 to 1.8.2. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.7.7...v1.8.2) --- updated-dependencies: - dependency-name: axios dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Print scheduler call stack when SIGUSR2 received * Remove duplicate lines --------- Co-authored-by: Jeffrey Harrison <jeffrey.harrison@headspace.com>
* Bump axios version in simple auth manager * Cleanup fab provider packagejson
* Adds validation on backfill dryrun Adds some date checks before triggering backfill dryrun * Review comments handled * Update datetime check
Add warning on using virtualenvs with "non-airflow-compliant" dependencies.
Add reference to the "sqlalchemy>0.2.0"-induced side-effects working on virtualenvs.
Member
|
This PR shoudl be done against v2-10-test not main. I don't think we want to add this description in Airflow 3 docs. |
tgrandje
requested review from
Fokko,
XD-DENG,
ashb,
bbovenzi,
bolkedebruin,
dstandish,
eladkal,
ephraimbuddy,
hussein-awala,
jedcunningham,
jhtimmins,
josh-fell,
kaxil,
mik-laj,
mobuchowski,
o-nikolas,
pierrejeambrun,
ryanahamilton and
uranusjr
as code owners
Author
|
Of course you're right: thank you for your vigilance. I tried to edit the PR and it ended up with 5,000+ changes. I'll close this one and set a new one. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed in #46266 there are (and will be more) usecases for
sqlalchemy>0.2.0usage. As far as I know, this will stay there untill Airflow 3 is released. Until then, (I think) there is a good point for documenting on how to deal with this, even if this is not intended behaviour:pandasfor instance cannot work with Airflow as of yet.The current PR :