Skip to content

Sync v2-4-stable with v2-4-test to release 2.4.0 #26371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jedcunningham merged 33 commits into from
Sep 15, 2022
Merged

Sync v2-4-stable with v2-4-test to release 2.4.0 #26371

jedcunningham merged 33 commits into from
Sep 15, 2022

Conversation

@ephraimbuddy
Copy link
Contributor

@ephraimbuddy ephraimbuddy commented Sep 13, 2022

No description provided.

potiuk and others added 17 commits September 8, 2022 21:06
@potiuk
Fix case when SHELL variable is not set in kubernetes tests (#26235)
7f6e0d7 
When SHELL variable is not set, kubernetes tests will fall back
to using 'bash'

(cherry picked from commit b7db311)
@potiuk
Make breeze works with latest docker-compose (#26233)
8c71dd8 
The latest docker-compose dropped an alias for `docker-compose` and
we need to detect it and use "docker compose" instead.

(cherry picked from commit 7f47006)
@potiuk
Limit eager upgrade of protobuf library to < 4.21.0 (#26243)
e60d2b5 
* Limit eager upgrade of protobuf library to < 4.21.0

Until all the Google client libraries get upgraded to >= 2.0.0, we need to
limit the protobuf version.

(cherry picked from commit c6053f5)
@potiuk
Skip overriding version from parameter with the one from tag (#26244)
047f9d9 
Since we are not always updating the version suffix in code
(rc1/rc2/b1/b2), the tag specified via --tag-build prefix should
override the one in code rathe than the other way round.

So what's left now - we will just print warning if the suffix does
not match.

(cherry picked from commit 9cf6f6a)
@potiuk
The PROD cache is pushed always in regular cache build step (#26254)
c6d3344 
In v2-4-test it turned out that PROD cache build was a bit
too limited - the cache has not been prepared if branch was not
main (copy&paste victim). This PR fixes this - cache is always build
in merge run regardless of the branch we are in.

(cherry picked from commit 4178545)
@jedcunningham @ashb
Reformat 2.4.0 release notes (#26247)
7b6da0a 
This makes them proper sections and reorders a couple of them as well.

(cherry picked from commit 49fb3ff)
@potiuk @ashb
Add better progress in CI for constraints generation. (#26253)
c06d8db 
Currently constraints generation is not really showing good progress
while the packages are being removed/installed. This adds progress
that shows that something happens.

(cherry picked from commit d6f473b)
@edithturn @ashb
D400 first line should end with period batch02 (#25268)
d36dcc1 
(cherry picked from commit 5066844)
@uranusjr @ashb
Fix 'from airflow import version' lazy import (#26239)
9fd23bb 
This incorrectly accesses the version *attribute* in airflow.version
instead of the airflow.version module itself, breaking compatibility.

I actually rewrote the entire __getattr__ hook; much of the logic seems
to be quite redundant.

(cherry picked from commit b7a603c)
@dstandish @ashb
Don't blow up when a task produces a dataset that is not consumed. (#…
ea22657 
…26257)

If you have a dataset outlet on a task, and no DAG was recorded as
consuming that dataset it failed with a null value constraint violation
in the db

(cherry picked from commit fe82e96)
@potiuk
Properly build URL to retrieve logs independently from system (#26337)
6f24836 
The previous way of building the path depended on the OS path
but it was really used to build the URL so we should use
urllib instead of os.path.join.

(cherry picked from commit 1838602)
@cra @jedcunningham
Add more weekday operator and sensor examples #26071 (#26098)
a10fc9a 
(cherry picked from commit dd6b2e4)
@DavidKatz-il @jedcunningham
Add the dag_id to AirflowDagCycleException message (#26204)
25f6011 
(cherry picked from commit aada79e)
@jedcunningham
Flush dataset events before queuing dagruns (#26276)
05a6fcf 
When we go to schedule dagruns from the dataset dagrun queue, we assume
the events will happen before the queue records, which isn't the case
unless we explicitly flush them first. This ensures that dagruns are
properly related to their upstream dataset events.

(cherry picked from commit 954349a)
@jedcunningham
Fix dags_needing_dagruns dataset info timestamp (#26288)
c642e1a 
(cherry picked from commit 3c9c0f9)
@ashb @jedcunningham
Update erd SHA to pass static checks (#26263)
91ebd75 
The cause of the failure was a PR race -- the PR that changed migrations
was opened and passing tests before the ERD sha was added.

(cherry picked from commit 12b5445)
@ephraimbuddy
Update Airflow version
95eda4b
@boring-cyborg boring-cyborg bot added area:core-operators area:dev-tools area:logging area:production-image Production image improvements and fixes area:providers area:Scheduler including HA (high availability) scheduler kind:documentation provider:amazon AWS/Amazon - related issues provider:databricks provider:google Google (including GCP) related issues labels Sep 13, 2022
@ephraimbuddy ephraimbuddy marked this pull request as ready for review September 14, 2022 17:48
jedcunningham
jedcunningham reviewed Sep 14, 2022
View reviewed changes
jedcunningham
jedcunningham reviewed Sep 14, 2022
View reviewed changes
Copy link
Member

@jedcunningham jedcunningham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need the AIP-48 catchall in new features.

@ephraimbuddy ephraimbuddy force-pushed the v2-4-test branch 3 times, most recently from 6bda3ae to a8ce509 Compare September 14, 2022 21:13
jedcunningham and others added 5 commits September 14, 2022 18:57
@jedcunningham @weddige
Fix UI redirect (#26409)
3871f00 
Co-authored-by: Konstantin Weddige <konstantin.weddige@lutrasecurity.com>
(cherry picked from commit 56e7555)
@ashb @jedcunningham
Better validation of Dataset URI during dag parse (#26389)
c7ea01d 
Previously we had the validation on the Dataset model, but we since
moved the "dag" facing class to a separate one. This adds validation to
the public class, and extends the validation to not allow space-only
strings

(cherry picked from commit bd181da)
@jedcunningham
Add min attrs version (#26408)
62322ef 
I'm not sure when we started using attrs directly, but we do and we need
>=22.1.0 as we use the min_length validator.

(cherry picked from commit fdecf12)
@jedcunningham
Clear autoregistered DAGs if there are any import errors (#26398)
0967259 
We need to clear any autoregistered DAGs that may have been already
registered if we encounter any import errors while parsing a given DAG
file.

This maintains the behavior before we autoregistered DAGs.

(cherry picked from commit 01e3fb7)
@jedcunningham
Require dag_id arg for dags list-runs (#26357)
38d3d4f 
While it would ideal to transition to a positional arg like was
attempted in #25978, this unfortunately does result in a breaking change
so we cannot do it now.

By instead marking the existing arg as required, we maintain backcompat
while also providing a helpful error message to the user if they forget
it.

This reverts commit ed6ea72.

(cherry picked from commit f6c579c)
@ephraimbuddy ephraimbuddy force-pushed the v2-4-test branch 2 times, most recently from 3348a3b to bef7857 Compare September 15, 2022 07:55
ashb
ashb reviewed Sep 15, 2022
View reviewed changes
Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. We need to add datasets to the release notes!

@ephraimbuddy
Copy link
Contributor Author

ephraimbuddy commented Sep 15, 2022

Oh. We need to add datasets to the release notes!

It's the first in new features?

@ashb
Copy link
Member

ashb commented Sep 15, 2022

Oh. We need to add datasets to the release notes!

It's the first in new features?

Sure, but it needs more than just a bullet point :D (i.e. a big call out like we have for the other Significant Changes) Working on it

potiuk
potiuk approved these changes Sep 15, 2022
View reviewed changes
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all, all looks familiar. I additionally cherry-picked one change #26392 which was missing (not sure where and how)

@ephraimbuddy
Copy link
Contributor Author

ephraimbuddy commented Sep 15, 2022

Reviewed all, all looks familiar. I additionally cherry-picked one change #26392 which was missing (not sure where and how)

Probably because it doesn't have a milestone

@potiuk @ephraimbuddy
Move the deserialization of custom XCom Backend to 2.4.0 (#26392)
a166fb8 
The change has been cherry-picked so we need to update the docs.

(cherry picked from commit 41e48aa)
jedcunningham
jedcunningham reviewed Sep 15, 2022
View reviewed changes
@potiuk
Copy link
Member

potiuk commented Sep 15, 2022

Reviewed all, all looks familiar. I additionally cherry-picked one change #26392 which was missing (not sure where and how)

Probably because it doesn't have a milestone

Likely 🤦

jedcunningham
jedcunningham reviewed Sep 15, 2022
View reviewed changes
@jedcunningham jedcunningham merged commit 0bcdba0 into v2-4-stable Sep 15, 2022
@potiuk
Copy link
Member

potiuk commented Sep 15, 2022

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashb ashb ashb left review comments

@jedcunningham jedcunningham jedcunningham approved these changes

@potiuk potiuk potiuk approved these changes

@mik-laj mik-laj Awaiting requested review from mik-laj mik-laj is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@turbaszek turbaszek Awaiting requested review from turbaszek turbaszek is a code owner

@dstandish dstandish Awaiting requested review from dstandish dstandish is a code owner

@uranusjr uranusjr Awaiting requested review from uranusjr uranusjr is a code owner

@jhtimmins jhtimmins Awaiting requested review from jhtimmins jhtimmins is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@bbovenzi bbovenzi Awaiting requested review from bbovenzi bbovenzi is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

Assignees

No one assigned

Labels

area:dev-tools area:logging area:production-image Production image improvements and fixes area:providers area:Scheduler including HA (high availability) scheduler kind:documentation provider:amazon AWS/Amazon - related issues provider:databricks provider:google Google (including GCP) related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@ephraimbuddy @ashb @potiuk @jedcunningham @edithturn @uranusjr @dstandish @cra @DavidKatz-il @blag