Introducing --latest for generate-issue-content-core command #38250

amoghrajesh · 2024-03-18T09:01:31Z

Follow ups to #38062, there should be mechanisms to find the latest airflow versions released to integrate seamlessly with the ci. Integrating --latest, that gets the latest version of airflow from https://pypi.org/pypi/apache-airflow/json, consumed VERSION env variable and falls back in case not set graciously.

Example:

➜  airflow git:(main) ✗ breeze release-management generate-issue-content-core --latest --verbose
/Users/adesai/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(

Environment variable VERSION not set, setting current release version as 2.8.3rc1

/Users/adesai/Documents/OSS/airflow/dev/breeze/src/airflow_breeze/commands /Users/adesai/Documents/OSS/airflow
Command to run: 'git log --pretty=format:%H %h %cd %s --date=short 2.8.3...2.8.3rc1 -- .'

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

amoghrajesh · 2024-03-18T15:09:56Z

Test failures look unrelated, @potiuk when you have some time, could you help in a review, this is as per our discussion in #38062

potiuk · 2024-03-19T13:01:28Z

Can you please close and re-open the PR from the airflow repository and add canary and default versions only labels? This will show what's going on - because this is the only place where the workflow testing airlfo w releases is tested now. I think.

amoghrajesh · 2024-03-25T06:19:33Z

@potiuk I just did that, lets see the run

amoghrajesh · 2024-03-25T13:13:06Z

The checks look fine here @potiuk.
Wdyt?

potiuk · 2024-03-25T15:29:27Z

Hmm. The output still looks strange:

I'd expect the actual list of issue since 2.8.3 (but maybe some other set of parameters is needed to generate these ?)

amoghrajesh · 2024-03-26T07:54:35Z

Hmm. The output still looks strange:
I'd expect the actual list of issue since 2.8.3 (but maybe some other set of parameters is needed to generate these ?)

I made it such that it would take the versions as latest and latest-rc1 if "VERSION" env is not set, since the latest and latest-rc1 wont have commits in between, it will show as 0 PRs.

I'd expect the actual list of issue since 2.8.3 (but maybe some other set of parameters is needed to generate these ?)

To accomodate this use case, I have made the changes for it to take 'main'. My only worry is that now it will have 2000+ PRs, is that ok?

potiuk · 2024-03-26T08:20:43Z

To accomodate this use case, I have made the changes for it to take 'main'. My only worry is that now it will have 2000+ PRs, is that ok?

We can use HEAD~40 for example explicitly as the base version. This will always take last 40 commits.

potiuk · 2024-03-26T08:21:38Z

https://git-scm.com/docs/gitrevisions

amoghrajesh · 2024-03-26T08:25:53Z

To accomodate this use case, I have made the changes for it to take 'main'. My only worry is that now it will have 2000+ PRs, is that ok?

We can use HEAD~40 for example explicitly as the base version. This will always take last 40 commits.

Right, OR i can use the --max-count too.

potiuk · 2024-03-26T08:27:56Z

Yep. --max-count is better.

amoghrajesh · 2024-03-26T08:35:27Z

@potiuk thanks, its much better now!

➜  airflow git:(latestCore) ✗ breeze release-management generate-issue-content-core --latest --verbose
/Users/adesai/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(

Environment variable VERSION not set, setting current release version as 'main'

/Users/adesai/Documents/OSS/airflow/dev/breeze/src/airflow_breeze/commands /Users/adesai/Documents/OSS/airflow
Command to run: 'git log --pretty=format:%H %h %cd %s --date=short --max-count 30 2.8.4...main -- .'
Retrieving PR#38478: https://github.com/apache/airflow/pull/38478
Retrieving PR#38477: https://github.com/apache/airflow/pull/38477
Retrieving PR#38440: https://github.com/apache/airflow/pull/38440
Retrieving Linked issue PR#32471: https://github.com/apache/airflow/issue/32471
Failed to retrieve linked issue #32471: Unknown Issue
Retrieving PR#38469: https://github.com/apache/airflow/pull/38469
Retrieving PR#38472: https://github.com/apache/airflow/pull/38472
Retrieving PR#38467: https://github.com/apache/airflow/pull/38467
Retrieving 30 PRs  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  17% 0:00:23

dev/breeze/src/airflow_breeze/commands/release_management_commands.py

amoghrajesh · 2024-03-26T09:48:14Z

@potiuk here is the output with the latest changes. Looks nice now!

➜  airflow git:(latestCore) ✗ breeze release-management generate-issue-content-core --limit-pr-count 25 --latest --verbose
/Users/adesai/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(

Environment variable VERSION not set, setting current release version as 'main'

/Users/adesai/Documents/OSS/airflow/dev/breeze/src/airflow_breeze/commands /Users/adesai/Documents/OSS/airflow
Command to run: 'git log --pretty=format:%H %h %cd %s --date=short 2.8.4...main -- .'
Retrieving PR#38478: https://github.com/apache/airflow/pull/38478
Retrieving PR#38477: https://github.com/apache/airflow/pull/38477
Retrieving PR#38440: https://github.com/apache/airflow/pull/38440
Retrieving Linked issue PR#32471: https://github.com/apache/airflow/issue/32471
Failed to retrieve linked issue #32471: Unknown Issue
Retrieving PR#38469: https://github.com/apache/airflow/pull/38469
Retrieving PR#38472: https://github.com/apache/airflow/pull/38472
Retrieving PR#38467: https://github.com/apache/airflow/pull/38467
Retrieving PR#38463: https://github.com/apache/airflow/pull/38463
Retrieving PR#38054: https://github.com/apache/airflow/pull/38054
Retrieving PR#38022: https://github.com/apache/airflow/pull/38022
Retrieving Linked issue PR#38005: https://github.com/apache/airflow/issue/38005
Retrieving PR#38434: https://github.com/apache/airflow/pull/38434
Retrieving Linked issue PR#38420: https://github.com/apache/airflow/issue/38420
Retrieving PR#38468: https://github.com/apache/airflow/pull/38468
Retrieving PR#38462: https://github.com/apache/airflow/pull/38462
Retrieving PR#38340: https://github.com/apache/airflow/pull/38340
Retrieving PR#38442: https://github.com/apache/airflow/pull/38442
Retrieving PR#38464: https://github.com/apache/airflow/pull/38464
Retrieving PR#38217: https://github.com/apache/airflow/pull/38217
Retrieving Linked issue PR#38031: https://github.com/apache/airflow/issue/38031
Retrieving PR#38456: https://github.com/apache/airflow/pull/38456
Retrieving PR#37988: https://github.com/apache/airflow/pull/37988
Retrieving PR#38451: https://github.com/apache/airflow/pull/38451
Retrieving PR#38374: https://github.com/apache/airflow/pull/38374
Retrieving Linked issue PR#38371: https://github.com/apache/airflow/issue/38371
Retrieving PR#38033: https://github.com/apache/airflow/pull/38033
Retrieving PR#38404: https://github.com/apache/airflow/pull/38404
Retrieving PR#38452: https://github.com/apache/airflow/pull/38452
Retrieving PR#38094: https://github.com/apache/airflow/pull/38094
Retrieving Linked issue PR#28199: https://github.com/apache/airflow/issue/28199
Retrieving PR#38445: https://github.com/apache/airflow/pull/38445
Retrieving 25 PRs  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
We are kindly requesting that contributors to [Apache Airflow RC None](https://pypi.org/project/apache-airflow/None/) help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

- [ ] [Delete all old dag pages and redirect to grid view (#37988)](https://github.com/apache/airflow/pull/37988): @bbovenzi
- [ ] [fix: EmrServerlessStartJobOperator not serializing DAGs correctly whe… (#38022)](https://github.com/apache/airflow/pull/38022): @jliu0812
     Linked issues:
     - [EMRServerlessStartJobOperator Expand/Expand Kwargs not Serializing properly (#38005)](https://github.com/apache/airflow/issues/38005)
- [ ] [[OpenLineage] fix: disabled_for_operators now stops whole event emission  (#38033)](https://github.com/apache/airflow/pull/38033): @kacpermuda
- [ ] [Add executor field to the DB and parameter to the operators (#38054)](https://github.com/apache/airflow/pull/38054): @o-nikolas
- [ ] [Prevent large objects from being stored in the RTIF (#38094)](https://github.com/apache/airflow/pull/38094): @PatrickfBraz @ephraimbuddy
     Linked issues:
     - [Prevent large objects from being stored in RenderedTaskInstanceFields (#28199)](https://github.com/apache/airflow/issues/28199)
- [ ] [Add option to clear only failed tasks in a graph view (#38217)](https://github.com/apache/airflow/pull/38217): @Pad71 @tirkarthi
     Linked issues:
     - [Add the new option "Clear Failed Tasks" to the DagRun Clear option button (#38031)](https://github.com/apache/airflow/issues/38031)
- [ ] [Only show latest dataset event timestamp after last run (#38340)](https://github.com/apache/airflow/pull/38340): @bbovenzi
- [ ] [Store duration in seconds and scale to handle case when a value in the series has a larger unit than the preceding durations (#38374)](https://github.com/apache/airflow/pull/38374): @tirkarthi
     Linked issues:
     - [Task and dagrun duration y-axis is incorrect if the last task has a different duration unit (#38371)](https://github.com/apache/airflow/issues/38371)
- [ ] [Refactor GKE hooks (#38404)](https://github.com/apache/airflow/pull/38404): @moiseenkov
- [ ] [Add legend for duration markline. (#38434)](https://github.com/apache/airflow/pull/38434): @simond @tirkarthi
     Linked issues:
     - [Add legend for average task runtime graph (#38420)](https://github.com/apache/airflow/issues/38420)
- [ ] [Apply task instance mutation hook consistently (#38440)](https://github.com/apache/airflow/pull/38440): @jscheffl
- [ ] [Fix image cache optimizations - speeding up the build (#38442)](https://github.com/apache/airflow/pull/38442): @potiuk
- [ ] [Fix failing MinSQLAlchemy test after moving sqlalchemy spec (#38445)](https://github.com/apache/airflow/pull/38445): @potiuk
- [ ] [Prepare fab provider RC1 (#38451)](https://github.com/apache/airflow/pull/38451): @eladkal
- [ ] [Remove D105 from pyproject.toml (#38452)](https://github.com/apache/airflow/pull/38452): @potiuk
- [ ] [Override chakra styles to keep dropdowns in filter bar (#38456)](https://github.com/apache/airflow/pull/38456): @bbovenzi
- [ ] [Handling edge case for generate-issue-content for providers (#38462)](https://github.com/apache/airflow/pull/38462): @amoghrajesh
- [ ] [Cleans up runs-on in workflows  (#38463)](https://github.com/apache/airflow/pull/38463): @potiuk
- [ ] [Updates trove-classifiers released today in build dependencies. (#38464)](https://github.com/apache/airflow/pull/38464): @potiuk
- [ ] [Add UV_REQUEST_TIMEOUT argument/envvar to building CI/PROD images (#38467)](https://github.com/apache/airflow/pull/38467): @potiuk
- [ ] [Revert "Fix image cache optimizations - speeding up the build (#38442)" (#38468)](https://github.com/apache/airflow/pull/38468): @potiuk
- [ ] [Helm chart 1.13.1 has been released (#38469)](https://github.com/apache/airflow/pull/38469): @jedcunningham
- [ ] [Revert "Add executor field to the DB and parameter to the operators" (#38472)](https://github.com/apache/airflow/pull/38472): @ferruzzi
- [ ] [Airflow 2.8.4 has been released (#38477)](https://github.com/apache/airflow/pull/38477): @jedcunningham
- [ ] [Chart: Default airflow version to 2.8.4 (#38478)](https://github.com/apache/airflow/pull/38478): @jedcunningham


Thanks to all who contributed to the release (probably not a complete list!):
@ephraimbuddy @ferruzzi @jliu0812 @PatrickfBraz @jscheffl @Pad71 @simond @tirkarthi @moiseenkov @bbovenzi @eladkal @kacpermuda @jedcunningham @amoghrajesh @potiuk @o-nikolas

amoghrajesh · 2024-03-26T14:53:48Z

@potiuk finally was able to get this green. Could you take another look?

potiuk · 2024-03-26T14:57:19Z

potiuk · 2024-03-26T14:58:32Z

🎉

amoghrajesh requested review from ashb, jedcunningham, kaxil and potiuk as code owners March 18, 2024 09:01

boring-cyborg bot added the area:dev-tools label Mar 18, 2024

amoghrajesh closed this Mar 25, 2024

amoghrajesh reopened this Mar 25, 2024

amoghrajesh added canary When set on PR running from apache repo - behave as canary run default versions only When assigned to PR - only default python version is used for CI tests labels Mar 25, 2024

Introducing --latest for generate-issue-content-core command

72f71a8

potiuk force-pushed the latestCore branch from 9468b29 to 72f71a8 Compare March 25, 2024 08:54

updating the fallback to main

437751f

Adding max-count

30d3e8c

potiuk reviewed Mar 26, 2024

View reviewed changes

dev/breeze/src/airflow_breeze/commands/release_management_commands.py Outdated Show resolved Hide resolved

using limit-pr-count instead

73e00b2

making it as HEAD

f21727d

amoghrajesh requested a review from potiuk March 26, 2024 12:23

potiuk approved these changes Mar 26, 2024

View reviewed changes

potiuk merged commit 9924bb6 into apache:main Mar 26, 2024

amoghrajesh mentioned this pull request Apr 1, 2024

Introducing --latest for generate-issue-content-helm-chart command #38648

Merged

Introducing --latest for generate-issue-content-core command #38250

Introducing --latest for generate-issue-content-core command #38250

Uh oh!

Conversation

amoghrajesh commented Mar 18, 2024

Uh oh!

amoghrajesh commented Mar 18, 2024

Uh oh!

potiuk commented Mar 19, 2024

Uh oh!

amoghrajesh commented Mar 25, 2024

Uh oh!

amoghrajesh commented Mar 25, 2024

Uh oh!

potiuk commented Mar 25, 2024

Uh oh!

amoghrajesh commented Mar 26, 2024

Uh oh!

potiuk commented Mar 26, 2024

Uh oh!

potiuk commented Mar 26, 2024

Uh oh!

amoghrajesh commented Mar 26, 2024

Uh oh!

potiuk commented Mar 26, 2024

Uh oh!

amoghrajesh commented Mar 26, 2024

Uh oh!

Uh oh!

amoghrajesh commented Mar 26, 2024

Uh oh!

amoghrajesh commented Mar 26, 2024

Uh oh!

potiuk commented Mar 26, 2024

Uh oh!

potiuk commented Mar 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants