ARROW-15693: [Dev] Update crossbow templates to use master or main #13975

kevingurney · 2022-08-25T15:47:16Z

Overview

This pull request:

Removes hard-coded dependencies on "master" as the default branch name in the crossbow infrastructure and CI template files.

Implementation

Removed comment/text references to "master" branch, including URLs to other repositories.
Modified core.py to add a new default_branch property and a new method is_default_branch, for checking whether on the default branch, to the Target class.
Modified CI template files to use the new is_default_branch function to check whether on the default branch.

Testing

Using lafiona/crossbow as a queue repository for qualification.
Ran modified template jobs. All failures appeared to be unrelated to the changes.
The branch names for all relevant qualification jobs are prefixed with build-34-*.
Example of a passing job: https://github.com/lafiona/crossbow/actions/runs/2920227769
Example of a failing job: https://github.com/lafiona/crossbow/runs/7998190113 - in this example, the "Push Docker Image" workflow step is not included, since we are not on the default branch. The failure appears to be related to issues fetching R package resources and not related to the default branch checking logic. There were a variety of other kinds of failures, but none of them appear related to the default branch checking logic.

Future Directions

Remove "master" from default_branch name property of Target class.
Remove all remaining uses of "master" terminology in crossbow.
ARROW-17512: Address minor issues with crossbow documentation.

Notes

Thank you to @lafiona for her help with this pull request!
Due to unexpected technical issues, we opened this pull request as a follow up to WIP: ARROW-15693: [Dev] Update crossbow templates to use master or main #13750. Please see WIP: ARROW-15693: [Dev] Update crossbow templates to use master or main #13750 for more discussion regarding qualification efforts.

… to other repositories. 2. Modified core.py to add default_branch property and a function forchecking whether on default branch, to Target. 3. Modified template files to use the python function for checking whether on default branch. 4. Updated macros.jinja to use python function for checking whether on default branch. Co-Authored-By: Kevin Gurney <kgurney@mathworks.com>

github-actions · 2022-08-25T20:21:51Z

https://issues.apache.org/jira/browse/ARROW-15693

kevingurney · 2022-08-29T15:27:13Z

These changes should be ready for review now.

Our apologies for the delay - it took us some time to qualify our changes in a robust way using @lafiona's queue repository.

@assignUser - if you have any bandwidth to review these changes, that would be greatly appreciated. Thank you!

…om default_branch property of Target class. Co-authored-by: Fiona La <fionala7@gmail.com>

raulcd

Thanks for the PR @kevingurney !
There are a couple of linting issues:

/arrow/dev/archery/archery/crossbow/core.py:744:80: E501 line too long (91 > 79 characters)
/arrow/dev/archery/archery/crossbow/core.py:784:9: E266 too many leading '#' for block comment
/arrow/dev/archery/archery/crossbow/core.py:784:80: E501 line too long (109 > 79 characters)

kevingurney · 2022-08-30T18:57:14Z

I'm very sorry @raulcd - I am not sure why I didn't notice the linting issues before! Thanks for pointing them out! I'll fix them as soon as possible.

…n from "is" to "in" after "master" is removed from the default_branch property. 2. Fix linting issues in core.py. Co-authored-by: Fiona La <fionala7@gmail.com>

kevingurney · 2022-08-30T20:01:37Z

I just pushed a fix for the linting issues. My apologies again for not noticing them sooner!

raulcd

Thanks @kevingurney ! The PR looks good to me just a comment.

dev/tasks/conda-recipes/azure.clean.yml

2. Add missing license header text. Co-authored-by: Fiona La <fionala7@gmail.com>

raulcd

Thanks, looks good to me. @kszucs can you take a look on this one?

kszucs

LGTM

assignUser

Looks good, thanks @kevingurney !
Sorry for the late review, I was out.

@kszucs I think this should be mergable now?

ursabot · 2022-09-06T13:42:39Z

Benchmark runs are scheduled for baseline = 23c5cee and contender = 3e40cd3. 3e40cd3 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Failed ⬇️1.37% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.53% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 3e40cd36 ec2-t3-xlarge-us-east-2
[Failed] 3e40cd36 test-mac-arm
[Failed] 3e40cd36 ursa-i9-9960x
[Finished] 3e40cd36 ursa-thinkcentre-m75q
[Finished] 23c5ceea ec2-t3-xlarge-us-east-2
[Failed] 23c5ceea test-mac-arm
[Failed] 23c5ceea ursa-i9-9960x
[Finished] 23c5ceea ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-09-06T13:42:57Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

kevingurney · 2022-09-06T14:40:15Z

Thanks for the code reviews everyone!

Regarding the performance regressions reported by ursabot - these seem a bit surprising. My initial guess would be that they are unrelated to the infrastructural changes in this PR.

Perhaps, we should re-run the benchmarks to make sure these regressions aren't false positives.

raulcd · 2022-09-06T14:56:35Z

@ursabot please benchmark

ursabot · 2022-09-06T14:56:41Z

Benchmark runs are scheduled for baseline = 7e7b8e1 and contender = 4575b26. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.58% ⬆️0.2%] test-mac-arm
[Failed ⬇️0.27% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.14% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 4575b269 ec2-t3-xlarge-us-east-2
[Failed] 4575b269 test-mac-arm
[Failed] 4575b269 ursa-i9-9960x
[Finished] 4575b269 ursa-thinkcentre-m75q
[Finished] 7e7b8e1f ec2-t3-xlarge-us-east-2
[Finished] 7e7b8e1f test-mac-arm
[Failed] 7e7b8e1f ursa-i9-9960x
[Finished] 7e7b8e1f ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

kevingurney · 2022-09-06T15:36:13Z

Thanks for re-running the benchmarks @raulcd! I was just about to re-run them myself, but had to step away.

raulcd · 2022-09-06T16:16:22Z

hi @kevingurney I don't think the benchmark issues are related to this PR, I've seen this other PR reporting the same issues:
#14019 (comment)

@lafiona

…pache#13975) # Overview This pull request: 1. Removes hard-coded dependencies on "master" as the default branch name in the crossbow infrastructure and CI template files. # Implementation 1. Removed comment/text references to "master" branch, including URLs to other repositories. 2. Modified `core.py` to add a new `default_branch` property and a new method `is_default_branch`, for checking whether on the default branch, to the `Target` class. 3. Modified CI template files to use the new `is_default_branch` function to check whether on the default branch. # Testing 1. Using [lafiona/crossbow](https://github.com/lafiona/crossbow) as a queue repository for qualification. 2. Ran modified template jobs. All failures appeared to be unrelated to the changes. 3. The branch names for all relevant qualification jobs are prefixed with `build-34-*`. 4. Example of a passing job: [https://github.com/lafiona/crossbow/actions/runs/2920227769](https://github.com/lafiona/crossbow/actions/runs/2920227769) 5. Example of a failing job: [https://github.com/lafiona/crossbow/runs/7998190113](https://github.com/lafiona/crossbow/runs/7998190113) - in this example, the *"Push Docker Image"* workflow step is not included, since we are not on the default branch. The failure appears to be related to issues fetching R package resources and not related to the default branch checking logic. There were a variety of other kinds of failures, but none of them appear related to the default branch checking logic. # Future Directions 1. Remove "master" from `default_branch` name property of `Target` class. 2. Remove all remaining uses of "master" terminology in crossbow. 3. [ARROW-17512](https://issues.apache.org/jira/browse/ARROW-17512): Address minor issues with crossbow documentation. # Notes 1. Thank you to @lafiona for her help with this pull request! 2. Due to unexpected technical issues, we opened this pull request as a follow up to apache#13750. Please see apache#13750 for more discussion regarding qualification efforts. Lead-authored-by: Kevin Gurney <kgurney@mathworks.com> Co-authored-by: Fiona La <fionala7@gmail.com> Signed-off-by: Alessandro Molina <amol@turbogears.org>

kevingurney mentioned this pull request Aug 25, 2022

WIP: ARROW-15693: [Dev] Update crossbow templates to use master or main #13750

Merged

github-actions bot added the Component: Documentation label Aug 25, 2022

kevingurney changed the title ~~WIP: ARROW-15693: [Dev] Update crossbow templates to use master or main~~ ARROW-15693: [Dev] Update crossbow templates to use master or main Aug 29, 2022

kevingurney marked this pull request as ready for review August 29, 2022 15:24

Add Jira task ID ARROW-17552 to TODO comment for removing "master" fr…

55a7e5d

…om default_branch property of Target class. Co-authored-by: Fiona La <fionala7@gmail.com>

raulcd reviewed Aug 30, 2022

View reviewed changes

1. Add Jira task ID ARROW-17552 to TODO comment for updating conditio…

f49073e

…n from "is" to "in" after "master" is removed from the default_branch property. 2. Fix linting issues in core.py. Co-authored-by: Fiona La <fionala7@gmail.com>

raulcd reviewed Aug 31, 2022

View reviewed changes

dev/tasks/conda-recipes/azure.clean.yml Outdated Show resolved Hide resolved

1. Remove unnecessary import statement for jinja macros.

4575b26

2. Add missing license header text. Co-authored-by: Fiona La <fionala7@gmail.com>

raulcd approved these changes Aug 31, 2022

View reviewed changes

kszucs approved these changes Aug 31, 2022

View reviewed changes

assignUser approved these changes Sep 5, 2022

View reviewed changes

amol- merged commit 3e40cd3 into apache:master Sep 6, 2022

asfimport mentioned this pull request Feb 16, 2023

[Dev] Remove "master" from default_branch property of Target class in core.py after migration to "main" as the default Git branch #32804

Closed

kevingurney deleted the ARROW-15693 branch August 21, 2023 18:05

ARROW-15693: [Dev] Update crossbow templates to use master or main #13975

ARROW-15693: [Dev] Update crossbow templates to use master or main #13975

Uh oh!

Conversation

kevingurney commented Aug 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Implementation

Testing

Future Directions

Notes

Uh oh!

github-actions bot commented Aug 25, 2022

Uh oh!

kevingurney commented Aug 29, 2022

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

kevingurney commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevingurney commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

kszucs left a comment

Choose a reason for hiding this comment

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented Sep 6, 2022

Uh oh!

ursabot commented Sep 6, 2022

Uh oh!

kevingurney commented Sep 6, 2022

Uh oh!

raulcd commented Sep 6, 2022

Uh oh!

ursabot commented Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevingurney commented Sep 6, 2022

Uh oh!

raulcd commented Sep 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kevingurney commented Aug 25, 2022 •

edited

Loading

kevingurney commented Aug 30, 2022 •

edited

Loading

kevingurney commented Aug 30, 2022 •

edited

Loading

ursabot commented Sep 6, 2022 •

edited

Loading