OSX CI failing on install-native-dependencies.sh

[josalem]

OSX jobs are failing inside install-native-dependencies.sh when it calls brew update/brew upgrade.

link to AzDO job example: https://dev.azure.com/dnceng/public/_build/results?buildId=652102&view=logs&jobId=557db079-59cb-586e-d002-d89183ae853e&j=557db079-59cb-586e-d002-d89183ae853e&t=31e7116d-7806-5d9c-f684-dbe999657fff

Homebrew attempts to upgrade installed packages and one of the packages must have a broken formula:

...
==> Downloading https://github.com/awslabs/aws-sam-cli/releases/download/v0.50.0//aws-sam-cli-0.50.0.sierra.bottle.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "aws-sam-cli"
...

@dotnet/runtime-infrastructure

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[josalem]

Tried hitting the link in the log manually and it seems to be valid, so this might be a networking issue.

[hoyosjs]

From runfo over 100 builds (same for 250 builds):

runfo search-timeline -p public -d 686 --pr -t "Install Build Dependencies" -c 100 -m -v "Bash exited with code '1'."

Build	Kind	Timeline Record
652038	Rolling	Install Build Dependencies
652038	Rolling	Install Build Dependencies
652038	Rolling	Install Build Dependencies
652038	Rolling	Install Build Dependencies
652042	PR #36702	Install Build Dependencies
652016	PR #36711	Install Build Dependencies
652027	PR #36289	Install Build Dependencies
652027	PR #36289	Install Build Dependencies
651956	PR #36201	Install Build Dependencies

Evaluated 100 builds
Impacted 5 bulids
Impacted 9 jobs

For some reason John's PR is not here.

[safern]

That is weird... @wfurt any insights on this?

[jashook]

This is definitely highly impactful and very recent.

[wfurt]

Thet location sends back redirect

> GET /awslabs/aws-sam-cli/releases/download/v0.50.0//aws-sam-cli-0.50.0.sierra.bottle.tar.gz HTTP/1.1
> Host: github.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 302 Found
< date: Wed, 20 May 2020 00:57:31 GMT
< content-type: text/html; charset=utf-8
< server: GitHub.com
< status: 302 Found
< vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding
< location: https://github-production-release-asset-2e65be.s3.amazonaws.com/92205085/09d90700-99e5-11ea-9e00-bb180368cad0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200520%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200520T005731Z&X-Amz-Expires=300&X-Amz-Signature=558dcb0cf43857d498dc6ca80b8c1326bda693ea32bdc7b590febd8984d1b2d3&X-Amz-SignedHeaders=host&actor_id=0&repo_id=92205085&response-content-disposition=attachment%3B%20filename%3Daws-sam-cli-0.50.0.sierra.bottle.tar.gz&response-content-type=application%2Foctet-stream
< cache-control: no-cache

I'm wondering if something was inconsistent for a while or if the AWS had some transient outages.
It can also load balance to different locations based on various aspect.

[wfurt]

BTW where the aws-sam-cli comes from? I don/t see it in eng/install-native-dependencies.sh

[wfurt]

elif [ "$1" = "OSX" ]; then
    brew update
    brew upgrade

I'm wondering if this is too broad. Looking through the logs, we tried to install Java, PHP & other crap. I'm wondering if we should only install/upgrade packages we need for the build.

Also it seems like tvOS & iOS are identical and there is only one package different with OSX.

[hoyosjs]

elif [ "$1" = "OSX" ]; then
    brew update
    brew upgrade

I'm wondering if this is too broad. Looking through the logs, we tried to install Java, PHP & other crap. I'm wondering if we should only install/upgrade packages we need for the build.

Also it seems like tvOS & iOS are identical and there is only one package different with OSX.

It's broad, but it was for pretty silly issues that we had no good way of working around (brew didn't allow us to be too selective). See #33481 cc: @directhex

[jashook]

This was recently added to avoid other issues @steveisok