Helix SDK: execute.sh not found on Mac Queue starting Helix SDK 2.0.0-beta.19151.5

[hoyosjs]

See dotnet/coreclr#22967.

In coreclr we haven't been able to get updates from Arcade. We were seeing some issues with downloading the test payload in Helix and reported it in dotnet/core-eng#5391. After #2147 got merged I tried to manually update the SDK as well as using Darc to consume the fix, but we haven't been able to move off Arcade/Helix SDK 19128.3 as every time we get the same error:

Executed on dci-mac-build-042.local
/bin/sh: /Users/dotnet-bot/dotnetbuild/work/4c18df46-2060-4732-8bc8-b93b4b02663f/Payload/scripts/756b76df1ae54fbd9dc8f31f9e9b0d22/execute.sh: No such file or directory

Things observed:

• Always runs on dci-mac-build-042.local
• Talking to @alexperovich, he said "It looks like the payload that is supposed to have that file got removed somehow. Nothing jumps out to me as the problem."

I tried to check the repro environment, but I can't get to http://mlskvmweb.

@riarenas any idea why this might be the case? Not sure if the error is on our side, but is there anything we can do to narrow down the failure?

Hard to tell at a glance what is going wrong here. I couldn't find binlogs from the helix sdk which would let us take a better look.

I tried to check the repro environment, but I can't get to http://mlskvmweb.

I believe the correct site is https://mlskvmui (make sure to use IE, apparently that's the only browser that works)

@MattGal any ideas on what's going on?

[MattGal]

I'm taking a look at payloads and logs, will update if I find anything useful.

Related Question: Where did the reference to mlskvmweb come from? @garath changed the only place I know where this was listed a couple weeks back.

[MattGal]

Something's definitely awry on dci-mac-build-042. Someone took it offline and removed its heartbeat from the table so I expect it just got thoroughly horked.

I'll try to connect to it and give a better explanation, but it's clearly dead and not taking work so I would expect whatever caused this problem will not repro on subsequent attempts.

[MattGal]

I got on the machine for a bit but couldn't find anything useful. The main Helix log starts @ 3/7/2019 @ 1:43 PM for unknown reasons and from grepping the available logs no mention of the outcome of unzipping 93e4958d-cdc7-4c22-b708-851fff898a47.zip.

@ilyas1974 might be able to look this up if someone created a ticket for it, but otherwise you may as well close this as it's a rogue machine problem and not an Arcade issue.

[hoyosjs]

The main issue is we keep getting the same machine on all our testing for No-tiered, checked OSX testing.

This is for the release 3.0 branch 2: https://github.com/dotnet/coreclr/pull/23110/checks?check_run_id=74602787

I tried manually doing it and I get the same machine as well. Regardless of closing the PR, making a new one, or re-triggering.

[hoyosjs]

As for the mlskvmweb, it's the instructions I get from the repro tool on mc. Tells me to go to that site to look for the resource.

The main issue is we keep getting the same machine on all our testing for No-tiered, checked OSX testing.

I see some workitems succeeded in the jobs from that build on different machines. As long as the machine was in rotation I would definitely expect some of the workitems to be picked up by the rogue machine and fail with the exact same error, but now that the machine is gone from that queue, I would be really surprised if it took any work. Did you retry the job and saw that dci-mac-build-042 took failed work items?

[MattGal]

Ah I didn't realize it was still taking work but not heartbeating. Looks like some kind of wacky DNS stuff. It's directly offline for real now and is clearly a very troubled machine.

The thing about a broken machine is that it does what it does with work items WAY faster than one that actually, you know, does stuff. Not much we can do about that other than identifying the bad machines and taking them offline as fast as possible.

Ah, I see Matt found that the machine was still there sporadically. Hopefully subsequent builds won't hit the same problem.

[hoyosjs]

I'll let it run again and verify. Thanks! I'll close this issue tehn.

[hoyosjs]

This issue was moved to dotnet/core-eng#5441