VSCode/Brackets + Santa cause Xcode autoconfiguration to fail on macOS

[vicb]

Description of the problem / feature request:

I got an error when trying to build angular (I'm on the Angular team berchet@)

The error is:

$ bazel test packages/core/test/render3
ERROR: /private/var/tmp/_bazel_berchet/136114fe9f12514cf56a27652da0b4c4/external/local_config_cc/BUILD:50:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL
ERROR: Analysis of target '//packages/core/test/render3:render3' failed; build aborted: Analysis of target '@local_config_cc//:cc-compiler-darwin_x86_64' failed; build aborted
INFO: Elapsed time: 2.277s
FAILED: Build did NOT complete successfully (34 packages loaded)
ERROR: Couldn't start the build. Unable to run tests

Note: I had this working in the past few weeks (& using multiple times a day).

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

On a gMac:

1. clone the angular repo at https://github.com/angular/angular
2. execute yarn into the root folder to install deps
3. execute bazel test packages/core/test/render3 at the root

yarn and bazel must be installed via brew

What operating system are you running Bazel on?

This seems to be the issue, 4 of us ran into this bug after an High Sierra 10.13.3 update

What's the output of bazel info release?

 bazel version
Build label: 0.9.0-homebrew
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Sun Jul 12 12:24:01 +49936 (1513677414241)
Build timestamp: 1513677414241
Build timestamp as int: 1513677414241

Have you found anything relevant by searching the web?

I have searched the web and my team-mates, misc info

• rebooting help,
• xcode-select -s /Applications/Xcode.app/Contents/Developer before rebooting seems to help,
• when you are in a working state, never ever bazel clean you could get into the broken state

The error seems to originate in this BUILD file:

# /private/var/tmp/_bazel_berchet/136114fe9f12514cf56a27652da0b4c4/external/local_config_xcode/BUILD

package(default_visibility = ['//visibility:public'])

xcode_config(name = 'host_xcodes')
# Error: Invoking xcode-locator failed, return code 256, stderr: java.io.IOException: Cannot run program "/private/var/tmp/_bazel_berchet/136114fe9f12514cf56a27652da0b4c4/external/local_config_xcode/./xcode-locator-bin" (in directory "/private/var/tmp/_bazel_berchet/136114fe9f12514cf56a27652da0b4c4/external/local_config_xcode"): error=1, Operation not permitted, stdout: 

executing /private/var/tmp/_bazel_berchet/136114fe9f12514cf56a27652da0b4c4/external/local_config_xcode/./xcode-locator-bin fromthe CLI returns

{
	"9.2.0": "/Applications/Xcode.app/Contents/Developer",
	"9.2": "/Applications/Xcode.app/Contents/Developer",
	"9": "/Applications/Xcode.app/Contents/Developer",
}

permissions are -rwxr-xr-x

EDIT: Turned out to be an interacction with VSCode, see the fix below

[vicb]

Just tried to update bazel to

Build label: 0.10.0-homebrew
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Jan 10 02:02:06 +50057 (1517480013726)
Build timestamp: 1517480013726
Build timestamp as int: 1517480013726

I get the same exact error (after a bazel clean)

[sergiocampama]

I've seen this multiple times before, but haven't been able to reproduce consistently.

I believe there's some sort of global state that prevents xcode-locator-bin from being invoked within java. If you cat /private/var/tmp/_bazel_berchet/136114fe9f12514cf56a27652da0b4c4/external/local_config_xcode/BUILD you'll see the error.

Can you try this just to diagnose?

1. bazel clean --expunge
2. if 1 doesn't work, reboot

[sergiocampama]

fyi @c-parsons @jmmv

[vicb]

@sergiocampama

If you cat /private/[...]/local_config_xcode/BUILD you'll see the error.

true, I even did that in my initial report !

bazel clean --expunge

1. starting from a working state (after a reboot),
   1a. bazel test ... -> ok
   1b. bazel test ... -> ok
   ...
   1Z. bazel test ... -> ok
2. bazel clean
3. bazel test ... -> fails (~3s)
4. bazel clean --expunge
5. bazel test ... -> fails (~25s)

It takes longer to fail which doesn't look like an improvement to me ;)

[sergiocampama]

My bad, I missed that part of your report :)

Yeah, that seems to match my experience, after a reboot, everything works, but once you clean it gets messed up. I believe that bazel clean just removes the output directories, but bazel clean --expunge deletes all of the execution root stuff, which is why it takes longer to start.

For now the only workaround I know of is to reboot.

What I find most interesting about this is that xcode-locator-bin gets called twice, once from xcode_configure.bzl and another time from osx_cc_configure.bzl. By tracing the code and the output of local_cc_config/BUILD, it appears that the call from osx_cc_configure.bzl does indeed work, but the one in xcode_configure.bzl doesn't.

[jmmv]

It'd be that sandboxing is blocking the execution of the xcode-locator-bin, which results in the obscure error written to the BUILD file. Could you maybe try your reproduction steps after sticking:

build --genrule_strategy=standalone
build --spawn_strategy=standalone

into ~/.blazerc and see if you can trigger the problem?

Just a thought. The fact that an invalid BUILD file is generated at all and left behind is concerning.

[sergiocampama]

I looked into that, but repository_ctx.execute does not use any strategy, it just runs the commands passed through JavaSubprocess, so not sure if a sandbox is used at all for repository execute actions

[vicb]

@jmmv just tried didn't help

I started from a broken state:

• bazel test ... -> fails
• bazel clean
• bazel test ... -> fails
• bazel clean --expunge
• bazel test ... -> fails

[alexeagle]

interestingly, I did High Sierra upgrade on my personal Mac and didn't see this, so it could be something about our Managed Software Update version of XCode in Google's corp Mac setup

[alexeagle]

@vicb @jmmv your instructions said to put the options in .blazerc rather than .bazelrc so you may not have tested what you thought...

[jmmv]

@sergiocampama confirmed by code inspection that the xcode-locator-bin process is being executed directly via Java's subprocess facilities and that no sandboxing is involved... so that's not the case.

However, we have just discovered something that may be getting in the way. Think... antivirus software and Santa. No good explanation on what exactly is wrong yet though.

[jmmv]

Alright, so it could be that, or... something more crazy: VSCode.

It seems that VSCode holds an open file handle on the xcode-locator-bin file as soon as it's created, which could be blocking executions until whatever VSCode wants to do completes. This may or may not be a problem in itself, or it may be a problem only in combination with Santa.

However, @sergiocampama and myself have confirmed that launching and closing VSCode makes the problem appear and vanish consistently.

Add:

"files.exclude": {"bazel-*": true}

to your workspace configuration to ignore the Bazel trees and things will work. (That's why I had never seen this, because I've always had this setting. Removing it made the problem appear pretty quickly.)

[alexeagle]

Awesome debugging. To be fair, I had `build symlink_prefix=/` in our shared bazel.rc for a while, but that made it harder to see outputs and follow error messages, so we changed it to `symlink_prefix=dist/`. Everyone on the team already has `dist` ignored in VScode. But for some reason (bug?) the `bazel-out` folder is not governed by the `symlink_prefix` setting.

…

On Thu, Feb 8, 2018 at 2:14 PM Julio Merino ***@***.***> wrote: Alright, so it could be that, or... something more crazy: VSCode. It seems that VSCode holds an open file handle on the xcode-locator-bin file as soon as it's created, which could be blocking executions until whatever VSCode wants to do completes. This may or may not be a problem in itself, or it may be a problem only in combination with Santa. However, @sergiocampama <https://github.com/sergiocampama> and myself have confirmed that launching and closing VSCode makes the problem appear and vanish consistently. Add: "files.exclude": {"bazel-*": true} to your workspace configuration to ignore the Bazel trees and things will work. (That's why I had never seen this, because I've always had this setting. Removing it made the problem appear pretty quickly.) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4603 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC5IwfgHRjNwwJmHcOpPtFpyf4gcFWPks5tS3G2gaJpZM4R9mUp> .