[SR-1676] Build SourceKit on Linux

[briancroom]

Previous ID	SR-1676
Radar	None
Original Reporter	@briancroom
Type	New Feature
Status	Closed
Resolution	Done

Additional Detail from JIRA

Votes	5
Component/s	Source Tooling
Labels	New Feature, Linux
Assignee	@alblue
Priority	Medium

md5: 2d76208c2c35984e0b1d18a6d27299b5

Sub-Tasks:

• SR-3698 Ensure dry run works for SourceKit
• SR-3699 Configure builtbots so that source kit is included in default install

blocks:

• SR-710 Generate XCTestCaseProvider entries on Linux

is blocked by:

• SR-1613 Require blocks runtime when compiling SourceKit on Linux
• SR-1639 [SourceKit] Add in-process implementations of sourcekitd functions only defined for XPC
• SR-1677 Link with libdispatch when building SourceKit on Linux

Issue Description:

SourceKit is currently disabled altogether on Linux, but the plan for resolving SR-710 requires that it be available. The InProc version of sourcekitd should be sufficient for this purpose, but there is still a bit of work required to get this building. See the linked sub-tasks for a (partial?) list of outstanding work.

[briancroom]

For reference, here is a list of related PRs that @modocache posted on SR-710 recently:

• [SR-1613][SourceKit] Include libdispatch #2704 – Includes the swift-corelibs-libdispatch headers, so the SourceKit Linux build at least gets to the linking stage (which fails).

• [SourceKit] Conditionally use in-proc for repl #2763 – When building SourceKit using an in-process model (as opposed to XPC), the linking phase of sourcekitd-repl fails, because we haven't build sourcekitd (instead we build sourcekitdInProc. This fixes that linker error.

• [SR-1639][SourceKit] Link SwiftLang libraries to sourcekitd #2766 – Fixes another linker error for SourceKit on Linux.

• [SR-1639] Fix linker errors for sourcekitd when in-process #2769 – Provides stub implementations for sourcekitdAPI functions that will need non-XPC implementations. The idea here is to at least get SourceKit to compile on Linux, and to leave the implementation for later. This probably shouldn't be merged, but I'd appreciate feedback.
  [SR-1639] Move sourcekitdInProc to std::mutex #2793 – After applying the above pull requests, the only remaining linker errors for SourceKit on Linux are for unresolved libdispatch symbols. Which begs the question: do we link libdispatch to Linux SourceKit, or do we replace libdispatch with something else (like std::async)? This pull request is the first in a series of steps that would replace libdispatch.

[briancroom]

I've opened some other pull requests related to this work:

#3019 - Annotate some tests that fail on Linux
#2862 - Link with libdispatch on Linux

[briancroom]

Here is some additional outstanding work on this front:

#3026 - In-Proc implementation of sourcekitd API functions
#3079 - Only build sourcekitd-repl if a unicode-aware libedit is present
#3080 - CMake dependency fixes
#3085 - Add Semaphore to SourceKitSupport's concurrency module

[briancroom]

Unassigning this from myself as I am unable to continue working on it currently (and haven't been for some time already.)

[alblue]

The build script has been changed to allow source kit to be built on Linux. It does not yet exist in the standard installation however, which means that it's not in the downloadable packages that come from swift.org.

[norio-nomura]

opened #8485 - Enable SourceKit tests if building SourceKit

[norio-nomura]

opened #9025 - Enable SourceKit tests if building SourceKit other than ubuntu 14.04

[alblue]

Since we're building SourceKit on Linux, and the other task isn't blocking this one, I believe we can close this. Norio, can you confirm?

[norio-nomura]

👍
Should we open another issue about tests failed and are disabled on ubuntu 14.04?

[alblue]

Sounds like a good idea. The problem is, I wasn't able to reproduce it locally and it's difficult to know how the 14.04 buildbot is set up.