Swift Static Build

[mredig]

There doesn't appear to be compatibility with building with static linking on linux. I assume this might have something to do with the static linking sdk's use of musl.

Anyways, the project builds perfectly fine with swift build and swift build -c release. However, after installing the static linking sdk and running swift build -c release --swift-sdk aarch64-swift-linux-musl I get the following:

root@74397c1150f2:~/Lighter# swift build -c release --swift-sdk aarch64-swift-linux-musl
Building for production...
<module-includes>:1:10: note: in file included from <module-includes>:1:
1 | #include "shim.h"
  |          `- note: in file included from <module-includes>:1:
2 |

/root/Lighter/Sources/SQLite3-Linux/shim.h:1:10: error: 'sqlite3.h' file not found
1 | #include <sqlite3.h>
  |          `- error: 'sqlite3.h' file not found
2 |

/root/Lighter/Sources/SQLite3Schema/CatalogObject.swift:89:8: error: could not build C module 'SQLite3'
 87 | // MARK: - Fetching
 88 |
 89 | import SQLite3
    |        `- error: could not build C module 'SQLite3'
 90 |
 91 | public extension Schema.CatalogObject {
<module-includes>:1:10: note: in file included from <module-includes>:1:
1 | #include "shim.h"
  |          `- note: in file included from <module-includes>:1:
2 |

/root/Lighter/Sources/SQLite3-Linux/shim.h:1:10: error: 'sqlite3.h' file not found
1 | #include <sqlite3.h>
  |          `- error: 'sqlite3.h' file not found
2 |

I'd post a PR, but I really don't know much about the difference between musl and glibc and if my hunch is even in the right direction.

I'm just going to use another package. I wish I could use this one, but alas, it doesn't appear to be destined. Anyways, I just wanted to let you know in case that's something you'd want to address in the future. I'll understand if it's just not on the roadmap :)

[helje5]

That most likely just means that the static SDK doesn't include SQLite? Does it work with any other SQLite lib?
I'd like to see this working, using Lighter on the server is absolutely on my list of things (I already have an example for that in the Examples repo).

Can you reproduce the issue in a small GitHub repository, along with instructions on how to build it and when it fails?

[mredig]

That most likely just means that the static SDK doesn't include SQLite?

That would seem logical, actually.

Does it work with any other SQLite lib?

I tried GRDB, but ran into the exact same problem (I glossed over that it had the same dependency until building - I was up late last night!) I ultimately just gave up and decided to output my data to csv, as it wasn't a complicated set at all.

I'd like to see this working, using Lighter on the server is absolutely on my list of things (I already have an example for that in the Examples repo).

I'm pretty sure it does! I just didn't get the static linking to work, but it was building just fine without it.

(continued)

[mredig]

Can you reproduce the issue in a small GitHub repository, along with instructions on how to build it and when it fails?

I don't think it needs a whole repo - here are the steps:

1. On a linux machine with swift installed, install the static linking sdk
   1. Go to swift downloads and get the static sdk link (the Download Linux Static SDK button) and its checksum
   2. run swift sdk install <URL-or-filename-here> [--checksum <checksum-for-archive-URL>]
      • if you run into any issues, you can manually download the sdk file and even unarchive it, just put the path to whichever option you choose after install
   3. confirm it installed with swift sdk list
2. Clone a copy of Lighter
3. Test the base build first, I suppose, with swift build and swift build -c release. This should work (assuming you have all the swift/sqlite3 deps installed on linux)
4. Just to be safe, clear the build cache rm -rf .build
5. Build with the static sdk: swift build -c release --swift-sdk aarch64-swift-linux-musl or swift build -c release --swift-sdk x86_64-swift-linux-musl (I'm guessing on the x86 string, tbf)
   • this one should give you the issue

[helje5]

I suppose one option would be to package SQLite as a Swift package vs just referencing the system one. Not sure how one would switch between the two, as you usually would want the system one.

[mredig]

I was thinking along the same lines (but didn't think as far as making choices).

Is the traits feature released yet? That might be a good solution.

Alternatively, I wonder if there's a way (in Package.swift) to detect if we are building statically or not?

[mredig]

Here's a thought (ironically, I think it was you who discussed a similar idea with me in the past) - Since the static sdk is really only a linux/spm command line thing, what if you included it with an env var?

In Package.swift, do something like

import Foundation

if ProcessInfo.processInfo.environment["UseBundledSQLITE"] == "true" {
    // setup deps one way
} else {
    // setup deps normal way
}

and when building with the static SDK, it would just be required to prepend this to the command:

UseBundledSQLITE="true" swift build --swift-sdk aarch64-swift-linux-musl

I don't think this would work with an Xcode build at all, but I don't think that really matters in this case.

[mredig]

Jeeze I must have been sleepy last night. I didn't even notice I posted this in the wrong repo! Sorry about that!

[mredig]

Here's a thought (ironically, I think it was you who discussed a similar idea with me in the past) - Since the static sdk is really only a linux/spm command line thing, what if you included it with an env var?

In Package.swift, do something like

import Foundation

if ProcessInfo.processInfo.environment["UseBundledSQLITE"] == "true" {
    // setup deps one way
} else {
    // setup deps normal way
}

and when building with the static SDK, it would just be required to prepend this to the command:

UseBundledSQLITE="true" swift build --swift-sdk aarch64-swift-linux-musl

I don't think this would work with an Xcode build at all, but I don't think that really matters in this case.

I made a small demo repo here that works in my very limited testing:

https://github.com/mredig/SPM_Env_Test

[mredig]

I just made an experiment build wrapping sqlite code in a swift package, replaced SQLite3 in Lighter, and successfully ran swift build --swift-sdk aarch64-swift-linux-musl!

It's clunky, but it's a working proof of concept: static lighter

[mredig]

Well, I went ahead and basically did the whole env-var-mode-switch.

I didn't directly fork Lighter, tho, so it's not so easy to make a PR. It would need to be doctored up, regardless though. It's pointing to a personal repo for the SPM SQLite build. I think it might be a good start, apart from that!

forked lighter

edit: I apparently neglected to update the readme, so if you do SPM_SQLITE=true swift build, it'll add the dependency and use that, otherwise it'll use the system sqlite.