windows build

[pm100]

after much work I have got a windows version running. Many issues were found, some nothing to do with windows

• there is no list of prerequisites. protobuf, llvm (for bindgen), pnpm, vite
• if pnpm is missing then the build silently fails (fixed now - ty)
• sqlean does not build if sqlite dev tools are not installed. Nothing says where to look for the sqlite header files . If sqlite headers are already installed in a sensible place then they are found, other wise the build fails, I fixed this by adding

.include("../sqlite-loadable/sqlite3ext-sys/sqlite3")

to point cc at the headers included in the github dirs.

Windows specific

• google protobuf includes are not found. They are installed in /usr/include on linux (you note this in build.rs of core) but they are not in any standard location on windows. Temp hack

change
let includes = vec![PathBuf::from("/usr/include"), PathBuf::from(PROTO_PATH)];

to

let includes = vec![
    PathBuf::from("c:/tools/proto/include"),
    PathBuf::from(PROTO_PATH),
  ];

• stack overflow when booting up. I fixed this using editbin tool to change the stack size. I need to track down the correct linker flags to make it compile and link correctly.

But now it runs ok - woo hoo.

I need to track down why any minor tweak to anything causes a rebuild of all 700+ crates. I suspect there is something being emitted by the build.rs files.

[pm100]

update - I now know how to set the stack size at compile time using RUSTFLAGS, but it needs to be set in the cargo config.toml file. Working on that

[ignatz]

I'm very supportive of having windows builds, I just haven't gotten around to it. Your effort is greatly appreciated.

One requirement would be that it can build in a GitHub workflow, See https://github.com/trailbaseio/trailbase/blob/main/.github/workflows/release.yml, for the Mac and Linux builds.

sqlean does not build if sqlite dev tools are not installed. Nothing says where to look for the sqlite header files . If sqlite headers are already installed in a sensible place then they are found, other wise the build fails, I fixed this by adding

Sorry, that's a known issue. It should pick up the headers from the bundles sqlite to ensure consistency (though sqlite's APIs are incredibly stable). I didn't point it at sqlite-loadable because I am still considering to get rid of sqlite-loadable. I was also considering to remove sqlean entirely. Not sure how strong folks feel about stored procedures at this point. Either way, the correct crate would be https://crates.io/crates/libsqlite3-sys.

stack overflow when booting up. I fixed this using editbin tool to change the stack size. I need to track down the correct linker flags to make it compile and link correctly.

The stack size is fundamentally a runtime property: https://www.man7.org/linux/man-pages/man3/pthread_attr_setstacksize.3.html. When setting compiler flags you might only be able to override the default.

I need to track down why any minor tweak to anything causes a rebuild of all 700+ crates. I suspect there is something being emitted by the build.rs files.

cargo reads the stdout of build.rs and is looking for commands like

println!("cargo::rerun-if-changed={PROTO_PATH}");

Note that missing directories will always trigger a rebuild.

[ignatz]

google protobuf includes are not found. They are installed in /usr/include on linux (you note this in build.rs of core) but they are not in any standard location on windows. Temp hack

I can't really speak to Window's way of discovering includes but the workaround I had added was due to a broken "libprotobuf-dev" debian package at the time. Don't remember the details, presumably the pkgconf definition was incorrect. Either way, removed it now. I would expect that windows has a similar mechanism. If it doesn't build could it be:

• a broken install, e.g. not a discoverable location for windows?
• an issue with the prost compiler on windows?

We can always add another workaround but would be helpful to understand what an idiomatic solution would look like.

[pm100]

google protobuf windows install just says 'unzip this file and place the include dir somewhere where your compiler will find it'. There is no standard location, or discovery mechanism.

Maybe you could add protobuf as a github dependency, to get the include dir into the build , rather than depending on the location of the files.

[ignatz]

google protobuf windows install just says 'unzip this file and place the include dir somewhere where your compiler will find it'

Can this be read as unzip the files to your compiler's default include path even if Windows itself doesn't provide a default one?

EDIT: I'm also not opposed to making the build more hermetic but I would first have to look into the prost build. Ideally it would do that already.

[ignatz]

Windows specific

• google protobuf includes are not found. They are installed in /usr/include on linux (you note this in build.rs of core) but they are not in any standard location on windows. Temp hack

change
let includes = vec![PathBuf::from("/usr/include"), PathBuf::from(PROTO_PATH)];

Of the originally reported issues, this is the only one not yet addressed. That said, I took the holidays to play a bit with Windows and while being far from knowledgeable in that space, I installed protobuf using winget and had no issues, which furthers my previous suspicion that this is an insufficient manual install

[pm100]

windows needs a larger stack

need a .cargo/config.toml with

[target.x86_64-pc-windows-msvc]
rustflags = ["-C", "link-arg=/STACK:4000000"]

I can do a PR

[ignatz]

What you're saying is that Window's default stack size of 1MB isn't enough for TrailBase (Linux' default is 8MB). I was surprised that 1MB isn't enough but was able to reproduce this by limiting my stack size. It seems to be related to sqlite3_parser:

Thread 1 "trail" received signal SIGSEGV, Segmentation fault.
0x0000555559eda8f1 in sqlite3_parser::parser::parse::yyParser::yy_reduce (self=0x55555d4d59f8, yyruleno=<error reading variable: Cannot access memory at address 0x7fffffeda06e>,
    yy_look_ahead=0, yy_lookahead_token=0x5f454160180) at /home/sebastian/projects/trailbase/target/debug/build/sqlite3-parser-4bdba99858e43dc0/out/parse.rs:4166
4166        fn yy_reduce(
(gdb) bt
#0  0x0000555559eda8f1 in sqlite3_parser::parser::parse::yyParser::yy_reduce (self=0x55555d4d59f8, yyruleno=<error reading variable: Cannot access memory at address 0x7fffffeda06e>,
    yy_look_ahead=0, yy_lookahead_token=0x5f454160180) at /home/sebastian/projects/trailbase/target/debug/build/sqlite3-parser-4bdba99858e43dc0/out/parse.rs:4166
#1  0x0000555559f087d9 in sqlite3_parser::parser::parse::yyParser::sqlite3Parser (self=0x7ffffffdf9a8, yymajor=sqlite3_parser::dialect::token::TokenType::TK_TABLE, yyminor=...)
    at /home/sebastian/projects/trailbase/target/debug/build/sqlite3-parser-4bdba99858e43dc0/out/parse.rs:6571
#2  0x0000555559eb465f in sqlite3_parser::lexer::sql::{impl#1}::next (self=0x7ffffffdf9a8) at src/lexer/sql/mod.rs:205

I'll take a look at the parser

[ignatz]

I did document my findings here: gwenn/lemon-rs#78 . Let's see. Hopefully the author has a better approach. It's not very satisfying but as a temporary work around your stack increase should work. Alternatively and even worse for development, you could also build and run in release mode.

[ignatz]

Ok, 44e0bb6 should have addressed the issue also for dev builds selectively bumping the optimization level. Let me know if it didn't fix your issue.

[ignatz]

sqlean does not build if sqlite dev tools are not installed. Nothing says where to look for the sqlite header files . If sqlite headers are already installed in a sensible place then they are found, other wise the build fails, I fixed this by adding

.include("../sqlite-loadable/sqlite3ext-sys/sqlite3")

sqlite-loadable has since been removed and sqlean includes the headers from the bundled sqlite3 from libsqlite3-sys.

[ignatz]

The latest release v0.3.3 includes a windows prebuilt binary serving serving both as an easy default as well as the workflow as a reference for building on windows. Was pleasantly surprised by how far windows dep management has gotten