wasm build?

[tantaman]

Just a question -- have you been able to compile a Rust loadable extension into the WASM build of SQLite?

[tantaman]

Ah, I just saw: https://github.com/asg017/sqlite-loadable-rs#probably-cant-be-compiled-into-wasm

josephg seemed to think compiling a rust dependency into a c library and targeting WASM would be possible: vlcn-io/cr-sqlite#65 (comment)

Haven't tried myself yet, however.

[asg017]

From what I've tried, compiling it to a staticlib (aka .a file) works and can be used in C projects, but emcc only really works with raw .c files. If you try passing in a pre-build .a file into emcc while compiling sqlite into WASM, there's some cryptic errors, but essentially emcc only works with flat .c files or .a files that are generated by emcc.

What I think is the solution is the wasm32-unknown-emscripten target. With that, I believe you can compile a .a file that's in the format emcc expects, and would theoretically compile nicely into sql.js or some other sqlite WASM module.

But I think wasm32-unknown-emscripten isn't maintained anymore, or at least I couldn't find much info or documentation about it.

Also, the problem with WASM SQLite extensions is that they must be statically linked into the SQLite WASM module to be used - which means forking sql.js or forking the official SQLite WASM module. There's no way to dynamically load extensions into an already-compiled SQLite module. It's not a dealbreaker necessarily, but it does make it very difficult to work with.

I also imagine that the binary size of an extension written with sqlite-loadable-rs and compiled to WASM would be quite larger than a bare-bones SQLite library, probably a few MBs.

[tantaman]

Also, the problem with WASM SQLite extensions is that they must be statically

Yep, I've forked the official SQLite build in order to bundle my extensions into a WASM distribution. Wish there was a better way..

would be quite larger than a bare-bones SQLite library

mainly just due to rust standard library inclusion? How big is a bare bones hello world Rust WASM binary?

[asg017]

A compiled "hello world" in Rust with sqlite-loadable-rs is 496KB, compared to C's 17KB. That's on my mac tho, as a dynamic library, so I'm unsure how big a static library in WASM would be. I think SQLite WASM is something like 700-900KB by itself

I dont think Rust includes the entire standard library by default (rather sqlite-loadable-rs dependencies + the subset of Rust's stdlib that is used) totals the 496KB. But when you add crates and dependencies to actually do things, it ballons quick - sqlite-xsv (which just depends on the csv crate and reads from files) is 1.9MB on my mac (and 5.38MB on linux, for some reason).

[josephg]

But when you add crates and dependencies to actually do things, it ballons quick - sqlite-xsv (which just depends on the csv crate and reads from files) is 1.9MB on my mac (and 5.38MB on linux, for some reason).

If you compile to wasm with -Oz, strip the result and run it through wasm-opt, wasm size can stay much smaller than that.

Diamond types in wasm is currently 255KB. That includes a lot of stuff from rust's std (including malloc), my custom b-tree implementation, jumprope (a skip list implementation) and my custom binary encoder & decoder for text CRDTs (which is bigger than it should be).

With brotli compression that drops to 83kb.

Its also worth remembering that modern browsers decode wasm binaries much faster than they decode javascript - usually faster than the network can go.

[asg017]

@josephg this is extemely useful information, thanks for sharing! Really cool to here about wasm-opt and -0z, and I've never considering decoding performance of wasm vs js

Will give compiling sqlite-loadable-rs extensions into WASM another shot

[tantaman]

I got WASM LLVM bitcode generated by cargo to link to a SQLite WASM build today. Going to try actually invoking the function next 🤞

Here is what I did:

1. create a new library crate
2. add the function I intend to expose

use std::ffi::c_void;
#[no_mangle]
pub extern "C" fn rs_test_commit_hook(_user_data: c_void) -> i32 {
    0
}

3. compile and emit LLVM bitcode for the wasm-unknown-unknown target

RUSTFLAGS="--emit=llvm-bc" cargo build --target wasm32-unknown-unknown

4. compile sqlite to bitcode
5. link the rust generated bitcode to the emscripten generated bitcode

emcc -s ALLOW_MEMORY_GROWTH=1 -s WASM=1 -s INVOKE_RUN -s WASM_BIGINT=1 -Oz -flto --closure 1 \
	  -s EXPORTED_FUNCTIONS=@src/exported_functions.json -s EXPORTED_RUNTIME_METHODS=@src/extra_exported_runtime_methods.json \
	  --js-library src/libfunction.js --js-library src/libmodule.js --js-library src/libvfs.js \
	  -s ASYNCIFY -s ASYNCIFY_IMPORTS=@src/asyncify_imports.json -s ASYNCIFY_STACK_SIZE=12288 \
		-I'deps/sqlite-amalgamation-3400000' -I./crsql -Wno-non-literal-null-conversion -Oz -flto \
	  tmp/bc/dist/extension-functions.bc tmp/bc/dist/libfunction.bc tmp/bc/dist/libmodule.bc tmp/bc/dist/libvfs.bc rs/crsqlite_replication_client/target/wasm32-unknown-unknown/debug/deps/crsqlite_replication_client.bc *.o -o dist/wa-sqlite-async.mjs

which compiled with no errors 🎉

I'll see if I can strip this down to minimal bits ahead of our conversation and actually get that function invoked from sqlite.

Also -- should we move this discussion to https://github.com/asg017/sqlite-loadable-rs ?

[tantaman]

minimal demo repo: https://github.com/tantaman/sqlite-rust-wasm

Seems like it all works :)

Now to try building sqlite-loadable-rs in the same way.

[maikonweber]

Hi, i Have interesse in this project.