Webassembly linker does not import symbols

[josephg]

Related to #1570...

Consider this zig code:

extern fn inc(a: i32) i32;

export fn add(a: i32, b: i32) i32 {
    return inc(a) + b;
}

... Which expects the symbol inc to be imported from javascript (or whatever environment is embedding the wasm build).

It won't compile, because the linker is (correctly) identifying that it has no idea where inc will come from in the built artifact:

sephsmac:zig josephg$ zig build-lib --release-small --target-arch wasm32 wasm.zig 
lld: error: zig-cache/wasm.o: undefined symbol: inc

sephsmac:zig josephg$ zig build-exe --release-small --target-arch wasm32 wasm.zig 
lld: error: zig-cache/wasm.o: undefined symbol: inc

The wasm linker supports 2 option arguments for dealing with this:

sephsmac:zig josephg$ wasm-ld --help
OVERVIEW: LLVM Linker

USAGE: wasm-ld [options] <inputs>

OPTIONS:
  --allow-undefined-file=<value>
                         Allow symbols listed in <file> to be undefined in linked binary
  --allow-undefined      Allow undefined symbols in linked binary
...

So either this:

$ zig build-obj --release-small --target-arch wasm32 wasm.zig 
$ wasm-ld wasm.o -o xyz.wasm -O2 --no-entry --allow-undefined

or this:

$ zig build-obj --release-small --target-arch wasm32 wasm.zig 
$ echo 'inc' > symbols.txt 
$ wasm-ld wasm.o -o xyz -O2 --no-entry --allow-undefined-file=symbols.txt 

(module
...
  (import "env" "inc" (func $inc (type 0)))
  (func $add (type 2) (param i32 i32) (result i32)
    get_local 0
    call $inc
    get_local 1
    i32.add)
  (export "add" (func $add)))

How should zig import functions from JS? For now maybe just pass --allow-undefined to wasm-ld?

But that said, larger projects we want that link error if you forget to add a function. We could have extern "wasm" fn blah() void;, or something like that. Then compilation could drop all those function names into a text file and pass them via --allow-undefined-file= to the linker.

?? Thoughts?

[andrewrk]

Thanks for trying this out. I wonder, how is the linker expecting to find symbols? Normally you would pass in .o files or .a files which have the symbols, or for dynamically linked libraries it may make sense to allow undefined symbols. For webassembly, it seems that you should be able to pass in other webassembly modules to the linker to resolve the symbols. Does it let you do this?

If so then I think the export "foo" makes sense. That says that the symbol is expected to be found dynamically in a module named foo, and that would add it to the allow-undef list.

[josephg]

In a sense, all wasm modules are dynamically linked. But all the external symbols need to be injected from javascript (or from whatever environment embeds it:

const fs = require('fs')
const m = new WebAssembly.Module(fs.readFileSync('xyz'))
const env = {
  inc(x) { return x+1 }
}
const i = new WebAssembly.Instance(m, {env})
console.log(i.exports.add(1,2)) // returns 4

It looks like you can hook up multiple wasm binaries and link their symbols together if you compile libraries with --relocatable.

With wasm2.zig:

export fn inc(a: i32) i32 {
  return a+1;
}

You can build if you pass --relocatable to wasm-ld:

$ zig build-obj --release-small --target-arch wasm32 wasm2.zig 
$ wasm-ld wasm2.o -o wasm2r.wasm -O2 --relocatable
$ wasm-ld wasm.o wasm2r.wasm -o xyz -O2 --no-entry

(module
...
  (func $add (type 1) (param i32 i32) (result i32)
    get_local 0
    call $inc
    get_local 1
    i32.add)
  (func $inc (type 2) (param i32) (result i32)
    get_local 0
    i32.const 1
    i32.add)
...
  (export "add" (func $add))
  (export "inc" (func $inc)))

[josephg]

... Ideally you'd want to be able to make a library which imports a bunch of functions from javascript (eg cargo's web-sys crate ) and then you could link to that & call functions from zig. You still need the undefined symbol list when linking the final result though - the linker wants to know which functions your javascript runtime is expected to provide.

[andrewrk]

Here's what I think should happen:

• zig build-lib --static passes the --relocatable flag. Also zig build-obj does this. The reason they both do the same thing is that I'm considering deprecating zig build-obj in favor of zig build-lib --static.
• extern "wasm" as you mentioned, will mark the symbol as --allow-undefined.
• --object foo will let you add relocatable webassembly modules to the linker line. I believe this already works. This can resolve extern symbols that are not expected to be found at runtime.

Sound good?

[josephg]

Points 2 and 3 I agree with. That sounds good 👍

I'm not sure about deprecating build-obj - My spider sense is telling me that build-obj will be useful for something, or that there's probably things you can do with a set of object files that you can't do with a relocatable library. Creating a relocatable library certainly changes the contents - but I'm not sure if those changes matter.

@bnoordhuis thoughts?

sephsmac:zig josephg$ wasm2wat wasm.o 
(module
  (type (;0;) (func (param i32 i32) (result i32)))
  (type (;1;) (func (param i32) (result i32)))
  (import "env" "__linear_memory" (memory (;0;) 0))
  (import "env" "__indirect_function_table" (table (;0;) 0 anyfunc))
  (import "env" "inc" (func (;0;) (type 1)))
  (func (;1;) (type 0) (param i32 i32) (result i32)
    get_local 0
    call 0
    get_local 1
    i32.add))

sephsmac:zig josephg$ wasm-ld wasm.o -o wasmolib.wasm --relocatable
sephsmac:zig josephg$ wasm2wat wasmolib.wasm 
(module
  (type (;0;) (func (param i32) (result i32)))
  (type (;1;) (func (param i32 i32) (result i32)))
  (import "env" "inc" (func $inc (type 0)))
  (func $add (type 1) (param i32 i32) (result i32)
    get_local 0
    call $inc
    get_local 1
    i32.add)
  (table (;0;) 1 1 anyfunc)
  (memory (;0;) 0))

[bnoordhuis]

Andrew's proposal sounds good. I don't have a strong opinion on zig build-obj.

One thing about --relocatable though: it affects link-time dead code elimination since it disables --gc-sections and --merge-data-segments. Is that what people expect from zig build-lib --static?

(Not a leading question. I genuinely don't know.)

[andrewrk]

One thing about --relocatable though: it affects link-time dead code elimination since it disables --gc-sections and --merge-data-segments. Is that what people expect from zig build-lib --static?

Yes. When you build a static library it has all the symbols available for use, and then the linker does a final garbage collection when you link against the static library, omitting unused symbols.

[andrewrk]

@josephg regarding build-obj, the self hosted compiler puts a lot of effort into breaking up a compilation into many .o files to parallelize the build process. My thoughts here are that, in general, a static library accomplishes what an object file does (potentially even better since it allows garbage collection as described above; see also #54), plus a static library can have multiple object files in it, so that we can have this parallelized build process.