How do data segments interact with dynamic linking?

[titzer]

Forked discussion from #301

How can data segments (which specify the initial contents of memory at startup) be useful and integrate with dynamic linking? Should programs be able to dynamically load data segments into memory?

[kg]

If you can load a module at runtime, that implies that you can load a data segment at runtime - unless we prohibit non-main-modules from having data segments. I think that prohibition would be a difficult sell, though.

This implies some sort of address space management. If we want to do it in user space, we should expose a single primitive that user code can use to request that a module's data segment be mapped (loaded) into the heap at offset X.

[lukewagner]

Definitely agreed that shared modules should have global data sections; it's a basic need. Copying from comment in #301 to hopefully continue this line of discussion:

A main module knows it has the whole [0, memory_size) range to itself and can put anything anywhere infallibly (an invariant that could be leveraged for interesting optimizations). For a shared module, since wasm semantics don't say what memory in [0, memory_size) is already in use, I've been assuming that we'd want to specify an allocate_global_data_section(length) callback that is specified to be called by the engine when loading a shared module and allows the app to decide exactly how it wants to lay out global data. FWIW, the same issue comes up with aliased thread-local state (which needs to go in the heap) and could have the same callback solution. (There's a lot of symmetry between thread-local state and dynamically linked global state.)

[jfbastien]

As discussed in the PR, it would be nice if the basic user-side allocator (potentially provided by the toolchain) didn't have to magically know about where the module's data is loaded (because it wasn't asked "where would you want this?").

Furthermore, it would be nice if this basic allocator provided ASLR by default. Developers could still do whatever they want, but wouldn't need to reinvent ASLR since they'd get it by default. As things stand that would require copying the data section to a new location on load, which seems silly.

[lukewagner]

@jfbastien The toolchain gets to place the data so I don't understand "magically know"; the data is where the toolchain put the data.

You do make an interesting point that, if we require main module data segments to be loaded at a constant address, this would require doing something suboptimal for a toolchain that wanted to provide ASLR. Now, for shared modules, if we have the above-proposed allocator hook, the hook could easily do the randomization. This would suggest doing the same thing for the main module. The only quirk is that the main module's allocator hook would run with a zero'd heap (the heap was correctly-sized, but not yet initialized) so the hook's code would have to be written not to depend on any global heap state; but that's fine if all the hook wants to do is call random. This would provide more symmetry between main and shared modules, so I'm in favor.

[lskyum]

Just throwing something into the discussion here...

There could be cases where a wasm module is loaded into many browser tabs, for example a .NET/Java VM or some other popular component. Such modules will often have a bunch of static data that will be dublicated between the browser tabs - unless there is a mechanism to share that data without violating security. (For example by making it read-only.)

An example of static data could be language and encoding data in a .NET runtime.

[lukewagner]

@lskyum I agree that's an important scenario. In such cases, we'll also want browser caching to share machine code. The cache file won't be a literal wasm file, but something browser internal (or perhaps a native executable format like ELF). Given this, it seems like we could achieve sharing of the underlying pages by COW-mmap'ing the data segments from the cache file. This does rely on the destination address being a multiple of page size; we could consider mandating this (as already proposed for memory resizing).

[jfbastien]

@jfbastien The toolchain gets to place the data so I don't understand "magically know"; the data is where the toolchain put the data.

@lukewagner Seems like we agree on the overall idea, but I just wanted to clarify this: I just meant that the allocator's code will have to "know" about some magical extern globals for main module start and end location. Something akin to extern const void *__main_module_start, *__main_module_end;. It's not the end of the world, but it's pretty ugly compared to just asking the base allocator (enable ASLR and other things), though indeed the code then has to be able to run without an initialized heap!

[lukewagner]

@jfbastien Ah, I see. There is actually a small design question here: is there one allocate_global_data_section callback defined by the main module and called for all subsequent shared module loads or does each shared module define its own allocate_global_data_section callback? If we did the former, then everything happens in the main module which has all the knowledge and can integrate with the malloc/mmap impls. The latter sounds vaguely more powerful and encapsulated, but it needs the magic imports you were talking about (which would just be part of the ABI that we need to define anyway). Either strategy could be mapped to the other via ABI convention.

[jfbastien]

I think we want to have the following layering:

1. Developers can use any regular malloc they want.
   • These mallocs work as-is (no code change if they're compiled for a POSIX-y system!).
   • These mallocs use mmap.
   • There's one malloc per wasm module (dsos use the main module's allocator).
2. The toolchain provides a global allocator named mmap which serves a shim between malloc and wasm's magical innards.
   • This is specific to the wasm platform, but developers can roll their own.
   • There's again just one such function per wasm module.
   • This function is known to the wasm platform, and is called by the loader to perform initialization of the heap (including allocating data sections, and when loading dsos). It effectively keeps its own "page table" of the internal heap.
3. The wasm platform has magical syscalls for heap allocation and growth.
   • These call into the trusted runtime.

[lukewagner]

@jfbastien Sounds right to me.

[AndrewScheidecker]

I think what @jfbastien said works is about as good as you can do within the current memory model, but to me it seems inevitable that wasm must abandon the idea that a module has absolute control over its address space. I think a model closer to OS processes can be efficiently polyfilled and will more easily accommodate dynamic linking.

Some things we can take from OSes:

• Processes as isolated address spaces.
• Modules don't get their own address space, or any isolation from other modules running in the same process, or from the OS.
• The OS manages the address space of the process, and the module must go through it for page-level allocations.

Some things we shouldn't take from OSes:

• The idea of modules being loaded at static addresses.
• You only have control over the direct dependencies of your module, making it easy to get conflicts in indirect dependencies. I think that can be solved by a process-wide dictionary from semantic names (e.g. libc v34) to URIs, but I don't want to derail this thread with that half-baked idea.

If we applied these ideas to WebAssembly:

• We can define a wasm process with an address space isolated from other processes. Make minimal guarantees about the address space: maybe just a minimum size and a protected first page.
• Define a platform API for allocating virtual address space and committing physical pages to it. mmap(pointer?,I64) -> pointer and munmap(pointer,I64) -> bool are all that's really needed.
• The platform loader needs to cooperate with the map/unmap API to reserve address space for module data segments.

This could all be implemented efficiently in the polyfill as a page allocator for the asm.js linear memory. The polyfill could also require the application to provide an initial size for linear memory, a hint that would no longer be necessary for the wasm module itself.

I expect native implementations would reserve a fixed range of addresses for a wasm process, and generate memory access code using an immutable base and bounds check. However you do it, supporting a true 64-bit address space would likely require a separate OS process for each wasm process, which has its own implications for things like APIs to talk to the browser.

Thoughts?

[sunfishcode]

@AndrewScheidecker This issue here is just about working out the details of how to load data segments into memory within the current design. What you're proposing sounds like a pretty major change to the design. Would you mind filing a new issue so that it can get top-level visibility and discussion, rather than being buried here in the comments for this issue? Thanks!

[lukewagner]

@AndrewScheidecker Actually, what we're calling a "wasm module" is almost exactly the wasm process you described. For background, read discussion in #285 (especially about the contiguity issue) and the resulting FAQ. If you still have specific suggestions, feel free to open a new issue as @sunfishcode suggested. [Edit: oops, I didn't see you already did create #306]