Missing epoll.h when compiling grpc library to wasm

[Han-zb]

I want to compile grpc library to wasm in order to build standalone wasm applications linking against libgrpc.a, and run it in wasmtime.

tools version:
emsdk: 3.1.31
cmake: 3.25.0

Run emcmake cmake .. and emmake make in grpc source directory with compile arguments STANDALONE_WASM and --no-entry, then got the error "epoll.h not found":

In file included from /home/xxx/grpc_emscripten/grpc/src/core/lib/event_engine/posix_engine/ev_epoll1_linux.cc:16:
/home/xxx/grpc_emscripten/grpc/src/core/lib/event_engine/posix_engine/ev_epoll1_linux.h:38:10: fatal error: 'sys/epoll.h' file not found
#include <sys/epoll.h>
         ^~~~~~~~~~~~~
25 warnings and 1 error generated.
em++: error: '/home/xxx/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -I/home/xxx/emsdk/upstream/emscripten/cache/sysroot/include/SDL --sysroot=/home/xxx/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -DCARES_STATICLIB -I/home/xxx/grpc_emscripten/grpc/third_party/re2 -I/home/xxx/grpc_emscripten/grpc/third_party/zlib -I/home/xxx/grpc_emscripten/grpc/include -I/home/xxx/grpc_emscripten/grpc -I/home/xxx/grpc_emscripten/grpc/third_party/address_sorting/include -I/home/xxx/grpc_emscripten/grpc/build/third_party/re2 -I/home/xxx/grpc_emscripten/grpc/third_party/boringssl-with-bazel/src/include -I/home/xxx/grpc_emscripten/grpc/src/core/ext/upb-generated -I/home/xxx/grpc_emscripten/grpc/src/core/ext/upbdefs-generated -I/home/xxx/grpc_emscripten/grpc/third_party/upb -I/home/xxx/grpc_emscripten/grpc/third_party/xxhash -I/home/xxx/grpc_emscripten/grpc/build/third_party/zlib -I/home/xxx/grpc_emscripten/grpc/build/third_party/cares/cares -I/home/xxx/grpc_emscripten/grpc/third_party/cares/cares -I/home/xxx/grpc_emscripten/grpc/third_party/cares/cares/include -I/home/xxx/grpc_emscripten/grpc/third_party/abseil-cpp -fPIC -std=gnu++14 -MD -MT CMakeFiles/grpc.dir/src/core/lib/event_engine/posix_engine/ev_epoll1_linux.cc.o -MF CMakeFiles/grpc.dir/src/core/lib/event_engine/posix_engine/ev_epoll1_linux.cc.o.d -c /home/xxx/grpc_emscripten/grpc/src/core/lib/event_engine/posix_engine/ev_epoll1_linux.cc -o CMakeFiles/grpc.dir/src/core/lib/event_engine/posix_engine/ev_epoll1_linux.cc.o' failed (returned 1)
make[2]: *** [CMakeFiles/grpc.dir/build.make:7112: CMakeFiles/grpc.dir/src/core/lib/event_engine/posix_engine/ev_epoll1_linux.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1124: CMakeFiles/grpc.dir/all] Error 2
make: *** [Makefile:156: all] Error 2
emmake: error: 'make' failed (returned 2)

In order to compile successfully, I just added epoll.h and eventfd.h in emsdk include/sys directory and got libgrpc.a, although I know it may fail In the phase of link.

As expected, when built my demo.cpp, I got error when link against libgrpc:

configure: cmake .. -DCMAKE_CXX_FLAGS=-std=c++17 -DCMAKE_TOOLCHAIN_FILE=/home/xxx/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=/home/xxx/emsdk/node/14.18.2_64bit/bin/node;--experimental-wasm-bulk-memory;--experimental-wasm-threads
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /home/xxx/code_test/grpc_test_wasm/build
make: make
[ 14%] Building CXX object CMakeFiles/hw_grpc_proto.dir/protobuf/grpc_test.grpc.pb.cc.o
[ 28%] Building CXX object CMakeFiles/hw_grpc_proto.dir/protobuf/grpc_test.pb.cc.o
[ 42%] Linking CXX static library libhw_grpc_proto.a
[ 42%] Built target hw_grpc_proto
[ 57%] Building CXX object CMakeFiles/server.dir/server.cpp.o
[ 71%] Linking CXX executable server.wasm
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(wakeup_fd_eventfd.cc.o): undefined symbol: eventfd
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(wakeup_fd_eventfd.cc.o): undefined symbol: eventfd_read
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(wakeup_fd_eventfd.cc.o): undefined symbol: eventfd_write
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(wakeup_fd_eventfd.cc.o): undefined symbol: eventfd
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_ctl
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_ctl
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_wait
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_ctl
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_create1
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(wakeup_fd_eventfd.cc.o): undefined symbol: eventfd
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(wakeup_fd_eventfd.cc.o): undefined symbol: eventfd_read
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(wakeup_fd_eventfd.cc.o): undefined symbol: eventfd_write
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_ctl
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_ctl
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_ctl
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_create1
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_ctl
wasm-ld: error: /home/xxx/grpc_emscripten/grpc/build/libgrpc.a(ev_epoll1_linux.cc.o): undefined symbol: epoll_wait
em++: error: '/home/xxx/emsdk/upstream/bin/wasm-ld @/tmp/emscripten_k16or6kb.rsp.utf-8' failed (returned 1)
make[2]: *** [CMakeFiles/server.dir/build.make:101: server.wasm] Error 1
make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/server.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
emmake: error: 'make' failed (returned 2)

I think the reason is that there is no epoll.h functions in libc.a. So I try to recompile the musl/libc in emsdk with epoll files, but it turns out that the directories and Make files in /home/hzb/emsdk/upstream/emscripten/system/lib/libc/musl missing some files and doesn't support for wasm32

root@pekphis292586:/home/xxx/emsdk/upstream/emscripten/system/lib/libc/musl# emconfigure ./configure --prefix=./my_install --exec-prefix=./my_binary --syslibdir=./my_lib/
configure: ./configure --prefix=./my_install --exec-prefix=./my_binary --syslibdir=./my_lib/
./configure: error: chosen malloc implementation 'mallocng' does not exist
emconfigure: error: './configure --prefix=./my_install --exec-prefix=./my_binary --syslibdir=./my_lib/' failed (returned 1)

learned from #17638, emscripten doesn't supprt epoll system calls, but I found poll system calls in libc.a:

poll.o:
00000000 d .debug_abbrev
00000000 d .debug_line
00000000 d .debug_str
         U __stack_pointer
         U __syscall_poll
         U __syscall_ret
00000001 T poll

I wonder,

1. If epoll.h is added to the musl/libc source and recompiled, can epoll system calls be supported? (which I failed recompile) or why not?
2. If not, maybe I need to change epoll to poll in grpc source code to avoid epoll.

I appreciate any reply and suggestions. Thanks.

[sbc100]

epoll is a linux-specific syscall which is not currently supported by emscripten. Implementing might be possible, but I imagine would be quite a lot of work.

I imagine that grpc supports other mechanisms and that you can configure to avoid epoll when building for emscripten?

[Han-zb]

epoll is a linux-specific syscall which is not currently supported by emscripten. Implementing might be possible, but I imagine would be quite a lot of work.

I imagine that grpc supports other mechanisms and that you can configure to avoid epoll when building for emscripten?

Thank you so much @sbc100 . I just undefined the epoll macro in grpc source code like #define GRPC_LINUX_EPOLL 1 and compiled successfully.

But here is another error about socket(websocket). I run the server.wasm (standalone wasm module) which is a simple grpc server demo in wasmtime, the compile arguments are:

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -sSTANDALONE_WASM -pthread -lwebsocket.js -sPROXY_POSIX_SOCKETS -sUSE_PTHREADS -sPROXY_TO_PTHREAD -Wl,--error-limit=0 -Wl,--no-check-features")

(from https://emscripten.org/docs/porting/networking.html#full-posix-sockets-over-websocket-proxy-server)

and run with wasmtime --wasm-features threads server.wasm, which occurs

root@pekphis292586:/home/xxx/code_test/grpc_test_wasm# wasmtime --wasm-features threads server.wasm
Error: failed to run main module `server.wasm`

Caused by:
    0: failed to instantiate "server.wasm"
    1: unknown import: `env::invoke_iiiii` has not been defined

I don't know if it is related with -lwebsocket.js, and is it possible that run wasm socket(websocket) in standalone mode?

THANKS~

[sbc100]

IIRC the invoke_XXX functions are used for longjmp and exception handling. Can you try rebuilding with -sSUPPORT_LONGJMP=0.

I wouldn't hold out much hope of running this kind of code in wasmtime though, there are going to be a lot of emscripten-specific imports I imagine.

[moondayyy]

epoll is a linux-specific syscall which is not currently supported by emscripten. Implementing might be possible, but I imagine would be quite a lot of work.
I imagine that grpc supports other mechanisms and that you can configure to avoid epoll when building for emscripten?

Thank you so much @sbc100 . I just undefined the epoll macro in grpc source code like #define GRPC_LINUX_EPOLL 1 and compiled successfully.

But here is another error about socket(websocket). I run the server.wasm (standalone wasm module) which is a simple grpc server demo in wasmtime, the compile arguments are:

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -sSTANDALONE_WASM -pthread -lwebsocket.js -sPROXY_POSIX_SOCKETS -sUSE_PTHREADS -sPROXY_TO_PTHREAD -Wl,--error-limit=0 -Wl,--no-check-features")

(from https://emscripten.org/docs/porting/networking.html#full-posix-sockets-over-websocket-proxy-server)

and run with wasmtime --wasm-features threads server.wasm, which occurs

root@pekphis292586:/home/xxx/code_test/grpc_test_wasm# wasmtime --wasm-features threads server.wasm
Error: failed to run main module `server.wasm`

Caused by:
    0: failed to instantiate "server.wasm"
    1: unknown import: `env::invoke_iiiii` has not been defined

I don't know if it is related with -lwebsocket.js, and is it possible that run wasm socket(websocket) in standalone mode?

THANKS~

How to fixed this error "fatal error: 'sys/eventfd.h' file not found"

[sbc100]

eventfd.h is not supported by emscirpten. You would need to somehow disable or modify this code to make ti work under emscripten. eventfd.h is very much linux specific and won't work for other targets (such as emscripten).