From 22b1d89e64940aca83db1ace44cfddcbde2c2f41 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 9 Mar 2023 15:06:30 -0800 Subject: [PATCH 001/165] ci: document and script the release process (#295) Previously, the release process for wasi-sdk was undocumented and manual. The `RELEASING.md` document in this commit describes the steps necessary to publish a new version of `wasi-sdk` as a GitHub release and provides helpful scripts to automate some of the steps. With this in place, a future change could add a workflow trigger that allows running the steps automatically in GitHub actions. Keeping the steps as scripts in the repository, however, allows developers to re-run steps themselves in the (likely) case that something goes awry. --- README.md | 19 ++++-- RELEASING.md | 48 ++++++++++++++ ci/download-workflow-artifacts.sh | 102 ++++++++++++++++++++++++++++++ ci/draft-release.sh | 71 +++++++++++++++++++++ ci/get-workflows-for-tag.sh | 43 +++++++++++++ 5 files changed, 276 insertions(+), 7 deletions(-) create mode 100644 RELEASING.md create mode 100755 ci/download-workflow-artifacts.sh create mode 100755 ci/draft-release.sh create mode 100755 ci/get-workflows-for-tag.sh diff --git a/README.md b/README.md index 6e6ecbfea..e9ab245c8 100644 --- a/README.md +++ b/README.md @@ -38,22 +38,24 @@ The Wasm-sdk's build process needs some packages : * `clang` * `ninja` -Please refer to your OS documentation to install those packages. +Please refer to your OS documentation to install those packages. -## Build +## Build -To build the full package +To build the full package: ```shell script cd wasi-sdk NINJA_FLAGS=-v make package ``` -The built package can be found into `dist` directory. +The built package can be found into `dist` directory. For releasing a new +version of the package on GitHub, see [RELEASING.md](RELEASING.md). ## Install A typical installation from the release binaries might look like the following: + ```shell script export WASI_VERSION=14 export WASI_VERSION_FULL=${WASI_VERSION}.0 @@ -64,13 +66,16 @@ tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz ## Use Use the clang installed in the wasi-sdk directory: + ```shell script export WASI_SDK_PATH=`pwd`/wasi-sdk-${WASI_VERSION_FULL} CC="${WASI_SDK_PATH}/bin/clang --sysroot=${WASI_SDK_PATH}/share/wasi-sysroot" $CC foo.c -o foo.wasm ``` -Note: `${WASI_SDK_PATH}/share/wasi-sysroot` contains the WASI-specific includes/libraries/etc. The `--sysroot=...` option -is not necessary if `WASI_SDK_PATH` is `/opt/wasi-sdk`. + +Note: `${WASI_SDK_PATH}/share/wasi-sysroot` contains the WASI-specific +includes/libraries/etc. The `--sysroot=...` option is not necessary if +`WASI_SDK_PATH` is `/opt/wasi-sdk`. ## Notes for Autoconf @@ -104,7 +109,7 @@ disabled in a configure step before building with wasi-sdk. This repository does not yet support C++ exceptions. C++ code is supported only with -fno-exceptions for now. Similarly, there is not -yet support for setjmp/longjmp. Work on support for [exception handling] +yet support for setjmp/longjmp. Work on support for [exception handling] is underway at the language level which will support both of these features. diff --git a/RELEASING.md b/RELEASING.md new file mode 100644 index 000000000..9f8dcf794 --- /dev/null +++ b/RELEASING.md @@ -0,0 +1,48 @@ +# Release Process + +To publish a new version of `wasi-sdk` as a GitHub release: + +1. Tag a commit with an annotated tag. Note that this must be an annotated tag, + not a lightweight tag, so that `version.sh` can use it for calculating the + package version (use `git show wasi-sdk-...` to show other tag messages). + + ```shell script + TAG=wasi-sdk-1 + git tag -a $TAG + git push origin $TAG + ``` + +2. Find a successful workflow that CI has run for the tag. That successful + workflow run will have build artifacts that need to be attached to the + release. One could search around in the GitHub [actions], but the following + script will list completed workflows for a tag (get a token [here][tokens]): + + ```shell script + ci/get-workflows-for-tag.sh $TAG $GITHUB_TOKEN + ``` + + [actions]: https://github.com/WebAssembly/wasi-sdk/actions + [tokens]: https://github.com/settings/tokens + +3. Download and unzip the workflow artifacts. Note that artifacts with `+m` or + `.m` suffixes indicate that the Git tree was modified. Expect some duplicates + since some of the same artifacts are built on multiple CI runners (e.g., + Windows, MacOS, Linux). The following script does all of this automatically: + + ```shell script + ci/download-workflow-artifacts.sh $TAG $WORKFLOW_RUN_ID $GITHUB_TOKEN + ``` + +4. Draft a new release. This could be done [manually][releases] but the + following script simplifies the uploading of all the files and auto-generates + the release description: + + ```shell script + ci/draft-release.sh $TAG $ARTIFACTS_DIR $GITHUB_TOKEN + ``` + + [releases]: https://github.com/WebAssembly/wasi-sdk/releases + +5. Publish the release; the previous step only creates a draft. Follow the link + in the previous step or navigate to the GitHub [releases] to review the + description, commit, tag, and assets before clicking "Publish" diff --git a/ci/download-workflow-artifacts.sh b/ci/download-workflow-artifacts.sh new file mode 100755 index 000000000..3fccd587e --- /dev/null +++ b/ci/download-workflow-artifacts.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +set -e + +# This script downloads and unzips the artifacts produced in a workflow run. It +# also checks that the workflow commit corresponds to the tag commit that these +# artifacts will be released under. The script has several pre-requisites: +# - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) +# - an already-created tag in the repository (this marks the code to release) +# - the ID of a workflow run that has run successfully--this is where we +# retrieve the artifacts from +# - a GitHub access token, see https://github.com/settings/tokens +# +# Usage: download-workflow-artifacts.sh + +TAG=$1 +WORKFLOW_RUN_ID=$2 +GITHUB_TOKEN=$3 +GITHUB_API_VERSION=2022-11-28 +GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk +TMP_DIR=$(mktemp -d -t wasi-sdk-artifacts.XXXXXXX) + +if [ -z "${TAG}" ] || [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then + >&2 echo "Missing parameter; exiting..." + >&2 echo "Usage: download-worfklow-artifacts.sh " + exit 1 +fi + +# Get the commit SHA for the passed tag. +# See https://docs.github.com/en/rest/commits/commits#get-a-commit +MATCHING_COMMIT=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/commits/${TAG}") +COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') +>&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" + +# Check that the commit of the workflow run matches the tag commit and that the +# workflow was successful. +# See https://docs.github.com/en/rest/actions/workflow-runs#get-a-workflow-run +WORKFLOW_RUN=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/actions/runs/${WORKFLOW_RUN_ID}") +WORKFLOW_COMMIT=$(echo $WORKFLOW_RUN | jq -r '.head_sha') +WORKFLOW_STATUS=$(echo $WORKFLOW_RUN | jq -r '.status') +>&2 echo "===== Found commit for workflow ${WORKFLOW_RUN_ID}: ${WORKFLOW_COMMIT} =====" +if [ "${COMMIT}" != "${WORKFLOW_COMMIT}" ]; then + >&2 echo "Commit at tag ${TAG} did not match the commit for workflow ${WORKFLOW_RUN_ID}, exiting...:" + >&2 echo " ${COMMIT} != ${WORKFLOW_COMMIT}" + exit 1 +fi +if [ "${WORKFLOW_STATUS}" != "completed" ]; then + >&2 echo "Workflow ${WORKFLOW_RUN_ID} did not end successfully, exiting...:" + >&2 echo " status = ${WORKFLOW_STATUS}" + exit 1 +fi + +# List out the artifacts in the given workflow run. +# See https://docs.github.com/en/rest/actions/artifacts#list-workflow-run-artifacts +ARTIFACTS=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/actions/runs/${WORKFLOW_RUN_ID}/artifacts" \ + | jq -r '.artifacts[] | [(.id|tostring), .name, .archive_download_url] | join(",")') + +for A in $ARTIFACTS; do + ID=$(echo $A | cut -d ',' -f 1) + NAME=$(echo $A | cut -d ',' -f 2) + URL=$(echo $A | cut -d ',' -f 3) + TO=$TMP_DIR/$NAME.zip + >&2 echo "===== Downloading: ${TO} =====" + + # Download the artifacts to the temporary directory. + # See https://docs.github.com/en/rest/actions/artifacts#download-an-artifact + curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + --location --output "${TO}" \ + "${GITHUB_API_URL}/actions/artifacts/${ID}/zip" +done + +# Unzip the workflow artifacts into a `release` directory. +pushd $TMP_DIR > /dev/null +mkdir release +ls -1 *.zip | xargs -n1 unzip -q -o -d release +# Some explanation: +# -1 prints each file on a separate line +# -n1 runs the command once for each item +# -q means quietly +# -o allows unzip to overwrite existing files (e.g., multiple copies of `libclang_rt.builtins-wasm32-wasi-...`) +# -d tells unzip which directory to place things in +>&2 echo "===== Files to release: ${TMP_DIR}/release =====" +>&2 ls -1 release +popd > /dev/null + +>&2 echo +>&2 echo "Ensure the above artifacts look correct, then run \`draft-release.sh\` with the following directory:" +echo "${TMP_DIR}/release" diff --git a/ci/draft-release.sh b/ci/draft-release.sh new file mode 100755 index 000000000..207da9796 --- /dev/null +++ b/ci/draft-release.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +set -e + +# This script creates a draft pre-release with the artifacts produced in a +# workflow run (see `download-workflow-artifacts.sh`). Note that the pre-release +# is not published publicly--this is kept as a manual step as a safeguard. The +# script has several pre-requisites: +# - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) +# - an already-created tag in the repository (this marks the code to release) +# - a directory containing the artifacts to attach to the release. +# - a GitHub access token, see https://github.com/settings/tokens +# +# Usage: draft-release.sh + +TAG=$1 +ARTIFACTS_DIR=$2 +GITHUB_TOKEN=$3 +GITHUB_API_VERSION=2022-11-28 +GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk +TMP_DIR=$(mktemp -d -t release.sh.XXXXXXX) + +if [ -z "${TAG}" ] || [ -z "${ARTIFACTS_DIR}" ] || [ -z "${GITHUB_TOKEN}" ]; then + >&2 echo "Missing parameter; exiting..." + >&2 echo "Usage: draft-release.sh " + exit 1 +fi + +# Get the commit SHA for the passed tag. +# See https://docs.github.com/en/rest/commits/commits#get-a-commit +MATCHING_COMMIT=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/commits/${TAG}") +COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') +>&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" + +# Create a draft pre-release for this commit. +# See https://docs.github.com/en/rest/releases/releases#create-a-release +RELEASE_JSON=$(curl \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}"\ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/releases" \ + -d '{"tag_name":"'${TAG}'","target_commitish":"'${COMMIT}'","name":"'${TAG}'","draft":true,"prerelease":true,"generate_release_notes":true}') +UPLOAD_URL=$(echo $RELEASE_JSON | jq -r '.upload_url') +# Remove the "helpful" but invalid URL suffix that GitHub adds: +UPLOAD_URL=${UPLOAD_URL/\{?name,label\}} +HTML_URL=$(echo $RELEASE_JSON | jq -r '.html_url') +>&2 echo "===== Created draft release: ${HTML_URL} =====" + +# Upload the unzipped artifact files to the release. +# See https://docs.github.com/en/rest/releases/assets#upload-a-release-asset +for FILE in $(ls "${ARTIFACTS_DIR}"); do + FROM=$ARTIFACTS_DIR/$FILE + >&2 echo "===== Uploading: ${FROM} =====" + UPLOADED=$(curl \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}"\ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + -H "Content-Type: application/octet-stream" \ + "${UPLOAD_URL}?name=${FILE}" \ + --data-binary "@${FROM}") +done + +>&2 echo +>&2 echo "===== Completed =====" +>&2 echo "This created a draft release, do not forget to manually publish it at:" +echo "${HTML_URL}" diff --git a/ci/get-workflows-for-tag.sh b/ci/get-workflows-for-tag.sh new file mode 100755 index 000000000..1b117cb72 --- /dev/null +++ b/ci/get-workflows-for-tag.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -e + +# This script retrieves a list of GitHub workflow runs that have successfully completed for a given +# Git tag. The list is printed to stdout (all other output to stderr). It has several +# pre-requisites: +# - some standard Bash tools (curl) and one slightly more rare one (jq) +# - an already-created tag in the repository (this marks the code to release) +# - a GitHub access token, see https://github.com/settings/tokens +# +# Usage: get-workflows-for-tag.sh + +TAG=$1 +GITHUB_TOKEN=$2 +GITHUB_API_VERSION=2022-11-28 +GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk + +if [ -z "${TAG}" ] || [ -z "${GITHUB_TOKEN}" ]; then + >&2 echo "Missing parameter; exiting..." + >&2 echo "Usage: get-workflows-for-tag.sh " + exit 1 +fi + +# Get the commit SHA for the passed tag. +# See https://docs.github.com/en/rest/commits/commits#get-a-commit +MATCHING_COMMIT=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/commits/${TAG}") +COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') +>&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" + +# Find the workflow runs matching the tag commit. +# See https://docs.github.com/en/rest/actions/workflow-runs#list-workflow-runs-for-a-repository +MATCHING_WORKFLOWS=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/actions/runs?head_sha=${COMMIT}&status=success") +WORKFLOW_RUN_IDS=$(echo $MATCHING_WORKFLOWS | jq -r '.workflow_runs[].id') +>&2 echo "===== Matching workflow run IDs: =====" +echo "$WORKFLOW_RUN_IDS" From a433a25ca701783c2304e70aab425a0ace051105 Mon Sep 17 00:00:00 2001 From: Johannes Lundberg Date: Mon, 13 Mar 2023 08:55:08 -0700 Subject: [PATCH 002/165] Get arch from system when building debian package (#298) --- deb_from_installation.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/deb_from_installation.sh b/deb_from_installation.sh index 053de421c..2c1596163 100755 --- a/deb_from_installation.sh +++ b/deb_from_installation.sh @@ -32,10 +32,12 @@ if [ ! -d $INSTALL_DIR ] ; then exit 1 fi +ARCH=$(dpkg --print-architecture) + rm -rf build/pkg mkdir -p build/pkg/opt mkdir -p build/pkg/DEBIAN sed -e s/VERSION/$VERSION/ wasi-sdk.control > build/pkg/DEBIAN/control cp -R $INSTALL_DIR build/pkg/opt/ -cd build && dpkg-deb -b pkg wasi-sdk_$VERSION\_amd64.deb && cd .. -mv build/wasi-sdk_$VERSION\_amd64.deb $OUTDIR/ +cd build && dpkg-deb -b pkg wasi-sdk_$VERSION\_$ARCH\.deb && cd .. +mv build/wasi-sdk_$VERSION\_$ARCH\.deb $OUTDIR/ From d21c04076890f83ed37c4c6321348a7ecfd6b1e0 Mon Sep 17 00:00:00 2001 From: Catherine Date: Thu, 16 Mar 2023 22:15:09 +0000 Subject: [PATCH 003/165] Build libcxxabi and libcxx with threads enabled (#301) --- Makefile | 21 ++++++++++++++++----- src/wasi-libc | 2 +- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 761ccbb8e..5d078adf0 100644 --- a/Makefile +++ b/Makefile @@ -153,8 +153,8 @@ LIBCXX_CMAKE_FLAGS = \ -DLLVM_CONFIG_PATH=$(ROOT_DIR)/build/llvm/bin/llvm-config \ -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ -DCXX_SUPPORTS_CXX11=ON \ - -DLIBCXX_ENABLE_THREADS:BOOL=OFF \ - -DLIBCXX_HAS_PTHREAD_API:BOOL=OFF \ + -DLIBCXX_ENABLE_THREADS:BOOL=@PTHREAD@ \ + -DLIBCXX_HAS_PTHREAD_API:BOOL=@PTHREAD@ \ -DLIBCXX_HAS_EXTERNAL_THREAD_API:BOOL=OFF \ -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF \ @@ -171,8 +171,8 @@ LIBCXX_CMAKE_FLAGS = \ -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF \ -DLIBCXXABI_ENABLE_SHARED:BOOL=OFF \ -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON \ - -DLIBCXXABI_ENABLE_THREADS:BOOL=OFF \ - -DLIBCXXABI_HAS_PTHREAD_API:BOOL=OFF \ + -DLIBCXXABI_ENABLE_THREADS:BOOL=@PTHREAD@ \ + -DLIBCXXABI_HAS_PTHREAD_API:BOOL=@PTHREAD@ \ -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF \ -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF \ @@ -184,7 +184,7 @@ LIBCXX_CMAKE_FLAGS = \ build/libcxx.BUILT: build/llvm.BUILT build/compiler-rt.BUILT build/wasi-libc.BUILT # Do the build. mkdir -p build/libcxx - cd build/libcxx && cmake -G Ninja $(LIBCXX_CMAKE_FLAGS) \ + cd build/libcxx && cmake -G Ninja $(LIBCXX_CMAKE_FLAGS:@PTHREAD@=OFF) \ -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP)" \ -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP)" \ @@ -193,8 +193,19 @@ build/libcxx.BUILT: build/llvm.BUILT build/compiler-rt.BUILT build/wasi-libc.BUI -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ $(LLVM_PROJ_DIR)/runtimes ninja $(NINJA_FLAGS) -C build/libcxx + mkdir -p build/libcxx-threads + cd build/libcxx-threads && cmake -G Ninja $(LIBCXX_CMAKE_FLAGS:@PTHREAD@=ON) \ + -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ + -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) -pthread" \ + -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) -pthread" \ + -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-threads \ + -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-threads \ + -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ + $(LLVM_PROJ_DIR)/runtimes + ninja $(NINJA_FLAGS) -C build/libcxx-threads # Do the install. DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx install + DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx-threads install touch build/libcxx.BUILT build/config.BUILT: diff --git a/src/wasi-libc b/src/wasi-libc index b4814997f..a29c349a9 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit b4814997f61ee352d8c1ae397561a813fb30c701 +Subproject commit a29c349a9868fc8dc2b0a6dd5ff6694bf8b7297c From 128245614306970dca5614291d71737e8a8aa2f9 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Sat, 18 Mar 2023 02:09:35 +0900 Subject: [PATCH 004/165] download-workflow-artifacts.sh: skip dist-ubuntu-latest (#307) Exclude dist-ubuntu-latest to prefer dist-ubuntu-bionic, which is compatible with wider distributions. cf. https://github.com/WebAssembly/wasi-sdk/pull/273#issuecomment-1373879491 https://github.com/WebAssembly/wasi-sdk/issues/303 --- ci/download-workflow-artifacts.sh | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/ci/download-workflow-artifacts.sh b/ci/download-workflow-artifacts.sh index 3fccd587e..a4849cca0 100755 --- a/ci/download-workflow-artifacts.sh +++ b/ci/download-workflow-artifacts.sh @@ -71,6 +71,14 @@ for A in $ARTIFACTS; do NAME=$(echo $A | cut -d ',' -f 2) URL=$(echo $A | cut -d ',' -f 3) TO=$TMP_DIR/$NAME.zip + # Exclude dist-ubuntu-latest to prefer dist-ubuntu-bionic, which is + # compatible with wider distributions. + # cf. + # https://github.com/WebAssembly/wasi-sdk/pull/273#issuecomment-1373879491 + # https://github.com/WebAssembly/wasi-sdk/issues/303 + if [ "${NAME}" = "dist-ubuntu-latest" ]; then + continue + fi >&2 echo "===== Downloading: ${TO} =====" # Download the artifacts to the temporary directory. From 4293ded25742b90e7ab88a140a5dda212de436b2 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Mon, 20 Mar 2023 02:33:05 -0700 Subject: [PATCH 005/165] Downgrade OSX deployment target to 10.12 (#308) This change switches the MacOS build to target 10.12, an older version than currently supported. This would close #127. @sbc100 mentions that this is the setting used by Emscripten and related projects. --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 5d078adf0..bf16fa35b 100644 --- a/Makefile +++ b/Makefile @@ -62,7 +62,7 @@ build/llvm.BUILT: -DLLVM_STATIC_LINK_CXX_STDLIB=ON \ -DLLVM_HAVE_LIBXAR=OFF \ -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ - -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0 \ + -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 \ -DCMAKE_INSTALL_PREFIX=$(PREFIX) \ -DLLVM_TARGETS_TO_BUILD=WebAssembly \ -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi \ From e2666e548b4aee11a1c4df967611160347e63b22 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Wed, 22 Mar 2023 11:00:47 -0700 Subject: [PATCH 006/165] wasi-libc: upgrade to latest `main` branch (#313) This change brings in several helpful PRs (e.g., WebAssembly/wasi-libc#397, WebAssembly/wasi-libc#399, WebAssembly/wasi-libc#401) in anticipation of creating a release based on LLVM 16. --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index a29c349a9..f2a35a454 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit a29c349a9868fc8dc2b0a6dd5ff6694bf8b7297c +Subproject commit f2a35a454e8472b63831885daacc7f5fadd46747 From 46fe12f3a12e2334e39a6f104793cec37accd299 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 23 Mar 2023 16:52:45 +0100 Subject: [PATCH 007/165] llvm: update to LLVM 16.0.0 (#314) * llvm-project: update to 16.0.0 release This changes updates the `src/llvm-project` submodule to the `HEAD` of `release/16.x`, the same commit used to [release] the LLVM 16.0.0 binaries. [release]: https://github.com/llvm/llvm-project/releases/tag/llvmorg-16.0.0 * fix: use only Clang's major version in install prefix Due to [a change] in LLVM, Clang will expect to find the `libclang_rt.builtins-wasm32.a` file in a path that only contains the major version (`16`) instead of the entire version (`16.0.0`) as was previously the case. This change modifies the `CMAKE_INSTALL_PREFIX` to use Clang's major version only. [a change]: https://reviews.llvm.org/D125860 * review: only use `llvm_version_major.sh` Since the `Makefile` can get by with only knowing Clang's major version, this change removes `llvm_version.sh` and sets `CLANG_VERSION` to use only the major part. --- Makefile | 7 ++++--- llvm_version.sh | 6 ------ llvm_version_major.sh | 4 ++++ src/llvm-project | 2 +- 4 files changed, 9 insertions(+), 10 deletions(-) delete mode 100755 llvm_version.sh create mode 100755 llvm_version_major.sh diff --git a/Makefile b/Makefile index bf16fa35b..d2ccff68a 100644 --- a/Makefile +++ b/Makefile @@ -37,7 +37,8 @@ BASH= endif -CLANG_VERSION=$(shell $(BASH) ./llvm_version.sh $(LLVM_PROJ_DIR)) +# Only the major version is needed for Clang, see https://reviews.llvm.org/D125860. +CLANG_VERSION=$(shell $(BASH) ./llvm_version_major.sh $(LLVM_PROJ_DIR)) VERSION:=$(shell $(BASH) ./version.sh) DEBUG_PREFIX_MAP=-fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION) @@ -67,8 +68,8 @@ build/llvm.BUILT: -DLLVM_TARGETS_TO_BUILD=WebAssembly \ -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi \ -DLLVM_ENABLE_PROJECTS="lld;clang;clang-tools-extra" \ - $(if $(patsubst 9.%,,$(CLANG_VERSION)), \ - $(if $(patsubst 10.%,,$(CLANG_VERSION)), \ + $(if $(patsubst 9,,$(CLANG_VERSION)), \ + $(if $(patsubst 10,,$(CLANG_VERSION)), \ -DDEFAULT_SYSROOT=../share/wasi-sysroot, \ -DDEFAULT_SYSROOT=$(PREFIX)/share/wasi-sysroot), \ -DDEFAULT_SYSROOT=$(PREFIX)/share/wasi-sysroot) \ diff --git a/llvm_version.sh b/llvm_version.sh deleted file mode 100755 index 41434eb79..000000000 --- a/llvm_version.sh +++ /dev/null @@ -1,6 +0,0 @@ -#/bin/bash -LLVM_PROJ_DIR=${1:-./src/llvm-project} -MAJOR=`grep "set(LLVM_VERSION_MAJOR" $LLVM_PROJ_DIR/llvm/CMakeLists.txt | awk '{print substr($2, 1, length($2) - 1)}'` -MINOR=`grep "set(LLVM_VERSION_MINOR" $LLVM_PROJ_DIR/llvm/CMakeLists.txt | awk '{print substr($2, 1, length($2) - 1)}'` -PATCH=`grep "set(LLVM_VERSION_PATCH" $LLVM_PROJ_DIR/llvm/CMakeLists.txt | awk '{print substr($2, 1, length($2) - 1)}'` -echo $MAJOR.$MINOR.$PATCH diff --git a/llvm_version_major.sh b/llvm_version_major.sh new file mode 100755 index 000000000..2e5d973ca --- /dev/null +++ b/llvm_version_major.sh @@ -0,0 +1,4 @@ +#/bin/bash +LLVM_PROJ_DIR=${1:-./src/llvm-project} +MAJOR=`grep "set(LLVM_VERSION_MAJOR" $LLVM_PROJ_DIR/llvm/CMakeLists.txt | awk '{print substr($2, 1, length($2) - 1)}'` +echo $MAJOR diff --git a/src/llvm-project b/src/llvm-project index 8dfdcc7b7..08d094a0e 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a +Subproject commit 08d094a0e457360ad8b94b017d2dc277e697ca76 From 314eeacb186dff8540eeac6f9a6172da01d16ad0 Mon Sep 17 00:00:00 2001 From: Jo Shields Date: Thu, 23 Mar 2023 15:04:42 -0400 Subject: [PATCH 008/165] Force clang as CMake Assembly compiler, with wasm32-wasi triple (#315) Sometimes a C codebase might have the odd file written in Assembly. We should force the use of Clang as the assembly compiler for these cases (which it is perfectly capable of doing), and make sure we pass on the WASI compiler triple the same way we do for C and C++. --- wasi-sdk-pthread.cmake | 2 ++ wasi-sdk.cmake | 2 ++ 2 files changed, 4 insertions(+) diff --git a/wasi-sdk-pthread.cmake b/wasi-sdk-pthread.cmake index e82622f97..c7f665215 100644 --- a/wasi-sdk-pthread.cmake +++ b/wasi-sdk-pthread.cmake @@ -18,10 +18,12 @@ endif() set(CMAKE_C_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) set(CMAKE_CXX_COMPILER ${WASI_SDK_PREFIX}/bin/clang++${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_ASM_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) set(CMAKE_AR ${WASI_SDK_PREFIX}/bin/llvm-ar${WASI_HOST_EXE_SUFFIX}) set(CMAKE_RANLIB ${WASI_SDK_PREFIX}/bin/llvm-ranlib${WASI_HOST_EXE_SUFFIX}) set(CMAKE_C_COMPILER_TARGET ${triple}) set(CMAKE_CXX_COMPILER_TARGET ${triple}) +set(CMAKE_ASM_COMPILER_TARGET ${triple}) # Don't look in the sysroot for executables to run during the build set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) diff --git a/wasi-sdk.cmake b/wasi-sdk.cmake index 3245a4c29..83ee9b0ac 100644 --- a/wasi-sdk.cmake +++ b/wasi-sdk.cmake @@ -16,10 +16,12 @@ endif() set(CMAKE_C_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) set(CMAKE_CXX_COMPILER ${WASI_SDK_PREFIX}/bin/clang++${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_ASM_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) set(CMAKE_AR ${WASI_SDK_PREFIX}/bin/llvm-ar${WASI_HOST_EXE_SUFFIX}) set(CMAKE_RANLIB ${WASI_SDK_PREFIX}/bin/llvm-ranlib${WASI_HOST_EXE_SUFFIX}) set(CMAKE_C_COMPILER_TARGET ${triple}) set(CMAKE_CXX_COMPILER_TARGET ${triple}) +set(CMAKE_ASM_COMPILER_TARGET ${triple}) # Don't look in the sysroot for executables to run during the build set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) From 15a18dc374f9dbabcaf078a4b6f7218c3387c5b6 Mon Sep 17 00:00:00 2001 From: Anuraag Agrawal Date: Sat, 25 Mar 2023 16:37:49 +0900 Subject: [PATCH 009/165] Update docker build to use LLVM 16 (#316) --- docker/Dockerfile | 9 +++------ docker/wasi-sdk.cmake | 9 +++++---- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index 12312c8d6..610de47bc 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -9,6 +9,7 @@ FROM ubuntu:22.04 as dist ADD dist/wasi-sdk-*.*-linux.tar.gz / +ADD dist/libclang_rt.builtins-wasm32-wasi-*.*.tar.gz /wasi-sysroot-clang_rt # Move versioned folder to unversioned to using bash glob to allow # this file to be independent of major version number. @@ -16,7 +17,7 @@ RUN mv /wasi-sdk-* /wasi-sdk FROM ubuntu:22.04 -ENV LLVM_VERSION 15 +ENV LLVM_VERSION 16 # Install build toolchain including clang, ld, make, autotools, ninja, and cmake RUN apt-get update && \ @@ -33,11 +34,7 @@ RUN apt-get update && \ rm -rf /var/lib/apt/lists/* COPY --from=dist /wasi-sdk/share/wasi-sysroot/ /wasi-sysroot/ -# The path to the rt directory contains the LLVM patch version which is not reflected in the LLVM apt repository -# or package. To make adding the RT robust to changing patch versions without needing to duplicate the folder -# content, we symlink after extracting using a bash glob to resolve the patch version -ADD dist/libclang_rt.builtins-wasm32-wasi-*.*.tar.gz /wasi-sysroot-clang_rt -RUN ln -s /wasi-sysroot-clang_rt/lib/wasi/ $(echo /usr/lib/llvm-${LLVM_VERSION}/lib/clang/${LLVM_VERSION}.*)/lib/wasi +COPY --from=dist /wasi-sysroot-clang_rt/lib/wasi /usr/lib/llvm-${LLVM_VERSION}/lib/clang/${LLVM_VERSION}/lib/wasi ADD docker/wasi-sdk.cmake /usr/share/cmake/wasi-sdk.cmake ENV CMAKE_TOOLCHAIN_FILE /usr/share/cmake/wasi-sdk.cmake diff --git a/docker/wasi-sdk.cmake b/docker/wasi-sdk.cmake index b87c94838..f2f3229a0 100644 --- a/docker/wasi-sdk.cmake +++ b/docker/wasi-sdk.cmake @@ -11,10 +11,11 @@ set(CMAKE_SYSTEM_VERSION 1) set(CMAKE_SYSTEM_PROCESSOR wasm32) set(triple wasm32-wasi) -set(CMAKE_C_COMPILER /usr/bin/clang-15) -set(CMAKE_CXX_COMPILER /usr/bin/clang++-15) -set(CMAKE_AR /usr/bin/llvm-ar-15) -set(CMAKE_RANLIB /usr/bin/llvm-ranlib-15) +set(CMAKE_C_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) +set(CMAKE_CXX_COMPILER /usr/bin/clang++-$ENV{LLVM_VERSION}) +set(CMAKE_ASM_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) +set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) +set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) set(CMAKE_C_COMPILER_TARGET ${triple}) set(CMAKE_CXX_COMPILER_TARGET ${triple}) SET(CMAKE_SYSROOT /wasi-sysroot) From 935fe1acd2fcd7ea4aed2d5ee4527482862b6344 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Tue, 28 Mar 2023 20:55:23 +0200 Subject: [PATCH 010/165] wasi-libc: update to include atomic barrier fix (#317) WebAssembly/wasi-libc#403 fixed an issue with `a_barrier` that should be included in the next release of wasi-sdk. This change updates wasi-libc to the latest `HEAD` of `main` to include it. --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index f2a35a454..1dfe5c302 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit f2a35a454e8472b63831885daacc7f5fadd46747 +Subproject commit 1dfe5c302d1c5ab621f7abf04620fae92700fd22 From ccb99d45be8840415a2b37a71c9f770964d649e0 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 6 Apr 2023 08:52:30 -0500 Subject: [PATCH 011/165] Pass through extra flags to libcxx build (#322) This commit passes through `EXTRA_CFLAGS` to the libcxx build, along with `EXTRA_CXXFLAGS`. This models after the pattern in `wasi-libc` for passing these flags to thread more flags through. --- Makefile | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index d2ccff68a..039c8d4bb 100644 --- a/Makefile +++ b/Makefile @@ -187,8 +187,8 @@ build/libcxx.BUILT: build/llvm.BUILT build/compiler-rt.BUILT build/wasi-libc.BUI mkdir -p build/libcxx cd build/libcxx && cmake -G Ninja $(LIBCXX_CMAKE_FLAGS:@PTHREAD@=OFF) \ -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP)" \ - -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP)" \ + -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS)" \ + -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS)" \ -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi \ -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi \ -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ @@ -197,8 +197,8 @@ build/libcxx.BUILT: build/llvm.BUILT build/compiler-rt.BUILT build/wasi-libc.BUI mkdir -p build/libcxx-threads cd build/libcxx-threads && cmake -G Ninja $(LIBCXX_CMAKE_FLAGS:@PTHREAD@=ON) \ -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) -pthread" \ - -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) -pthread" \ + -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CFLAGS)" \ + -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CXXFLAGS)" \ -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-threads \ -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-threads \ -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ From 4f678be2b757e6a4f9c8ab945de4568410941662 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Wed, 29 Mar 2023 19:13:54 +0200 Subject: [PATCH 012/165] ci: split out release check into `is-workflow-valid.sh` This change refactors the script logic that checks if a workflow in fact built the commit matching our release tag out into a separate script. This is mainly an improvement in clarity. --- RELEASING.md | 13 +++++-- ci/download-workflow-artifacts.sh | 56 ++++++----------------------- ci/is-workflow-valid.sh | 58 +++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+), 49 deletions(-) create mode 100755 ci/is-workflow-valid.sh diff --git a/RELEASING.md b/RELEASING.md index 9f8dcf794..80f7b6e6b 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -24,7 +24,14 @@ To publish a new version of `wasi-sdk` as a GitHub release: [actions]: https://github.com/WebAssembly/wasi-sdk/actions [tokens]: https://github.com/settings/tokens -3. Download and unzip the workflow artifacts. Note that artifacts with `+m` or +3. Check that the workflow built the artifacts for the given tag and that the + workflow completed successfully: + + ```shell script + ci/is-worfklow-valid.sh $TAG $WORKFLOW_RUN_ID $GITHUB_TOKEN + ``` + +4. Download and unzip the workflow artifacts. Note that artifacts with `+m` or `.m` suffixes indicate that the Git tree was modified. Expect some duplicates since some of the same artifacts are built on multiple CI runners (e.g., Windows, MacOS, Linux). The following script does all of this automatically: @@ -33,7 +40,7 @@ To publish a new version of `wasi-sdk` as a GitHub release: ci/download-workflow-artifacts.sh $TAG $WORKFLOW_RUN_ID $GITHUB_TOKEN ``` -4. Draft a new release. This could be done [manually][releases] but the +5. Draft a new release. This could be done [manually][releases] but the following script simplifies the uploading of all the files and auto-generates the release description: @@ -43,6 +50,6 @@ To publish a new version of `wasi-sdk` as a GitHub release: [releases]: https://github.com/WebAssembly/wasi-sdk/releases -5. Publish the release; the previous step only creates a draft. Follow the link +6. Publish the release; the previous step only creates a draft. Follow the link in the previous step or navigate to the GitHub [releases] to review the description, commit, tag, and assets before clicking "Publish" diff --git a/ci/download-workflow-artifacts.sh b/ci/download-workflow-artifacts.sh index a4849cca0..5b53868f6 100755 --- a/ci/download-workflow-artifacts.sh +++ b/ci/download-workflow-artifacts.sh @@ -1,59 +1,24 @@ #!/usr/bin/env bash set -e -# This script downloads and unzips the artifacts produced in a workflow run. It -# also checks that the workflow commit corresponds to the tag commit that these -# artifacts will be released under. The script has several pre-requisites: +# This script downloads and unzips the artifacts produced in a workflow run. The +# script has several pre-requisites: # - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) -# - an already-created tag in the repository (this marks the code to release) # - the ID of a workflow run that has run successfully--this is where we # retrieve the artifacts from # - a GitHub access token, see https://github.com/settings/tokens # -# Usage: download-workflow-artifacts.sh +# Usage: download-workflow-artifacts.sh -TAG=$1 -WORKFLOW_RUN_ID=$2 -GITHUB_TOKEN=$3 +WORKFLOW_RUN_ID=$1 +GITHUB_TOKEN=$2 GITHUB_API_VERSION=2022-11-28 GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk TMP_DIR=$(mktemp -d -t wasi-sdk-artifacts.XXXXXXX) -if [ -z "${TAG}" ] || [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then +if [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then >&2 echo "Missing parameter; exiting..." - >&2 echo "Usage: download-worfklow-artifacts.sh " - exit 1 -fi - -# Get the commit SHA for the passed tag. -# See https://docs.github.com/en/rest/commits/commits#get-a-commit -MATCHING_COMMIT=$(curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/commits/${TAG}") -COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') ->&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" - -# Check that the commit of the workflow run matches the tag commit and that the -# workflow was successful. -# See https://docs.github.com/en/rest/actions/workflow-runs#get-a-workflow-run -WORKFLOW_RUN=$(curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/actions/runs/${WORKFLOW_RUN_ID}") -WORKFLOW_COMMIT=$(echo $WORKFLOW_RUN | jq -r '.head_sha') -WORKFLOW_STATUS=$(echo $WORKFLOW_RUN | jq -r '.status') ->&2 echo "===== Found commit for workflow ${WORKFLOW_RUN_ID}: ${WORKFLOW_COMMIT} =====" -if [ "${COMMIT}" != "${WORKFLOW_COMMIT}" ]; then - >&2 echo "Commit at tag ${TAG} did not match the commit for workflow ${WORKFLOW_RUN_ID}, exiting...:" - >&2 echo " ${COMMIT} != ${WORKFLOW_COMMIT}" - exit 1 -fi -if [ "${WORKFLOW_STATUS}" != "completed" ]; then - >&2 echo "Workflow ${WORKFLOW_RUN_ID} did not end successfully, exiting...:" - >&2 echo " status = ${WORKFLOW_STATUS}" + >&2 echo "Usage: download-worfklow-artifacts.sh " exit 1 fi @@ -72,10 +37,9 @@ for A in $ARTIFACTS; do URL=$(echo $A | cut -d ',' -f 3) TO=$TMP_DIR/$NAME.zip # Exclude dist-ubuntu-latest to prefer dist-ubuntu-bionic, which is - # compatible with wider distributions. - # cf. - # https://github.com/WebAssembly/wasi-sdk/pull/273#issuecomment-1373879491 - # https://github.com/WebAssembly/wasi-sdk/issues/303 + # compatible with wider distributions. See: + # - https://github.com/WebAssembly/wasi-sdk/pull/273#issuecomment-1373879491 + # - https://github.com/WebAssembly/wasi-sdk/issues/303 if [ "${NAME}" = "dist-ubuntu-latest" ]; then continue fi diff --git a/ci/is-workflow-valid.sh b/ci/is-workflow-valid.sh new file mode 100755 index 000000000..b4097f899 --- /dev/null +++ b/ci/is-workflow-valid.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +set -e + +# This script checks 1) that the workflow commit corresponds to the commit for +# the given tag and 2) that the workflow has completed. This is a sanity check +# to ensure the artifacts we are about to publish are in fact built from the +# commit/tag we think. The script has several pre-requisites: +# - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) +# - an already-created tag in the repository (this marks the code to release) +# - the ID of a workflow run that has run successfully--this is where we +# retrieve the artifacts from +# - a GitHub access token, see https://github.com/settings/tokens +# +# Usage: is-workflow-valid.sh + +TAG=$1 +WORKFLOW_RUN_ID=$2 +GITHUB_TOKEN=$3 +GITHUB_API_VERSION=2022-11-28 +GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk + +if [ -z "${TAG}" ] || [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then + >&2 echo "Missing parameter; exiting..." + >&2 echo "Usage: is-workflow-valid.sh " + exit 1 +fi + +# Get the commit SHA for the passed tag. +# See https://docs.github.com/en/rest/commits/commits#get-a-commit +MATCHING_COMMIT=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/commits/${TAG}") +COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') +>&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" + +# Check that the commit of the workflow run matches the tag commit and that the +# workflow was successful. +# See https://docs.github.com/en/rest/actions/workflow-runs#get-a-workflow-run +WORKFLOW_RUN=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/actions/runs/${WORKFLOW_RUN_ID}") +WORKFLOW_COMMIT=$(echo $WORKFLOW_RUN | jq -r '.head_sha') +WORKFLOW_STATUS=$(echo $WORKFLOW_RUN | jq -r '.status') +>&2 echo "===== Found commit for workflow ${WORKFLOW_RUN_ID}: ${WORKFLOW_COMMIT} =====" +if [ "${COMMIT}" != "${WORKFLOW_COMMIT}" ]; then + >&2 echo "Commit at tag ${TAG} did not match the commit for workflow ${WORKFLOW_RUN_ID}, exiting...:" + >&2 echo " ${COMMIT} != ${WORKFLOW_COMMIT}" + exit 1 +fi +if [ "${WORKFLOW_STATUS}" != "completed" ]; then + >&2 echo "Workflow ${WORKFLOW_RUN_ID} did not end successfully, exiting...:" + >&2 echo " status = ${WORKFLOW_STATUS}" + exit 1 +fi From 2345285702961fcdac06f4a853ea50795a9fc5af Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Wed, 29 Mar 2023 19:20:21 +0200 Subject: [PATCH 013/165] ci: allow overriding script inputs from the environment When testing out these scripts on a fork, I realized that it was more convenient to use a single environment variable (e.g., `GITHUB_API_URL=... ci/draft-release.sh`) to change some of the script's input parameters. --- ci/download-workflow-artifacts.sh | 8 ++++---- ci/draft-release.sh | 10 +++++----- ci/get-workflows-for-tag.sh | 8 ++++---- ci/is-workflow-valid.sh | 10 +++++----- 4 files changed, 18 insertions(+), 18 deletions(-) diff --git a/ci/download-workflow-artifacts.sh b/ci/download-workflow-artifacts.sh index 5b53868f6..409b484e0 100755 --- a/ci/download-workflow-artifacts.sh +++ b/ci/download-workflow-artifacts.sh @@ -10,10 +10,10 @@ set -e # # Usage: download-workflow-artifacts.sh -WORKFLOW_RUN_ID=$1 -GITHUB_TOKEN=$2 -GITHUB_API_VERSION=2022-11-28 -GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk +WORKFLOW_RUN_ID=${WORKFLOW_RUN_ID:-$1} +GITHUB_TOKEN=${GITHUB_TOKEN:-$2} +GITHUB_API_VERSION=${GITHUB_API_VERSION:-2022-11-28} +GITHUB_API_URL=${GITHUB_API_URL:-https://api.github.com/repos/WebAssembly/wasi-sdk} TMP_DIR=$(mktemp -d -t wasi-sdk-artifacts.XXXXXXX) if [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then diff --git a/ci/draft-release.sh b/ci/draft-release.sh index 207da9796..905ecb9a7 100755 --- a/ci/draft-release.sh +++ b/ci/draft-release.sh @@ -12,11 +12,11 @@ set -e # # Usage: draft-release.sh -TAG=$1 -ARTIFACTS_DIR=$2 -GITHUB_TOKEN=$3 -GITHUB_API_VERSION=2022-11-28 -GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk +TAG=${TAG:-$1} +ARTIFACTS_DIR=${ARTIFACTS_DIR:-$2} +GITHUB_TOKEN=${GITHUB_TOKEN:-$3} +GITHUB_API_VERSION=${GITHUB_API_VERSION:-2022-11-28} +GITHUB_API_URL=${GITHUB_API_URL:-https://api.github.com/repos/WebAssembly/wasi-sdk} TMP_DIR=$(mktemp -d -t release.sh.XXXXXXX) if [ -z "${TAG}" ] || [ -z "${ARTIFACTS_DIR}" ] || [ -z "${GITHUB_TOKEN}" ]; then diff --git a/ci/get-workflows-for-tag.sh b/ci/get-workflows-for-tag.sh index 1b117cb72..879204184 100755 --- a/ci/get-workflows-for-tag.sh +++ b/ci/get-workflows-for-tag.sh @@ -10,10 +10,10 @@ set -e # # Usage: get-workflows-for-tag.sh -TAG=$1 -GITHUB_TOKEN=$2 -GITHUB_API_VERSION=2022-11-28 -GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk +TAG=${TAG:-$1} +GITHUB_TOKEN=${GITHUB_TOKEN:-$2} +GITHUB_API_VERSION=${GITHUB_API_VERSION:-2022-11-28} +GITHUB_API_URL=${GITHUB_API_URL:-https://api.github.com/repos/WebAssembly/wasi-sdk} if [ -z "${TAG}" ] || [ -z "${GITHUB_TOKEN}" ]; then >&2 echo "Missing parameter; exiting..." diff --git a/ci/is-workflow-valid.sh b/ci/is-workflow-valid.sh index b4097f899..a7f0c80ab 100755 --- a/ci/is-workflow-valid.sh +++ b/ci/is-workflow-valid.sh @@ -13,11 +13,11 @@ set -e # # Usage: is-workflow-valid.sh -TAG=$1 -WORKFLOW_RUN_ID=$2 -GITHUB_TOKEN=$3 -GITHUB_API_VERSION=2022-11-28 -GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk +TAG=${TAG:-$1} +WORKFLOW_RUN_ID=${WORKFLOW_RUN_ID:-$2} +GITHUB_TOKEN=${GITHUB_TOKEN:-$3} +GITHUB_API_VERSION=${GITHUB_API_VERSION:-2022-11-28} +GITHUB_API_URL=${GITHUB_API_URL:-https://api.github.com/repos/WebAssembly/wasi-sdk} if [ -z "${TAG}" ] || [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then >&2 echo "Missing parameter; exiting..." From 95af0b8ec2c3c65d66e42cb17f18a9c0d10aeff5 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 30 Mar 2023 08:24:42 +0200 Subject: [PATCH 014/165] ci: add note about clearing the cache I found the artifacts created to have incorrect version numbers; clearing the Actions cache should resolve this. --- RELEASING.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/RELEASING.md b/RELEASING.md index 80f7b6e6b..8c65cc8e8 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -5,6 +5,8 @@ To publish a new version of `wasi-sdk` as a GitHub release: 1. Tag a commit with an annotated tag. Note that this must be an annotated tag, not a lightweight tag, so that `version.sh` can use it for calculating the package version (use `git show wasi-sdk-...` to show other tag messages). + Note that you may need to clear the repository cache to avoid problems with + cached artifacts [^cache]. ```shell script TAG=wasi-sdk-1 @@ -53,3 +55,11 @@ To publish a new version of `wasi-sdk` as a GitHub release: 6. Publish the release; the previous step only creates a draft. Follow the link in the previous step or navigate to the GitHub [releases] to review the description, commit, tag, and assets before clicking "Publish" + +[^cache]: Here is an example of how to clear a cache with the GitHub CLI: + + ```shell script + URL=/repos/WebAssembly/wasi-sdk/actions/caches + gh api $URL -q '.actions_caches[].id' \ + | xargs -I {} gh api --method DELETE $URL/{} + ``` From c891cd2036d5630e67826f70aec9c0ba117c67e9 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Fri, 7 Apr 2023 13:17:13 +0900 Subject: [PATCH 015/165] wasi-sdk-pthread.cmake: add --import-memory (#297) --- wasi-sdk-pthread.cmake | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/wasi-sdk-pthread.cmake b/wasi-sdk-pthread.cmake index c7f665215..0e5f6ff00 100644 --- a/wasi-sdk-pthread.cmake +++ b/wasi-sdk-pthread.cmake @@ -9,6 +9,11 @@ set(CMAKE_SYSTEM_PROCESSOR wasm32) set(triple wasm32-wasi-threads) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -pthread") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread") +# wasi-threads requires --import-memory. +# wasi requires --export-memory. +# (--export-memory is implicit unless --import-memory is given) +set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--import-memory") +set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--export-memory") if(WIN32) set(WASI_HOST_EXE_SUFFIX ".exe") From 6dde7bb8d639cafe94daf3f946cffeeec68b9f3f Mon Sep 17 00:00:00 2001 From: Cheng Shao Date: Mon, 8 May 2023 20:13:06 +0200 Subject: [PATCH 016/165] tests: use latest wasmtime for testing (#327) This patch enables using latest version of wasmtime for testing. This should also make it possible to running tests for wasm32-wasi-threads in the future. --- .github/workflows/main.yml | 2 +- tests/general/abort.c.stderr.expected | 4 ++-- tests/general/abort.c.stderr.expected.filter | 5 ++--- tests/general/assert-fail.c.stderr.expected | 4 ++-- tests/general/assert-fail.c.stderr.expected.filter | 5 ++--- 5 files changed, 9 insertions(+), 11 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 718cb423f..fb3bf377a 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -39,7 +39,7 @@ jobs: 0-cache-macos-latest if: matrix.os == 'macos-latest' - name: Install wasmtime for tests - run: curl -f -L --retry 5 https://wasmtime.dev/install.sh | bash -s -- --version v2.0.2 + run: curl -f -L --retry 5 https://wasmtime.dev/install.sh | bash -s -- --version v8.0.1 - uses: actions/checkout@v3 with: fetch-depth: 0 diff --git a/tests/general/abort.c.stderr.expected b/tests/general/abort.c.stderr.expected index bfba4982d..c6ecf6a93 100644 --- a/tests/general/abort.c.stderr.expected +++ b/tests/general/abort.c.stderr.expected @@ -2,5 +2,5 @@ Error: failed to run main module `abort.c.---.wasm` Caused by: 0: failed to invoke command default - 1: wasm trap: wasm `unreachable` instruction executed - wasm backtrace: + 1: error while executing at wasm backtrace: + 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/abort.c.stderr.expected.filter b/tests/general/abort.c.stderr.expected.filter index b9f9c6665..742503a5c 100755 --- a/tests/general/abort.c.stderr.expected.filter +++ b/tests/general/abort.c.stderr.expected.filter @@ -1,7 +1,6 @@ -#!/bin/bash +#!/usr/bin/env bash set -euo pipefail cat \ | sed -e 's/main module `abort\.c\.[^`]*\.wasm`/main module `abort.c.---.wasm`/' \ - | sed -e 's/source location: @[[:xdigit:]]*$/source location: @----/' \ - | head -n 6 + | sed -E '/0x[[:xdigit:]]+/d' diff --git a/tests/general/assert-fail.c.stderr.expected b/tests/general/assert-fail.c.stderr.expected index 806abf9f4..c179f09ad 100644 --- a/tests/general/assert-fail.c.stderr.expected +++ b/tests/general/assert-fail.c.stderr.expected @@ -3,5 +3,5 @@ Error: failed to run main module `assert-fail.c.---.wasm` Caused by: 0: failed to invoke command default - 1: wasm trap: wasm `unreachable` instruction executed - wasm backtrace: + 1: error while executing at wasm backtrace: + 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/assert-fail.c.stderr.expected.filter b/tests/general/assert-fail.c.stderr.expected.filter index df075b130..d2213dd6b 100755 --- a/tests/general/assert-fail.c.stderr.expected.filter +++ b/tests/general/assert-fail.c.stderr.expected.filter @@ -1,7 +1,6 @@ -#!/bin/bash +#!/usr/bin/env bash set -euo pipefail cat \ | sed -e 's/main module `assert-fail\.c\.[^`]*\.wasm`/main module `assert-fail.c.---.wasm`/' \ - | sed -e 's/source location: @[[:xdigit:]]*$/source location: @----/' \ - | head -n 7 + | sed -E '/0x[[:xdigit:]]+/d' From 3fb0057a6da0313d18dad241ef7a87aa43f1485c Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Thu, 11 May 2023 16:49:30 -0700 Subject: [PATCH 017/165] ci: build fewer LLVM components (#309) This is a recommendation from #304 to build less of LLVM in CI. Perhaps it can speed up build times. --- Makefile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Makefile b/Makefile index 039c8d4bb..86082894a 100644 --- a/Makefile +++ b/Makefile @@ -65,6 +65,10 @@ build/llvm.BUILT: -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 \ -DCMAKE_INSTALL_PREFIX=$(PREFIX) \ + -DLLVM_INCLUDE_TESTS=OFF \ + -DLLVM_INCLUDE_UTILS=OFF \ + -DLLVM_INCLUDE_BENCHMARKS=OFF \ + -DLLVM_INCLUDE_EXAMPLES=OFF \ -DLLVM_TARGETS_TO_BUILD=WebAssembly \ -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi \ -DLLVM_ENABLE_PROJECTS="lld;clang;clang-tools-extra" \ From 9a271da9378b407e7234dcbc26ae60fdd80b6d1a Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 9 Jun 2023 12:35:13 -0500 Subject: [PATCH 018/165] Fix build on non-x86_64 Linux (#321) It looks like some macOS-specific CMake options to LLVM end up forcing an x86_64 build of some blake3 intrinsics (or something like that) which causes the build to fail on AArch64 Linux, for example. I found, though, that when removing these options it was then possible to build a toolchain for AArch64 Linux. --- Makefile | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 86082894a..9376249e8 100644 --- a/Makefile +++ b/Makefile @@ -37,6 +37,11 @@ BASH= endif +ifeq ($(shell uname),Darwin) +LLVM_CMAKE_FLAGS += -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ + -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 +endif + # Only the major version is needed for Clang, see https://reviews.llvm.org/D125860. CLANG_VERSION=$(shell $(BASH) ./llvm_version_major.sh $(LLVM_PROJ_DIR)) VERSION:=$(shell $(BASH) ./version.sh) @@ -62,8 +67,6 @@ build/llvm.BUILT: -DLLVM_ENABLE_ZSTD=OFF \ -DLLVM_STATIC_LINK_CXX_STDLIB=ON \ -DLLVM_HAVE_LIBXAR=OFF \ - -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ - -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 \ -DCMAKE_INSTALL_PREFIX=$(PREFIX) \ -DLLVM_INCLUDE_TESTS=OFF \ -DLLVM_INCLUDE_UTILS=OFF \ From 00cb3b2e5ffc9c747311f5d22cc9cf49f3be12e5 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Fri, 23 Jun 2023 14:45:31 -0700 Subject: [PATCH 019/165] Improve `README.md` (#331) This changes the front-page documentation to: - use out-of-line links in more places - mention the need for `libclang_rt.builtins-wasm32.a` when using standard Clang - mention how to use a `wasi-libc` sysroot - explain the experimental status of `wasm32-wasi-threads --- README.md | 68 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 40 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index e9ab245c8..f00a95319 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,9 @@ ## Quick Start -[Download SDK packages here.](https://github.com/WebAssembly/wasi-sdk/releases) +[Download SDK packages here.][releases] + +[releases]: https://github.com/WebAssembly/wasi-sdk/releases ## About this repository @@ -10,8 +12,9 @@ This repository contains no compiler or library code itself; it uses git submodules to pull in the upstream Clang and LLVM tree, as well as the wasi-libc tree. -The libc portion of this SDK is the -[wasi-libc](https://github.com/WebAssembly/wasi-libc). +The libc portion of this SDK is maintained in [wasi-libc]. + +[wasi-libc]: https://github.com/WebAssembly/wasi-libc Upstream Clang and LLVM (from 9.0 onwards) can compile for WASI out of the box, and WebAssembly support is included in them by default. So, all that's done here @@ -19,8 +22,11 @@ is to provide builds configured to set the default target and sysroot for convenience. One could also use a standard Clang installation, build a sysroot from the -sources mentioned above, and compile with -"--target=wasm32-wasi --sysroot=/path/to/sysroot". +sources mentioned above, and compile with `--target=wasm32-wasi +--sysroot=/path/to/sysroot`. In this scenario, one would also need the +`libclang_rt.builtins-wasm32.a` objects available separately in the [release +downloads][releases] which must be extracted into +`$CLANG_INSTALL_DIR/$CLANG_VERSION/lib/wasi/`. ## Clone @@ -57,7 +63,7 @@ version of the package on GitHub, see [RELEASING.md](RELEASING.md). A typical installation from the release binaries might look like the following: ```shell script -export WASI_VERSION=14 +export WASI_VERSION=20 export WASI_VERSION_FULL=${WASI_VERSION}.0 wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz @@ -75,12 +81,15 @@ $CC foo.c -o foo.wasm Note: `${WASI_SDK_PATH}/share/wasi-sysroot` contains the WASI-specific includes/libraries/etc. The `--sysroot=...` option is not necessary if -`WASI_SDK_PATH` is `/opt/wasi-sdk`. +`WASI_SDK_PATH` is `/opt/wasi-sdk`. For troubleshooting, one can replace the +`--sysroot` path with a manual build of [wasi-libc]. ## Notes for Autoconf -[Autoconf](https://www.gnu.org/software/autoconf/autoconf.html) 2.70 now -[recognizes WASI](https://git.savannah.gnu.org/gitweb/?p=autoconf.git;a=blob;f=build-aux/config.sub;h=19c9553b1825cafb182115513bc628e0ee801bd0;hb=97fbc5c184acc6fa591ad094eae86917f03459fa#l1723). +[Autoconf] 2.70 now [recognizes WASI]. + +[Autoconf]: https://www.gnu.org/software/autoconf/autoconf.html +[recognizes WASI]: https://git.savannah.gnu.org/gitweb/?p=autoconf.git;a=blob;f=build-aux/config.sub;h=19c9553b1825cafb182115513bc628e0ee801bd0;hb=97fbc5c184acc6fa591ad094eae86917f03459fa#l1723 For convenience when building packages that aren't yet updated, updated config.sub and config.guess files are installed at `share/misc/config.*` @@ -88,11 +97,12 @@ in the install directory. ## Docker Image -We provide a [docker image](https://github.com/WebAssembly/wasi-sdk/pkgs/container/wasi-sdk) -including wasi-sdk that can be used for building projects without a -separate installation of the SDK. Autotools, CMake, and Ninja are included -in this image, and standard environment variables are set to use wask-sdk -for building. +We provide a [docker image] including wasi-sdk that can be used for building +projects without a separate installation of the SDK. Autotools, CMake, and Ninja +are included in this image, and standard environment variables are set to use +wask-sdk for building. + +[docker image]: https://github.com/WebAssembly/wasi-sdk/pkgs/container/wasi-sdk For example, this command can build a make-based project with the Docker image. @@ -107,24 +117,26 @@ disabled in a configure step before building with wasi-sdk. ## Notable Limitations -This repository does not yet support C++ exceptions. C++ code is -supported only with -fno-exceptions for now. Similarly, there is not -yet support for setjmp/longjmp. Work on support for [exception handling] -is underway at the language level which will support both of these -features. +This repository does not yet support __C++ exceptions__. C++ code is supported +only with -fno-exceptions for now. Similarly, there is not yet support for +setjmp/longjmp. Work on support for [exception handling] is underway at the +language level which will support both of these features. [exception handling]: https://github.com/WebAssembly/exception-handling/ -This repository does not yet support [threads]. Specifically, WASI does -not yet have an API for creating and managing threads yet, and WASI libc -does not yet have pthread support. +This repository experimentally supports __threads__ with +`--target=wasm32-wasi-threads`. It uses WebAssembly's [threads] primitives +(atomics, `wait`/`notify`, shared memory) and [wasi-threads] for spawning +threads. Note: this is experimental — do not expect long-term stability! [threads]: https://github.com/WebAssembly/threads +[wasi-threads]: https://github.com/WebAssembly/wasi-threads + +This repository does not yet support __dynamic libraries__. While there are +[some efforts] to design a system for dynamic libraries in wasm, it is still in +development and not yet generally usable. -This repository does not yet support dynamic libraries. While there are -[some efforts](https://github.com/WebAssembly/tool-conventions/blob/master/DynamicLinking.md) -to design a system for dynamic libraries in wasm, it is still in development -and not yet generally usable. +[some efforts]: https://github.com/WebAssembly/tool-conventions/blob/master/DynamicLinking.md -There is no support for networking. It is a goal of WASI to support networking -in the future though. +There is no support for __networking__. It is a goal of WASI to support +networking in the future though. From 7c0558ba41594c0a54b7341a210e667fc5abd6c1 Mon Sep 17 00:00:00 2001 From: Casper Beyer Date: Wed, 26 Jul 2023 09:02:07 +0800 Subject: [PATCH 020/165] Add a testcase for getentropy (#169) --- tests/general/getentropy.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 tests/general/getentropy.c diff --git a/tests/general/getentropy.c b/tests/general/getentropy.c new file mode 100644 index 000000000..3b7fe0e6c --- /dev/null +++ b/tests/general/getentropy.c @@ -0,0 +1,17 @@ +#include +#include + +int main() { + char buf[256] = {0}; + int ret = getentropy(buf, 256); + assert(ret == 0); + + int sum = 0; + for (int i = 0; i < 256; i++) { + sum += buf[i]; + } + + assert(sum != 0); + + return 0; +} From f3b43c703f1a3b6a93fcbdac9cb2b8439adcf0c1 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Fri, 28 Jul 2023 08:45:29 -0700 Subject: [PATCH 021/165] ci: ensure `LLVM_CMAKE_FLAGS` additions are kept (#337) In #321, some OSX-specific `Makefile` additions to `LLVM_CMAKE_FLAGS` were skipped unless `make` is run on a Darwin OS. This allowed building wasi-sdk for aarch64. But, as reported in #336, this also broke arm64/x86_64 universal binaries that are built during CI. The reason for this is that CI's `main.yml` overrides `LLVM_CMAKE_FLAGS` to add caching but `make` will not append to a variable set on the command line. This changes uses the `override` keyword to append to such a variable, as suggested [here]. [here]: https://www.gnu.org/software/make/manual/html_node/Override-Directive.html --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 9376249e8..3aab204ee 100644 --- a/Makefile +++ b/Makefile @@ -38,7 +38,7 @@ BASH= endif ifeq ($(shell uname),Darwin) -LLVM_CMAKE_FLAGS += -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ +override LLVM_CMAKE_FLAGS += -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 endif From fd20cf5a649bf3a1609a837e45f2c1fe1baad435 Mon Sep 17 00:00:00 2001 From: Anuraag Agrawal Date: Wed, 23 Aug 2023 07:17:31 +0900 Subject: [PATCH 022/165] Add wasi-sdk-pthread.cmake to docker image to allow enabling it with env (#339) --- docker/Dockerfile | 1 + docker/wasi-sdk-pthread.cmake | 36 +++++++++++++++++++++++++++++++++++ docker/wasi-sdk.cmake | 1 + 3 files changed, 38 insertions(+) create mode 100644 docker/wasi-sdk-pthread.cmake diff --git a/docker/Dockerfile b/docker/Dockerfile index 610de47bc..15658eedb 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -37,6 +37,7 @@ COPY --from=dist /wasi-sdk/share/wasi-sysroot/ /wasi-sysroot/ COPY --from=dist /wasi-sysroot-clang_rt/lib/wasi /usr/lib/llvm-${LLVM_VERSION}/lib/clang/${LLVM_VERSION}/lib/wasi ADD docker/wasi-sdk.cmake /usr/share/cmake/wasi-sdk.cmake +ADD docker/wasi-sdk-pthread.cmake /usr/share/cmake/wasi-sdk-pthread.cmake ENV CMAKE_TOOLCHAIN_FILE /usr/share/cmake/wasi-sdk.cmake ADD cmake/Platform/WASI.cmake /usr/share/cmake/Modules/Platform/WASI.cmake diff --git a/docker/wasi-sdk-pthread.cmake b/docker/wasi-sdk-pthread.cmake new file mode 100644 index 000000000..61354451a --- /dev/null +++ b/docker/wasi-sdk-pthread.cmake @@ -0,0 +1,36 @@ +# Cmake toolchain description file for the wasi-sdk docker image + +# This is arbitrary, AFAIK, for now. +cmake_minimum_required(VERSION 3.4.0) + +# To make sure it recognizes the WASI platform +list(APPEND CMAKE_MODULE_PATH /usr/share/cmake/Modules) + +set(CMAKE_SYSTEM_NAME WASI) +set(CMAKE_SYSTEM_VERSION 1) +set(CMAKE_SYSTEM_PROCESSOR wasm32) +set(triple wasm32-wasi-threads) +set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -pthread") +set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread") +# wasi-threads requires --import-memory. +# wasi requires --export-memory. +# (--export-memory is implicit unless --import-memory is given) +set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--import-memory") +set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--export-memory") + +set(CMAKE_C_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) +set(CMAKE_CXX_COMPILER /usr/bin/clang++-$ENV{LLVM_VERSION}) +set(CMAKE_ASM_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) +set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) +set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) +set(CMAKE_C_COMPILER_TARGET ${triple}) +set(CMAKE_CXX_COMPILER_TARGET ${triple}) +set(CMAKE_ASM_COMPILER_TARGET ${triple}) +SET(CMAKE_SYSROOT /wasi-sysroot) + +# Don't look in the sysroot for executables to run during the build +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) +# Only look in the sysroot (not in the host paths) for the rest +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/docker/wasi-sdk.cmake b/docker/wasi-sdk.cmake index f2f3229a0..eabb877ca 100644 --- a/docker/wasi-sdk.cmake +++ b/docker/wasi-sdk.cmake @@ -18,6 +18,7 @@ set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) set(CMAKE_C_COMPILER_TARGET ${triple}) set(CMAKE_CXX_COMPILER_TARGET ${triple}) +set(CMAKE_ASM_COMPILER_TARGET ${triple}) SET(CMAKE_SYSROOT /wasi-sysroot) # Don't look in the sysroot for executables to run during the build From 4025e95fac91b49e892cbaa5211163cf6f00c839 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Wed, 23 Aug 2023 07:18:19 +0900 Subject: [PATCH 023/165] Bump wasi-libc (#340) I want to include https://github.com/WebAssembly/wasi-libc/pull/409 which is rather easy to hit when people are experimenting wasi-threads. --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index 1dfe5c302..9f51a7102 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 1dfe5c302d1c5ab621f7abf04620fae92700fd22 +Subproject commit 9f51a7102085ec6a6ced5778f0864c9af9f50000 From e85250b1c42d0ed797d52de143edf9b250ae70a8 Mon Sep 17 00:00:00 2001 From: Michael Lill Date: Mon, 28 Aug 2023 19:35:44 +0200 Subject: [PATCH 024/165] Update README.md (#343) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f00a95319..38d646443 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ in the install directory. We provide a [docker image] including wasi-sdk that can be used for building projects without a separate installation of the SDK. Autotools, CMake, and Ninja are included in this image, and standard environment variables are set to use -wask-sdk for building. +wasi-sdk for building. [docker image]: https://github.com/WebAssembly/wasi-sdk/pkgs/container/wasi-sdk From 19fb80bcd80ee4381e0b1ba1c480020ca69e7320 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Wed, 30 Aug 2023 11:13:24 +0900 Subject: [PATCH 025/165] Bump wasi-libc (#344) To include: https://github.com/WebAssembly/wasi-libc/pull/433 --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index 9f51a7102..ec4566bea 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 9f51a7102085ec6a6ced5778f0864c9af9f50000 +Subproject commit ec4566beae84e54952637f0bf61bee4b4cacc087 From bbfb9736af53ad40197a92058ec77ac7248c5ac7 Mon Sep 17 00:00:00 2001 From: Kian Cross Date: Wed, 6 Sep 2023 15:45:27 +0100 Subject: [PATCH 026/165] Do not use GNU find features on FreeBSD (#345) FreeBSD (like MacOS) does not support the `-executable` argument for `find`. This commit checks if the OS is FreeBSD, and if so, uses the same workaround as already in place on MacOS. --- strip_symbols.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/strip_symbols.sh b/strip_symbols.sh index c161f60aa..effd30090 100755 --- a/strip_symbols.sh +++ b/strip_symbols.sh @@ -2,9 +2,9 @@ set -e DIRECTORY=${1:-/opt/wasi-sdk/bin} -if [[ "$OSTYPE" == "darwin"* ]]; then -# macos find doesnt support -executable so we fall back on having a permission -# bit to execute: +if [[ "$OSTYPE" == "darwin"* ]] || [[ "$OSTYPE" == "freebsd"* ]]; then +# macos and freebsd find do not support -executable so we fall back on +# having a permission bit to execute: EXECUTABLES=$(find ${DIRECTORY} -type f -perm +111) else EXECUTABLES=$(find ${DIRECTORY} -type f -executable) From 8d4dbb19fbda67ec227d33c4058b889ccfaee5ce Mon Sep 17 00:00:00 2001 From: Arvid Norlander <132438847+arvid-norlander@users.noreply.github.com> Date: Wed, 6 Sep 2023 21:11:31 +0200 Subject: [PATCH 027/165] Set CMAKE_MODULE_PATH in the cmake toolchain file (#335) Also add some basic testing for the cmake toolchain file. Fixes: #181 --- Makefile | 5 +++-- tests/cmake/CMakeLists.txt | 34 ++++++++++++++++++++++++++++++ tests/cmake/test_driver.sh | 17 +++++++++++++++ tests/run.sh | 42 +++++++++++++++++++++++++++++++++----- wasi-sdk.cmake | 4 ++++ 5 files changed, 95 insertions(+), 7 deletions(-) create mode 100644 tests/cmake/CMakeLists.txt create mode 100755 tests/cmake/test_driver.sh diff --git a/Makefile b/Makefile index 3aab204ee..319cabd90 100644 --- a/Makefile +++ b/Makefile @@ -53,7 +53,7 @@ default: build check: CC="clang --sysroot=$(BUILD_PREFIX)/share/wasi-sysroot" \ CXX="clang++ --sysroot=$(BUILD_PREFIX)/share/wasi-sysroot -fno-exceptions" \ - PATH="$(PATH_PREFIX)/bin:$$PATH" tests/run.sh $(RUNTIME) + PATH="$(PATH_PREFIX)/bin:$$PATH" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" clean: rm -rf build $(DESTDIR) @@ -219,9 +219,10 @@ build/libcxx.BUILT: build/llvm.BUILT build/compiler-rt.BUILT build/wasi-libc.BUI build/config.BUILT: mkdir -p $(BUILD_PREFIX)/share/misc cp src/config/config.sub src/config/config.guess $(BUILD_PREFIX)/share/misc - mkdir -p $(BUILD_PREFIX)/share/cmake + mkdir -p $(BUILD_PREFIX)/share/cmake/Platform cp wasi-sdk.cmake $(BUILD_PREFIX)/share/cmake cp wasi-sdk-pthread.cmake $(BUILD_PREFIX)/share/cmake + cp cmake/Platform/WASI.cmake $(BUILD_PREFIX)/share/cmake/Platform touch build/config.BUILT build: build/llvm.BUILT build/wasi-libc.BUILT build/compiler-rt.BUILT build/libcxx.BUILT build/config.BUILT diff --git a/tests/cmake/CMakeLists.txt b/tests/cmake/CMakeLists.txt new file mode 100644 index 000000000..28c9b839b --- /dev/null +++ b/tests/cmake/CMakeLists.txt @@ -0,0 +1,34 @@ +cmake_minimum_required(VERSION 3.22) + +project(wasi-sdk-test) + +# Sanity check setup +if (NOT ${CMAKE_SYSTEM_NAME} STREQUAL WASI) + message(FATAL_ERROR "Wrong system name (${CMAKE_SYSTEM_NAME}), wrong toolchain file in use?") +endif() + +if(NOT DEFINED WASI) + message(FATAL_ERROR "WASI is not set, platform file likely not loaded") +endif() + +set(RUNWASI "" CACHE STRING "Path to or name of WASM runner") + +# Test build a C and C++ target respectively +add_executable(void_main_c ../general/void_main.c) +add_executable(void_main_cc ../general/void_main.cc) + +include(CTest) +enable_testing() + +add_test(NAME void_main_c + COMMAND + ${CMAKE_CURRENT_SOURCE_DIR}/test_driver.sh + ${RUNWASI} + $ + ${CMAKE_CURRENT_SOURCE_DIR}/../general/void_main.c.stdout.expected) +add_test(NAME void_main_cc + COMMAND + ${CMAKE_CURRENT_SOURCE_DIR}/test_driver.sh + ${RUNWASI} + $ + ${CMAKE_CURRENT_SOURCE_DIR}/../general/void_main.cc.stdout.expected) diff --git a/tests/cmake/test_driver.sh b/tests/cmake/test_driver.sh new file mode 100755 index 000000000..6e469a1ae --- /dev/null +++ b/tests/cmake/test_driver.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# Simplified runner for cmake + +set -ex + +runwasi="$1" +target="$2" +stdout_expected="$3" +stderr_expected="/dev/null" + +stdout_observed="$target.stdout.observed" +stderr_observed="$target.stderr.observed" + +"$runwasi" "$target" > "$stdout_observed" 2> "$stderr_observed" + +diff -u "$stderr_expected" "$stderr_observed" +diff -u "$stdout_expected" "$stdout_observed" diff --git a/tests/run.sh b/tests/run.sh index 6379b581c..36bcda55f 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -1,9 +1,10 @@ #!/bin/bash set -ueo pipefail -# Top-level test runner. Usage is "run.sh" to run tests in compile-only mode, -# or "run.sh " where is a WASI-capable runtime to run the -# tests in full compile and execute mode. +# Top-level test runner. Usage is "run.sh " to run tests +# in compile-only mode, or "run.sh " where +# is a WASI-capable runtime to run the tests in full compile and +# execute mode. # # By default this script will look for `clang` and `clang++` in $PATH and # assume that they are correctly configured with the sysroot in the default @@ -12,10 +13,16 @@ set -ueo pipefail # export CXX="/bin/clang++ --sysroot /share/wasi-sysroot" # export CC="/bin/clang --sysroot /share/wasi-sysroot" # +if [ $# -lt 1 ]; then + echo "Path to WASI SDK is required" + exit 1 +fi + +wasi_sdk="$1" # Determine the wasm runtime to use, if one is provided. -if [ $# -gt 0 ]; then - runwasi="$1" +if [ $# -gt 1 ]; then + runwasi="$2" else runwasi="" fi @@ -26,6 +33,7 @@ CXX=${CXX:=clang++} echo $CC echo $CXX +echo "SDK: $wasi_sdk" cd $testdir/compile-only for options in -O0 -O2 "-O2 -flto"; do @@ -54,3 +62,27 @@ for options in -O0 -O2 "-O2 -flto"; do done done cd - >/dev/null + +# Test cmake build system for wasi-sdk +test_cmake() { + local option + for option in Debug Release; do + rm -rf "$testdir/cmake/build/$option" + mkdir -p "$testdir/cmake/build/$option" + cd "$testdir/cmake/build/$option" + cmake \ + -G "Unix Makefiles" \ + -DCMAKE_BUILD_TYPE="$option" \ + -DRUNWASI="$runwasi" \ + -DWASI_SDK_PREFIX="$wasi_sdk" \ + -DCMAKE_TOOLCHAIN_FILE="$wasi_sdk/share/cmake/wasi-sdk.cmake" \ + ../.. + make + if [[ -n "$runwasi" ]]; then + ctest --output-on-failure + fi + cd - >/dev/null + done +} + +test_cmake diff --git a/wasi-sdk.cmake b/wasi-sdk.cmake index 83ee9b0ac..f5021a5e1 100644 --- a/wasi-sdk.cmake +++ b/wasi-sdk.cmake @@ -3,6 +3,10 @@ # This is arbitrary, AFAIK, for now. cmake_minimum_required(VERSION 3.4.0) +# Until Platform/WASI.cmake is upstream we need to inject the path to it +# into CMAKE_MODULE_PATH. +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}") + set(CMAKE_SYSTEM_NAME WASI) set(CMAKE_SYSTEM_VERSION 1) set(CMAKE_SYSTEM_PROCESSOR wasm32) From 9b9708e9ebc354b4fe34904e84fa413a3951e238 Mon Sep 17 00:00:00 2001 From: "liang.he" Date: Wed, 13 Sep 2023 02:21:16 +0800 Subject: [PATCH 028/165] Enhancement of WASI_SDK_PREFIX in toolchain files (#349) No need to pass `WASI_SDK_PREFIX` value from the command line. Use `CMAKE_CURRENT_LIST_DIR` to assemble the value of `WASI_SDK_PREFIX` now. --- README.md | 14 ++++++++++++++ wasi-sdk-pthread.cmake | 5 +++++ wasi-sdk.cmake | 5 +++++ 3 files changed, 24 insertions(+) diff --git a/README.md b/README.md index 38d646443..b85471100 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,20 @@ includes/libraries/etc. The `--sysroot=...` option is not necessary if `WASI_SDK_PATH` is `/opt/wasi-sdk`. For troubleshooting, one can replace the `--sysroot` path with a manual build of [wasi-libc]. +### Integrating with a CMake build system + +Use a toolchain file to setup the *wasi-sdk* platform. + +``` +$ cmake -DCMAKE_TOOLCHAIN_FILE=${WASI_SDK_PATH}/share/cmake/wasi-sdk.cmake ... +``` + +or the *wasi-sdk-thread* platform + +``` +$ cmake -DCMAKE_TOOLCHAIN_FILE=${WASI_SDK_PATH}/share/cmake/wasi-sdk-pthread.cmake ... +``` + ## Notes for Autoconf [Autoconf] 2.70 now [recognizes WASI]. diff --git a/wasi-sdk-pthread.cmake b/wasi-sdk-pthread.cmake index 0e5f6ff00..ba660234b 100644 --- a/wasi-sdk-pthread.cmake +++ b/wasi-sdk-pthread.cmake @@ -21,6 +21,11 @@ else() set(WASI_HOST_EXE_SUFFIX "") endif() +# When building from source, WASI_SDK_PREFIX represents the generated directory +if(NOT WASI_SDK_PREFIX) + set(WASI_SDK_PREFIX ${CMAKE_CURRENT_LIST_DIR}/../../) +endif() + set(CMAKE_C_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) set(CMAKE_CXX_COMPILER ${WASI_SDK_PREFIX}/bin/clang++${WASI_HOST_EXE_SUFFIX}) set(CMAKE_ASM_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) diff --git a/wasi-sdk.cmake b/wasi-sdk.cmake index f5021a5e1..f6a0059c8 100644 --- a/wasi-sdk.cmake +++ b/wasi-sdk.cmake @@ -18,6 +18,11 @@ else() set(WASI_HOST_EXE_SUFFIX "") endif() +# When building from source, WASI_SDK_PREFIX represents the generated directory +if(NOT WASI_SDK_PREFIX) + set(WASI_SDK_PREFIX ${CMAKE_CURRENT_LIST_DIR}/../../) +endif() + set(CMAKE_C_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) set(CMAKE_CXX_COMPILER ${WASI_SDK_PREFIX}/bin/clang++${WASI_HOST_EXE_SUFFIX}) set(CMAKE_ASM_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) From fe76aa8a6ec227d2c5b8d7bac025c9c2289aa6a8 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Thu, 28 Sep 2023 04:24:04 +0900 Subject: [PATCH 029/165] Fix submodule update (#354) ``` error: Server does not allow request for unadvertised object f992bcc08219edb283d2ab31dd3871a4a0e8220e fatal: Fetched in submodule path 'src/config', but it did not contain f992bcc08219edb283d2ab31dd3871a4a0e8220e. Direct fetching of that commit failed. ``` --- .github/workflows/main.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index fb3bf377a..8055b5b4d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -48,7 +48,7 @@ jobs: # bump depth (or even better, the submodule), in case of "error: # Server does not allow request for unadvertised object" in the # future. - - run: git submodule update --init --depth 16 --jobs 3 + - run: git submodule update --init --depth 32 --jobs 3 - name: Install ccache, ninja (macOS) run: brew install ccache ninja if: matrix.os == 'macos-latest' @@ -102,7 +102,7 @@ jobs: - uses: actions/checkout@v3 with: fetch-depth: 0 - - run: git submodule update --init --depth 16 --jobs 3 + - run: git submodule update --init --depth 32 --jobs 3 - name: Build shell: msys2 {0} run: | @@ -135,7 +135,7 @@ jobs: with: fetch-depth: 0 - - run: git submodule update --init --depth 16 --jobs 3 + - run: git submodule update --init --depth 32 --jobs 3 - uses: docker/login-action@v2 with: From 2393be41c8df1fc2ad6fe7299ba7edb99d5699d1 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Fri, 29 Sep 2023 09:59:19 +0900 Subject: [PATCH 030/165] Update LLVM to 17 (#352) * Bump llvm version to 17.0.1 * docker/Dockerfile: bump LLVM_VERSION to 17 --- docker/Dockerfile | 2 +- src/llvm-project | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index 15658eedb..cbb503bfa 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -17,7 +17,7 @@ RUN mv /wasi-sdk-* /wasi-sdk FROM ubuntu:22.04 -ENV LLVM_VERSION 16 +ENV LLVM_VERSION 17 # Install build toolchain including clang, ld, make, autotools, ninja, and cmake RUN apt-get update && \ diff --git a/src/llvm-project b/src/llvm-project index 08d094a0e..e19b7dc36 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 08d094a0e457360ad8b94b017d2dc277e697ca76 +Subproject commit e19b7dc36bc047b9eb72078d034596be766da350 From 71fd9cb741aa3b36e027cbcadebe99457ea7d59f Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Thu, 7 Dec 2023 11:53:40 -0700 Subject: [PATCH 031/165] Bump LLVM version to 17.0.6 (#362) Signed-off-by: Joel Dice --- src/llvm-project | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/llvm-project b/src/llvm-project index e19b7dc36..6009708b4 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit e19b7dc36bc047b9eb72078d034596be766da350 +Subproject commit 6009708b4367171ccdbf4b5905cb6a803753fe18 From a35b453a87314f126bdd3f554b084d273e3265fb Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Mon, 11 Dec 2023 17:20:52 -0700 Subject: [PATCH 032/165] add shared library support (#338) This adds support for building WASI shared libraries per https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md. For the time being, the goal is to allow "pseudo-dynamic" linking using the Component Model per https://github.com/WebAssembly/component-model/blob/main/design/mvp/examples/SharedEverythingDynamicLinking.md. This requires all libraries to be available when the component is created, but still allows runtime symbol resolution via `dlopen`/`dlsym` backed by a static lookup table. This is sufficient to support Python native extensions, for example. A complete demo using `wit-component` is available at https://github.com/dicej/component-linking-demo. Signed-off-by: Joel Dice --- .gitmodules | 2 +- Makefile | 31 ++++++++++++++++++------------- src/wasi-libc | 2 +- 3 files changed, 20 insertions(+), 15 deletions(-) diff --git a/.gitmodules b/.gitmodules index e8b2eace4..0691635ce 100644 --- a/.gitmodules +++ b/.gitmodules @@ -3,7 +3,7 @@ url = https://github.com/llvm/llvm-project [submodule "src/wasi-libc"] path = src/wasi-libc - url = https://github.com/CraneStation/wasi-libc + url = https://github.com/WebAssembly/wasi-libc [submodule "src/config"] path = src/config url = https://git.savannah.gnu.org/git/config.git diff --git a/Makefile b/Makefile index 319cabd90..fd705b3a1 100644 --- a/Makefile +++ b/Makefile @@ -107,12 +107,13 @@ build/llvm.BUILT: llvm-config touch build/llvm.BUILT -build/wasi-libc.BUILT: build/llvm.BUILT +build/wasi-libc.BUILT: build/compiler-rt.BUILT $(MAKE) -C $(ROOT_DIR)/src/wasi-libc \ CC=$(BUILD_PREFIX)/bin/clang \ AR=$(BUILD_PREFIX)/bin/llvm-ar \ NM=$(BUILD_PREFIX)/bin/llvm-nm \ - SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot + SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ + default libc_so $(MAKE) -C $(ROOT_DIR)/src/wasi-libc \ CC=$(BUILD_PREFIX)/bin/clang \ AR=$(BUILD_PREFIX)/bin/llvm-ar \ @@ -121,7 +122,7 @@ build/wasi-libc.BUILT: build/llvm.BUILT THREAD_MODEL=posix touch build/wasi-libc.BUILT -build/compiler-rt.BUILT: build/llvm.BUILT build/wasi-libc.BUILT +build/compiler-rt.BUILT: build/llvm.BUILT # Do the build, and install it. mkdir -p build/compiler-rt cd build/compiler-rt && cmake -G Ninja \ @@ -151,6 +152,8 @@ build/compiler-rt.BUILT: build/llvm.BUILT build/wasi-libc.BUILT touch build/compiler-rt.BUILT # Flags for libcxx and libcxxabi. +# $(1): pthreads ON or OFF +# $(2): shared libraries ON or OFF LIBCXX_CMAKE_FLAGS = \ -DCMAKE_C_COMPILER_WORKS=ON \ -DCMAKE_CXX_COMPILER_WORKS=ON \ @@ -158,41 +161,43 @@ LIBCXX_CMAKE_FLAGS = \ -DCMAKE_MODULE_PATH=$(ROOT_DIR)/cmake \ -DCMAKE_TOOLCHAIN_FILE=$(ROOT_DIR)/wasi-sdk.cmake \ -DCMAKE_STAGING_PREFIX=$(PREFIX)/share/wasi-sysroot \ + -DCMAKE_POSITION_INDEPENDENT_CODE=$(2) \ -DLLVM_CONFIG_PATH=$(ROOT_DIR)/build/llvm/bin/llvm-config \ -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ -DCXX_SUPPORTS_CXX11=ON \ - -DLIBCXX_ENABLE_THREADS:BOOL=@PTHREAD@ \ - -DLIBCXX_HAS_PTHREAD_API:BOOL=@PTHREAD@ \ + -DLIBCXX_ENABLE_THREADS:BOOL=$(1) \ + -DLIBCXX_HAS_PTHREAD_API:BOOL=$(1) \ -DLIBCXX_HAS_EXTERNAL_THREAD_API:BOOL=OFF \ -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF \ -DLLVM_COMPILER_CHECKED=ON \ -DCMAKE_BUILD_TYPE=RelWithDebugInfo \ - -DLIBCXX_ENABLE_SHARED:BOOL=OFF \ + -DLIBCXX_ENABLE_SHARED:BOOL=$(2) \ -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF \ -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF \ -DLIBCXX_ENABLE_FILESYSTEM:BOOL=OFF \ + -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF \ -DLIBCXX_CXX_ABI=libcxxabi \ -DLIBCXX_CXX_ABI_INCLUDE_PATHS=$(LLVM_PROJ_DIR)/libcxxabi/include \ -DLIBCXX_HAS_MUSL_LIBC:BOOL=ON \ -DLIBCXX_ABI_VERSION=2 \ -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF \ - -DLIBCXXABI_ENABLE_SHARED:BOOL=OFF \ + -DLIBCXXABI_ENABLE_SHARED:BOOL=$(2) \ -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON \ - -DLIBCXXABI_ENABLE_THREADS:BOOL=@PTHREAD@ \ - -DLIBCXXABI_HAS_PTHREAD_API:BOOL=@PTHREAD@ \ + -DLIBCXXABI_ENABLE_THREADS:BOOL=$(1) \ + -DLIBCXXABI_HAS_PTHREAD_API:BOOL=$(1) \ -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF \ -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF \ - -DLIBCXXABI_ENABLE_PIC:BOOL=OFF \ + -DLIBCXXABI_ENABLE_PIC:BOOL=$(2) \ -DWASI_SDK_PREFIX=$(BUILD_PREFIX) \ -DUNIX:BOOL=ON \ --debug-trycompile -build/libcxx.BUILT: build/llvm.BUILT build/compiler-rt.BUILT build/wasi-libc.BUILT +build/libcxx.BUILT: build/llvm.BUILT build/wasi-libc.BUILT # Do the build. mkdir -p build/libcxx - cd build/libcxx && cmake -G Ninja $(LIBCXX_CMAKE_FLAGS:@PTHREAD@=OFF) \ + cd build/libcxx && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,OFF,ON) \ -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS)" \ -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS)" \ @@ -202,7 +207,7 @@ build/libcxx.BUILT: build/llvm.BUILT build/compiler-rt.BUILT build/wasi-libc.BUI $(LLVM_PROJ_DIR)/runtimes ninja $(NINJA_FLAGS) -C build/libcxx mkdir -p build/libcxx-threads - cd build/libcxx-threads && cmake -G Ninja $(LIBCXX_CMAKE_FLAGS:@PTHREAD@=ON) \ + cd build/libcxx-threads && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,ON,OFF) \ -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CFLAGS)" \ -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CXXFLAGS)" \ diff --git a/src/wasi-libc b/src/wasi-libc index ec4566bea..9c17f5235 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit ec4566beae84e54952637f0bf61bee4b4cacc087 +Subproject commit 9c17f5235c7977cb2a000990eb8c605a25a80adf From 317548590b402d622194416094fe0e392b725fe2 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Fri, 15 Dec 2023 15:07:49 -0800 Subject: [PATCH 033/165] Update wasi-libc (#366) This includes [wasi-libc#450], a last minute fix before releasing `wasi-sdk-21`. [wasi-libc#450]: https://github.com/WebAssembly/wasi-libc/pull/450 --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index 9c17f5235..c5264e2bb 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 9c17f5235c7977cb2a000990eb8c605a25a80adf +Subproject commit c5264e2bbe532994d06b039005f2af91bedcc1a6 From 8f465debf66628a646aaa00a0185aa48764fb8ef Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Wed, 10 Jan 2024 09:41:01 -0700 Subject: [PATCH 034/165] build and test `wasm32-wasi-preview2` target (#370) * build and test `wasm32-wasi-preview2` target This updates `wasi-libc` to include https://github.com/WebAssembly/wasi-libc/pull/457, which adds preliminary support for the new `wasm32-wasi-preview2` target. It also adds support for testing the new target using Wasmtime 16.0.0 and `wit-component`. Note that Wasmtime produces different output when reporting errors for Preview 2 components than it does for Preview 1 modules, so I've added a few .expected files specific to Preview 2. Signed-off-by: Joel Dice * test all three targets Signed-off-by: Joel Dice --------- Signed-off-by: Joel Dice --- .github/workflows/main.yml | 17 ++++- Makefile | 33 ++++++++-- src/wasi-libc | 2 +- ...ort.c.wasm32-wasi-preview2.stderr.expected | 6 ++ ...ail.c.wasm32-wasi-preview2.stderr.expected | 7 +++ ...brt.c.wasm32-wasi-preview2.stderr.expected | 6 ++ tests/run.sh | 62 +++++++++++-------- tests/testcase.sh | 40 +++++++++--- 8 files changed, 132 insertions(+), 41 deletions(-) create mode 100644 tests/general/abort.c.wasm32-wasi-preview2.stderr.expected create mode 100644 tests/general/assert-fail.c.wasm32-wasi-preview2.stderr.expected create mode 100644 tests/general/sigabrt.c.wasm32-wasi-preview2.stderr.expected diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 8055b5b4d..7489419f9 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -39,7 +39,20 @@ jobs: 0-cache-macos-latest if: matrix.os == 'macos-latest' - name: Install wasmtime for tests - run: curl -f -L --retry 5 https://wasmtime.dev/install.sh | bash -s -- --version v8.0.1 + # TODO: switch to Wasmtime 17 once it's released, which will include https://github.com/bytecodealliance/wasmtime/pull/7750 + run: | + curl -f -L --retry 5 https://wasmtime.dev/install.sh | bash -s -- --version dev + ~/.wasmtime/bin/wasmtime --version + curl -f -L --retry 5 -o ~/.wasmtime/bin/wasi_snapshot_preview1.command.wasm https://github.com/bytecodealliance/wasmtime/releases/download/v16.0.0/wasi_snapshot_preview1.command.wasm + if [ "${{ matrix.os }}" == "ubuntu-latest" ]; then + curl -f -OL --retry 5 https://github.com/bytecodealliance/wasm-tools/releases/download/wasm-tools-1.0.54/wasm-tools-1.0.54-x86_64-linux.tar.gz + tar xf wasm-tools-1.0.54-x86_64-linux.tar.gz + cp wasm-tools-1.0.54-x86_64-linux/wasm-tools ~/.wasmtime/bin/ + else + curl -f -OL --retry 5 https://github.com/bytecodealliance/wasm-tools/releases/download/wasm-tools-1.0.54/wasm-tools-1.0.54-x86_64-macos.tar.gz + tar xf wasm-tools-1.0.54-x86_64-macos.tar.gz + cp wasm-tools-1.0.54-x86_64-macos/wasm-tools ~/.wasmtime/bin/ + fi - uses: actions/checkout@v3 with: fetch-depth: 0 @@ -59,7 +72,7 @@ jobs: run: NINJA_FLAGS=-v make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON shell: bash - name: Run the testsuite - run: NINJA_FLAGS=-v make check RUNTIME=~/.wasmtime/bin/wasmtime + run: NINJA_FLAGS=-v make check RUNTIME=~/.wasmtime/bin/wasmtime ADAPTER=~/.wasmtime/bin/wasi_snapshot_preview1.command.wasm WASM_TOOLS=~/.wasmtime/bin/wasm-tools - name: Upload artifacts uses: actions/upload-artifact@v1 with: diff --git a/Makefile b/Makefile index fd705b3a1..76d428141 100644 --- a/Makefile +++ b/Makefile @@ -53,7 +53,7 @@ default: build check: CC="clang --sysroot=$(BUILD_PREFIX)/share/wasi-sysroot" \ CXX="clang++ --sysroot=$(BUILD_PREFIX)/share/wasi-sysroot -fno-exceptions" \ - PATH="$(PATH_PREFIX)/bin:$$PATH" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" + PATH="$(PATH_PREFIX)/bin:$$PATH" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" "$(ADAPTER)" "$(WASM_TOOLS)" clean: rm -rf build $(DESTDIR) @@ -108,6 +108,13 @@ build/llvm.BUILT: touch build/llvm.BUILT build/wasi-libc.BUILT: build/compiler-rt.BUILT + $(MAKE) -C $(ROOT_DIR)/src/wasi-libc \ + CC=$(BUILD_PREFIX)/bin/clang \ + AR=$(BUILD_PREFIX)/bin/llvm-ar \ + NM=$(BUILD_PREFIX)/bin/llvm-nm \ + SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ + WASI_SNAPSHOT=preview2 \ + default libc_so $(MAKE) -C $(ROOT_DIR)/src/wasi-libc \ CC=$(BUILD_PREFIX)/bin/clang \ AR=$(BUILD_PREFIX)/bin/llvm-ar \ @@ -196,11 +203,21 @@ LIBCXX_CMAKE_FLAGS = \ build/libcxx.BUILT: build/llvm.BUILT build/wasi-libc.BUILT # Do the build. + mkdir -p build/libcxx-preview2 + cd build/libcxx-preview2 && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,OFF,ON) \ + -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ + -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS) --target=wasm32-wasi-preview2" \ + -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS) --target=wasm32-wasi-preview2" \ + -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-preview2 \ + -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-preview2 \ + -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ + $(LLVM_PROJ_DIR)/runtimes + ninja $(NINJA_FLAGS) -C build/libcxx-preview2 mkdir -p build/libcxx cd build/libcxx && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,OFF,ON) \ -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS)" \ - -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS)" \ + -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS) --target=wasm32-wasi" \ + -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS) --target=wasm32-wasi" \ -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi \ -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi \ -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ @@ -209,16 +226,22 @@ build/libcxx.BUILT: build/llvm.BUILT build/wasi-libc.BUILT mkdir -p build/libcxx-threads cd build/libcxx-threads && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,ON,OFF) \ -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CFLAGS)" \ - -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CXXFLAGS)" \ + -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CFLAGS) --target=wasm32-wasi-threads" \ + -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CXXFLAGS) --target=wasm32-wasi-threads" \ -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-threads \ -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-threads \ -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ $(LLVM_PROJ_DIR)/runtimes ninja $(NINJA_FLAGS) -C build/libcxx-threads # Do the install. + DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx-preview2 install + mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/wasm32-wasi-preview2/ DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx install + mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/wasm32-wasi/ DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx-threads install + mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/wasm32-wasi-threads/ + # As of this writing, `clang++` will ignore the above include dirs unless this one also exists: + mkdir -p $(BUILD_PREFIX)/share/wasi-sysroot/include/c++/v1 touch build/libcxx.BUILT build/config.BUILT: diff --git a/src/wasi-libc b/src/wasi-libc index c5264e2bb..925ad6d75 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit c5264e2bbe532994d06b039005f2af91bedcc1a6 +Subproject commit 925ad6d75899397d26b9f09a6f195dbf5eb35814 diff --git a/tests/general/abort.c.wasm32-wasi-preview2.stderr.expected b/tests/general/abort.c.wasm32-wasi-preview2.stderr.expected new file mode 100644 index 000000000..26fe376cf --- /dev/null +++ b/tests/general/abort.c.wasm32-wasi-preview2.stderr.expected @@ -0,0 +1,6 @@ +Error: failed to run main module `abort.c.---.wasm` + +Caused by: + 0: failed to invoke `run` function + 1: error while executing at wasm backtrace: + 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/assert-fail.c.wasm32-wasi-preview2.stderr.expected b/tests/general/assert-fail.c.wasm32-wasi-preview2.stderr.expected new file mode 100644 index 000000000..3702431bd --- /dev/null +++ b/tests/general/assert-fail.c.wasm32-wasi-preview2.stderr.expected @@ -0,0 +1,7 @@ +Assertion failed: false (assert-fail.c: main: 5) +Error: failed to run main module `assert-fail.c.---.wasm` + +Caused by: + 0: failed to invoke `run` function + 1: error while executing at wasm backtrace: + 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/sigabrt.c.wasm32-wasi-preview2.stderr.expected b/tests/general/sigabrt.c.wasm32-wasi-preview2.stderr.expected new file mode 100644 index 000000000..f437a8214 --- /dev/null +++ b/tests/general/sigabrt.c.wasm32-wasi-preview2.stderr.expected @@ -0,0 +1,6 @@ +raising SIGABRT... +Program received fatal signal: Aborted +Error: failed to run main module `sigabrt.c.---.wasm` + +Caused by: + 0: failed to invoke `run` function diff --git a/tests/run.sh b/tests/run.sh index 36bcda55f..62d05e9ce 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -23,8 +23,17 @@ wasi_sdk="$1" # Determine the wasm runtime to use, if one is provided. if [ $# -gt 1 ]; then runwasi="$2" + if [ $# -gt 3 ]; then + adapter="$3" + wasm_tools="$4" + else + adapter="" + wasm_tools="" + fi else runwasi="" + adapter="" + wasm_tools="" fi testdir=$(dirname $0) @@ -35,34 +44,37 @@ echo $CC echo $CXX echo "SDK: $wasi_sdk" -cd $testdir/compile-only -for options in -O0 -O2 "-O2 -flto"; do - echo "===== Testing compile-only with $options =====" - for file in *.c; do - echo "Testing compile-only $file..." - ../testcase.sh "" "$CC" "$options" "$file" +for target in wasm32-wasi wasm32-wasi-threads wasm32-wasi-preview2; do + echo "===== Testing target $target =====" + cd $testdir/compile-only + for options in -O0 -O2 "-O2 -flto"; do + echo "===== Testing compile-only with $options =====" + for file in *.c; do + echo "Testing compile-only $file..." + ../testcase.sh "$target" "" "" "" "$CC" "$options" "$file" + done + for file in *.cc; do + echo "Testing compile-only $file..." + ../testcase.sh "$target" "" "" "" "$CXX" "$options" "$file" + done done - for file in *.cc; do - echo "Testing compile-only $file..." - ../testcase.sh "" "$CXX" "$options" "$file" + cd - >/dev/null + + cd $testdir/general + for options in -O0 -O2 "-O2 -flto"; do + echo "===== Testing with $options =====" + for file in *.c; do + echo "Testing $file..." + ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$CC" "$options" "$file" + done + for file in *.cc; do + echo "Testing $file..." + ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$CXX" "$options" "$file" + done done + cd - >/dev/null done -cd - >/dev/null - -cd $testdir/general -for options in -O0 -O2 "-O2 -flto"; do - echo "===== Testing with $options =====" - for file in *.c; do - echo "Testing $file..." - ../testcase.sh "$runwasi" "$CC" "$options" "$file" - done - for file in *.cc; do - echo "Testing $file..." - ../testcase.sh "$runwasi" "$CXX" "$options" "$file" - done -done -cd - >/dev/null - + # Test cmake build system for wasi-sdk test_cmake() { local option diff --git a/tests/testcase.sh b/tests/testcase.sh index 38db9a37e..d7368e09c 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -6,16 +6,20 @@ set -ueo pipefail # Command-line parsing; this script is meant to be run from a higher-level # script, so don't do anything fancy. -runwasi="$1" -compiler="$2" -options="$3" -input="$4" +target="$1" +runwasi="$2" +adapter="$3" +wasm_tools="$4" +compiler="$5" +options="$6" +input="$7" # Compile names for generated files. wasm="$input.$options.wasm" stdout_observed="$input.$options.stdout.observed" stderr_observed="$input.$options.stderr.observed" exit_status_observed="$input.$options.exit_status.observed" +run_args="" # Optionally load compiler options from a file. if [ -e "$input.options" ]; then @@ -24,16 +28,27 @@ else file_options= fi +if [ "$target" == "wasm32-wasi-threads" ]; then + pthread_options="-pthread" +else + pthread_options= +fi + echo "Testing $input..." # Compile the testcase. -$compiler $options $file_options "$input" -o "$wasm" +$compiler --target=$target $pthread_options $options $file_options "$input" -o "$wasm" # If we don't have a runwasi command, we're just doing compile-only testing. if [ "$runwasi" == "" ]; then exit 0 fi +if [ "$target" == "wasm32-wasi-preview2" -a -n "$adapter" -a -n "$wasm_tools" ]; then + "$wasm_tools" component new --adapt "$adapter" "$wasm" -o "$wasm" + run_args="--wasm component-model" +fi + # Determine the input file to write to stdin. if [ -e "$input.stdin" ]; then stdin="$input.stdin" @@ -59,7 +74,7 @@ fi # Run the test, capturing stdout, stderr, and the exit status. exit_status=0 -"$runwasi" $env $dir "$wasm" $dirarg \ +"$runwasi" $run_args $env $dir "$wasm" $dirarg \ < "$stdin" \ > "$stdout_observed" \ 2> "$stderr_observed" \ @@ -68,7 +83,11 @@ echo $exit_status > "$exit_status_observed" # Determine the reference files to compare with. if [ -e "$input.stdout.expected" ]; then - stdout_expected="$input.stdout.expected" + if [ -e "$input.$target.stdout.expected" ]; then + stdout_expected="$input.$target.stdout.expected" + else + stdout_expected="$input.stdout.expected" + fi # Apply output filters. if [ -e "$input.stdout.expected.filter" ]; then @@ -80,8 +99,13 @@ if [ -e "$input.stdout.expected" ]; then else stdout_expected="/dev/null" fi + if [ -e "$input.stderr.expected" ]; then - stderr_expected="$input.stderr.expected" + if [ -e "$input.$target.stderr.expected" ]; then + stderr_expected="$input.$target.stderr.expected" + else + stderr_expected="$input.stderr.expected" + fi # Apply output filters. if [ -e "$input.stderr.expected.filter" ]; then From f1858468e2a5fb9d4f2c7dcfc118324ae2263c9c Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Wed, 10 Jan 2024 13:29:41 -0800 Subject: [PATCH 035/165] doc: remember to tag wasi-libc during a release (#374) This change documents a final step called out in [wasi-libc#461]: tag the wasi-libc repository with the same wasi-sdk version tag. [wasi-libc#461]: https://github.com/WebAssembly/wasi-libc/issues/461 --- RELEASING.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/RELEASING.md b/RELEASING.md index 8c65cc8e8..9ad2cbbe4 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -46,15 +46,24 @@ To publish a new version of `wasi-sdk` as a GitHub release: following script simplifies the uploading of all the files and auto-generates the release description: - ```shell script - ci/draft-release.sh $TAG $ARTIFACTS_DIR $GITHUB_TOKEN - ``` + ```shell script + ci/draft-release.sh $TAG $ARTIFACTS_DIR $GITHUB_TOKEN + ``` [releases]: https://github.com/WebAssembly/wasi-sdk/releases 6. Publish the release; the previous step only creates a draft. Follow the link in the previous step or navigate to the GitHub [releases] to review the - description, commit, tag, and assets before clicking "Publish" + description, commit, tag, and assets before clicking "Publish." + +7. Remember to tag the wasi-libc repository with the new `$TAG` version. + + ```shell script + git submodule status -- src/wasi-libc # grab $WASI_LIBC_COMMIT from the output + cd $WASI_LIBC_REPO_DIR + git tag $TAG $WASI_LIBC_COMMIT + git push origin $TAG + ``` [^cache]: Here is an example of how to clear a cache with the GitHub CLI: From 9fbc7b5972950836a1bf2befb94d7d63cf9ead20 Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Thu, 11 Jan 2024 17:02:17 -0700 Subject: [PATCH 036/165] update wasi-libc and Makefile to fix `` (#375) As of LLVM 17, which includes https://reviews.llvm.org/D152168, libcxx has combined the old `_LIBCPP_HAS_NO_FILESYSTEM_LIBRARY` and `_LIBCPP_HAS_NO_FSTREAM` preprocessor symbols into a single `_LIBCPP_HAS_NO_FILESYSTEM` symbol, which means there's no longer any way to enable `` without enabling ``. The solution is to set `-DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON` and update `wasi-libc`, which includes stubs for the functions required by libcxx's `` implementation. Fixes #373 Signed-off-by: Joel Dice --- Makefile | 2 +- src/wasi-libc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 76d428141..4d26fd1c8 100644 --- a/Makefile +++ b/Makefile @@ -182,7 +182,7 @@ LIBCXX_CMAKE_FLAGS = \ -DLIBCXX_ENABLE_SHARED:BOOL=$(2) \ -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF \ -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF \ - -DLIBCXX_ENABLE_FILESYSTEM:BOOL=OFF \ + -DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON \ -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF \ -DLIBCXX_CXX_ABI=libcxxabi \ -DLIBCXX_CXX_ABI_INCLUDE_PATHS=$(LLVM_PROJ_DIR)/libcxxabi/include \ diff --git a/src/wasi-libc b/src/wasi-libc index 925ad6d75..cc62fa82c 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 925ad6d75899397d26b9f09a6f195dbf5eb35814 +Subproject commit cc62fa82c29bfced82d280cbccc231667ee80822 From a1b80857c4904d43e3823afd07480f64e62fab6d Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Fri, 12 Jan 2024 11:41:42 -0800 Subject: [PATCH 037/165] doc: fix release docs (#368) --- RELEASING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/RELEASING.md b/RELEASING.md index 9ad2cbbe4..d7eaef2f7 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -30,7 +30,7 @@ To publish a new version of `wasi-sdk` as a GitHub release: workflow completed successfully: ```shell script - ci/is-worfklow-valid.sh $TAG $WORKFLOW_RUN_ID $GITHUB_TOKEN + ci/is-workflow-valid.sh $TAG $WORKFLOW_RUN_ID $GITHUB_TOKEN ``` 4. Download and unzip the workflow artifacts. Note that artifacts with `+m` or @@ -39,7 +39,7 @@ To publish a new version of `wasi-sdk` as a GitHub release: Windows, MacOS, Linux). The following script does all of this automatically: ```shell script - ci/download-workflow-artifacts.sh $TAG $WORKFLOW_RUN_ID $GITHUB_TOKEN + ci/download-workflow-artifacts.sh $WORKFLOW_RUN_ID $GITHUB_TOKEN ``` 5. Draft a new release. This could be done [manually][releases] but the From bd96bf018c4ce346bb5abc6bff735827972a887c Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 9 Feb 2024 15:23:37 -0800 Subject: [PATCH 038/165] Update README to use the name "WASI SDK" consistently (#384) --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index b85471100..41adc2ffe 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz ## Use -Use the clang installed in the wasi-sdk directory: +Use the clang installed in the `wasi-sdk` directory: ```shell script export WASI_SDK_PATH=`pwd`/wasi-sdk-${WASI_VERSION_FULL} @@ -111,10 +111,10 @@ in the install directory. ## Docker Image -We provide a [docker image] including wasi-sdk that can be used for building +We provide a [docker image] including WASI SDK that can be used for building projects without a separate installation of the SDK. Autotools, CMake, and Ninja are included in this image, and standard environment variables are set to use -wasi-sdk for building. +WASI SDK for building. [docker image]: https://github.com/WebAssembly/wasi-sdk/pkgs/container/wasi-sdk @@ -127,7 +127,7 @@ docker run -v `pwd`:/src -w /src ghcr.io/webassembly/wasi-sdk make Take note of the [notable limitations](#notable-limitations) below when building projects, for example many projects will need threads support -disabled in a configure step before building with wasi-sdk. +disabled in a configure step before building with WASI SDK. ## Notable Limitations From 52536f2f690ef3773127a37ccfcd47c0b7d31724 Mon Sep 17 00:00:00 2001 From: Changqing Jing Date: Sat, 10 Feb 2024 09:22:32 +0800 Subject: [PATCH 039/165] Fix cmake_build_type typo (#382) * Fix cmake_build_type typo * fix abort.c test failed due to wasmtime version mismatch on CI --- .github/workflows/main.yml | 2 +- Makefile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 7489419f9..e50929b08 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -43,7 +43,7 @@ jobs: run: | curl -f -L --retry 5 https://wasmtime.dev/install.sh | bash -s -- --version dev ~/.wasmtime/bin/wasmtime --version - curl -f -L --retry 5 -o ~/.wasmtime/bin/wasi_snapshot_preview1.command.wasm https://github.com/bytecodealliance/wasmtime/releases/download/v16.0.0/wasi_snapshot_preview1.command.wasm + curl -f -L --retry 5 -o ~/.wasmtime/bin/wasi_snapshot_preview1.command.wasm https://github.com/bytecodealliance/wasmtime/releases/download/dev/wasi_snapshot_preview1.command.wasm if [ "${{ matrix.os }}" == "ubuntu-latest" ]; then curl -f -OL --retry 5 https://github.com/bytecodealliance/wasm-tools/releases/download/wasm-tools-1.0.54/wasm-tools-1.0.54-x86_64-linux.tar.gz tar xf wasm-tools-1.0.54-x86_64-linux.tar.gz diff --git a/Makefile b/Makefile index 4d26fd1c8..6afaa734f 100644 --- a/Makefile +++ b/Makefile @@ -178,7 +178,7 @@ LIBCXX_CMAKE_FLAGS = \ -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF \ -DLLVM_COMPILER_CHECKED=ON \ - -DCMAKE_BUILD_TYPE=RelWithDebugInfo \ + -DCMAKE_BUILD_TYPE=RelWithDebInfo \ -DLIBCXX_ENABLE_SHARED:BOOL=$(2) \ -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF \ -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF \ From f1ebc52a74394cdf885d03bfde13899b3d5c6d2d Mon Sep 17 00:00:00 2001 From: Andy Li Date: Sat, 10 Feb 2024 14:13:50 +0800 Subject: [PATCH 040/165] Clean up dependency packages in Docker (#380) --- docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index cbb503bfa..9fda47064 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -30,7 +30,7 @@ RUN apt-get update && \ \ apt-get update && \ apt-get install -y clang-${LLVM_VERSION} lld-${LLVM_VERSION} cmake ninja-build make autoconf autogen automake libtool && \ - apt-get remove -y curl gnupg && \ + apt-get autoremove -y curl gnupg && \ rm -rf /var/lib/apt/lists/* COPY --from=dist /wasi-sdk/share/wasi-sysroot/ /wasi-sysroot/ From c2f9c100b0346fb17e9b264448203310c0652921 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 6 Mar 2024 16:41:52 -0600 Subject: [PATCH 041/165] Start renaming preview1 to p1 and preview2 to p2 (#386) * Start renaming preview1 to p1 and preview2 to p2 This commit is a reflection of WebAssembly/wasi-libc#478 into this repository where a few changes are happening: * A new `wasm32-wasip1` sysroot is prepared matching `wasm32-wasi` * A new `wasm32-wasip1-threads` sysroot is prepared matching `wasm32-wasi-threads` * The `wasm32-wasi-preview2` target is renamed `wasm32-wasip2` I've done a bit of makefile refactoring to deduplicate things a bit now that there's a number of targets being built. The long-term goal would be to remove the `wasm32-wasi` and `wasm32-wasip1-threads` targets, but that's not proposed just yet at this time. * Use $(CLANG_VERSION) instead of hardcoding --- Makefile | 103 ++++++++++++++++++++-------------------------- src/wasi-libc | 2 +- tests/run.sh | 2 +- tests/testcase.sh | 2 +- 4 files changed, 48 insertions(+), 61 deletions(-) diff --git a/Makefile b/Makefile index 6afaa734f..03858eaa6 100644 --- a/Makefile +++ b/Makefile @@ -107,26 +107,22 @@ build/llvm.BUILT: llvm-config touch build/llvm.BUILT +# Flags for running `make` in wasi-libc +# $(1): the target that's being built +WASI_LIBC_MAKEFLAGS = \ + -C $(ROOT_DIR)/src/wasi-libc \ + CC=$(BUILD_PREFIX)/bin/clang \ + AR=$(BUILD_PREFIX)/bin/llvm-ar \ + NM=$(BUILD_PREFIX)/bin/llvm-nm \ + SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ + TARGET_TRIPLE=$(1) + build/wasi-libc.BUILT: build/compiler-rt.BUILT - $(MAKE) -C $(ROOT_DIR)/src/wasi-libc \ - CC=$(BUILD_PREFIX)/bin/clang \ - AR=$(BUILD_PREFIX)/bin/llvm-ar \ - NM=$(BUILD_PREFIX)/bin/llvm-nm \ - SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - WASI_SNAPSHOT=preview2 \ - default libc_so - $(MAKE) -C $(ROOT_DIR)/src/wasi-libc \ - CC=$(BUILD_PREFIX)/bin/clang \ - AR=$(BUILD_PREFIX)/bin/llvm-ar \ - NM=$(BUILD_PREFIX)/bin/llvm-nm \ - SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - default libc_so - $(MAKE) -C $(ROOT_DIR)/src/wasi-libc \ - CC=$(BUILD_PREFIX)/bin/clang \ - AR=$(BUILD_PREFIX)/bin/llvm-ar \ - NM=$(BUILD_PREFIX)/bin/llvm-nm \ - SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - THREAD_MODEL=posix + $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasi) default libc_so + $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip1) default libc_so + $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip2) WASI_SNAPSHOT=p2 default libc_so + $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasi-threads) THREAD_MODEL=posix + $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip1-threads) THREAD_MODEL=posix touch build/wasi-libc.BUILT build/compiler-rt.BUILT: build/llvm.BUILT @@ -156,11 +152,15 @@ build/compiler-rt.BUILT: build/llvm.BUILT DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/compiler-rt install # Install clang-provided headers. cp -R $(ROOT_DIR)/build/llvm/lib/clang $(BUILD_PREFIX)/lib/ + cp -R $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasi $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasip1 + cp -R $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasi $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasip2 touch build/compiler-rt.BUILT # Flags for libcxx and libcxxabi. # $(1): pthreads ON or OFF # $(2): shared libraries ON or OFF +# $(3): the name of the target being built for +# $(4): extra compiler flags to pass LIBCXX_CMAKE_FLAGS = \ -DCMAKE_C_COMPILER_WORKS=ON \ -DCMAKE_CXX_COMPILER_WORKS=ON \ @@ -199,47 +199,34 @@ LIBCXX_CMAKE_FLAGS = \ -DLIBCXXABI_ENABLE_PIC:BOOL=$(2) \ -DWASI_SDK_PREFIX=$(BUILD_PREFIX) \ -DUNIX:BOOL=ON \ - --debug-trycompile + --debug-trycompile \ + -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ + -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS) $(4) --target=$(3)" \ + -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS) $(4) --target=$(3)" \ + -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/$(3) \ + -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/$(3) \ + -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ + $(LLVM_PROJ_DIR)/runtimes + +# Rules to build libcxx, factored out here to deduplicate the below +# $(1): pthreads ON or OFF +# $(2): shared libraries ON or OFF +# $(3): the name of the target being built for +define BUILD_LIBCXX + mkdir -p build/libcxx-$(3) + cd build/libcxx-$(3) && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,$(1),$(2),$(3),$(4)) + ninja $(NINJA_FLAGS) -C build/libcxx-$(3) + DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx-$(3) install + rm -rf $(BUILD_PREFIX)/share/wasi-sysroot/include/$(3)/c++ + mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/$(3)/ +endef build/libcxx.BUILT: build/llvm.BUILT build/wasi-libc.BUILT - # Do the build. - mkdir -p build/libcxx-preview2 - cd build/libcxx-preview2 && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,OFF,ON) \ - -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS) --target=wasm32-wasi-preview2" \ - -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS) --target=wasm32-wasi-preview2" \ - -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-preview2 \ - -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-preview2 \ - -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ - $(LLVM_PROJ_DIR)/runtimes - ninja $(NINJA_FLAGS) -C build/libcxx-preview2 - mkdir -p build/libcxx - cd build/libcxx && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,OFF,ON) \ - -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS) --target=wasm32-wasi" \ - -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS) --target=wasm32-wasi" \ - -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi \ - -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi \ - -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ - $(LLVM_PROJ_DIR)/runtimes - ninja $(NINJA_FLAGS) -C build/libcxx - mkdir -p build/libcxx-threads - cd build/libcxx-threads && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,ON,OFF) \ - -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CFLAGS) --target=wasm32-wasi-threads" \ - -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) -pthread $(EXTRA_CXXFLAGS) --target=wasm32-wasi-threads" \ - -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-threads \ - -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/wasm32-wasi-threads \ - -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ - $(LLVM_PROJ_DIR)/runtimes - ninja $(NINJA_FLAGS) -C build/libcxx-threads - # Do the install. - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx-preview2 install - mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/wasm32-wasi-preview2/ - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx install - mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/wasm32-wasi/ - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx-threads install - mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/wasm32-wasi-threads/ + $(call BUILD_LIBCXX,OFF,ON,wasm32-wasi) + $(call BUILD_LIBCXX,OFF,ON,wasm32-wasip1) + $(call BUILD_LIBCXX,OFF,ON,wasm32-wasip2) + $(call BUILD_LIBCXX,ON,OFF,wasm32-wasi-threads,-pthread) + $(call BUILD_LIBCXX,ON,OFF,wasm32-wasip1-threads,-pthread) # As of this writing, `clang++` will ignore the above include dirs unless this one also exists: mkdir -p $(BUILD_PREFIX)/share/wasi-sysroot/include/c++/v1 touch build/libcxx.BUILT diff --git a/src/wasi-libc b/src/wasi-libc index cc62fa82c..c9c7d0616 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit cc62fa82c29bfced82d280cbccc231667ee80822 +Subproject commit c9c7d0616e0f7fe4d0d0fa54372b6d1f5689f91d diff --git a/tests/run.sh b/tests/run.sh index 62d05e9ce..1d41ff1b5 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -44,7 +44,7 @@ echo $CC echo $CXX echo "SDK: $wasi_sdk" -for target in wasm32-wasi wasm32-wasi-threads wasm32-wasi-preview2; do +for target in wasm32-wasi wasm32-wasip1 wasm32-wasi-threads wasm32-wasip1-threads wasm32-wasip2; do echo "===== Testing target $target =====" cd $testdir/compile-only for options in -O0 -O2 "-O2 -flto"; do diff --git a/tests/testcase.sh b/tests/testcase.sh index d7368e09c..285424ff0 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -28,7 +28,7 @@ else file_options= fi -if [ "$target" == "wasm32-wasi-threads" ]; then +if echo "$target" | grep -q -- '-threads$'; then pthread_options="-pthread" else pthread_options= From 73dbb2fe8971bc7c04fd4ecbd2d5359b2f29c465 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 8 Mar 2024 08:11:04 -0600 Subject: [PATCH 042/165] Refactor installation of wasmtime/wasm-tools on CI (#387) * Refactor installation of wasmtime/wasm-tools on CI We've got official actions for installation Wasmtime and wasm-tools in the bytecodealliance organization which should help make this installation step a bit more readable. * Change the directory of the adapter * Update descriptions --- .github/workflows/main.yml | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e50929b08..17ab9885e 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -38,21 +38,16 @@ jobs: restore-keys: | 0-cache-macos-latest if: matrix.os == 'macos-latest' - - name: Install wasmtime for tests - # TODO: switch to Wasmtime 17 once it's released, which will include https://github.com/bytecodealliance/wasmtime/pull/7750 - run: | - curl -f -L --retry 5 https://wasmtime.dev/install.sh | bash -s -- --version dev - ~/.wasmtime/bin/wasmtime --version - curl -f -L --retry 5 -o ~/.wasmtime/bin/wasi_snapshot_preview1.command.wasm https://github.com/bytecodealliance/wasmtime/releases/download/dev/wasi_snapshot_preview1.command.wasm - if [ "${{ matrix.os }}" == "ubuntu-latest" ]; then - curl -f -OL --retry 5 https://github.com/bytecodealliance/wasm-tools/releases/download/wasm-tools-1.0.54/wasm-tools-1.0.54-x86_64-linux.tar.gz - tar xf wasm-tools-1.0.54-x86_64-linux.tar.gz - cp wasm-tools-1.0.54-x86_64-linux/wasm-tools ~/.wasmtime/bin/ - else - curl -f -OL --retry 5 https://github.com/bytecodealliance/wasm-tools/releases/download/wasm-tools-1.0.54/wasm-tools-1.0.54-x86_64-macos.tar.gz - tar xf wasm-tools-1.0.54-x86_64-macos.tar.gz - cp wasm-tools-1.0.54-x86_64-macos/wasm-tools ~/.wasmtime/bin/ - fi + - name: Setup `wasmtime` for tests + uses: bytecodealliance/actions/wasmtime/setup@v1 + with: + version: "18.0.2" + - name: Setup `wasm-tools` for tests + uses: bytecodealliance/actions/wasm-tools/setup@v1 + with: + version: "1.201.0" + - name: Download command adapter for tests + run: curl -f -L --retry 5 -o ${{ runner.temp }}/wasi_snapshot_preview1.command.wasm https://github.com/bytecodealliance/wasmtime/releases/download/v18.0.2/wasi_snapshot_preview1.command.wasm - uses: actions/checkout@v3 with: fetch-depth: 0 @@ -72,7 +67,7 @@ jobs: run: NINJA_FLAGS=-v make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON shell: bash - name: Run the testsuite - run: NINJA_FLAGS=-v make check RUNTIME=~/.wasmtime/bin/wasmtime ADAPTER=~/.wasmtime/bin/wasi_snapshot_preview1.command.wasm WASM_TOOLS=~/.wasmtime/bin/wasm-tools + run: NINJA_FLAGS=-v make check RUNTIME=wasmtime ADAPTER=${{ runner.temp }}/wasi_snapshot_preview1.command.wasm WASM_TOOLS=wasm-tools - name: Upload artifacts uses: actions/upload-artifact@v1 with: From 3e93db0b189741fe32733fbb00383bff8c0c9d08 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 8 Mar 2024 13:36:29 -0600 Subject: [PATCH 043/165] Add target-prefixed executables to the installation `bin` directory (#388) Currently the `bin` directory installed by wasi-sdk is not currently suitable for putting in `$PATH` in all cases because it can shadow a system-installed `clang` executable which is intended for native binaries. For Linux distributions with gcc-based cross-compilers I've often seen the pattern where they're installed as `$target-gcc` and so I've taken a leaf out of their books to do that here as well. This commit adds, currently alongside the preexisting `clang` executable, target-prefixed executables such as `wasm32-wasi-clang` and `wasm32-wasi-clang++`. These executables are symlinks to the `clang` executable itself in the same manner that `clang++` is a symlink to `clang` itself. I'll also note that this doesn't fix the problem of "add the wasi-sdk bin dir to PATH" because `clang` and other prominent executables are still there. My hope though is that this opens up a future refactoring for doing so. --- Makefile | 21 ++++++++++++++++++--- tests/run.sh | 34 ++++++++++++++++------------------ tests/testcase.sh | 6 +++--- 3 files changed, 37 insertions(+), 24 deletions(-) diff --git a/Makefile b/Makefile index 03858eaa6..c17094da3 100644 --- a/Makefile +++ b/Makefile @@ -42,6 +42,8 @@ override LLVM_CMAKE_FLAGS += -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 endif +TARGETS = wasm32-wasi wasm32-wasip1 wasm32-wasip2 wasm32-wasip1-threads wasm32-wasi-threads + # Only the major version is needed for Clang, see https://reviews.llvm.org/D125860. CLANG_VERSION=$(shell $(BASH) ./llvm_version_major.sh $(LLVM_PROJ_DIR)) VERSION:=$(shell $(BASH) ./version.sh) @@ -51,16 +53,29 @@ default: build @echo "Use -fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION)" check: - CC="clang --sysroot=$(BUILD_PREFIX)/share/wasi-sysroot" \ - CXX="clang++ --sysroot=$(BUILD_PREFIX)/share/wasi-sysroot -fno-exceptions" \ - PATH="$(PATH_PREFIX)/bin:$$PATH" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" "$(ADAPTER)" "$(WASM_TOOLS)" + TARGETS="$(TARGETS)" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" "$(ADAPTER)" "$(WASM_TOOLS)" clean: rm -rf build $(DESTDIR) +# Default symlinks that clang creates to the `clang` executable +CLANG_LINKS_TO_CREATE = clang++ clang-cl clang-cpp + +# Add target-prefixed versions of `clang` and `clang++` so they can be used +# without `--target` as it's auto-inferred from the executable name by clang. +CLANG_LINKS_TO_CREATE += $(foreach target,$(TARGETS),$(target)-clang) +CLANG_LINKS_TO_CREATE += $(foreach target,$(TARGETS),$(target)-clang++) + +# Small helper to create a `join-with` function that can join elements of a +# list with a defined separator. +noop = +space = $(noop) $(noop) +join-with = $(subst $(space),$1,$(strip $2)) + build/llvm.BUILT: mkdir -p build/llvm cd build/llvm && cmake -G Ninja \ + -DCLANG_LINKS_TO_CREATE="$(call join-with,;,$(CLANG_LINKS_TO_CREATE))" \ -DCMAKE_BUILD_TYPE=MinSizeRel \ -DLLVM_ENABLE_TERMINFO=OFF \ -DLLVM_ENABLE_ZLIB=OFF \ diff --git a/tests/run.sh b/tests/run.sh index 1d41ff1b5..080eeaee5 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -6,13 +6,7 @@ set -ueo pipefail # is a WASI-capable runtime to run the tests in full compile and # execute mode. # -# By default this script will look for `clang` and `clang++` in $PATH and -# assume that they are correctly configured with the sysroot in the default -# location. Alternatively, exporting $CC and $CXX allow more flexibility. e.g: -# -# export CXX="/bin/clang++ --sysroot /share/wasi-sysroot" -# export CC="/bin/clang --sysroot /share/wasi-sysroot" -# +# The compiler used during testing is loaded from ``. if [ $# -lt 1 ]; then echo "Path to WASI SDK is required" exit 1 @@ -37,44 +31,48 @@ else fi testdir=$(dirname $0) -CC=${CC:=clang} -CXX=${CXX:=clang++} -echo $CC -echo $CXX echo "SDK: $wasi_sdk" -for target in wasm32-wasi wasm32-wasip1 wasm32-wasi-threads wasm32-wasip1-threads wasm32-wasip2; do +# NB: all tests are run with the default `clang` and `clang++` executables +# but they're also executed with the target-prefixed `clang` executables to +# ensure that those work as well when the `--target` option is omitted. + +for target in $TARGETS; do echo "===== Testing target $target =====" cd $testdir/compile-only for options in -O0 -O2 "-O2 -flto"; do echo "===== Testing compile-only with $options =====" for file in *.c; do echo "Testing compile-only $file..." - ../testcase.sh "$target" "" "" "" "$CC" "$options" "$file" + ../testcase.sh "$target" "" "" "" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" + ../testcase.sh "$target" "" "" "" "$wasi_sdk/bin/$target-clang" "$options" "$file" done for file in *.cc; do echo "Testing compile-only $file..." - ../testcase.sh "$target" "" "" "" "$CXX" "$options" "$file" + ../testcase.sh "$target" "" "" "" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" + ../testcase.sh "$target" "" "" "" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" done done cd - >/dev/null - + cd $testdir/general for options in -O0 -O2 "-O2 -flto"; do echo "===== Testing with $options =====" for file in *.c; do echo "Testing $file..." - ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$CC" "$options" "$file" + ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" + ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$wasi_sdk/bin/$target-clang" "$options" "$file" done for file in *.cc; do echo "Testing $file..." - ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$CXX" "$options" "$file" + ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" + ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" done done cd - >/dev/null done - + # Test cmake build system for wasi-sdk test_cmake() { local option diff --git a/tests/testcase.sh b/tests/testcase.sh index 285424ff0..b646e2d4a 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -37,7 +37,7 @@ fi echo "Testing $input..." # Compile the testcase. -$compiler --target=$target $pthread_options $options $file_options "$input" -o "$wasm" +$compiler $pthread_options $options $file_options "$input" -o "$wasm" # If we don't have a runwasi command, we're just doing compile-only testing. if [ "$runwasi" == "" ]; then @@ -87,7 +87,7 @@ if [ -e "$input.stdout.expected" ]; then stdout_expected="$input.$target.stdout.expected" else stdout_expected="$input.stdout.expected" - fi + fi # Apply output filters. if [ -e "$input.stdout.expected.filter" ]; then @@ -105,7 +105,7 @@ if [ -e "$input.stderr.expected" ]; then stderr_expected="$input.$target.stderr.expected" else stderr_expected="$input.stderr.expected" - fi + fi # Apply output filters. if [ -e "$input.stderr.expected.filter" ]; then From 2d7853c64b0b118d71764b132df1593c2c7545bf Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 8 Mar 2024 17:36:40 -0600 Subject: [PATCH 044/165] Bump to LLVM 18.1.0 (#393) * Explicitly set LIBCXXABI_USE_LLVM_UNWINDER off Clang trunk has recently turned it on, and also now fails the build when libunwind is not built at the same time. * Bump LLVM to 18.1.0 --------- Co-authored-by: Mike Hommey --- Makefile | 2 +- src/llvm-project | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index c17094da3..b30a62a3b 100644 --- a/Makefile +++ b/Makefile @@ -81,7 +81,6 @@ build/llvm.BUILT: -DLLVM_ENABLE_ZLIB=OFF \ -DLLVM_ENABLE_ZSTD=OFF \ -DLLVM_STATIC_LINK_CXX_STDLIB=ON \ - -DLLVM_HAVE_LIBXAR=OFF \ -DCMAKE_INSTALL_PREFIX=$(PREFIX) \ -DLLVM_INCLUDE_TESTS=OFF \ -DLLVM_INCLUDE_UTILS=OFF \ @@ -212,6 +211,7 @@ LIBCXX_CMAKE_FLAGS = \ -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF \ -DLIBCXXABI_ENABLE_PIC:BOOL=$(2) \ + -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=OFF \ -DWASI_SDK_PREFIX=$(BUILD_PREFIX) \ -DUNIX:BOOL=ON \ --debug-trycompile \ diff --git a/src/llvm-project b/src/llvm-project index 6009708b4..461274b81 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 6009708b4367171ccdbf4b5905cb6a803753fe18 +Subproject commit 461274b81d8641eab64d494accddc81d7db8a09e From 1f63274f2c857d1b230b0d2d5d7bec7c47ccfc61 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 8 Mar 2024 17:47:36 -0600 Subject: [PATCH 045/165] Update versions of actions used on CI (#396) Resolve some warnings about using Node 16 in actions as it's being deprecated. The new versions should be all on Node 20 now. --- .github/workflows/main.yml | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 17ab9885e..4303d08d6 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -20,7 +20,7 @@ jobs: - ubuntu-latest - macos-latest steps: - - uses: actions/cache@v3 + - uses: actions/cache@v4 with: path: ~/.cache/ccache # Bump the prefix number to evict all previous caches and @@ -31,7 +31,7 @@ jobs: restore-keys: | 0-cache-ubuntu-latest if: matrix.os == 'ubuntu-latest' - - uses: actions/cache@v3 + - uses: actions/cache@v4 with: path: ~/Library/Caches/ccache key: 0-cache-macos-latest-${{ github.run_id }} @@ -48,7 +48,7 @@ jobs: version: "1.201.0" - name: Download command adapter for tests run: curl -f -L --retry 5 -o ${{ runner.temp }}/wasi_snapshot_preview1.command.wasm https://github.com/bytecodealliance/wasmtime/releases/download/v18.0.2/wasi_snapshot_preview1.command.wasm - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: fetch-depth: 0 # We can't use `--depth 1` here sadly because the GNU config @@ -69,7 +69,7 @@ jobs: - name: Run the testsuite run: NINJA_FLAGS=-v make check RUNTIME=wasmtime ADAPTER=${{ runner.temp }}/wasi_snapshot_preview1.command.wasm WASM_TOOLS=wasm-tools - name: Upload artifacts - uses: actions/upload-artifact@v1 + uses: actions/upload-artifact@v4 with: # Upload the dist folder. Give it a name according to the OS it was built for. name: ${{ format( 'dist-{0}', matrix.os) }} @@ -89,7 +89,7 @@ jobs: sys: clang32 env: clang-i686 steps: - - uses: actions/cache@v3 + - uses: actions/cache@v4 with: path: ~/AppData/Local/ccache key: 0-${{ format( 'cache-windows-latest-{0}', matrix.arch) }}-${{ github.run_id }} @@ -107,7 +107,7 @@ jobs: msystem: ${{ matrix.sys }} update: true release: false - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: fetch-depth: 0 - run: git submodule update --init --depth 32 --jobs 3 @@ -122,7 +122,7 @@ jobs: C:\wasi-sdk\bin\llvm-ar.exe --version C:\wasi-sdk\bin\wasm-ld.exe --version - name: Upload artifacts - uses: actions/upload-artifact@v3 + uses: actions/upload-artifact@v4 with: # Upload the dist folder. Give it a name according to the OS it was built for. name: ${{ format( 'dist-windows-latest-{0}', matrix.arch) }} @@ -132,14 +132,14 @@ jobs: name: Docker Build runs-on: ubuntu-latest steps: - - uses: actions/cache@v3 + - uses: actions/cache@v4 with: path: ~/.ccache key: 0-cache-ubuntu-bionic-${{ github.run_id }} restore-keys: | 0-cache-ubuntu-bionic - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: fetch-depth: 0 @@ -168,7 +168,7 @@ jobs: - name: Run docker_build script run: ./docker_build.sh - name: Upload artifacts - uses: actions/upload-artifact@v1 + uses: actions/upload-artifact@v4 with: # Upload the dist folder. Give it a name according to the OS it was built for. name: dist-ubuntu-bionic From 9389ea5eeec98afc61039683ae92c6147fee9c54 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Sat, 9 Mar 2024 15:49:15 -0600 Subject: [PATCH 046/165] Fix a flaky test failure in getentropy.c (#398) This test looks to be asserting that `getrandom` never returns 256 consecutive zeros, but the way it's asserting that is summing up the bytes and asserting the sum is nonzero. Due to this being a signed addition, however, it's possible for the bytes to be nonzero and still trigger the assert. Locally running this test in a loop took 30 or so seconds before it triggered a failure. I've updated the test to instead hunt for any entry which is not equal to zero and then assert that something is not zero. --- tests/general/getentropy.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/general/getentropy.c b/tests/general/getentropy.c index 3b7fe0e6c..d52bfbc7a 100644 --- a/tests/general/getentropy.c +++ b/tests/general/getentropy.c @@ -1,17 +1,19 @@ #include #include +#include int main() { char buf[256] = {0}; int ret = getentropy(buf, 256); assert(ret == 0); - int sum = 0; + bool something_nonzero = false; for (int i = 0; i < 256; i++) { - sum += buf[i]; + if (buf[i] != 0) + something_nonzero = true; } - assert(sum != 0); + assert(something_nonzero); return 0; } From 96bbd95197cf24cf070a04aa4a7bbc6115ae5df3 Mon Sep 17 00:00:00 2001 From: Mike Hommey Date: Wed, 13 Mar 2024 08:37:09 +0900 Subject: [PATCH 047/165] Support getting LLVM version from its new location (#399) https://github.com/llvm/llvm-project/pull/84641 changed its location. --- llvm_version_major.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/llvm_version_major.sh b/llvm_version_major.sh index 2e5d973ca..93fe9b01b 100755 --- a/llvm_version_major.sh +++ b/llvm_version_major.sh @@ -1,4 +1,4 @@ #/bin/bash LLVM_PROJ_DIR=${1:-./src/llvm-project} -MAJOR=`grep "set(LLVM_VERSION_MAJOR" $LLVM_PROJ_DIR/llvm/CMakeLists.txt | awk '{print substr($2, 1, length($2) - 1)}'` +MAJOR=`(grep "set(LLVM_VERSION_MAJOR" $LLVM_PROJ_DIR/llvm/CMakeLists.txt || grep "set(LLVM_VERSION_MAJOR" $LLVM_PROJ_DIR/cmake/Modules/LLVMVersion.cmake) | awk '{print substr($2, 1, length($2) - 1)}'` echo $MAJOR From 2436b99f1d59becb2d2e6b3e11dd6e48903a6adb Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Mon, 18 Mar 2024 13:50:58 -0700 Subject: [PATCH 048/165] Replace multiple version scripts with a Python script In the past, I've had trouble keeping the current set of version scripts to correctly output the version, especially in light of how we append Git suffixes for a non-tagged commit and dirty state. This change replaces those scripts with a single Python one which, though much more wordy than the previous one, may be easier for contributors to read and modify. The original scripts relied on Perl; this one relies on Python, which seems like a fair exchange. Having a single script also makes it easier to solve #372. --- Makefile | 5 +- RELEASING.md | 2 +- deb_from_installation.sh | 2 +- llvm_version_major.sh | 4 -- tar_from_installation.sh | 2 +- version.py | 122 +++++++++++++++++++++++++++++++++++++++ version.sh | 6 -- 7 files changed, 128 insertions(+), 15 deletions(-) delete mode 100755 llvm_version_major.sh create mode 100755 version.py delete mode 100755 version.sh diff --git a/Makefile b/Makefile index b30a62a3b..cf3ccb17c 100644 --- a/Makefile +++ b/Makefile @@ -3,6 +3,7 @@ ROOT_DIR=${CURDIR} LLVM_PROJ_DIR?=$(ROOT_DIR)/src/llvm-project +VERSION_SCRIPT=python3 ./version.py # Windows needs munging ifeq ($(OS),Windows_NT) @@ -45,8 +46,8 @@ endif TARGETS = wasm32-wasi wasm32-wasip1 wasm32-wasip2 wasm32-wasip1-threads wasm32-wasi-threads # Only the major version is needed for Clang, see https://reviews.llvm.org/D125860. -CLANG_VERSION=$(shell $(BASH) ./llvm_version_major.sh $(LLVM_PROJ_DIR)) -VERSION:=$(shell $(BASH) ./version.sh) +CLANG_VERSION=$(shell $(VERSION_SCRIPT) llvm-major --llvm-dir=$(LLVM_PROJ_DIR)) +VERSION:=$(shell $(VERSION_SCRIPT)) DEBUG_PREFIX_MAP=-fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION) default: build diff --git a/RELEASING.md b/RELEASING.md index d7eaef2f7..c684149be 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -3,7 +3,7 @@ To publish a new version of `wasi-sdk` as a GitHub release: 1. Tag a commit with an annotated tag. Note that this must be an annotated tag, - not a lightweight tag, so that `version.sh` can use it for calculating the + not a lightweight tag, so that `version.py` can use it for calculating the package version (use `git show wasi-sdk-...` to show other tag messages). Note that you may need to clear the repository cache to avoid problems with cached artifacts [^cache]. diff --git a/deb_from_installation.sh b/deb_from_installation.sh index 2c1596163..2699c819c 100755 --- a/deb_from_installation.sh +++ b/deb_from_installation.sh @@ -18,7 +18,7 @@ fi if [ -n "$2" ]; then VERSION="$2" else - VERSION=`./version.sh` + VERSION=`./version.py` fi if [ -n "$3" ]; then diff --git a/llvm_version_major.sh b/llvm_version_major.sh deleted file mode 100755 index 93fe9b01b..000000000 --- a/llvm_version_major.sh +++ /dev/null @@ -1,4 +0,0 @@ -#/bin/bash -LLVM_PROJ_DIR=${1:-./src/llvm-project} -MAJOR=`(grep "set(LLVM_VERSION_MAJOR" $LLVM_PROJ_DIR/llvm/CMakeLists.txt || grep "set(LLVM_VERSION_MAJOR" $LLVM_PROJ_DIR/cmake/Modules/LLVMVersion.cmake) | awk '{print substr($2, 1, length($2) - 1)}'` -echo $MAJOR diff --git a/tar_from_installation.sh b/tar_from_installation.sh index f90000bf5..7d0943266 100755 --- a/tar_from_installation.sh +++ b/tar_from_installation.sh @@ -10,7 +10,7 @@ fi if [ -n "$2" ]; then VERSION="$2" else - VERSION=`./version.sh` + VERSION=`./version.py` fi if [ -n "$3" ]; then diff --git a/version.py b/version.py new file mode 100755 index 000000000..52dc07ab1 --- /dev/null +++ b/version.py @@ -0,0 +1,122 @@ +#!/usr/bin/env python3 + +# This script finds and prints the various versions in this project: wasi-sdk +# itself, LLVM, and the Git revisions of dependencies. +# +# Usage: version [wasi-sdk|llvm|llvm-major|dump] [--llvm-dir=] + +import sys +import argparse +import subprocess + +# The number of characters to use for the abbreviated Git revision. +GIT_REF_LEN = 12 + + +def exec(command, cwd=None): + result = subprocess.run(command, stdout=subprocess.PIPE, + universal_newlines=True, check=True, cwd=cwd) + return result.stdout.strip() + + +def git_commit(dir='.'): + return exec(['git', 'rev-parse', f'--short={GIT_REF_LEN}', 'HEAD'], dir) + + +def parse_git_version(version): + # Parse, e.g.: wasi-sdk-21-0-g317548590b40+m + parts = version.replace('+', '-').split('-') + assert parts.pop(0) == 'wasi' + assert parts.pop(0) == 'sdk' + + major, minor = parts.pop(0), parts.pop(0) + git = None + dirty = False + + if parts: + # Check: git|dirty. + next = parts.pop(0) + if next == 'm': + dirty = True + else: + git = next[1:] + + # Check: dirty. + if parts: + assert parts.pop(0) == 'm', f'expected dirty flag: +m' + dirty = True + + assert not parts, f'unexpected suffixes: {parts}' + return major, minor, git, dirty + + +# Some inline tests to check Git version parsing: +assert parse_git_version( + 'wasi-sdk-21-0-g317548590b40+m') == ('21', '0', '317548590b40', True) +assert parse_git_version('wasi-sdk-21-2+m') == ('21', '2', None, True) +assert parse_git_version( + 'wasi-sdk-23-0-g317548590b40') == ('23', '0', '317548590b40', False) + + +def git_version(): + version = exec(['git', 'describe', '--long', '--candidates=999', + '--match=wasi-sdk-*', '--dirty=+m', f'--abbrev={GIT_REF_LEN}']) + major, minor, git, dirty = parse_git_version(version) + version = f'{major}.{minor}' + if git: + version += f'g{git}' + if dirty: + version += '+m' + return version + + +def parse_cmake_set(line): + return line.split(' ')[1].split(')')[0] + + +def llvm_cmake_version(llvm_dir): + with open(f'{llvm_dir}/llvm/CMakeLists.txt') as file: + for line in file: + line = line.strip() + if line.startswith('set(LLVM_VERSION_MAJOR'): + llvm_version_major = parse_cmake_set(line) + elif line.startswith('set(LLVM_VERSION_MINOR'): + llvm_version_minor = parse_cmake_set(line) + elif line.startswith('set(LLVM_VERSION_PATCH'): + llvm_version_patch = parse_cmake_set(line) + return llvm_version_major, llvm_version_minor, llvm_version_patch + + +def main(action, llvm_dir): + if action == 'wasi-sdk': + print(git_version()) + elif action == 'llvm': + major, minor, path = llvm_cmake_version(llvm_dir) + print(f'{major}.{minor}.{path}') + elif action == 'llvm-major': + major, _, _ = llvm_cmake_version(llvm_dir) + print(major) + elif action == 'dump': + print(git_version()) + print(f'wasi-libc: {git_commit("src/wasi-libc")}') + print(f'llvm: {git_commit(llvm_dir)}') + major, minor, path = llvm_cmake_version(llvm_dir) + print(f'llvm-version: {major}.{minor}.{path}') + print(f'config: {git_commit("src/config")}') + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description='Print the various kinds of versions in wasi-sdk') + parser.add_argument('action', + choices=['wasi-sdk', 'llvm', 'llvm-major', 'dump'], + nargs='?', + default='wasi-sdk', + help='Which kind of version to print (default: wasi-sdk).') + parser.add_argument('--llvm-dir', + nargs='?', + default='src/llvm-project', + help='Override the location of the LLVM source directory (default: src/llvm-project).') + args = parser.parse_args() + main(args.action, args.llvm_dir) + sys.exit(0) diff --git a/version.sh b/version.sh deleted file mode 100755 index 35c8a908e..000000000 --- a/version.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/usr/bin/env bash -set -e -git config --global --add safe.directory "/workspace" -GIT_DESCR=$(git describe --long --candidates=999 --match='wasi-sdk-*' --dirty='+m' --abbrev=12) -GIT_PACKAGE_VERSION=$(echo $GIT_DESCR | perl -ne 'if(/^wasi-sdk-(\d+)-(\d+)-g([0-9a-f]{7,12})([+]m)?$/) { if($2 == 0) { print "$1.$2$4" } else { print "$1.$2g$3$4" } exit } else { print "could not parse git description"; exit 1 }';) -echo $GIT_PACKAGE_VERSION From 0409839729fb4348764905a7e8c2bb53000087a3 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Mon, 18 Mar 2024 13:55:57 -0700 Subject: [PATCH 049/165] Dump a `VERSION` file in the build directory As noted in #372, some users may need to identify which version of wasi-sdk they are using after it has been downloaded. There are many ways to solve this, but the one I chose here is to dump the wasi-sdk version into a `VERSION` file and follow that up with several dependency versions that may be helpful to note when troubleshooting. Ideally someone could paste the contents of that file when filing a bug. If we adopt this approach, this fixes #372. --- Makefile | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index cf3ccb17c..9c7acffbf 100644 --- a/Makefile +++ b/Makefile @@ -256,7 +256,11 @@ build/config.BUILT: cp cmake/Platform/WASI.cmake $(BUILD_PREFIX)/share/cmake/Platform touch build/config.BUILT -build: build/llvm.BUILT build/wasi-libc.BUILT build/compiler-rt.BUILT build/libcxx.BUILT build/config.BUILT +build/version.BUILT: + $(VERSION_SCRIPT) dump > $(BUILD_PREFIX)/VERSION + touch build/version.BUILT + +build: build/llvm.BUILT build/wasi-libc.BUILT build/compiler-rt.BUILT build/libcxx.BUILT build/config.BUILT build/version.BUILT strip: build/llvm.BUILT ./strip_symbols.sh $(BUILD_PREFIX)/bin From c376c9cfbd9fa6d12716e78ef5d6b4a58bab6bf5 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Mon, 18 Mar 2024 13:57:42 -0700 Subject: [PATCH 050/165] Thread UID/GID through Docker When running Git commands inside this Docker container (i.e., commands that the `version.py` script needs for determining version information), the Docker build would run into issues like: ``` fatal: detected dubious ownership in repository at '/workspace' To add an exception for this directory, call: git config --global --add safe.directory /workspace ``` This is due to an extra Git check that detects that the Docker user is not the same one who owns the `.git` directory of this project. After looking into this, the best solution the internet has to offer is to thread the current user's UID and GID through the Docker image (i.e., the new `builder` user) and then `docker run --user ...`. This both avoids the Git check but also seems to be considered a best practice in some circles (?). --- Dockerfile | 32 +++++++++++++++++++++----------- docker_build.sh | 15 +++++++++++++-- 2 files changed, 34 insertions(+), 13 deletions(-) diff --git a/Dockerfile b/Dockerfile index 558c1b285..9a864e63e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -3,18 +3,23 @@ # Here we choose Bionic 18.04. FROM ubuntu:bionic +# We want to use the same UID/GID of the external user to avoid permission +# issues. See the user setup at the end of the file. +ARG UID=1000 +ARG GID=1000 + RUN apt-get update \ - && apt-get install -y --no-install-recommends \ - ccache \ - curl \ - ca-certificates \ - build-essential \ - clang \ - python3 \ - git \ - ninja-build \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* + && apt-get install -y --no-install-recommends \ + ccache \ + curl \ + ca-certificates \ + build-essential \ + clang \ + python3 \ + git \ + ninja-build \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.25.1/cmake-3.25.1-linux-x86_64.tar.gz \ && tar xf cmake-3.25.1-linux-x86_64.tar.gz \ @@ -22,3 +27,8 @@ RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.25.1/cmake- && mkdir -p /opt \ && mv cmake-3.25.1-linux-x86_64 /opt/cmake ENV PATH /opt/cmake/bin:$PATH + +RUN groupadd -g ${GID} builder && \ + useradd --create-home --uid ${UID} --gid ${GID} builder +USER builder +WORKDIR /workspace diff --git a/docker_build.sh b/docker_build.sh index 38845277f..050862fac 100755 --- a/docker_build.sh +++ b/docker_build.sh @@ -1,7 +1,18 @@ #!/bin/sh set -ex + echo "Building the docker image" -docker build -t wasi-sdk-builder:latest . +docker build \ + --build-arg UID=$(id -u) --build-arg GID=$(id -g) \ + -t wasi-sdk-builder:latest . + echo "Building the package in docker image" mkdir -p ~/.ccache -docker run --rm -v "$PWD":/workspace -v ~/.ccache:/root/.ccache -e NINJA_FLAGS=-v --workdir /workspace --tmpfs /tmp:exec wasi-sdk-builder:latest make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON +docker run --rm \ + --user $(id -u):$(id -g) \ + -v "$PWD":/workspace:Z \ + -v ~/.ccache:/home/builder/.ccache:Z \ + -e NINJA_FLAGS=-v \ + --tmpfs /tmp:exec \ + wasi-sdk-builder:latest \ + make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON From a50a641f4b5a1398fcc811264272aae5bdd3b763 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Mon, 18 Mar 2024 14:08:40 -0700 Subject: [PATCH 051/165] Handle new LLVM version path In #399, @glandium points out that newer versions of LLVM will place their version information at a different path. This change adapts #399 to the new Python version script. --- version.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/version.py b/version.py index 52dc07ab1..2d9de6d5b 100755 --- a/version.py +++ b/version.py @@ -5,9 +5,10 @@ # # Usage: version [wasi-sdk|llvm|llvm-major|dump] [--llvm-dir=] -import sys import argparse +import os import subprocess +import sys # The number of characters to use for the abbreviated Git revision. GIT_REF_LEN = 12 @@ -75,7 +76,11 @@ def parse_cmake_set(line): def llvm_cmake_version(llvm_dir): - with open(f'{llvm_dir}/llvm/CMakeLists.txt') as file: + path = f'{llvm_dir}/cmake/Modules/LLVMVersion.cmake' + if not os.path.exists(path): + # Handle older LLVM versions; see #399. + path = f'{llvm_dir}/llvm/CMakeLists.txt' + with open(path) as file: for line in file: line = line.strip() if line.startswith('set(LLVM_VERSION_MAJOR'): From a7b2182949d932b3a6dc36f1f4de5e19b245712a Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Wed, 3 Apr 2024 08:42:06 -0600 Subject: [PATCH 052/165] update wasi-libc and fix out-of-date tests (#403) This updates the `wasi-libc` submodule to point to `main` and also: - update tests/testcase.sh to use new wasm32-wasip2 target name - update various tests/**/*.expected files to match Wasmtime output Signed-off-by: Joel Dice --- src/wasi-libc | 2 +- tests/general/abort.c.wasm32-wasi.stderr.expected | 6 ++++++ tests/general/abort.c.wasm32-wasip2.stderr.expected | 6 ++++++ tests/general/assert-fail.c.wasm32-wasip2.stderr.expected | 7 +++++++ tests/general/sigabrt.c.wasm32-wasip2.stderr.expected | 6 ++++++ tests/testcase.sh | 2 +- 6 files changed, 27 insertions(+), 2 deletions(-) create mode 100644 tests/general/abort.c.wasm32-wasi.stderr.expected create mode 100644 tests/general/abort.c.wasm32-wasip2.stderr.expected create mode 100644 tests/general/assert-fail.c.wasm32-wasip2.stderr.expected create mode 100644 tests/general/sigabrt.c.wasm32-wasip2.stderr.expected diff --git a/src/wasi-libc b/src/wasi-libc index c9c7d0616..d03829489 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit c9c7d0616e0f7fe4d0d0fa54372b6d1f5689f91d +Subproject commit d03829489904d38c624f6de9983190f1e5e7c9c5 diff --git a/tests/general/abort.c.wasm32-wasi.stderr.expected b/tests/general/abort.c.wasm32-wasi.stderr.expected new file mode 100644 index 000000000..c6ecf6a93 --- /dev/null +++ b/tests/general/abort.c.wasm32-wasi.stderr.expected @@ -0,0 +1,6 @@ +Error: failed to run main module `abort.c.---.wasm` + +Caused by: + 0: failed to invoke command default + 1: error while executing at wasm backtrace: + 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/abort.c.wasm32-wasip2.stderr.expected b/tests/general/abort.c.wasm32-wasip2.stderr.expected new file mode 100644 index 000000000..26fe376cf --- /dev/null +++ b/tests/general/abort.c.wasm32-wasip2.stderr.expected @@ -0,0 +1,6 @@ +Error: failed to run main module `abort.c.---.wasm` + +Caused by: + 0: failed to invoke `run` function + 1: error while executing at wasm backtrace: + 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/assert-fail.c.wasm32-wasip2.stderr.expected b/tests/general/assert-fail.c.wasm32-wasip2.stderr.expected new file mode 100644 index 000000000..3702431bd --- /dev/null +++ b/tests/general/assert-fail.c.wasm32-wasip2.stderr.expected @@ -0,0 +1,7 @@ +Assertion failed: false (assert-fail.c: main: 5) +Error: failed to run main module `assert-fail.c.---.wasm` + +Caused by: + 0: failed to invoke `run` function + 1: error while executing at wasm backtrace: + 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/sigabrt.c.wasm32-wasip2.stderr.expected b/tests/general/sigabrt.c.wasm32-wasip2.stderr.expected new file mode 100644 index 000000000..f437a8214 --- /dev/null +++ b/tests/general/sigabrt.c.wasm32-wasip2.stderr.expected @@ -0,0 +1,6 @@ +raising SIGABRT... +Program received fatal signal: Aborted +Error: failed to run main module `sigabrt.c.---.wasm` + +Caused by: + 0: failed to invoke `run` function diff --git a/tests/testcase.sh b/tests/testcase.sh index b646e2d4a..61c6612e5 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -44,7 +44,7 @@ if [ "$runwasi" == "" ]; then exit 0 fi -if [ "$target" == "wasm32-wasi-preview2" -a -n "$adapter" -a -n "$wasm_tools" ]; then +if [ "$target" == "wasm32-wasip2" -a -n "$adapter" -a -n "$wasm_tools" ]; then "$wasm_tools" component new --adapt "$adapter" "$wasm" -o "$wasm" run_args="--wasm component-model" fi From 392035c16d5bbae37345ce729ddb2cc45e50ad87 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 11 Apr 2024 12:30:23 -0500 Subject: [PATCH 053/165] Update LLVM to 18.1.2 and add wasm-component-ld (#402) * Update LLVM to 18.1.2 and add wasm-component-ld This commit has two intertwined changes within it. First the LLVM submodule is updated to the 18.1.2 release branch. This alone would cause CI and tests to fail due to differing behavior for the `wasm32-wasip2` target. To fix these test failures the `wasm-component-ld` tool is added to the build. This tool is a Rust-written tool and installed via `cargo install` as part of the build at a pinned version written in the `Makefile`. This linker, used for components, is then used for the `wasm32-wasip2` target. Tests and CI are then updated to skip the need to have `wasm-tools` or the adapter for WASI when making components since that's now the job of `wasm-component-ld`. This then necessitated some changes to tests too. * Add back accidentally deleted CI * Inherit tools on PATH on Windows * Add Rust to docker build * Put rust in a different location in docker * Try to fix permissions * Review comments * Revert changes to test outputs --- .github/workflows/main.yml | 9 ++------- Dockerfile | 5 +++++ Makefile | 12 ++++++++++-- src/llvm-project | 2 +- tests/run.sh | 25 ++++++++----------------- tests/testcase.sh | 16 ++++------------ 6 files changed, 30 insertions(+), 39 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 4303d08d6..0a2d00398 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -42,12 +42,6 @@ jobs: uses: bytecodealliance/actions/wasmtime/setup@v1 with: version: "18.0.2" - - name: Setup `wasm-tools` for tests - uses: bytecodealliance/actions/wasm-tools/setup@v1 - with: - version: "1.201.0" - - name: Download command adapter for tests - run: curl -f -L --retry 5 -o ${{ runner.temp }}/wasi_snapshot_preview1.command.wasm https://github.com/bytecodealliance/wasmtime/releases/download/v18.0.2/wasi_snapshot_preview1.command.wasm - uses: actions/checkout@v4 with: fetch-depth: 0 @@ -67,7 +61,7 @@ jobs: run: NINJA_FLAGS=-v make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON shell: bash - name: Run the testsuite - run: NINJA_FLAGS=-v make check RUNTIME=wasmtime ADAPTER=${{ runner.temp }}/wasi_snapshot_preview1.command.wasm WASM_TOOLS=wasm-tools + run: NINJA_FLAGS=-v make check RUNTIME=wasmtime - name: Upload artifacts uses: actions/upload-artifact@v4 with: @@ -107,6 +101,7 @@ jobs: msystem: ${{ matrix.sys }} update: true release: false + path-type: inherit - uses: actions/checkout@v4 with: fetch-depth: 0 diff --git a/Dockerfile b/Dockerfile index 9a864e63e..adddd029c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -28,6 +28,11 @@ RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.25.1/cmake- && mv cmake-3.25.1-linux-x86_64 /opt/cmake ENV PATH /opt/cmake/bin:$PATH +ENV RUSTUP_HOME=/rust/rustup CARGO_HOME=/rust/cargo PATH=$PATH:/rust/cargo/bin +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \ + sh -s -- -y --profile=minimal && \ + chmod -R a+w /rust + RUN groupadd -g ${GID} builder && \ useradd --create-home --uid ${UID} --gid ${GID} builder USER builder diff --git a/Makefile b/Makefile index 9c7acffbf..fbe6af583 100644 --- a/Makefile +++ b/Makefile @@ -54,7 +54,7 @@ default: build @echo "Use -fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION)" check: - TARGETS="$(TARGETS)" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" "$(ADAPTER)" "$(WASM_TOOLS)" + TARGETS="$(TARGETS)" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" clean: rm -rf build $(DESTDIR) @@ -122,6 +122,14 @@ build/llvm.BUILT: llvm-config touch build/llvm.BUILT +# Build the `wasm-component-ld` linker from source via `cargo install`. This is +# used for the `wasm32-wasip2` target natively by Clang. Note that `--root` +# passed to `cargo install` will place it in the output directory automatically. +build/wasm-component-ld.BUILT: build/llvm.BUILT + cargo install wasm-component-ld@0.1.5 --root $(BUILD_PREFIX) + touch build/wasm-component-ld.BUILT + + # Flags for running `make` in wasi-libc # $(1): the target that's being built WASI_LIBC_MAKEFLAGS = \ @@ -132,7 +140,7 @@ WASI_LIBC_MAKEFLAGS = \ SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ TARGET_TRIPLE=$(1) -build/wasi-libc.BUILT: build/compiler-rt.BUILT +build/wasi-libc.BUILT: build/compiler-rt.BUILT build/wasm-component-ld.BUILT $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasi) default libc_so $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip1) default libc_so $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip2) WASI_SNAPSHOT=p2 default libc_so diff --git a/src/llvm-project b/src/llvm-project index 461274b81..26a1d6601 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 461274b81d8641eab64d494accddc81d7db8a09e +Subproject commit 26a1d6601d727a96f4301d0d8647b5a42760ae0c diff --git a/tests/run.sh b/tests/run.sh index 080eeaee5..bc8dd4d0b 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -17,17 +17,8 @@ wasi_sdk="$1" # Determine the wasm runtime to use, if one is provided. if [ $# -gt 1 ]; then runwasi="$2" - if [ $# -gt 3 ]; then - adapter="$3" - wasm_tools="$4" - else - adapter="" - wasm_tools="" - fi else runwasi="" - adapter="" - wasm_tools="" fi testdir=$(dirname $0) @@ -45,13 +36,13 @@ for target in $TARGETS; do echo "===== Testing compile-only with $options =====" for file in *.c; do echo "Testing compile-only $file..." - ../testcase.sh "$target" "" "" "" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" - ../testcase.sh "$target" "" "" "" "$wasi_sdk/bin/$target-clang" "$options" "$file" + ../testcase.sh "$target" "" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" + ../testcase.sh "$target" "" "$wasi_sdk/bin/$target-clang" "$options" "$file" done for file in *.cc; do echo "Testing compile-only $file..." - ../testcase.sh "$target" "" "" "" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" - ../testcase.sh "$target" "" "" "" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" + ../testcase.sh "$target" "" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" + ../testcase.sh "$target" "" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" done done cd - >/dev/null @@ -61,13 +52,13 @@ for target in $TARGETS; do echo "===== Testing with $options =====" for file in *.c; do echo "Testing $file..." - ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" - ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$wasi_sdk/bin/$target-clang" "$options" "$file" + ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" + ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/$target-clang" "$options" "$file" done for file in *.cc; do echo "Testing $file..." - ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" - ../testcase.sh "$target" "$runwasi" "$adapter" "$wasm_tools" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" + ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" + ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" done done cd - >/dev/null diff --git a/tests/testcase.sh b/tests/testcase.sh index 61c6612e5..3d8860464 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -8,18 +8,15 @@ set -ueo pipefail # script, so don't do anything fancy. target="$1" runwasi="$2" -adapter="$3" -wasm_tools="$4" -compiler="$5" -options="$6" -input="$7" +compiler="$3" +options="$4" +input="$5" # Compile names for generated files. wasm="$input.$options.wasm" stdout_observed="$input.$options.stdout.observed" stderr_observed="$input.$options.stderr.observed" exit_status_observed="$input.$options.exit_status.observed" -run_args="" # Optionally load compiler options from a file. if [ -e "$input.options" ]; then @@ -44,11 +41,6 @@ if [ "$runwasi" == "" ]; then exit 0 fi -if [ "$target" == "wasm32-wasip2" -a -n "$adapter" -a -n "$wasm_tools" ]; then - "$wasm_tools" component new --adapt "$adapter" "$wasm" -o "$wasm" - run_args="--wasm component-model" -fi - # Determine the input file to write to stdin. if [ -e "$input.stdin" ]; then stdin="$input.stdin" @@ -74,7 +66,7 @@ fi # Run the test, capturing stdout, stderr, and the exit status. exit_status=0 -"$runwasi" $run_args $env $dir "$wasm" $dirarg \ +"$runwasi" $env $dir "$wasm" $dirarg \ < "$stdin" \ > "$stdout_observed" \ 2> "$stderr_observed" \ From 5bde500a86666c69f2a8adec27fd85725d4768d9 Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Thu, 11 Apr 2024 17:33:17 -0600 Subject: [PATCH 054/165] update wasi-libc submodule (#407) Signed-off-by: Joel Dice --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index d03829489..9e8c54231 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit d03829489904d38c624f6de9983190f1e5e7c9c5 +Subproject commit 9e8c542319242a5e536e14e6046de5968d298038 From b4f426a328ac1dee54fa31ff4d53b4e17217a8e0 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 12 Apr 2024 10:53:09 -0700 Subject: [PATCH 055/165] Force-fetch tags in CI to fix `git describe` My release attempt in #408 did not go well because the release artifacts were not named correctly. This fixes an issue described by actions/checkout#290 where checkouts of annotated tags overwrite the annotation which breaks `git describe`. That means that version detection is broken in CI during releases which causes artifacts to have the wrong information. This applies the workaround described in that issue to `git fetch --tags --force` after the checkout step to undo the overwrite done in the checkout step. --- .github/workflows/main.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 0a2d00398..9ccef2620 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -45,6 +45,8 @@ jobs: - uses: actions/checkout@v4 with: fetch-depth: 0 + - run: git fetch --tags --force + name: Force-fetch tags to work around actions/checkout#290 # We can't use `--depth 1` here sadly because the GNU config # submodule is not pinned to a particular tag/branch. Please # bump depth (or even better, the submodule), in case of "error: @@ -105,6 +107,8 @@ jobs: - uses: actions/checkout@v4 with: fetch-depth: 0 + - run: git fetch --tags --force + name: Force-fetch tags to work around actions/checkout#290 - run: git submodule update --init --depth 32 --jobs 3 - name: Build shell: msys2 {0} @@ -137,6 +141,8 @@ jobs: - uses: actions/checkout@v4 with: fetch-depth: 0 + - run: git fetch --tags --force + name: Force-fetch tags to work around actions/checkout#290 - run: git submodule update --init --depth 32 --jobs 3 From 99abb024209dc7a9a8d8f3eb6d03908a18bb0594 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 15 Apr 2024 09:04:55 -0500 Subject: [PATCH 056/165] Don't add git hash to version if minor==0 (#409) This commit fixes a minor mistake from #392 where the previous perl script had a special case that if the "minor" version was zero then the git hash information wouldn't be printed. --- version.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/version.py b/version.py index 2d9de6d5b..a3199f12c 100755 --- a/version.py +++ b/version.py @@ -39,7 +39,7 @@ def parse_git_version(version): next = parts.pop(0) if next == 'm': dirty = True - else: + elif minor != '0': git = next[1:] # Check: dirty. @@ -53,10 +53,10 @@ def parse_git_version(version): # Some inline tests to check Git version parsing: assert parse_git_version( - 'wasi-sdk-21-0-g317548590b40+m') == ('21', '0', '317548590b40', True) + 'wasi-sdk-21-1-g317548590b40+m') == ('21', '1', '317548590b40', True) assert parse_git_version('wasi-sdk-21-2+m') == ('21', '2', None, True) assert parse_git_version( - 'wasi-sdk-23-0-g317548590b40') == ('23', '0', '317548590b40', False) + 'wasi-sdk-23-0-g317548590b40') == ('23', '0', None, False) def git_version(): From 34d77e5e39e9f000944122c91795f844a18b04e1 Mon Sep 17 00:00:00 2001 From: Philip Rying <36996706+imrying@users.noreply.github.com> Date: Tue, 23 Apr 2024 20:13:17 +0200 Subject: [PATCH 057/165] add python3 as build requirement (#415) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 41adc2ffe..fd26ddb4e 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ The Wasm-sdk's build process needs some packages : * `cmake` * `clang` * `ninja` +* `python3` Please refer to your OS documentation to install those packages. From 95e3c2c6c79367eae412db5833155f4a36582eed Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Sat, 4 May 2024 20:05:04 -0700 Subject: [PATCH 058/165] Update the version of `wasm-component-ld` used (#416) Recent changes have added various flags for customizing the component-creation process in addition to supporting more flags being forwarded to `wasm-ld`. --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index fbe6af583..eeac2dbff 100644 --- a/Makefile +++ b/Makefile @@ -126,7 +126,7 @@ build/llvm.BUILT: # used for the `wasm32-wasip2` target natively by Clang. Note that `--root` # passed to `cargo install` will place it in the output directory automatically. build/wasm-component-ld.BUILT: build/llvm.BUILT - cargo install wasm-component-ld@0.1.5 --root $(BUILD_PREFIX) + cargo install wasm-component-ld@0.5.0 --root $(BUILD_PREFIX) touch build/wasm-component-ld.BUILT From af429ed1573783473d2835d35c10eb94a462d979 Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Thu, 9 May 2024 12:54:13 -0600 Subject: [PATCH 059/165] exclude 32-bit Windows artifact in download-workflow-artifacts.sh (#417) This helps ensure the identically-named 32-bit tarfile doesn't overwrite the 64-bit one when we extract everything into a single directory. I'll follow this up by making an updated wasi-sdk-22.0.m-mingw.tar.gz and adding it to the wasi-sdk-22 release. Fixes #326 Signed-off-by: Joel Dice --- ci/download-workflow-artifacts.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/ci/download-workflow-artifacts.sh b/ci/download-workflow-artifacts.sh index 409b484e0..543f93cad 100755 --- a/ci/download-workflow-artifacts.sh +++ b/ci/download-workflow-artifacts.sh @@ -43,6 +43,13 @@ for A in $ARTIFACTS; do if [ "${NAME}" = "dist-ubuntu-latest" ]; then continue fi + # Exclude the 32-bit Windows artifact in favor of the 64-bit one. This helps + # ensure the identically-named 32-bit tarfile doesn't overwrite the 64-bit + # one below. See: + # - https://github.com/WebAssembly/wasi-sdk/issues/326 + if [ "${NAME}" = "dist-windows-latest-x86" ]; then + continue + fi >&2 echo "===== Downloading: ${TO} =====" # Download the artifacts to the temporary directory. From 307694b2fb679105281f30c4c55ff69d68e983a1 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 22 May 2024 11:11:21 -0500 Subject: [PATCH 060/165] Add DWARF debugging information to all artifacts by default (#422) * Add DWARF debugging information to all artifacts by default This commit adds DWARF debugging information with the `-g` compiler flag to all WASI artifacts for wasi-sdk. The LLVM build itself does not have debugging information, only the sysroot artifacts. This is intended to assist with debugging. The main downside to this is binary size of generated artifacts will, by default, be larger. Stripping debug information from an artifact though involves removing custom sections which is generally pretty easy to do through wasm tooling. * Pass extra cflags to wasi-libc * Fix tests from previous commit * Update some expected error messages and remove some files with duplicate error messages that are no longer needed. * Remove undefined behavior in `stat.c` where padding bytes were being compared. --- Makefile | 12 +++++++++--- src/wasi-libc | 2 +- tests/general/abort.c.stderr.expected | 3 ++- tests/general/abort.c.stderr.expected.filter | 1 + ...bort.c.wasm32-wasi-preview2.stderr.expected | 6 ------ .../abort.c.wasm32-wasi.stderr.expected | 6 ------ .../abort.c.wasm32-wasip2.stderr.expected | 6 ------ tests/general/assert-fail.c.stderr.expected | 3 ++- .../assert-fail.c.stderr.expected.filter | 1 + ...fail.c.wasm32-wasi-preview2.stderr.expected | 7 ------- ...assert-fail.c.wasm32-wasip2.stderr.expected | 7 ------- tests/general/stat.c | 18 ++++++++++++++---- 12 files changed, 30 insertions(+), 42 deletions(-) delete mode 100644 tests/general/abort.c.wasm32-wasi-preview2.stderr.expected delete mode 100644 tests/general/abort.c.wasm32-wasi.stderr.expected delete mode 100644 tests/general/abort.c.wasm32-wasip2.stderr.expected delete mode 100644 tests/general/assert-fail.c.wasm32-wasi-preview2.stderr.expected delete mode 100644 tests/general/assert-fail.c.wasm32-wasip2.stderr.expected diff --git a/Makefile b/Makefile index eeac2dbff..04fb23fa4 100644 --- a/Makefile +++ b/Makefile @@ -50,6 +50,11 @@ CLANG_VERSION=$(shell $(VERSION_SCRIPT) llvm-major --llvm-dir=$(LLVM_PROJ_DIR)) VERSION:=$(shell $(VERSION_SCRIPT)) DEBUG_PREFIX_MAP=-fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION) +# Generate debuginfo by default for wasi-sdk since it's easily strippable and +# otherwise quite useful for debugging. +WASI_SDK_CFLAGS := $(DEBUG_PREFIX_MAP) -g +WASI_SDK_CXXFLAGS := $(WASI_SDK_CFLAGS) + default: build @echo "Use -fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION)" @@ -138,6 +143,7 @@ WASI_LIBC_MAKEFLAGS = \ AR=$(BUILD_PREFIX)/bin/llvm-ar \ NM=$(BUILD_PREFIX)/bin/llvm-nm \ SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ + EXTRA_CFLAGS="$(WASI_SDK_CFLAGS)" \ TARGET_TRIPLE=$(1) build/wasi-libc.BUILT: build/compiler-rt.BUILT build/wasm-component-ld.BUILT @@ -166,7 +172,7 @@ build/compiler-rt.BUILT: build/llvm.BUILT -DCOMPILER_RT_ENABLE_IOS=OFF \ -DCOMPILER_RT_DEFAULT_TARGET_ONLY=On \ -DWASI_SDK_PREFIX=$(BUILD_PREFIX) \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP)" \ + -DCMAKE_C_FLAGS="$(WASI_SDK_CFLAGS)" \ -DLLVM_CONFIG_PATH=$(ROOT_DIR)/build/llvm/bin/llvm-config \ -DCOMPILER_RT_OS_DIR=wasi \ -DCMAKE_INSTALL_PREFIX=$(PREFIX)/lib/clang/$(CLANG_VERSION)/ \ @@ -225,8 +231,8 @@ LIBCXX_CMAKE_FLAGS = \ -DUNIX:BOOL=ON \ --debug-trycompile \ -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CFLAGS) $(4) --target=$(3)" \ - -DCMAKE_CXX_FLAGS="$(DEBUG_PREFIX_MAP) $(EXTRA_CXXFLAGS) $(4) --target=$(3)" \ + -DCMAKE_C_FLAGS="$(WASI_SDK_CFLAGS) $(EXTRA_CFLAGS) $(4) --target=$(3)" \ + -DCMAKE_CXX_FLAGS="$(WASI_SDK_CXXFLAGS) $(EXTRA_CXXFLAGS) $(4) --target=$(3)" \ -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/$(3) \ -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/$(3) \ -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ diff --git a/src/wasi-libc b/src/wasi-libc index 9e8c54231..44c4b1e3a 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 9e8c542319242a5e536e14e6046de5968d298038 +Subproject commit 44c4b1e3a58f5c3042fa26dff9c1b29b6fc695b2 diff --git a/tests/general/abort.c.stderr.expected b/tests/general/abort.c.stderr.expected index c6ecf6a93..d533e582f 100644 --- a/tests/general/abort.c.stderr.expected +++ b/tests/general/abort.c.stderr.expected @@ -1,6 +1,7 @@ Error: failed to run main module `abort.c.---.wasm` Caused by: - 0: failed to invoke command default + 0: failed to invoke --- 1: error while executing at wasm backtrace: + note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/abort.c.stderr.expected.filter b/tests/general/abort.c.stderr.expected.filter index 742503a5c..ecd16c053 100755 --- a/tests/general/abort.c.stderr.expected.filter +++ b/tests/general/abort.c.stderr.expected.filter @@ -3,4 +3,5 @@ set -euo pipefail cat \ | sed -e 's/main module `abort\.c\.[^`]*\.wasm`/main module `abort.c.---.wasm`/' \ + | sed -e 's/failed to invoke.*/failed to invoke ---/' \ | sed -E '/0x[[:xdigit:]]+/d' diff --git a/tests/general/abort.c.wasm32-wasi-preview2.stderr.expected b/tests/general/abort.c.wasm32-wasi-preview2.stderr.expected deleted file mode 100644 index 26fe376cf..000000000 --- a/tests/general/abort.c.wasm32-wasi-preview2.stderr.expected +++ /dev/null @@ -1,6 +0,0 @@ -Error: failed to run main module `abort.c.---.wasm` - -Caused by: - 0: failed to invoke `run` function - 1: error while executing at wasm backtrace: - 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/abort.c.wasm32-wasi.stderr.expected b/tests/general/abort.c.wasm32-wasi.stderr.expected deleted file mode 100644 index c6ecf6a93..000000000 --- a/tests/general/abort.c.wasm32-wasi.stderr.expected +++ /dev/null @@ -1,6 +0,0 @@ -Error: failed to run main module `abort.c.---.wasm` - -Caused by: - 0: failed to invoke command default - 1: error while executing at wasm backtrace: - 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/abort.c.wasm32-wasip2.stderr.expected b/tests/general/abort.c.wasm32-wasip2.stderr.expected deleted file mode 100644 index 26fe376cf..000000000 --- a/tests/general/abort.c.wasm32-wasip2.stderr.expected +++ /dev/null @@ -1,6 +0,0 @@ -Error: failed to run main module `abort.c.---.wasm` - -Caused by: - 0: failed to invoke `run` function - 1: error while executing at wasm backtrace: - 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/assert-fail.c.stderr.expected b/tests/general/assert-fail.c.stderr.expected index c179f09ad..9e0f44c5a 100644 --- a/tests/general/assert-fail.c.stderr.expected +++ b/tests/general/assert-fail.c.stderr.expected @@ -2,6 +2,7 @@ Assertion failed: false (assert-fail.c: main: 5) Error: failed to run main module `assert-fail.c.---.wasm` Caused by: - 0: failed to invoke command default + 0: failed to invoke --- 1: error while executing at wasm backtrace: + note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/assert-fail.c.stderr.expected.filter b/tests/general/assert-fail.c.stderr.expected.filter index d2213dd6b..a1d649de0 100755 --- a/tests/general/assert-fail.c.stderr.expected.filter +++ b/tests/general/assert-fail.c.stderr.expected.filter @@ -3,4 +3,5 @@ set -euo pipefail cat \ | sed -e 's/main module `assert-fail\.c\.[^`]*\.wasm`/main module `assert-fail.c.---.wasm`/' \ + | sed -e 's/failed to invoke.*/failed to invoke ---/' \ | sed -E '/0x[[:xdigit:]]+/d' diff --git a/tests/general/assert-fail.c.wasm32-wasi-preview2.stderr.expected b/tests/general/assert-fail.c.wasm32-wasi-preview2.stderr.expected deleted file mode 100644 index 3702431bd..000000000 --- a/tests/general/assert-fail.c.wasm32-wasi-preview2.stderr.expected +++ /dev/null @@ -1,7 +0,0 @@ -Assertion failed: false (assert-fail.c: main: 5) -Error: failed to run main module `assert-fail.c.---.wasm` - -Caused by: - 0: failed to invoke `run` function - 1: error while executing at wasm backtrace: - 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/assert-fail.c.wasm32-wasip2.stderr.expected b/tests/general/assert-fail.c.wasm32-wasip2.stderr.expected deleted file mode 100644 index 3702431bd..000000000 --- a/tests/general/assert-fail.c.wasm32-wasip2.stderr.expected +++ /dev/null @@ -1,7 +0,0 @@ -Assertion failed: false (assert-fail.c: main: 5) -Error: failed to run main module `assert-fail.c.---.wasm` - -Caused by: - 0: failed to invoke `run` function - 1: error while executing at wasm backtrace: - 2: wasm trap: wasm `unreachable` instruction executed diff --git a/tests/general/stat.c b/tests/general/stat.c index dc007c120..393921f70 100644 --- a/tests/general/stat.c +++ b/tests/general/stat.c @@ -43,10 +43,20 @@ int main(int argc, char *argv[]) { assert(file_statbuf.st_dev == link_statbuf.st_dev); - // Clear out the access time fields, and they should be the same. - memset(&file_statbuf.st_atim, 0, sizeof(struct timespec)); - memset(&link_statbuf.st_atim, 0, sizeof(struct timespec)); - assert(memcmp(&file_statbuf, &link_statbuf, sizeof(struct stat)) == 0); + assert(file_statbuf.st_dev == link_statbuf.st_dev); + assert(file_statbuf.st_ino == link_statbuf.st_ino); + assert(file_statbuf.st_mode == link_statbuf.st_mode); + assert(file_statbuf.st_uid == link_statbuf.st_uid); + assert(file_statbuf.st_gid == link_statbuf.st_gid); + assert(file_statbuf.st_rdev == link_statbuf.st_rdev); + assert(file_statbuf.st_size == link_statbuf.st_size); + assert(file_statbuf.st_blksize == link_statbuf.st_blksize); + assert(file_statbuf.st_blocks == link_statbuf.st_blocks); + // NB: `atim` is explicitly not compared here + assert(file_statbuf.st_mtim.tv_sec == link_statbuf.st_mtim.tv_sec); + assert(file_statbuf.st_mtim.tv_nsec == link_statbuf.st_mtim.tv_nsec); + assert(file_statbuf.st_ctim.tv_sec == link_statbuf.st_ctim.tv_sec); + assert(file_statbuf.st_ctim.tv_nsec == link_statbuf.st_ctim.tv_nsec); // Test lstat. From 68df37e7f68df251163e147e8512337535e8faca Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 22 May 2024 14:37:54 -0500 Subject: [PATCH 061/165] Pass `-O2 -DNDEBUG` to wasi-libc again (#423) I forgot in #422 that by specifying `EXTRA_CFLAGS` to the wasi-libc build that it would override the defaults of wasi-libc which is to pass these two flags in. This passes them back in to ensure the default build still has the same flags. --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 04fb23fa4..da92c1eab 100644 --- a/Makefile +++ b/Makefile @@ -143,7 +143,7 @@ WASI_LIBC_MAKEFLAGS = \ AR=$(BUILD_PREFIX)/bin/llvm-ar \ NM=$(BUILD_PREFIX)/bin/llvm-nm \ SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - EXTRA_CFLAGS="$(WASI_SDK_CFLAGS)" \ + EXTRA_CFLAGS="$(WASI_SDK_CFLAGS) -O2 -DNDEBUG" \ TARGET_TRIPLE=$(1) build/wasi-libc.BUILT: build/compiler-rt.BUILT build/wasm-component-ld.BUILT From 8827cdffd0ebb4271ee116291ae9398b5708438d Mon Sep 17 00:00:00 2001 From: "Anuraag (Rag) Agrawal" Date: Wed, 12 Jun 2024 00:32:12 +0900 Subject: [PATCH 062/165] Update docker image to LLVM 18 (#427) --- docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index 9fda47064..63c5a3cff 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -17,7 +17,7 @@ RUN mv /wasi-sdk-* /wasi-sdk FROM ubuntu:22.04 -ENV LLVM_VERSION 17 +ENV LLVM_VERSION 18 # Install build toolchain including clang, ld, make, autotools, ninja, and cmake RUN apt-get update && \ From 7ff81cb10c6181db8b57f24800f765c19b461c0a Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Thu, 20 Jun 2024 09:57:14 -0600 Subject: [PATCH 063/165] add wasi-sdk-p2.cmake (#430) This is analogous to wasi-sdk.cmake and wasi-sdk-pthread.cmake, but for WASIp2. Signed-off-by: Joel Dice --- Makefile | 1 + docker/Dockerfile | 1 + docker/wasi-sdk-p2.cmake | 29 +++++++++++++++++++++++++++++ wasi-sdk-p2.cmake | 40 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 71 insertions(+) create mode 100644 docker/wasi-sdk-p2.cmake create mode 100644 wasi-sdk-p2.cmake diff --git a/Makefile b/Makefile index da92c1eab..fa4a66998 100644 --- a/Makefile +++ b/Makefile @@ -267,6 +267,7 @@ build/config.BUILT: mkdir -p $(BUILD_PREFIX)/share/cmake/Platform cp wasi-sdk.cmake $(BUILD_PREFIX)/share/cmake cp wasi-sdk-pthread.cmake $(BUILD_PREFIX)/share/cmake + cp wasi-sdk-p2.cmake $(BUILD_PREFIX)/share/cmake cp cmake/Platform/WASI.cmake $(BUILD_PREFIX)/share/cmake/Platform touch build/config.BUILT diff --git a/docker/Dockerfile b/docker/Dockerfile index 63c5a3cff..080b503ec 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -38,6 +38,7 @@ COPY --from=dist /wasi-sysroot-clang_rt/lib/wasi /usr/lib/llvm-${LLVM_VERSION}/l ADD docker/wasi-sdk.cmake /usr/share/cmake/wasi-sdk.cmake ADD docker/wasi-sdk-pthread.cmake /usr/share/cmake/wasi-sdk-pthread.cmake +ADD docker/wasi-sdk-p2.cmake /usr/share/cmake/wasi-sdk-p2.cmake ENV CMAKE_TOOLCHAIN_FILE /usr/share/cmake/wasi-sdk.cmake ADD cmake/Platform/WASI.cmake /usr/share/cmake/Modules/Platform/WASI.cmake diff --git a/docker/wasi-sdk-p2.cmake b/docker/wasi-sdk-p2.cmake new file mode 100644 index 000000000..2d34eadbe --- /dev/null +++ b/docker/wasi-sdk-p2.cmake @@ -0,0 +1,29 @@ +# Cmake toolchain description file for the wasi-sdk docker image + +# This is arbitrary, AFAIK, for now. +cmake_minimum_required(VERSION 3.4.0) + +# To make sure it recognizes the WASI platform +list(APPEND CMAKE_MODULE_PATH /usr/share/cmake/Modules) + +set(CMAKE_SYSTEM_NAME WASI) +set(CMAKE_SYSTEM_VERSION 1) +set(CMAKE_SYSTEM_PROCESSOR wasm32) +set(triple wasm32-wasip2) + +set(CMAKE_C_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) +set(CMAKE_CXX_COMPILER /usr/bin/clang++-$ENV{LLVM_VERSION}) +set(CMAKE_ASM_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) +set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) +set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) +set(CMAKE_C_COMPILER_TARGET ${triple}) +set(CMAKE_CXX_COMPILER_TARGET ${triple}) +set(CMAKE_ASM_COMPILER_TARGET ${triple}) +SET(CMAKE_SYSROOT /wasi-sysroot) + +# Don't look in the sysroot for executables to run during the build +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) +# Only look in the sysroot (not in the host paths) for the rest +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/wasi-sdk-p2.cmake b/wasi-sdk-p2.cmake new file mode 100644 index 000000000..681f71dcd --- /dev/null +++ b/wasi-sdk-p2.cmake @@ -0,0 +1,40 @@ +# Cmake toolchain description file for the Makefile + +# This is arbitrary, AFAIK, for now. +cmake_minimum_required(VERSION 3.4.0) + +# Until Platform/WASI.cmake is upstream we need to inject the path to it +# into CMAKE_MODULE_PATH. +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}") + +set(CMAKE_SYSTEM_NAME WASI) +set(CMAKE_SYSTEM_VERSION 1) +set(CMAKE_SYSTEM_PROCESSOR wasm32) +set(triple wasm32-wasip2) + +if(WIN32) + set(WASI_HOST_EXE_SUFFIX ".exe") +else() + set(WASI_HOST_EXE_SUFFIX "") +endif() + +# When building from source, WASI_SDK_PREFIX represents the generated directory +if(NOT WASI_SDK_PREFIX) + set(WASI_SDK_PREFIX ${CMAKE_CURRENT_LIST_DIR}/../../) +endif() + +set(CMAKE_C_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_CXX_COMPILER ${WASI_SDK_PREFIX}/bin/clang++${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_ASM_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_AR ${WASI_SDK_PREFIX}/bin/llvm-ar${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_RANLIB ${WASI_SDK_PREFIX}/bin/llvm-ranlib${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_C_COMPILER_TARGET ${triple}) +set(CMAKE_CXX_COMPILER_TARGET ${triple}) +set(CMAKE_ASM_COMPILER_TARGET ${triple}) + +# Don't look in the sysroot for executables to run during the build +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) +# Only look in the sysroot (not in the host paths) for the rest +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) From 35c271e6e1e053547572ebbe31793985f6b89e62 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Mon, 8 Jul 2024 11:17:31 -0700 Subject: [PATCH 064/165] Avoid extra Cargo installation artifacts (#432) As @bjorn3 mentioned in #420, using `--no-track` should get rid of the extra files that show up after the `cargo install`. Fixes #420. --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index fa4a66998..eda2a7454 100644 --- a/Makefile +++ b/Makefile @@ -131,7 +131,7 @@ build/llvm.BUILT: # used for the `wasm32-wasip2` target natively by Clang. Note that `--root` # passed to `cargo install` will place it in the output directory automatically. build/wasm-component-ld.BUILT: build/llvm.BUILT - cargo install wasm-component-ld@0.5.0 --root $(BUILD_PREFIX) + cargo install --no-track wasm-component-ld@0.5.0 --root $(BUILD_PREFIX) touch build/wasm-component-ld.BUILT From 873163a2f082f0a84f2e547d1d9ad01cee98cff9 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 10 Jul 2024 11:09:50 -0700 Subject: [PATCH 065/165] Update to the latest wasi-libc. (#435) Update to wasi-libc 3f43ea9abb24ed8d24d760989e1d87ea385f8eaa. --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index 44c4b1e3a..3f43ea9ab 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 44c4b1e3a58f5c3042fa26dff9c1b29b6fc695b2 +Subproject commit 3f43ea9abb24ed8d24d760989e1d87ea385f8eaa From e29a3fef8b57e9bab2aacccea115cf0c8c9c47f7 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 11 Jul 2024 17:53:22 -0500 Subject: [PATCH 066/165] Rewrite the build system with CMake (#429) * Rewrite the build system with CMake This commit is an attempt to provide a concrete path forward on WebAssembly/wasi-sdk#425. I personally think it's pretty important to get the ability to have more architectures here but at the same time I also think it's important to to take this as an opportunity to refactor and improve the build system of this repository. To that end this represents my attempt to improve the status quo. This removes the old `Makefile` and replaces it with a CMake-based system to build all these projects. Overall this is intended to be a "no functional change" intended sort of refactoring. Changing build systems inevitably causes issues, however, so this change additionally has a very high likelihood of needing follow-up fixes. At a high enough level this commit introduces two major changes to how this repository is built: 1. The `make`-based system (the root `Makefile`) is replaced with CMake. This additionally updates tests to use CMake. 2. A single "build" is split into either building a toolchain or building a sysroot. This enables builds to only build one or the other as necessary. The first change, using CMake, is due to the fact that using `make` on Windows basically is not pleasant coupled with the fact that more advanced logic, such as changing flags, compilers, etc, is much easier with a CMake-based system. The second change is intended to cover the use case of #425 in addition to refactoring the current build. Throughout this change I have intentionally not tried to keep a 1:1 correspondance with behaviors in the old `Makefile` because much of this PR is intended to address shortcomings in the old build system. A list of changes, improvements, etc, made here are: * CMake provides a much nicer portability story to Windows than `make`. This is moving towards the direction of not needing `bash`, for example, to build an SDK. Currently `wasi-libc` still requires this, but that's now the only "hard" dependency. * The set of targets built can now be configured for smaller builds and/or debugging just a single target. All WASI targets are still built by default but it's much easier to add/remove them. * Different targets are now able to be built in parallel as opposed to the unconditional serial-nature of the `Makefile`. * Use of `ninja` is no longer required and separate build systems can be used if desired. * The sysroot and the toolchain can now be built with different CMake build profiles. For example the `Makefile` hardcoded `MinSizeRel` and `RelWithDebInfo` and this can now be much more easily customized by the SDK builder. * Tarballs are now more consistently produced and named. For a tarball of the name `foo.tar.gz` it's guaranteed that there's a single folder `foo` created when unpacking the tarball. * The macOS binaries are no longer hybrid x64/arm64 binaries which greatly inflates the size of the SDK. There's now a separate build for each architecture. * CI now produces arm64-linux binaries. The sysroot is not built on the arm64-linux builder and the sysroot from the x86_64-linux builder is used instead. * Tests are almost ready to execute on Windows, there's just a few minor issues related to exit statuses and probably line endings which need to be worked out. Will require someone with a Windows checkout, however. * Tests are now integrated into CMake. This means that the wasm binaries are able to be built in parallel and the tests are additionally executed in parallel with `ctest`. It is possible to build/run a single test. Tests no longer place all of their output in the source tree. * Out-of-tree builds are now possible and the build/installation directories can both be customized. * CI configuration of Windows/macOS/Linux is much more uniform by having everything in one build matrix instead of separate matrices. * Linux builds are exclusively done in docker containers in CI now. CI no longer produces two Linux builds only for one to be discarded when artifacts are published. * Windows 32-bit builds are no longer produced in CI since it's expected that everyone actually wants the 64-bit ones instead. * Use of `ccache` is now automatically enabled if it's detected on the system. * Many preexisting shell scripts are now translated to CMake one way or another. * There's no longer a separate build script for how to build wasi-sdk in docker and outside of docker which needs to be kept in sync, everything funnels through the same script. * The `docker/Dockerfile` build of wasi-sdk now uses the actual toolchain built from CI and additionally doesn't duplicate various CMake-based configuration files. Overall one thing I want to additionally point out is that I'm not CMake expert. I suspect there's lots of little stylistic and such improvements that can be made. * Fix skipping tests on windows * Fetch a full depth in the finalize workflow too * Fix multi-arch docker build * Fix assembling of sysroot * Fix script syntax * Clean up the merge script slightly * Remove Pat's email * Move configuration of CMAKE_EXECUTABLE_SUFFIX * Remove redundant sysroot option * Fix comment in testcase.sh * Update new p2 cmake files * Remove now-duplicate wasi-sdk-p2.cmake --- .github/workflows/main.yml | 238 +++++++------- CMakeLists.txt | 45 +++ Dockerfile | 39 --- Makefile | 291 ------------------ RELEASING.md | 3 + ci/build.sh | 49 +++ ci/docker-build.sh | 58 ++++ ci/docker/Dockerfile.arm64-linux | 23 ++ ci/docker/Dockerfile.common | 32 ++ ci/docker/Dockerfile.x86_64-linux | 4 + ci/docker/README.md | 10 + ci/merge-artifacts.sh | 79 +++++ cmake/wasi-sdk-dist.cmake | 37 +++ cmake/wasi-sdk-enable-ccache.cmake | 17 + cmake/wasi-sdk-sysroot.cmake | 276 +++++++++++++++++ cmake/wasi-sdk-toolchain.cmake | 148 +++++++++ deb_from_installation.sh | 43 --- docker/Dockerfile | 68 ++-- docker/wasi-sdk-p2.cmake | 29 -- docker/wasi-sdk-pthread.cmake | 36 --- docker/wasi-sdk.cmake | 29 -- docker_build.sh | 18 -- strip_symbols.sh | 15 - tar_from_installation.sh | 51 --- tests/CMakeLists.txt | 88 ++++++ tests/cmake/CMakeLists.txt | 34 -- tests/cmake/test_driver.sh | 17 - tests/compile-only/CMakeLists.txt | 9 + tests/general/CMakeLists.txt | 8 + tests/general/abort.c.stderr.expected.filter | 2 +- tests/general/assert-fail.c | 4 + tests/general/assert-fail.c.stderr.expected | 2 +- .../assert-fail.c.stderr.expected.filter | 3 +- tests/general/clocks.c.options | 1 - tests/general/iostream_main.cc.options | 1 - tests/general/mmap.c.options | 1 - .../printf-long-double-enabled.c.options | 1 - tests/general/sigabrt.c.options | 1 - .../general/sigabrt.c.stderr.expected.filter | 2 +- tests/general/signals.c | 7 + tests/general/signals.c.options | 1 - tests/run.sh | 89 ------ tests/testcase.sh | 56 +--- wasi-sdk-p2.cmake | 2 +- wasi-sdk.cmake | 2 +- wasi-sdk.control | 4 +- 46 files changed, 1067 insertions(+), 906 deletions(-) create mode 100644 CMakeLists.txt delete mode 100644 Dockerfile delete mode 100644 Makefile create mode 100755 ci/build.sh create mode 100755 ci/docker-build.sh create mode 100644 ci/docker/Dockerfile.arm64-linux create mode 100644 ci/docker/Dockerfile.common create mode 100644 ci/docker/Dockerfile.x86_64-linux create mode 100644 ci/docker/README.md create mode 100755 ci/merge-artifacts.sh create mode 100644 cmake/wasi-sdk-dist.cmake create mode 100644 cmake/wasi-sdk-enable-ccache.cmake create mode 100644 cmake/wasi-sdk-sysroot.cmake create mode 100644 cmake/wasi-sdk-toolchain.cmake delete mode 100755 deb_from_installation.sh delete mode 100644 docker/wasi-sdk-p2.cmake delete mode 100644 docker/wasi-sdk-pthread.cmake delete mode 100644 docker/wasi-sdk.cmake delete mode 100755 docker_build.sh delete mode 100755 strip_symbols.sh delete mode 100755 tar_from_installation.sh create mode 100644 tests/CMakeLists.txt delete mode 100644 tests/cmake/CMakeLists.txt delete mode 100755 tests/cmake/test_driver.sh create mode 100644 tests/compile-only/CMakeLists.txt create mode 100644 tests/general/CMakeLists.txt delete mode 100644 tests/general/clocks.c.options delete mode 100644 tests/general/iostream_main.cc.options delete mode 100644 tests/general/mmap.c.options delete mode 100644 tests/general/printf-long-double-enabled.c.options delete mode 100644 tests/general/sigabrt.c.options delete mode 100644 tests/general/signals.c.options delete mode 100755 tests/run.sh diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 9ccef2620..c0914e658 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -11,37 +11,39 @@ on: jobs: build: - name: Native Build + name: Build ${{ matrix.artifact }} runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: - os: - - ubuntu-latest - - macos-latest + include: + - artifact: x86_64-linux + os: ubuntu-latest + + - artifact: arm64-linux + os: ubuntu-latest + rust_target: aarch64-unknown-linux-gnu + + - artifact: arm64-macos + os: macos-latest + llvm_cmake_flags: -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DCMAKE_OSX_ARCHITECTURES=arm64 + rust_target: aarch64-apple-darwin + + - artifact: x86_64-macos + os: macos-latest + llvm_cmake_flags: -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DCMAKE_OSX_ARCHITECTURES=x86_64 + rust_target: x86_64-apple-darwin + skip_sysroot: 1 + + - artifact: x86_64-windows + os: windows-latest + # TODO: tests are pretty close to passing on Windows but need some + # final tweaks, namely testing the exit code doesn't work since + # exit codes are different on Windows and the `mmap.c` tests seems + # to have issues probably with line endings. Needs someone with a + # Windows checkout tot test further. + skip_tests: 1 steps: - - uses: actions/cache@v4 - with: - path: ~/.cache/ccache - # Bump the prefix number to evict all previous caches and - # enforce a clean build, in the unlikely case that some - # weird build error occur and ccache becomes a potential - # suspect. - key: 0-cache-ubuntu-latest-${{ github.run_id }} - restore-keys: | - 0-cache-ubuntu-latest - if: matrix.os == 'ubuntu-latest' - - uses: actions/cache@v4 - with: - path: ~/Library/Caches/ccache - key: 0-cache-macos-latest-${{ github.run_id }} - restore-keys: | - 0-cache-macos-latest - if: matrix.os == 'macos-latest' - - name: Setup `wasmtime` for tests - uses: bytecodealliance/actions/wasmtime/setup@v1 - with: - version: "18.0.2" - uses: actions/checkout@v4 with: fetch-depth: 0 @@ -53,108 +55,130 @@ jobs: # Server does not allow request for unadvertised object" in the # future. - run: git submodule update --init --depth 32 --jobs 3 + + # Persist ccache-based caches across builds. This directory is configured + # via the CCACHE_DIR env var below for ccache to use. + # + # Bump the prefix number to evict all previous caches and enforce a clean + # build, in the unlikely case that some weird build error occur and ccache + # becomes a potential suspect. + - uses: actions/cache@v4 + id: cache-restore + with: + path: ${{ runner.tool_cache }}/ccache + key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }} + restore-keys: | + 0-cache-${{ matrix.artifact }}- + - run: | + mkdir -p '${{ runner.tool_cache }}/ccache' + echo 'CCACHE_DIR=${{ runner.tool_cache }}/ccache' >> $GITHUB_ENV + shell: bash + + # Configure CMake flags for `ci/build.sh` as necessary for each + # matrix entry. + - run: echo WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS=${{ matrix.llvm_cmake_flags }} >> $GITHUB_ENV + if: matrix.llvm_cmake_flags != '' + shell: bash + - run: | + cmake_args=-DWASI_SDK_ARTIFACT=${{ matrix.artifact }} + if [ "${{ matrix.rust_target }}" != "" ]; then + rustup target add ${{ matrix.rust_target }} + cmake_args="$cmake_args -DRUST_TARGET=${{ matrix.rust_target }}" + fi + echo WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS="$cmake_args" >> $GITHUB_ENV + shell: bash + - run: echo WASI_SDK_CI_SKIP_SYSROOT=1 >> $GITHUB_ENV + shell: bash + if: matrix.skip_sysroot != '' + - run: echo WASI_SDK_CI_SKIP_TESTS=1 >> $GITHUB_ENV + shell: bash + if: matrix.skip_tests != '' + + # Add some extra installed software on each runner as necessary. + - name: Setup `wasmtime` for tests + uses: bytecodealliance/actions/wasmtime/setup@v1 + with: + version: "18.0.2" - name: Install ccache, ninja (macOS) run: brew install ccache ninja - if: matrix.os == 'macos-latest' + if: runner.os == 'macOS' + - name: Install ccache, ninja (Windows) + run: choco install ccache ninja + if: runner.os == 'Windows' - name: Install ccache, ninja (Linux) - run: sudo apt install ccache ninja-build - if: matrix.os == 'ubuntu-latest' - - name: Build - run: NINJA_FLAGS=-v make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON + run: sudo apt install ccache + if: runner.os == 'Linux' + + - name: Build and test (macOS) + run: ./ci/build.sh + if: runner.os == 'macOS' + + - name: Build and test (Linux) + run: ./ci/docker-build.sh ${{ matrix.artifact }} + if: runner.os == 'Linux' + + # Use a shorter build directory than the default on Windows to avoid + # hitting path length and command line length limits. + - name: Build and test (Windows) + run: ./ci/build.sh C:/wasi-sdk shell: bash - - name: Run the testsuite - run: NINJA_FLAGS=-v make check RUNTIME=wasmtime + if: runner.os == 'Windows' + + # Upload the `dist` folder from the build as the artifacts for this + # runner. - name: Upload artifacts uses: actions/upload-artifact@v4 with: - # Upload the dist folder. Give it a name according to the OS it was built for. - name: ${{ format( 'dist-{0}', matrix.os) }} - path: dist + name: ${{ format( 'dist-{0}', matrix.artifact) }} + path: build/dist - winbuild: - name: Windows Build - runs-on: windows-latest - strategy: - fail-fast: false - matrix: - include: - - arch: x64 - sys: clang64 - env: clang-x86_64 - - arch: x86 - sys: clang32 - env: clang-i686 - steps: - - uses: actions/cache@v4 - with: - path: ~/AppData/Local/ccache - key: 0-${{ format( 'cache-windows-latest-{0}', matrix.arch) }}-${{ github.run_id }} - restore-keys: | - 0-${{ format( 'cache-windows-latest-{0}', matrix.arch) }} - - uses: msys2/setup-msys2@v2 - with: - install: >- - base-devel - git - mingw-w64-${{ matrix.env }}-ccache - mingw-w64-${{ matrix.env }}-cmake - mingw-w64-${{ matrix.env }}-ninja - mingw-w64-${{ matrix.env }}-toolchain - msystem: ${{ matrix.sys }} - update: true - release: false - path-type: inherit - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - run: git fetch --tags --force - name: Force-fetch tags to work around actions/checkout#290 - - run: git submodule update --init --depth 32 --jobs 3 - - name: Build - shell: msys2 {0} - run: | - make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON - make check - - name: Does it work sans msys2? - run: | - C:\wasi-sdk\bin\clang.exe --version - C:\wasi-sdk\bin\llvm-ar.exe --version - C:\wasi-sdk\bin\wasm-ld.exe --version - - name: Upload artifacts - uses: actions/upload-artifact@v4 + # Help debug ccache issues by showing what happened. + - if: always() + name: Show ccache statistics + run: ccache --show-stats + + # Always save a cache, even if the build failed. This ensures that if + # live-debugging via CI the build gets to pick up where it left off last + # time instead of having to recreate everything each time a failure + # happens. + - if: always() && steps.cache-restore.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 with: - # Upload the dist folder. Give it a name according to the OS it was built for. - name: ${{ format( 'dist-windows-latest-{0}', matrix.arch) }} - path: dist + path: ${{ runner.tool_cache }}/ccache + key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }} - dockerbuild: - name: Docker Build + # Once all of the above matrix entries have completed this job will run and + # assemble the final `wasi-sdk-*` artifacts by fusing the toolchain/sysroot + # artifacts. + finalize: + name: Finalize wasi-sdk artifacts + needs: build runs-on: ubuntu-latest steps: - - uses: actions/cache@v4 - with: - path: ~/.ccache - key: 0-cache-ubuntu-bionic-${{ github.run_id }} - restore-keys: | - 0-cache-ubuntu-bionic - - uses: actions/checkout@v4 with: fetch-depth: 0 - run: git fetch --tags --force name: Force-fetch tags to work around actions/checkout#290 - - run: git submodule update --init --depth 32 --jobs 3 + # Download all artifacts from all platforms in `build`, merge them into + # final wasi-sdk-* artifacts, and then upload them. + - uses: actions/download-artifact@v4 + - run: ./ci/merge-artifacts.sh + - uses: actions/upload-artifact@v4 + with: + name: release-artifacts + path: dist + # Use the `wasi-sdk-*` artifacts just created to create a docker image + # with a toolchain pre-installed. - uses: docker/login-action@v2 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - - uses: docker/setup-qemu-action@v2 - uses: docker/setup-buildx-action@v2 - - uses: docker/metadata-action@v4 id: meta with: @@ -165,16 +189,6 @@ jobs: type=ref,event=tag type=ref,event=pr type=sha - - - name: Run docker_build script - run: ./docker_build.sh - - name: Upload artifacts - uses: actions/upload-artifact@v4 - with: - # Upload the dist folder. Give it a name according to the OS it was built for. - name: dist-ubuntu-bionic - path: dist - - name: Build and push wasi-sdk docker image uses: docker/build-push-action@v3 with: diff --git a/CMakeLists.txt b/CMakeLists.txt new file mode 100644 index 000000000..90abca435 --- /dev/null +++ b/CMakeLists.txt @@ -0,0 +1,45 @@ +# Build logic for building both a toolchain and a sysroot for WASI. +# +# This top level `CMakeLists.txt` file can be used either to build a clang +# toolchain or a WASI sysroot. Note that this can't be done at the same time. +# A toolchain build requires a compiler for the target architecture. A +# WASI sysroot build requires this previous compiler and must be runnable on +# the host. + +cmake_minimum_required(VERSION 3.26) +project(wasi-sdk) +include(ExternalProject) + +set(WASI_SDK_TARGETS "wasm32-wasi;wasm32-wasip1;wasm32-wasip2;wasm32-wasip1-threads;wasm32-wasi-threads" + CACHE STRING "List of WASI targets to build") +option(WASI_SDK_BUILD_TOOLCHAIN "Build a toolchain instead of the sysroot" OFF) + +set(llvm_proj_dir ${CMAKE_CURRENT_SOURCE_DIR}/src/llvm-project) +set(wasi_libc ${CMAKE_CURRENT_SOURCE_DIR}/src/wasi-libc) + +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake") +include(wasi-sdk-enable-ccache) + +find_program(PYTHON python3 python REQUIRED) + +# Set some variables based on the `version.py` script +set(version_script ${CMAKE_CURRENT_SOURCE_DIR}/version.py) +execute_process( + COMMAND ${PYTHON} ${version_script} llvm-major --llvm-dir=${llvm_proj_dir} + OUTPUT_VARIABLE clang_version + OUTPUT_STRIP_TRAILING_WHITESPACE) +execute_process( + COMMAND ${PYTHON} ${version_script} + OUTPUT_VARIABLE wasi_sdk_version + OUTPUT_STRIP_TRAILING_WHITESPACE) + +message(STATUS "wasi-sdk toolchain LLVM version is ${clang_version}") +message(STATUS "wasi-sdk version is ${wasi_sdk_version}") + +# Only include one version of the build logic as pulling in both isn't +# supported at this time. +if(WASI_SDK_BUILD_TOOLCHAIN) +include(wasi-sdk-toolchain) +else() +include(wasi-sdk-sysroot) +endif() diff --git a/Dockerfile b/Dockerfile deleted file mode 100644 index adddd029c..000000000 --- a/Dockerfile +++ /dev/null @@ -1,39 +0,0 @@ -# Use a relatively old/stable distro here to maximize the supported platforms -# and avoid depending on more recent version of, say, libc. -# Here we choose Bionic 18.04. -FROM ubuntu:bionic - -# We want to use the same UID/GID of the external user to avoid permission -# issues. See the user setup at the end of the file. -ARG UID=1000 -ARG GID=1000 - -RUN apt-get update \ - && apt-get install -y --no-install-recommends \ - ccache \ - curl \ - ca-certificates \ - build-essential \ - clang \ - python3 \ - git \ - ninja-build \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* - -RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.25.1/cmake-3.25.1-linux-x86_64.tar.gz \ - && tar xf cmake-3.25.1-linux-x86_64.tar.gz \ - && rm cmake-3.25.1-linux-x86_64.tar.gz \ - && mkdir -p /opt \ - && mv cmake-3.25.1-linux-x86_64 /opt/cmake -ENV PATH /opt/cmake/bin:$PATH - -ENV RUSTUP_HOME=/rust/rustup CARGO_HOME=/rust/cargo PATH=$PATH:/rust/cargo/bin -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \ - sh -s -- -y --profile=minimal && \ - chmod -R a+w /rust - -RUN groupadd -g ${GID} builder && \ - useradd --create-home --uid ${UID} --gid ${GID} builder -USER builder -WORKDIR /workspace diff --git a/Makefile b/Makefile deleted file mode 100644 index eda2a7454..000000000 --- a/Makefile +++ /dev/null @@ -1,291 +0,0 @@ -# Any copyright is dedicated to the Public Domain. -# http://creativecommons.org/publicdomain/zero/1.0/ - -ROOT_DIR=${CURDIR} -LLVM_PROJ_DIR?=$(ROOT_DIR)/src/llvm-project -VERSION_SCRIPT=python3 ./version.py - -# Windows needs munging -ifeq ($(OS),Windows_NT) - -PREFIX?=c:/wasi-sdk -# we need to explicitly call bash -c for makefile $(shell ...), otherwise we'll try under -# who knows what -BASH=bash -c - -ifeq (x$(MSYSTEM),x) -$(error On Windows, this Makefile only works in MSYS2 environments such as git-bash.) -endif - -# msys needs any /-prefixed arguments, or =/ containing, to turn into // -# because it tries to path-expand the / into the msys root. // escapes this. -ESCAPE_SLASH=/ - -BUILD_PREFIX=$(PREFIX) - -# assuming we're running under msys2 (git-bash), PATH needs /c/foo format directories (because -# it itself is :-delimited) -PATH_PREFIX=$(shell cygpath.exe -u $(BUILD_PREFIX)) - -else - -PREFIX?=/opt/wasi-sdk -DESTDIR=$(abspath build/install) -BUILD_PREFIX=$(DESTDIR)$(PREFIX) -PATH_PREFIX=$(BUILD_PREFIX) -ESCAPE_SLASH?= -BASH= - -endif - -ifeq ($(shell uname),Darwin) -override LLVM_CMAKE_FLAGS += -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ - -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -endif - -TARGETS = wasm32-wasi wasm32-wasip1 wasm32-wasip2 wasm32-wasip1-threads wasm32-wasi-threads - -# Only the major version is needed for Clang, see https://reviews.llvm.org/D125860. -CLANG_VERSION=$(shell $(VERSION_SCRIPT) llvm-major --llvm-dir=$(LLVM_PROJ_DIR)) -VERSION:=$(shell $(VERSION_SCRIPT)) -DEBUG_PREFIX_MAP=-fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION) - -# Generate debuginfo by default for wasi-sdk since it's easily strippable and -# otherwise quite useful for debugging. -WASI_SDK_CFLAGS := $(DEBUG_PREFIX_MAP) -g -WASI_SDK_CXXFLAGS := $(WASI_SDK_CFLAGS) - -default: build - @echo "Use -fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION)" - -check: - TARGETS="$(TARGETS)" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" - -clean: - rm -rf build $(DESTDIR) - -# Default symlinks that clang creates to the `clang` executable -CLANG_LINKS_TO_CREATE = clang++ clang-cl clang-cpp - -# Add target-prefixed versions of `clang` and `clang++` so they can be used -# without `--target` as it's auto-inferred from the executable name by clang. -CLANG_LINKS_TO_CREATE += $(foreach target,$(TARGETS),$(target)-clang) -CLANG_LINKS_TO_CREATE += $(foreach target,$(TARGETS),$(target)-clang++) - -# Small helper to create a `join-with` function that can join elements of a -# list with a defined separator. -noop = -space = $(noop) $(noop) -join-with = $(subst $(space),$1,$(strip $2)) - -build/llvm.BUILT: - mkdir -p build/llvm - cd build/llvm && cmake -G Ninja \ - -DCLANG_LINKS_TO_CREATE="$(call join-with,;,$(CLANG_LINKS_TO_CREATE))" \ - -DCMAKE_BUILD_TYPE=MinSizeRel \ - -DLLVM_ENABLE_TERMINFO=OFF \ - -DLLVM_ENABLE_ZLIB=OFF \ - -DLLVM_ENABLE_ZSTD=OFF \ - -DLLVM_STATIC_LINK_CXX_STDLIB=ON \ - -DCMAKE_INSTALL_PREFIX=$(PREFIX) \ - -DLLVM_INCLUDE_TESTS=OFF \ - -DLLVM_INCLUDE_UTILS=OFF \ - -DLLVM_INCLUDE_BENCHMARKS=OFF \ - -DLLVM_INCLUDE_EXAMPLES=OFF \ - -DLLVM_TARGETS_TO_BUILD=WebAssembly \ - -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi \ - -DLLVM_ENABLE_PROJECTS="lld;clang;clang-tools-extra" \ - $(if $(patsubst 9,,$(CLANG_VERSION)), \ - $(if $(patsubst 10,,$(CLANG_VERSION)), \ - -DDEFAULT_SYSROOT=../share/wasi-sysroot, \ - -DDEFAULT_SYSROOT=$(PREFIX)/share/wasi-sysroot), \ - -DDEFAULT_SYSROOT=$(PREFIX)/share/wasi-sysroot) \ - -DLLVM_INSTALL_BINUTILS_SYMLINKS=TRUE \ - -DLLVM_ENABLE_LIBXML2=OFF \ - $(LLVM_CMAKE_FLAGS) \ - $(LLVM_PROJ_DIR)/llvm - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/llvm \ - install-clang \ - install-clang-format \ - install-clang-tidy \ - install-clang-apply-replacements \ - install-lld \ - install-llvm-mc \ - install-llvm-ranlib \ - install-llvm-strip \ - install-llvm-dwarfdump \ - install-clang-resource-headers \ - install-ar \ - install-ranlib \ - install-strip \ - install-nm \ - install-size \ - install-strings \ - install-objdump \ - install-objcopy \ - install-c++filt \ - llvm-config - touch build/llvm.BUILT - -# Build the `wasm-component-ld` linker from source via `cargo install`. This is -# used for the `wasm32-wasip2` target natively by Clang. Note that `--root` -# passed to `cargo install` will place it in the output directory automatically. -build/wasm-component-ld.BUILT: build/llvm.BUILT - cargo install --no-track wasm-component-ld@0.5.0 --root $(BUILD_PREFIX) - touch build/wasm-component-ld.BUILT - - -# Flags for running `make` in wasi-libc -# $(1): the target that's being built -WASI_LIBC_MAKEFLAGS = \ - -C $(ROOT_DIR)/src/wasi-libc \ - CC=$(BUILD_PREFIX)/bin/clang \ - AR=$(BUILD_PREFIX)/bin/llvm-ar \ - NM=$(BUILD_PREFIX)/bin/llvm-nm \ - SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - EXTRA_CFLAGS="$(WASI_SDK_CFLAGS) -O2 -DNDEBUG" \ - TARGET_TRIPLE=$(1) - -build/wasi-libc.BUILT: build/compiler-rt.BUILT build/wasm-component-ld.BUILT - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasi) default libc_so - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip1) default libc_so - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip2) WASI_SNAPSHOT=p2 default libc_so - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasi-threads) THREAD_MODEL=posix - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip1-threads) THREAD_MODEL=posix - touch build/wasi-libc.BUILT - -build/compiler-rt.BUILT: build/llvm.BUILT - # Do the build, and install it. - mkdir -p build/compiler-rt - cd build/compiler-rt && cmake -G Ninja \ - -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_COMPILER_WORKS=ON \ - -DCMAKE_CXX_COMPILER_WORKS=ON \ - -DCMAKE_AR=$(BUILD_PREFIX)/bin/ar \ - -DCMAKE_MODULE_PATH=$(ROOT_DIR)/cmake \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DCMAKE_TOOLCHAIN_FILE=$(ROOT_DIR)/wasi-sdk.cmake \ - -DCOMPILER_RT_BAREMETAL_BUILD=On \ - -DCOMPILER_RT_BUILD_XRAY=OFF \ - -DCOMPILER_RT_INCLUDE_TESTS=OFF \ - -DCOMPILER_RT_HAS_FPIC_FLAG=OFF \ - -DCOMPILER_RT_ENABLE_IOS=OFF \ - -DCOMPILER_RT_DEFAULT_TARGET_ONLY=On \ - -DWASI_SDK_PREFIX=$(BUILD_PREFIX) \ - -DCMAKE_C_FLAGS="$(WASI_SDK_CFLAGS)" \ - -DLLVM_CONFIG_PATH=$(ROOT_DIR)/build/llvm/bin/llvm-config \ - -DCOMPILER_RT_OS_DIR=wasi \ - -DCMAKE_INSTALL_PREFIX=$(PREFIX)/lib/clang/$(CLANG_VERSION)/ \ - -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ - $(LLVM_PROJ_DIR)/compiler-rt/lib/builtins - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/compiler-rt install - # Install clang-provided headers. - cp -R $(ROOT_DIR)/build/llvm/lib/clang $(BUILD_PREFIX)/lib/ - cp -R $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasi $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasip1 - cp -R $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasi $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasip2 - touch build/compiler-rt.BUILT - -# Flags for libcxx and libcxxabi. -# $(1): pthreads ON or OFF -# $(2): shared libraries ON or OFF -# $(3): the name of the target being built for -# $(4): extra compiler flags to pass -LIBCXX_CMAKE_FLAGS = \ - -DCMAKE_C_COMPILER_WORKS=ON \ - -DCMAKE_CXX_COMPILER_WORKS=ON \ - -DCMAKE_AR=$(BUILD_PREFIX)/bin/ar \ - -DCMAKE_MODULE_PATH=$(ROOT_DIR)/cmake \ - -DCMAKE_TOOLCHAIN_FILE=$(ROOT_DIR)/wasi-sdk.cmake \ - -DCMAKE_STAGING_PREFIX=$(PREFIX)/share/wasi-sysroot \ - -DCMAKE_POSITION_INDEPENDENT_CODE=$(2) \ - -DLLVM_CONFIG_PATH=$(ROOT_DIR)/build/llvm/bin/llvm-config \ - -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ - -DCXX_SUPPORTS_CXX11=ON \ - -DLIBCXX_ENABLE_THREADS:BOOL=$(1) \ - -DLIBCXX_HAS_PTHREAD_API:BOOL=$(1) \ - -DLIBCXX_HAS_EXTERNAL_THREAD_API:BOOL=OFF \ - -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ - -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF \ - -DLLVM_COMPILER_CHECKED=ON \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DLIBCXX_ENABLE_SHARED:BOOL=$(2) \ - -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF \ - -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF \ - -DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON \ - -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF \ - -DLIBCXX_CXX_ABI=libcxxabi \ - -DLIBCXX_CXX_ABI_INCLUDE_PATHS=$(LLVM_PROJ_DIR)/libcxxabi/include \ - -DLIBCXX_HAS_MUSL_LIBC:BOOL=ON \ - -DLIBCXX_ABI_VERSION=2 \ - -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF \ - -DLIBCXXABI_ENABLE_SHARED:BOOL=$(2) \ - -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON \ - -DLIBCXXABI_ENABLE_THREADS:BOOL=$(1) \ - -DLIBCXXABI_HAS_PTHREAD_API:BOOL=$(1) \ - -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF \ - -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ - -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF \ - -DLIBCXXABI_ENABLE_PIC:BOOL=$(2) \ - -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=OFF \ - -DWASI_SDK_PREFIX=$(BUILD_PREFIX) \ - -DUNIX:BOOL=ON \ - --debug-trycompile \ - -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(WASI_SDK_CFLAGS) $(EXTRA_CFLAGS) $(4) --target=$(3)" \ - -DCMAKE_CXX_FLAGS="$(WASI_SDK_CXXFLAGS) $(EXTRA_CXXFLAGS) $(4) --target=$(3)" \ - -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/$(3) \ - -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/$(3) \ - -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ - $(LLVM_PROJ_DIR)/runtimes - -# Rules to build libcxx, factored out here to deduplicate the below -# $(1): pthreads ON or OFF -# $(2): shared libraries ON or OFF -# $(3): the name of the target being built for -define BUILD_LIBCXX - mkdir -p build/libcxx-$(3) - cd build/libcxx-$(3) && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,$(1),$(2),$(3),$(4)) - ninja $(NINJA_FLAGS) -C build/libcxx-$(3) - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx-$(3) install - rm -rf $(BUILD_PREFIX)/share/wasi-sysroot/include/$(3)/c++ - mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/$(3)/ -endef - -build/libcxx.BUILT: build/llvm.BUILT build/wasi-libc.BUILT - $(call BUILD_LIBCXX,OFF,ON,wasm32-wasi) - $(call BUILD_LIBCXX,OFF,ON,wasm32-wasip1) - $(call BUILD_LIBCXX,OFF,ON,wasm32-wasip2) - $(call BUILD_LIBCXX,ON,OFF,wasm32-wasi-threads,-pthread) - $(call BUILD_LIBCXX,ON,OFF,wasm32-wasip1-threads,-pthread) - # As of this writing, `clang++` will ignore the above include dirs unless this one also exists: - mkdir -p $(BUILD_PREFIX)/share/wasi-sysroot/include/c++/v1 - touch build/libcxx.BUILT - -build/config.BUILT: - mkdir -p $(BUILD_PREFIX)/share/misc - cp src/config/config.sub src/config/config.guess $(BUILD_PREFIX)/share/misc - mkdir -p $(BUILD_PREFIX)/share/cmake/Platform - cp wasi-sdk.cmake $(BUILD_PREFIX)/share/cmake - cp wasi-sdk-pthread.cmake $(BUILD_PREFIX)/share/cmake - cp wasi-sdk-p2.cmake $(BUILD_PREFIX)/share/cmake - cp cmake/Platform/WASI.cmake $(BUILD_PREFIX)/share/cmake/Platform - touch build/config.BUILT - -build/version.BUILT: - $(VERSION_SCRIPT) dump > $(BUILD_PREFIX)/VERSION - touch build/version.BUILT - -build: build/llvm.BUILT build/wasi-libc.BUILT build/compiler-rt.BUILT build/libcxx.BUILT build/config.BUILT build/version.BUILT - -strip: build/llvm.BUILT - ./strip_symbols.sh $(BUILD_PREFIX)/bin - -package: build/package.BUILT - -build/package.BUILT: build strip - mkdir -p dist - ./deb_from_installation.sh $(shell pwd)/dist "$(VERSION)" "$(BUILD_PREFIX)" - ./tar_from_installation.sh "$(shell pwd)/dist" "$(VERSION)" "$(BUILD_PREFIX)" - touch build/package.BUILT - -.PHONY: default clean build strip package check diff --git a/RELEASING.md b/RELEASING.md index c684149be..6bc387974 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,5 +1,8 @@ # Release Process +> **Note**: These instructions are out-of-date given the latest refactoring of +> the build system and should get updated before the next release. + To publish a new version of `wasi-sdk` as a GitHub release: 1. Tag a commit with an annotated tag. Note that this must be an annotated tag, diff --git a/ci/build.sh b/ci/build.sh new file mode 100755 index 000000000..f391e8ab4 --- /dev/null +++ b/ci/build.sh @@ -0,0 +1,49 @@ +#!/bin/bash + +# Build logic executed in CI. This is intentionally kept relatively minimal to +# one day not live in bash to have a bash-less build on Windows. For now though +# this will unconditionally build a toolchain and then optionally build a +# sysroot. Builders which can't actually execute the toolchain they produce +# skip the sysroot step below. + +set -ex + +# Optionally allow the first argument to this script to be the install +# location. +if [ "$1" = "" ]; then + install_dir=`pwd`/build/install +else + install_dir="$1" +fi + +cmake -G Ninja -B build/toolchain -S . \ + -DWASI_SDK_BUILD_TOOLCHAIN=ON \ + "-DCMAKE_INSTALL_PREFIX=$install_dir" \ + $WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ + "-DLLVM_CMAKE_FLAGS=$WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS" +ninja -C build/toolchain install dist -v + +mv build/toolchain/dist build/dist + +if [ "$WASI_SDK_CI_SKIP_SYSROOT" = "1" ]; then + exit 0 +fi + +# Use the just-built toolchain and its `CMAKE_TOOLCHAIN_FILE` to build a +# sysroot. +cmake -G Ninja -B build/sysroot -S . \ + "-DCMAKE_TOOLCHAIN_FILE=$install_dir/share/cmake/wasi-sdk.cmake" \ + -DCMAKE_C_COMPILER_WORKS=ON \ + -DCMAKE_CXX_COMPILER_WORKS=ON \ + -DWASI_SDK_INCLUDE_TESTS=ON \ + "-DCMAKE_INSTALL_PREFIX=$install_dir" +ninja -C build/sysroot install dist -v + +mv build/sysroot/dist/* build/dist + +if [ "$WASI_SDK_CI_SKIP_TESTS" = "1" ]; then + exit 0 +fi + +# Run tests to ensure that the sysroot works. +ctest --output-on-failure --parallel 10 --test-dir build/sysroot/tests diff --git a/ci/docker-build.sh b/ci/docker-build.sh new file mode 100755 index 000000000..a695d6f4a --- /dev/null +++ b/ci/docker-build.sh @@ -0,0 +1,58 @@ +#!/bin/sh + +# This is a helper script invoked from CI which will execute the `ci/build.sh` +# script within a docker container. This builds `ci/docker/Dockerfile.common` +# along with the specified `ci/docker/Dockerfile.$x` from the command line. +# This container is then used to execute `ci/build.sh`. + +set -e + +if [ "$1" = "" ]; then + echo "Usage: $0 " + echo "" + echo "example: $0 x86_64-linux" + exit 1 +fi + +set -x + +# Build the base image which the image below can used. +docker build \ + --file ci/docker/Dockerfile.common \ + --tag wasi-sdk-builder-base \ + ci/docker + +# Build the container that is going to be used +docker build \ + --file ci/docker/Dockerfile.$1 \ + --tag wasi-sdk-builder \ + ci/docker + +# Perform the build in `/src`. The current directory is mounted read-write at +# this location as well. To ensure that container-created files are reasonable +# on the host as well the `--user` is passed to configure various permissions. +args="--workdir /src --volume `pwd`:/src:Z" +args="$args --user $(id -u):$(id -g)" + +# Persist the ccache directory on the host to ensure repeated runs/debugging +# of this container don't take forever. Also enables caching in CI. +ccache_dir=$CCACHE_DIR +if [ "$ccache_dir" = "" ]; then + ccache_dir=$HOME/.ccache +fi +args="$args --volume $ccache_dir:/ccache:Z --env CCACHE_DIR=/ccache" + +# Inherit some tools from the host into this container. This ensures that the +# decision made on CI of what versions to use is the canonical source of truth +# for theset ools +args="$args --volume `rustc --print sysroot`:/rustc:ro" +args="$args --volume $(dirname $(which wasmtime)):/wasmtime:ro" + +# Before running `ci/build.sh` set up some rust/PATH related info to use what +# was just mounted above, and then execute the build. +docker run \ + $args \ + --tty \ + --init \ + wasi-sdk-builder \ + bash -c 'CARGO_HOME=/tmp/cargo-home PATH=$PATH:/rustc/bin:/wasmtime exec ci/build.sh' diff --git a/ci/docker/Dockerfile.arm64-linux b/ci/docker/Dockerfile.arm64-linux new file mode 100644 index 000000000..fa9a16ea4 --- /dev/null +++ b/ci/docker/Dockerfile.arm64-linux @@ -0,0 +1,23 @@ +FROM wasi-sdk-builder-base + +RUN apt-get install -y g++-aarch64-linux-gnu + +# Don't build a sysroot for this cross-compiled target since it would require a +# host compiler and the sysroot is otherwise already built on other CI builders. +ENV WASI_SDK_CI_SKIP_SYSROOT 1 + +ENV WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS \ + -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \ + -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \ + -DCMAKE_CROSSCOMPILING=True \ + -DCMAKE_CXX_FLAGS=-march=armv8-a \ + -DCMAKE_SYSTEM_PROCESSOR=arm64 \ + -DCMAKE_SYSTEM_NAME=Linux \ + -DLLVM_HOST_TRIPLE=aarch64-linux-gnu \ + -DRUST_TARGET=aarch64-unknown-linux-gnu + +ENV WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ + -DWASI_SDK_ARTIFACT=arm64-linux \ + -DRUST_TARGET=aarch64-unknown-linux-gnu + +ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER aarch64-linux-gnu-gcc diff --git a/ci/docker/Dockerfile.common b/ci/docker/Dockerfile.common new file mode 100644 index 000000000..1ab98a61a --- /dev/null +++ b/ci/docker/Dockerfile.common @@ -0,0 +1,32 @@ +# Use a relatively old/stable distro here to maximize the supported platforms +# and avoid depending on more recent version of, say, libc. +# Here we choose Bionic 18.04. + +FROM ubuntu:18.04 + +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + ccache \ + curl \ + ca-certificates \ + build-essential \ + clang \ + python3 \ + git \ + unzip \ + xz-utils + +RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.29.5/cmake-3.29.5-linux-x86_64.tar.gz \ + && tar xf cmake-3.29.5-linux-x86_64.tar.gz \ + && rm cmake-3.29.5-linux-x86_64.tar.gz \ + && mkdir -p /opt \ + && mv cmake-3.29.5-linux-x86_64 /opt/cmake + +ENV PATH /opt/cmake/bin:$PATH + +RUN curl -sSLO https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip \ + && unzip ninja-linux.zip \ + && rm *.zip \ + && mv ninja /opt/cmake/bin + +ENV XDG_CACHE_HOME /tmp/cache diff --git a/ci/docker/Dockerfile.x86_64-linux b/ci/docker/Dockerfile.x86_64-linux new file mode 100644 index 000000000..47473e691 --- /dev/null +++ b/ci/docker/Dockerfile.x86_64-linux @@ -0,0 +1,4 @@ +FROM wasi-sdk-builder-base + +ENV WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ + -DWASI_SDK_ARTIFACT=x86_64-linux diff --git a/ci/docker/README.md b/ci/docker/README.md new file mode 100644 index 000000000..09ed7bcf4 --- /dev/null +++ b/ci/docker/README.md @@ -0,0 +1,10 @@ +# About + +This folder contains the docker images that are used in CI to build the wasi-sdk +release toolchains. Docker is used to intentionally use older Linux +distributions to build the toolchain to have a more maximal set of glibc +compatibility. + +These images are intended to be used on an x86\_64 host. Images start from the +`Dockerfile.common` file and then layer on target-specific +toolchains/options/etc as necessary. diff --git a/ci/merge-artifacts.sh b/ci/merge-artifacts.sh new file mode 100755 index 000000000..0ef7e719d --- /dev/null +++ b/ci/merge-artifacts.sh @@ -0,0 +1,79 @@ +#!/bin/sh + +# Helper script executed on CI once all builds have completed. This takes +# `wasi-toolchain-*` artifacts and `wasi-sysroot-*` artifacts and merges +# them together into a single `wasi-sdk-*` artifact. Toolchains which don't +# have a sysroot that they themselves built use a sysroot from the x86_64-linux +# toolchain. + +set -ex + +rm -rf dist +mkdir dist +version=$(./version.py) + +make_deb() { + build=$1 + dir=$2 + + if ! command -v dpkg-deb >/dev/null; then + return + fi + + case $build in + dist-x86_64-linux) deb_arch=amd64 ;; + dist-arm64-linux) deb_arch=arm64 ;; + *) + echo "unknown build $build" + exit 1 + esac + + mkdir dist/pkg + mkdir dist/pkg/opt + mkdir dist/pkg/DEBIAN + sed s/VERSION/$version/ wasi-sdk.control | \ + sed s/ARCH/$deb_arch/ > dist/pkg/DEBIAN/control + cp -R $dir dist/pkg/opt/wasi-sdk + deb_name=$(echo $(basename $dir) | sed 's/.tar.gz//') + (cd dist && dpkg-deb -b pkg $deb_name.deb) + rm -rf dist/pkg +} + +compiler_rt=`ls dist-x86_64-linux/libclang_rt*` + +for build in dist-*; do + toolchain=`ls $build/wasi-toolchain-*` + if [ -f $build/wasi-sysroot-* ]; then + sysroot=`ls $build/wasi-sysroot-*` + else + sysroot=`ls dist-x86_64-linux/wasi-sysroot-*` + fi + + sdk_dir=`basename $toolchain | sed 's/.tar.gz//' | sed s/toolchain/sdk/` + mkdir dist/$sdk_dir + + # Start with the toolchain and then overlay the sysroot into + # `share/wasi-sysroot`, the default sysroot. + tar xf $toolchain -C dist/$sdk_dir --strip-components 1 + mkdir -p dist/$sdk_dir/share/wasi-sysroot + tar xf $sysroot -C dist/$sdk_dir/share/wasi-sysroot --strip-components 1 + + # Setup the compiler-rt library for wasi,wasip1,wasip2 + rtlibdir=$(dirname $(find dist/$sdk_dir/lib -name include))/lib + mkdir -p $rtlibdir/wasi + tar xf $compiler_rt -C $rtlibdir/wasi --strip-components 1 + cp -r $rtlibdir/wasi $rtlibdir/wasip1 + cp -r $rtlibdir/wasi $rtlibdir/wasip2 + + tar czf dist/$sdk_dir.tar.gz -C dist $sdk_dir + + if echo $build | grep -q linux; then + make_deb $build dist/$sdk_dir + fi + rm -rf dist/$sdk_dir +done + +# In addition to `wasi-sdk-*` also preserve artifacts for just the sysroot +# and just compiler-rt. +cp dist-x86_64-linux/wasi-sysroot-* dist +cp dist-x86_64-linux/libclang_rt* dist diff --git a/cmake/wasi-sdk-dist.cmake b/cmake/wasi-sdk-dist.cmake new file mode 100644 index 000000000..8a9b2b3a1 --- /dev/null +++ b/cmake/wasi-sdk-dist.cmake @@ -0,0 +1,37 @@ +# Helper function to create tarballs for wasi-sdk. +# +# The `target` is the name of the CMake target to create for the creation of +# this tarball. The `tarball` argument is where the final tarball will be +# located. The name of the tarball is also used for the name of the root folder +# in the tarball. The `dir` argument is is the directory that will get packaged +# up within the tarball. +function(wasi_sdk_add_tarball target tarball dir) + cmake_path(GET tarball PARENT_PATH tarball_dir) + + # Run STEM twice to chop of both `.gz` and `.tar` in `.tar.gz` + cmake_path(GET tarball STEM LAST_ONLY tarball_stem) + cmake_path(GET tarball_stem STEM LAST_ONLY tarball_stem) + + if(CMAKE_SYSTEM_NAME MATCHES Windows) + # Copy the contents of symlinks on Windows to avoid dealing with symlink + set(copy_dir ${CMAKE_COMMAND} -E copy_directory ${dir} ${tarball_stem}) + else() + # ... but on non-Windows copy symlinks themselves to cut down on + # distribution size. + set(copy_dir cp -R ${dir} ${tarball_stem}) + endif() + + add_custom_command( + OUTPUT ${tarball} + # First copy the directory under a different name, the filestem of the + # tarball. + COMMAND ${copy_dir} + # Next use CMake to create the tarball itself + COMMAND ${CMAKE_COMMAND} -E tar cfz ${tarball} ${tarball_stem} + # Finally delete the temporary directory created above. + COMMAND ${CMAKE_COMMAND} -E rm -rf ${tarball_stem} + WORKING_DIRECTORY ${tarball_dir} + COMMENT "Creating ${tarball}..." + ) + add_custom_target(${target} DEPENDS ${tarball}) +endfunction() diff --git a/cmake/wasi-sdk-enable-ccache.cmake b/cmake/wasi-sdk-enable-ccache.cmake new file mode 100644 index 000000000..f5e1f304e --- /dev/null +++ b/cmake/wasi-sdk-enable-ccache.cmake @@ -0,0 +1,17 @@ +# Helper module to auto-enable ccache if detected. + +find_program(CCACHE ccache) + +option(WASI_SDK_DISABLE_CCACHE "Force disable ccache even if it's found" OFF) + +if(NOT CMAKE_C_COMPILER_LAUNCHER) + if(NOT WASI_SDK_DISABLE_CCACHE) + if(CCACHE) + set(CMAKE_C_COMPILER_LAUNCHER ccache) + set(CMAKE_CXX_COMPILER_LAUNCHER ccache) + message(STATUS "Auto-enabling ccache") + else() + message(STATUS "Failed to auto-enable ccache, not found on system") + endif() + endif() +endif() diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake new file mode 100644 index 000000000..5c23fe610 --- /dev/null +++ b/cmake/wasi-sdk-sysroot.cmake @@ -0,0 +1,276 @@ +# Build logic for building a sysroot for wasi-sdk which includes compiler-rt, +# wasi-libc, libcxx, and libcxxabi. + +if(NOT CMAKE_BUILD_TYPE) + set(CMAKE_BUILD_TYPE RelWithDebInfo) +endif() + +find_program(MAKE make REQUIRED) + +option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" ON) +option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) + +set(wasi_sysroot ${CMAKE_INSTALL_PREFIX}/share/wasi-sysroot) + +if(WASI_SDK_DEBUG_PREFIX_MAP) + add_compile_options( + -fdebug-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=wasisdk://v${wasi_sdk_version}) +endif() + + +# Default arguments for builds of cmake projects (mostly LLVM-based) to forward +# along much of our own configuration into these projects. +set(default_cmake_args + -DCMAKE_SYSTEM_NAME=WASI + -DCMAKE_SYSTEM_VERSION=1 + -DCMAKE_SYSTEM_PROCESSOR=wasm32 + -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} + -DCMAKE_AR=${CMAKE_AR} + -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} + -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} + -DCMAKE_C_COMPILER_WORKS=ON + -DCMAKE_CXX_COMPILER_WORKS=ON + -DCMAKE_SYSROOT=${wasi_sysroot} + -DCMAKE_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}/cmake) + +if(CMAKE_C_COMPILER_LAUNCHER) + list(APPEND default_cmake_args -DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}) +endif() +if(CMAKE_CXX_COMPILER_LAUNCHER) + list(APPEND default_cmake_args -DCMAKE_CXX_COMPILER_LAUNCHER=${CMAKE_CXX_COMPILER_LAUNCHER}) +endif() + +# ============================================================================= +# compiler-rt build logic +# ============================================================================= + +set(compiler_rt_dst ${CMAKE_INSTALL_PREFIX}/lib/clang/${clang_version}) +ExternalProject_Add(compiler-rt-build + SOURCE_DIR "${llvm_proj_dir}/compiler-rt" + CMAKE_ARGS + ${default_cmake_args} + -DCOMPILER_RT_BAREMETAL_BUILD=ON + -DCOMPILER_RT_BUILD_XRAY=OFF + -DCOMPILER_RT_INCLUDE_TESTS=OFF + -DCOMPILER_RT_HAS_FPIC_FLAG=OFF + -DCOMPILER_RT_ENABLE_IOS=OFF + -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON + -DCMAKE_C_COMPILER_TARGET=wasm32-wasi + -DCOMPILER_RT_OS_DIR=wasi + -DCMAKE_INSTALL_PREFIX=${compiler_rt_dst} + EXCLUDE_FROM_ALL ON + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON +) + +# In addition to the default installation of `compiler-rt` itself also copy +# around some headers and make copies of the `wasi` directory as `wasip1` and +# `wasip2` +execute_process( + COMMAND ${CMAKE_C_COMPILER} -print-runtime-dir + OUTPUT_VARIABLE clang_runtime_dir + OUTPUT_STRIP_TRAILING_WHITESPACE) +cmake_path(GET clang_runtime_dir PARENT_PATH clang_runtime_libdir) # chop off `wasi` +cmake_path(GET clang_runtime_libdir PARENT_PATH clang_sysroot_dir) # chop off `lib` +add_custom_target(compiler-rt-post-build + COMMAND ${CMAKE_COMMAND} -E copy_directory + ${clang_sysroot_dir} ${compiler_rt_dst} + COMMAND ${CMAKE_COMMAND} -E copy_directory + ${compiler_rt_dst}/lib/wasi ${compiler_rt_dst}/lib/wasip1 + COMMAND ${CMAKE_COMMAND} -E copy_directory + ${compiler_rt_dst}/lib/wasi ${compiler_rt_dst}/lib/wasip2 + COMMAND ${CMAKE_COMMAND} -E copy_directory_if_different + ${compiler_rt_dst}/lib ${clang_runtime_libdir} + COMMENT "finalizing compiler-rt installation" +) +add_dependencies(compiler-rt-post-build compiler-rt-build) + +add_custom_target(compiler-rt DEPENDS compiler-rt-build compiler-rt-post-build) + +# ============================================================================= +# wasi-libc build logic +# ============================================================================= + +function(define_wasi_libc target) + set(build_dir ${CMAKE_CURRENT_BINARY_DIR}/wasi-libc-${target}) + + if(${target} MATCHES threads) + set(extra_make_flags THREAD_MODEL=posix) + elseif(${target} MATCHES p2) + set(extra_make_flags WASI_SNAPSHOT=p2 default libc_so) + else() + set(extra_make_flags default libc_so) + endif() + + string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPER) + get_property(directory_cflags DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) + set(extra_cflags_list + "${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") + list(JOIN extra_cflags_list " " extra_cflags) + + ExternalProject_Add(wasi-libc-${target} + # Currently wasi-libc doesn't support out-of-tree builds so feigh a + # "download command" which copies the source tree to a different location + # so out-of-tree builds are supported. + DOWNLOAD_COMMAND + ${CMAKE_COMMAND} -E copy_directory ${wasi_libc} ${build_dir} + SOURCE_DIR "${build_dir}" + CONFIGURE_COMMAND "" + BUILD_COMMAND + ${MAKE} -j8 -C ${build_dir} + CC=${CMAKE_C_COMPILER} + AR=${CMAKE_AR} + NM=${CMAKE_NM} + SYSROOT=${wasi_sysroot} + EXTRA_CFLAGS=${extra_cflags} + TARGET_TRIPLE=${target} + ${extra_make_flags} + INSTALL_COMMAND "" + DEPENDS compiler-rt + EXCLUDE_FROM_ALL ON + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON + ) +endfunction() + +foreach(target IN LISTS WASI_SDK_TARGETS) + define_wasi_libc(${target}) +endforeach() + +# ============================================================================= +# libcxx build logic +# ============================================================================= + +function(define_libcxx target) + if(${target} MATCHES threads) + set(threads ON) + set(pic OFF) + set(target_flags -pthread) + else() + set(threads OFF) + set(pic ON) + set(target_flags "") + endif() + + set(runtimes "libcxx;libcxxabi") + + get_property(dir_compile_opts DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) + get_property(dir_link_opts DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY LINK_OPTIONS) + set(extra_cflags_list ${CMAKE_C_FLAGS} ${target_flags} --target=${target} ${dir_compile_opts} ${dir_link_opts}) + list(JOIN extra_cflags_list " " extra_cflags) + set(extra_cxxflags_list ${CMAKE_CXX_FLAGS} ${target_flags} --target=${target} ${dir_compile_opts} ${dir_link_opts}) + list(JOIN extra_cxxflags_list " " extra_cxxflags) + + ExternalProject_Add(libcxx-${target}-build + SOURCE_DIR ${llvm_proj_dir}/runtimes + CMAKE_ARGS + ${default_cmake_args} + # Ensure headers are installed in a target-specific path instead of a + # target-generic path. + -DCMAKE_INSTALL_INCLUDEDIR=${wasi_sysroot}/include/${target} + -DCMAKE_STAGING_PREFIX=${wasi_sysroot} + -DCMAKE_POSITION_INDEPENDENT_CODE=${pic} + -DCXX_SUPPORTS_CXX11=ON + -DLIBCXX_ENABLE_THREADS:BOOL=${threads} + -DLIBCXX_HAS_PTHREAD_API:BOOL=${threads} + -DLIBCXX_HAS_EXTERNAL_THREAD_API:BOOL=OFF + -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF + -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF + -DLLVM_COMPILER_CHECKED=ON + -DLIBCXX_ENABLE_SHARED:BOOL=${pic} + -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF + -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF + -DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON + -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF + -DLIBCXX_CXX_ABI=libcxxabi + -DLIBCXX_CXX_ABI_INCLUDE_PATHS=${llvm_proj_dir}/libcxxabi/include + -DLIBCXX_HAS_MUSL_LIBC:BOOL=ON + -DLIBCXX_ABI_VERSION=2 + -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF + -DLIBCXXABI_ENABLE_SHARED:BOOL=${pic} + -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON + -DLIBCXXABI_ENABLE_THREADS:BOOL=${threads} + -DLIBCXXABI_HAS_PTHREAD_API:BOOL=${threads} + -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF + -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF + -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF + -DLIBCXXABI_ENABLE_PIC:BOOL=${pic} + -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=OFF + -DUNIX:BOOL=ON + -DCMAKE_C_FLAGS=${extra_cflags} + -DCMAKE_CXX_FLAGS=${extra_cxxflags} + -DLIBCXX_LIBDIR_SUFFIX=/${target} + -DLIBCXXABI_LIBDIR_SUFFIX=/${target} + + # See https://www.scivision.dev/cmake-externalproject-list-arguments/ for + # why this is in `CMAKE_CACHE_ARGS` instead of above + CMAKE_CACHE_ARGS + -DLLVM_ENABLE_RUNTIMES:STRING=${runtimes} + DEPENDS + wasi-libc-${target} + compiler-rt + EXCLUDE_FROM_ALL ON + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON + ) + + # As of this writing, `clang++` will ignore the target-specific include dirs + # unless this one also exists: + add_custom_target(libcxx-${target}-extra-dir + COMMAND ${CMAKE_COMMAND} -E make_directory ${wasi_sysroot}/include/c++/v1 + COMMENT "creating libcxx-specific header file folder") + add_custom_target(libcxx-${target} + DEPENDS libcxx-${target}-build libcxx-${target}-extra-dir) +endfunction() + +foreach(target IN LISTS WASI_SDK_TARGETS) + define_libcxx(${target}) +endforeach() + +# ============================================================================= +# misc build logic +# ============================================================================= + +# Add a top-level `build` target as well as `build-$target` targets. +add_custom_target(build ALL) +foreach(target IN LISTS WASI_SDK_TARGETS) + add_custom_target(build-${target}) + add_dependencies(build-${target} libcxx-${target} wasi-libc-${target} compiler-rt) + add_dependencies(build build-${target}) +endforeach() + +# Install a `VERSION` file in the output prefix with a dump of version +# information. +set(version_file_tmp ${CMAKE_CURRENT_BINARY_DIR}/VERSION) +execute_process( + COMMAND ${PYTHON} ${version_script} dump + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + OUTPUT_FILE ${version_file_tmp}) +install( + FILES ${version_file_tmp} + DESTINATION ${CMAKE_INSTALL_PREFIX}) + +if(WASI_SDK_INCLUDE_TESTS) + add_subdirectory(tests) +endif() + +include(wasi-sdk-dist) + +set(dist_dir ${CMAKE_CURRENT_BINARY_DIR}/dist) + +# Tarball with just `compiler-rt` builtins within it +wasi_sdk_add_tarball(dist-compiler-rt + ${dist_dir}/libclang_rt.builtins-wasm32-wasi-${wasi_sdk_version}.tar.gz + ${CMAKE_INSTALL_PREFIX}/lib/clang/${clang_version}/lib/wasi) +add_dependencies(dist-compiler-rt compiler-rt) + +# Tarball with the whole sysroot +wasi_sdk_add_tarball(dist-sysroot + ${dist_dir}/wasi-sysroot-${wasi_sdk_version}.tar.gz + ${CMAKE_INSTALL_PREFIX}/share/wasi-sysroot) +add_dependencies(dist-sysroot build install) + +add_custom_target(dist DEPENDS dist-compiler-rt dist-sysroot) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake new file mode 100644 index 000000000..f22861c3d --- /dev/null +++ b/cmake/wasi-sdk-toolchain.cmake @@ -0,0 +1,148 @@ +# Build logic and support for building a Clang toolchain that can target +# WebAssembly and build a WASI sysroot. + +set(LLVM_CMAKE_FLAGS "" CACHE STRING "Extra cmake flags to pass to LLVM's build") +set(RUST_TARGET "" CACHE STRING "Target to build Rust code for, if not the host") +set(WASI_SDK_ARTIFACT "" CACHE STRING "Name of the wasi-sdk artifact being produced") + +string(REGEX REPLACE "[ ]+" ";" llvm_cmake_flags_list "${LLVM_CMAKE_FLAGS}") + +if(NOT CMAKE_BUILD_TYPE) + set(CMAKE_BUILD_TYPE MinSizeRel) +endif() + +set(default_cmake_args + -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} + -DCMAKE_AR=${CMAKE_AR} + -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} + -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} + -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}) + +if(CMAKE_C_COMPILER_LAUNCHER) + list(APPEND default_cmake_args -DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}) +endif() +if(CMAKE_CXX_COMPILER_LAUNCHER) + list(APPEND default_cmake_args -DCMAKE_CXX_COMPILER_LAUNCHER=${CMAKE_CXX_COMPILER_LAUNCHER}) +endif() + +set(links_to_create clang-cl clang-cpp clang++) +foreach(target IN LISTS WASI_SDK_TARGETS) + list(APPEND links_to_create ${target}-clang) + list(APPEND links_to_create ${target}-clang++) +endforeach() + +set(projects "lld;clang;clang-tools-extra") + +set(tools + clang + clang-format + clang-tidy + clang-apply-replacements + lld + llvm-mc + llvm-ranlib + llvm-strip + llvm-dwarfdump + clang-resource-headers + ar + ranlib + strip + nm + size + strings + objdump + objcopy + c++filt + llvm-config) + +list(TRANSFORM tools PREPEND --target= OUTPUT_VARIABLE build_targets) +list(TRANSFORM tools PREPEND --target=install- OUTPUT_VARIABLE install_targets) + +ExternalProject_Add(llvm-build + SOURCE_DIR "${llvm_proj_dir}/llvm" + CMAKE_ARGS + ${default_cmake_args} + -DLLVM_ENABLE_TERMINFO=OFF + -DLLVM_ENABLE_ZLIB=OFF + -DLLVM_ENABLE_ZSTD=OFF + -DLLVM_STATIC_LINK_CXX_STDLIB=ON + -DLLVM_INCLUDE_TESTS=OFF + -DLLVM_INCLUDE_UTILS=OFF + -DLLVM_INCLUDE_BENCHMARKS=OFF + -DLLVM_INCLUDE_EXAMPLES=OFF + -DLLVM_TARGETS_TO_BUILD=WebAssembly + -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi + -DLLVM_INSTALL_BINUTILS_SYMLINKS=TRUE + -DLLVM_ENABLE_LIBXML2=OFF + -DDEFAULT_SYSROOT=../share/wasi-sysroot + # Pass `-s` to strip symbols by default and shrink the size of the + # distribution + -DCMAKE_EXE_LINKER_FLAGS=-s + ${llvm_cmake_flags_list} + # See https://www.scivision.dev/cmake-externalproject-list-arguments/ for + # why this is in `CMAKE_CACHE_ARGS` instead of above + CMAKE_CACHE_ARGS + -DLLVM_ENABLE_PROJECTS:STRING=${projects} + -DCLANG_LINKS_TO_CREATE:STRING=${links_to_create} + BUILD_COMMAND + cmake --build . ${build_targets} + INSTALL_COMMAND + cmake --build . ${install_targets} + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON +) + +# Build logic for `wasm-component-ld` installed from Rust code. +set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) +set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) +set(wasm_component_ld_version 0.5.0) +if(RUST_TARGET) + set(rust_target_flag --target=${RUST_TARGET}) +endif() +add_custom_command( + OUTPUT ${wasm_component_ld} + COMMAND + cargo install --root ${wasm_component_ld_root} ${rust_target_flag} + wasm-component-ld@${wasm_component_ld_version} + COMMENT "Building `wasm-component-ld` ...") + +add_custom_target(wasm-component-ld ALL DEPENDS ${wasm_component_ld}) + +install( + PROGRAMS ${wasm_component_ld} + DESTINATION ${CMAKE_INSTALL_PREFIX}/bin) + +# Setup installation logic for CMake support files. +install( + PROGRAMS src/config/config.sub src/config/config.guess + DESTINATION ${CMAKE_INSTALL_PREFIX}/share/misc) +install( + FILES wasi-sdk.cmake wasi-sdk-pthread.cmake wasi-sdk-p2.cmake + DESTINATION ${CMAKE_INSTALL_PREFIX}/share/cmake) +install( + DIRECTORY cmake/Platform + DESTINATION ${CMAKE_INSTALL_PREFIX}/share/cmake) + +include(wasi-sdk-dist) + +# Figure out the name of the artifact which is either explicitly specified or +# inferred from CMake default variables. +if(WASI_SDK_ARTIFACT) + set(wasi_sdk_artifact ${WASI_SDK_ARTIFACT}) +else() + if(APPLE) + set(wasi_sdk_os macos) + else() + string(TOLOWER ${CMAKE_SYSTEM_NAME} wasi_sdk_os) + endif() + set(wasi_sdk_arch ${CMAKE_SYSTEM_PROCESSOR}) + set(wasi_sdk_artifact ${wasi_sdk_arch}-${wasi_sdk_os}) +endif() + +set(dist_dir ${CMAKE_CURRENT_BINARY_DIR}/dist) +wasi_sdk_add_tarball(dist-toolchain + ${dist_dir}/wasi-toolchain-${wasi_sdk_version}-${wasi_sdk_artifact}.tar.gz + ${CMAKE_INSTALL_PREFIX}) +add_dependencies(dist-toolchain llvm-build install) +add_custom_target(dist DEPENDS dist-toolchain) diff --git a/deb_from_installation.sh b/deb_from_installation.sh deleted file mode 100755 index 2699c819c..000000000 --- a/deb_from_installation.sh +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env sh -set -x - -command -v dpkg-deb >/dev/null -if [ $? -ne 0 ]; then - echo "required tool dpkg-deb missing. exiting" - exit 0 -fi - -set -ex - -if [ -n "$1" ]; then - OUTDIR=$1 -else - OUTDIR=$PWD/dist -fi - -if [ -n "$2" ]; then - VERSION="$2" -else - VERSION=`./version.py` -fi - -if [ -n "$3" ]; then - INSTALL_DIR="$3" -else - INSTALL_DIR=/opt/wasi-sdk -fi - -if [ ! -d $INSTALL_DIR ] ; then - echo "Directory $INSTALL_DIR doesn't exist. Nothing to copy from." - exit 1 -fi - -ARCH=$(dpkg --print-architecture) - -rm -rf build/pkg -mkdir -p build/pkg/opt -mkdir -p build/pkg/DEBIAN -sed -e s/VERSION/$VERSION/ wasi-sdk.control > build/pkg/DEBIAN/control -cp -R $INSTALL_DIR build/pkg/opt/ -cd build && dpkg-deb -b pkg wasi-sdk_$VERSION\_$ARCH\.deb && cd .. -mv build/wasi-sdk_$VERSION\_$ARCH\.deb $OUTDIR/ diff --git a/docker/Dockerfile b/docker/Dockerfile index 080b503ec..d3b87f239 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -1,53 +1,27 @@ -# Docker image with a build toolchain and environment variables set to use -# the wasi-sdk sysroot. The SDK distribution must have first been built, -# for example using docker_build.sh - -# Extract built SDK archive to copy out the sysroot. We use an initial build -# stage to do this to make sure it is only the sysroot, not the entire SDK -# with binaries, that is included in the final image since we install those -# separately. -FROM ubuntu:22.04 as dist - -ADD dist/wasi-sdk-*.*-linux.tar.gz / -ADD dist/libclang_rt.builtins-wasm32-wasi-*.*.tar.gz /wasi-sysroot-clang_rt - -# Move versioned folder to unversioned to using bash glob to allow -# this file to be independent of major version number. -RUN mv /wasi-sdk-* /wasi-sdk +# A container which has a number of build tools pre-installed plus a build of +# `wasi-sdk` installed at `/opt/wasi-sdk`. This also has environment variables +# pre-configued to use the installed toolchain. +# +# This container is built as the last step on CI for this repository and +# pre-built versions of this container are pushed as a package to the repository +# as well. FROM ubuntu:22.04 -ENV LLVM_VERSION 18 - -# Install build toolchain including clang, ld, make, autotools, ninja, and cmake RUN apt-get update && \ - # Temporarily install to setup apt repositories - apt-get install -y curl gnupg && \ -\ - curl -sS https://apt.llvm.org/llvm-snapshot.gpg.key | gpg --dearmor > /etc/apt/trusted.gpg.d/llvm.gpg && \ - echo "deb [signed-by=/etc/apt/trusted.gpg.d/llvm.gpg] http://apt.llvm.org/jammy/ llvm-toolchain-jammy-${LLVM_VERSION} main" >> /etc/apt/sources.list.d/llvm.list && \ - echo "deb-src [signed-by=/etc/apt/trusted.gpg.d/llvm.gpg] http://apt.llvm.org/jammy/ llvm-toolchain-jammy-${LLVM_VERSION} main" >> /etc/apt/sources.list.d/llvm.list && \ -\ - apt-get update && \ - apt-get install -y clang-${LLVM_VERSION} lld-${LLVM_VERSION} cmake ninja-build make autoconf autogen automake libtool && \ - apt-get autoremove -y curl gnupg && \ + apt-get install -y cmake ninja-build make autoconf autogen automake libtool && \ rm -rf /var/lib/apt/lists/* -COPY --from=dist /wasi-sdk/share/wasi-sysroot/ /wasi-sysroot/ -COPY --from=dist /wasi-sysroot-clang_rt/lib/wasi /usr/lib/llvm-${LLVM_VERSION}/lib/clang/${LLVM_VERSION}/lib/wasi - -ADD docker/wasi-sdk.cmake /usr/share/cmake/wasi-sdk.cmake -ADD docker/wasi-sdk-pthread.cmake /usr/share/cmake/wasi-sdk-pthread.cmake -ADD docker/wasi-sdk-p2.cmake /usr/share/cmake/wasi-sdk-p2.cmake -ENV CMAKE_TOOLCHAIN_FILE /usr/share/cmake/wasi-sdk.cmake -ADD cmake/Platform/WASI.cmake /usr/share/cmake/Modules/Platform/WASI.cmake - -ENV CC clang-${LLVM_VERSION} -ENV CXX clang++-${LLVM_VERSION} -ENV LD wasm-ld-${LLVM_VERSION} -ENV AR llvm-ar-${LLVM_VERSION} -ENV RANLIB llvm-ranlib-${LLVM_VERSION} - -ENV CFLAGS --target=wasm32-wasi --sysroot=/wasi-sysroot -ENV CXXFLAGS --target=wasm32-wasi --sysroot=/wasi-sysroot -ENV LDFLAGS --target=wasm32-wasi --sysroot=/wasi-sysroot +ADD dist/wasi-sdk-*.deb . +RUN case `dpkg --print-architecture` in \ + amd64) dpkg -i wasi-sdk-*-x86_64-linux.deb ;; \ + arm64) dpkg -i wasi-sdk-*-arm64-linux.deb ;; \ + *) exit 1 ;; \ + esac && \ + rm wasi-sdk-*.deb + +ENV CC /opt/wasi-sdk/bin/clang +ENV CXX /opt/wasi-sdk/bin/clang++ +ENV LD /opt/wasi-sdk/bin/wasm-ld +ENV AR /opt/wasi-sdk/bin/llvm-ar +ENV RANLIB /opt/wasi-sdk/bin/llvm-ranlib diff --git a/docker/wasi-sdk-p2.cmake b/docker/wasi-sdk-p2.cmake deleted file mode 100644 index 2d34eadbe..000000000 --- a/docker/wasi-sdk-p2.cmake +++ /dev/null @@ -1,29 +0,0 @@ -# Cmake toolchain description file for the wasi-sdk docker image - -# This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) - -# To make sure it recognizes the WASI platform -list(APPEND CMAKE_MODULE_PATH /usr/share/cmake/Modules) - -set(CMAKE_SYSTEM_NAME WASI) -set(CMAKE_SYSTEM_VERSION 1) -set(CMAKE_SYSTEM_PROCESSOR wasm32) -set(triple wasm32-wasip2) - -set(CMAKE_C_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_CXX_COMPILER /usr/bin/clang++-$ENV{LLVM_VERSION}) -set(CMAKE_ASM_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) -set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) -set(CMAKE_C_COMPILER_TARGET ${triple}) -set(CMAKE_CXX_COMPILER_TARGET ${triple}) -set(CMAKE_ASM_COMPILER_TARGET ${triple}) -SET(CMAKE_SYSROOT /wasi-sysroot) - -# Don't look in the sysroot for executables to run during the build -set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) -# Only look in the sysroot (not in the host paths) for the rest -set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/docker/wasi-sdk-pthread.cmake b/docker/wasi-sdk-pthread.cmake deleted file mode 100644 index 61354451a..000000000 --- a/docker/wasi-sdk-pthread.cmake +++ /dev/null @@ -1,36 +0,0 @@ -# Cmake toolchain description file for the wasi-sdk docker image - -# This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) - -# To make sure it recognizes the WASI platform -list(APPEND CMAKE_MODULE_PATH /usr/share/cmake/Modules) - -set(CMAKE_SYSTEM_NAME WASI) -set(CMAKE_SYSTEM_VERSION 1) -set(CMAKE_SYSTEM_PROCESSOR wasm32) -set(triple wasm32-wasi-threads) -set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -pthread") -set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread") -# wasi-threads requires --import-memory. -# wasi requires --export-memory. -# (--export-memory is implicit unless --import-memory is given) -set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--import-memory") -set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--export-memory") - -set(CMAKE_C_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_CXX_COMPILER /usr/bin/clang++-$ENV{LLVM_VERSION}) -set(CMAKE_ASM_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) -set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) -set(CMAKE_C_COMPILER_TARGET ${triple}) -set(CMAKE_CXX_COMPILER_TARGET ${triple}) -set(CMAKE_ASM_COMPILER_TARGET ${triple}) -SET(CMAKE_SYSROOT /wasi-sysroot) - -# Don't look in the sysroot for executables to run during the build -set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) -# Only look in the sysroot (not in the host paths) for the rest -set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/docker/wasi-sdk.cmake b/docker/wasi-sdk.cmake deleted file mode 100644 index eabb877ca..000000000 --- a/docker/wasi-sdk.cmake +++ /dev/null @@ -1,29 +0,0 @@ -# Cmake toolchain description file for the wasi-sdk docker image - -# This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) - -# To make sure it recognizes the WASI platform -list(APPEND CMAKE_MODULE_PATH /usr/share/cmake/Modules) - -set(CMAKE_SYSTEM_NAME WASI) -set(CMAKE_SYSTEM_VERSION 1) -set(CMAKE_SYSTEM_PROCESSOR wasm32) -set(triple wasm32-wasi) - -set(CMAKE_C_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_CXX_COMPILER /usr/bin/clang++-$ENV{LLVM_VERSION}) -set(CMAKE_ASM_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) -set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) -set(CMAKE_C_COMPILER_TARGET ${triple}) -set(CMAKE_CXX_COMPILER_TARGET ${triple}) -set(CMAKE_ASM_COMPILER_TARGET ${triple}) -SET(CMAKE_SYSROOT /wasi-sysroot) - -# Don't look in the sysroot for executables to run during the build -set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) -# Only look in the sysroot (not in the host paths) for the rest -set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/docker_build.sh b/docker_build.sh deleted file mode 100755 index 050862fac..000000000 --- a/docker_build.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/sh -set -ex - -echo "Building the docker image" -docker build \ - --build-arg UID=$(id -u) --build-arg GID=$(id -g) \ - -t wasi-sdk-builder:latest . - -echo "Building the package in docker image" -mkdir -p ~/.ccache -docker run --rm \ - --user $(id -u):$(id -g) \ - -v "$PWD":/workspace:Z \ - -v ~/.ccache:/home/builder/.ccache:Z \ - -e NINJA_FLAGS=-v \ - --tmpfs /tmp:exec \ - wasi-sdk-builder:latest \ - make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON diff --git a/strip_symbols.sh b/strip_symbols.sh deleted file mode 100755 index effd30090..000000000 --- a/strip_symbols.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/usr/bin/env bash -set -e - -DIRECTORY=${1:-/opt/wasi-sdk/bin} -if [[ "$OSTYPE" == "darwin"* ]] || [[ "$OSTYPE" == "freebsd"* ]]; then -# macos and freebsd find do not support -executable so we fall back on -# having a permission bit to execute: -EXECUTABLES=$(find ${DIRECTORY} -type f -perm +111) -else -EXECUTABLES=$(find ${DIRECTORY} -type f -executable) -fi -for e in ${EXECUTABLES}; do - echo "Stripping symbols: ${e}" - strip ${e} || echo "Failed to strip symbols for ${e}; continuing on." -done diff --git a/tar_from_installation.sh b/tar_from_installation.sh deleted file mode 100755 index 7d0943266..000000000 --- a/tar_from_installation.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/usr/bin/env bash -set -ex - -if [ -n "$1" ]; then - OUTDIR=$1 -else - OUTDIR=$PWD/dist -fi - -if [ -n "$2" ]; then - VERSION="$2" -else - VERSION=`./version.py` -fi - -if [ -n "$3" ]; then - INSTALL_DIR="$3" -else - INSTALL_DIR=/opt/wasi-sdk -fi - -case "$(uname -s)" in - Linux*) MACHINE=linux;; - Darwin*) MACHINE=macos;; - CYGWIN*) MACHINE=cygwin;; - MINGW*) MACHINE=mingw;; - MSYS*) MACHINE=msys;; #MSYS_NT-10.0-19043 - *) MACHINE="UNKNOWN" -esac - -if [ ! -d $INSTALL_DIR ] ; then - echo "Directory $INSTALL_DIR doesn't exist. Nothing to copy from." - exit 1 -fi - -PKGDIR=build/wasi-sdk-$VERSION -rm -rf $PKGDIR -if [ "$MACHINE" == "cygwin" ] || [ "$MACHINE" == "mingw" ] || [ "$MACHINE" == "msys" ]; then - # Copy with -L to avoid trying to create symlinks on Windows. - cp -R -L $INSTALL_DIR $PKGDIR -else - cp -R $INSTALL_DIR $PKGDIR -fi -cd build -tar czf $OUTDIR/wasi-sdk-$VERSION\-$MACHINE.tar.gz wasi-sdk-$VERSION - -# As well as the full SDK package, also create archives of libclang_rt.builtins -# and the sysroot. These are made available for users who have an existing clang -# installation. -tar czf $OUTDIR/libclang_rt.builtins-wasm32-wasi-$VERSION.tar.gz -C compiler-rt lib/wasi -tar czf $OUTDIR/wasi-sysroot-$VERSION.tar.gz -C wasi-sdk-$VERSION/share wasi-sysroot diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt new file mode 100644 index 000000000..23010fa0b --- /dev/null +++ b/tests/CMakeLists.txt @@ -0,0 +1,88 @@ +# Support for running tests in the `tests/{compile-only,general}` folders +cmake_minimum_required(VERSION 3.22) +project(wasi-sdk-test) +include(CTest) +enable_testing() +set(CMAKE_EXECUTABLE_SUFFIX ".wasm") + +add_compile_options(--sysroot=${wasi_sysroot}) +add_link_options(--sysroot=${wasi_sysroot}) + +# Sanity check setup +if (NOT ${CMAKE_SYSTEM_NAME} STREQUAL WASI) + message(FATAL_ERROR "Wrong system name (${CMAKE_SYSTEM_NAME}), wrong toolchain file in use?") +endif() + +if(NOT DEFINED WASI) + message(FATAL_ERROR "WASI is not set, platform file likely not loaded") +endif() + +set(WASI_SDK_RUNWASI "wasmtime" CACHE STRING "Runner for tests") + +# Test everything at O0, O2, and O2+LTO +set(opt_flags -O0 -O2 "-O2 -flto") + +# Executes a single `test` specified. +# +# This will compile `test` for all the various targets and with various +# compiler options. If `runwasi` is non-empty then the test will be executed +# in that runner as well. +function(add_testcase runwasi test) + foreach(target IN LISTS WASI_SDK_TARGETS) + foreach(compile_flags IN LISTS opt_flags) + # Mangle the options into something appropriate for a CMake rule name + string(REGEX REPLACE " " "." target_name "${target}.${compile_flags}.${test}") + + # Add a new test executable based on `test` + add_executable(${target_name} ${test}) + + # Configure all the compile options necessary. For example `--target` here + # if the target doesn't look like it's already in the name of the compiler + # as well. + if(NOT(CMAKE_C_COMPILER MATCHES ${target})) + target_compile_options(${target_name} PRIVATE --target=${target}) + endif() + + # Apply test-specific compile options and link flags. + if(test MATCHES "clocks.c$") + target_compile_options(${target_name} PRIVATE -D_WASI_EMULATED_PROCESS_CLOCKS) + target_link_options(${target_name} PRIVATE -lwasi-emulated-process-clocks) + elseif(test MATCHES "mmap.c$") + target_compile_options(${target_name} PRIVATE -D_WASI_EMULATED_MMAN) + target_link_options(${target_name} PRIVATE -lwasi-emulated-mman) + elseif(test MATCHES "(sigabrt|signals).c$") + target_compile_options(${target_name} PRIVATE -D_WASI_EMULATED_SIGNAL) + target_link_options(${target_name} PRIVATE -lwasi-emulated-signal) + elseif(test MATCHES "printf-long-double-enabled.c$") + target_link_options(${target_name} PRIVATE -lc-printscan-long-double) + endif() + + # Apply language-specific options and dependencies. + if(test MATCHES "cc$") + target_compile_options(${target_name} PRIVATE -fno-exceptions) + add_dependencies(${target_name} libcxx-${target}) + else() + add_dependencies(${target_name} wasi-libc-${target}) + endif() + + # Apply target-specific options. + if(target MATCHES threads) + target_compile_options(${target_name} PRIVATE -pthread) + endif() + + if(runwasi) + add_test( + NAME test-${target_name} + COMMAND + bash ../testcase.sh + "${runwasi}" + ${test} + $ + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}) + endif() + endforeach() + endforeach() +endfunction() + +add_subdirectory(compile-only) +add_subdirectory(general) diff --git a/tests/cmake/CMakeLists.txt b/tests/cmake/CMakeLists.txt deleted file mode 100644 index 28c9b839b..000000000 --- a/tests/cmake/CMakeLists.txt +++ /dev/null @@ -1,34 +0,0 @@ -cmake_minimum_required(VERSION 3.22) - -project(wasi-sdk-test) - -# Sanity check setup -if (NOT ${CMAKE_SYSTEM_NAME} STREQUAL WASI) - message(FATAL_ERROR "Wrong system name (${CMAKE_SYSTEM_NAME}), wrong toolchain file in use?") -endif() - -if(NOT DEFINED WASI) - message(FATAL_ERROR "WASI is not set, platform file likely not loaded") -endif() - -set(RUNWASI "" CACHE STRING "Path to or name of WASM runner") - -# Test build a C and C++ target respectively -add_executable(void_main_c ../general/void_main.c) -add_executable(void_main_cc ../general/void_main.cc) - -include(CTest) -enable_testing() - -add_test(NAME void_main_c - COMMAND - ${CMAKE_CURRENT_SOURCE_DIR}/test_driver.sh - ${RUNWASI} - $ - ${CMAKE_CURRENT_SOURCE_DIR}/../general/void_main.c.stdout.expected) -add_test(NAME void_main_cc - COMMAND - ${CMAKE_CURRENT_SOURCE_DIR}/test_driver.sh - ${RUNWASI} - $ - ${CMAKE_CURRENT_SOURCE_DIR}/../general/void_main.cc.stdout.expected) diff --git a/tests/cmake/test_driver.sh b/tests/cmake/test_driver.sh deleted file mode 100755 index 6e469a1ae..000000000 --- a/tests/cmake/test_driver.sh +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/bash -# Simplified runner for cmake - -set -ex - -runwasi="$1" -target="$2" -stdout_expected="$3" -stderr_expected="/dev/null" - -stdout_observed="$target.stdout.observed" -stderr_observed="$target.stderr.observed" - -"$runwasi" "$target" > "$stdout_observed" 2> "$stderr_observed" - -diff -u "$stderr_expected" "$stderr_observed" -diff -u "$stdout_expected" "$stdout_observed" diff --git a/tests/compile-only/CMakeLists.txt b/tests/compile-only/CMakeLists.txt new file mode 100644 index 000000000..484a9cddc --- /dev/null +++ b/tests/compile-only/CMakeLists.txt @@ -0,0 +1,9 @@ +file(GLOB c_compile_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.c") +file(GLOB cxx_compile_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.cc") + +set(compile_tests ${c_compile_tests} ${cxx_compile_tests}) + +foreach(test IN LISTS compile_tests) + add_testcase("" ${test}) +endforeach() + diff --git a/tests/general/CMakeLists.txt b/tests/general/CMakeLists.txt new file mode 100644 index 000000000..ef0a5528d --- /dev/null +++ b/tests/general/CMakeLists.txt @@ -0,0 +1,8 @@ +file(GLOB c_general_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.c") +file(GLOB cxx_general_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.cc") + +set(general_tests ${c_general_tests} ${cxx_general_tests}) + +foreach(test IN LISTS general_tests) + add_testcase(${WASI_SDK_RUNWASI} ${test}) +endforeach() diff --git a/tests/general/abort.c.stderr.expected.filter b/tests/general/abort.c.stderr.expected.filter index ecd16c053..98233a1ee 100755 --- a/tests/general/abort.c.stderr.expected.filter +++ b/tests/general/abort.c.stderr.expected.filter @@ -2,6 +2,6 @@ set -euo pipefail cat \ - | sed -e 's/main module `abort\.c\.[^`]*\.wasm`/main module `abort.c.---.wasm`/' \ + | sed -e 's/main module `.*abort\.c\.wasm`/main module `abort.c.---.wasm`/' \ | sed -e 's/failed to invoke.*/failed to invoke ---/' \ | sed -E '/0x[[:xdigit:]]+/d' diff --git a/tests/general/assert-fail.c b/tests/general/assert-fail.c index 07f455784..54479f9b5 100644 --- a/tests/general/assert-fail.c +++ b/tests/general/assert-fail.c @@ -1,3 +1,7 @@ +#ifdef NDEBUG +#undef NDEBUG +#endif + #include #include diff --git a/tests/general/assert-fail.c.stderr.expected b/tests/general/assert-fail.c.stderr.expected index 9e0f44c5a..fcf1f75ac 100644 --- a/tests/general/assert-fail.c.stderr.expected +++ b/tests/general/assert-fail.c.stderr.expected @@ -1,4 +1,4 @@ -Assertion failed: false (assert-fail.c: main: 5) +Assertion failed: false (assert-fail.c: main: 9) Error: failed to run main module `assert-fail.c.---.wasm` Caused by: diff --git a/tests/general/assert-fail.c.stderr.expected.filter b/tests/general/assert-fail.c.stderr.expected.filter index a1d649de0..77849a1ca 100755 --- a/tests/general/assert-fail.c.stderr.expected.filter +++ b/tests/general/assert-fail.c.stderr.expected.filter @@ -2,6 +2,7 @@ set -euo pipefail cat \ - | sed -e 's/main module `assert-fail\.c\.[^`]*\.wasm`/main module `assert-fail.c.---.wasm`/' \ + | sed -e 's/main module `.*assert-fail\.c\.wasm`/main module `assert-fail.c.---.wasm`/' \ | sed -e 's/failed to invoke.*/failed to invoke ---/' \ + | sed -e 's/Assertion failed: false (.*assert-fail.c/Assertion failed: false (assert-fail.c/' \ | sed -E '/0x[[:xdigit:]]+/d' diff --git a/tests/general/clocks.c.options b/tests/general/clocks.c.options deleted file mode 100644 index 03953a9dc..000000000 --- a/tests/general/clocks.c.options +++ /dev/null @@ -1 +0,0 @@ --D_WASI_EMULATED_PROCESS_CLOCKS -lwasi-emulated-process-clocks diff --git a/tests/general/iostream_main.cc.options b/tests/general/iostream_main.cc.options deleted file mode 100644 index f7f5b39e8..000000000 --- a/tests/general/iostream_main.cc.options +++ /dev/null @@ -1 +0,0 @@ --fno-exceptions diff --git a/tests/general/mmap.c.options b/tests/general/mmap.c.options deleted file mode 100644 index 63236a027..000000000 --- a/tests/general/mmap.c.options +++ /dev/null @@ -1 +0,0 @@ --D_WASI_EMULATED_MMAN -lwasi-emulated-mman diff --git a/tests/general/printf-long-double-enabled.c.options b/tests/general/printf-long-double-enabled.c.options deleted file mode 100644 index 2b2ec8e65..000000000 --- a/tests/general/printf-long-double-enabled.c.options +++ /dev/null @@ -1 +0,0 @@ --lc-printscan-long-double diff --git a/tests/general/sigabrt.c.options b/tests/general/sigabrt.c.options deleted file mode 100644 index 4f07a3ec6..000000000 --- a/tests/general/sigabrt.c.options +++ /dev/null @@ -1 +0,0 @@ --D_WASI_EMULATED_SIGNAL -lwasi-emulated-signal diff --git a/tests/general/sigabrt.c.stderr.expected.filter b/tests/general/sigabrt.c.stderr.expected.filter index 425b060c2..e6b54c2b2 100755 --- a/tests/general/sigabrt.c.stderr.expected.filter +++ b/tests/general/sigabrt.c.stderr.expected.filter @@ -2,6 +2,6 @@ set -euo pipefail cat \ - | sed -e 's/main module `sigabrt\.c\.[^`]*\.wasm`/main module `sigabrt.c.---.wasm`/' \ + | sed -e 's/main module `.*sigabrt\.c\.wasm`/main module `sigabrt.c.---.wasm`/' \ | sed -e 's/source location: @[[:xdigit:]]*$/source location: @----/' \ | head -n 6 diff --git a/tests/general/signals.c b/tests/general/signals.c index 27c8e9eae..be4f54eef 100644 --- a/tests/general/signals.c +++ b/tests/general/signals.c @@ -1,3 +1,7 @@ +#ifdef NDEBUG +#undef NDEBUG +#endif + #include #include #include @@ -6,7 +10,10 @@ #include // Make sure this exists. +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-W#warnings" #include +#pragma clang diagnostic pop volatile sig_atomic_t flag = 0; diff --git a/tests/general/signals.c.options b/tests/general/signals.c.options deleted file mode 100644 index 4f07a3ec6..000000000 --- a/tests/general/signals.c.options +++ /dev/null @@ -1 +0,0 @@ --D_WASI_EMULATED_SIGNAL -lwasi-emulated-signal diff --git a/tests/run.sh b/tests/run.sh deleted file mode 100755 index bc8dd4d0b..000000000 --- a/tests/run.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/bin/bash -set -ueo pipefail - -# Top-level test runner. Usage is "run.sh " to run tests -# in compile-only mode, or "run.sh " where -# is a WASI-capable runtime to run the tests in full compile and -# execute mode. -# -# The compiler used during testing is loaded from ``. -if [ $# -lt 1 ]; then - echo "Path to WASI SDK is required" - exit 1 -fi - -wasi_sdk="$1" - -# Determine the wasm runtime to use, if one is provided. -if [ $# -gt 1 ]; then - runwasi="$2" -else - runwasi="" -fi - -testdir=$(dirname $0) - -echo "SDK: $wasi_sdk" - -# NB: all tests are run with the default `clang` and `clang++` executables -# but they're also executed with the target-prefixed `clang` executables to -# ensure that those work as well when the `--target` option is omitted. - -for target in $TARGETS; do - echo "===== Testing target $target =====" - cd $testdir/compile-only - for options in -O0 -O2 "-O2 -flto"; do - echo "===== Testing compile-only with $options =====" - for file in *.c; do - echo "Testing compile-only $file..." - ../testcase.sh "$target" "" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" - ../testcase.sh "$target" "" "$wasi_sdk/bin/$target-clang" "$options" "$file" - done - for file in *.cc; do - echo "Testing compile-only $file..." - ../testcase.sh "$target" "" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" - ../testcase.sh "$target" "" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" - done - done - cd - >/dev/null - - cd $testdir/general - for options in -O0 -O2 "-O2 -flto"; do - echo "===== Testing with $options =====" - for file in *.c; do - echo "Testing $file..." - ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" - ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/$target-clang" "$options" "$file" - done - for file in *.cc; do - echo "Testing $file..." - ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" - ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" - done - done - cd - >/dev/null -done - -# Test cmake build system for wasi-sdk -test_cmake() { - local option - for option in Debug Release; do - rm -rf "$testdir/cmake/build/$option" - mkdir -p "$testdir/cmake/build/$option" - cd "$testdir/cmake/build/$option" - cmake \ - -G "Unix Makefiles" \ - -DCMAKE_BUILD_TYPE="$option" \ - -DRUNWASI="$runwasi" \ - -DWASI_SDK_PREFIX="$wasi_sdk" \ - -DCMAKE_TOOLCHAIN_FILE="$wasi_sdk/share/cmake/wasi-sdk.cmake" \ - ../.. - make - if [[ -n "$runwasi" ]]; then - ctest --output-on-failure - fi - cd - >/dev/null - done -} - -test_cmake diff --git a/tests/testcase.sh b/tests/testcase.sh index 3d8860464..fae7443fa 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -6,39 +6,19 @@ set -ueo pipefail # Command-line parsing; this script is meant to be run from a higher-level # script, so don't do anything fancy. -target="$1" -runwasi="$2" -compiler="$3" -options="$4" -input="$5" +runwasi="$1" +input="$2" +wasm="$3" # Compile names for generated files. -wasm="$input.$options.wasm" -stdout_observed="$input.$options.stdout.observed" -stderr_observed="$input.$options.stderr.observed" -exit_status_observed="$input.$options.exit_status.observed" +stdout_observed="$wasm.stdout.observed" +stderr_observed="$wasm.stderr.observed" +exit_status_observed="$wasm.exit_status.observed" -# Optionally load compiler options from a file. -if [ -e "$input.options" ]; then - file_options=$(cat "$input.options") -else - file_options= -fi - -if echo "$target" | grep -q -- '-threads$'; then - pthread_options="-pthread" -else - pthread_options= -fi - -echo "Testing $input..." - -# Compile the testcase. -$compiler $pthread_options $options $file_options "$input" -o "$wasm" - -# If we don't have a runwasi command, we're just doing compile-only testing. +# Double-check that a runwasi command was specified since otherwise this script +# was invoked with no arguments which isn't as intended. if [ "$runwasi" == "" ]; then - exit 0 + exit 1 fi # Determine the input file to write to stdin. @@ -75,11 +55,7 @@ echo $exit_status > "$exit_status_observed" # Determine the reference files to compare with. if [ -e "$input.stdout.expected" ]; then - if [ -e "$input.$target.stdout.expected" ]; then - stdout_expected="$input.$target.stdout.expected" - else - stdout_expected="$input.stdout.expected" - fi + stdout_expected="$input.stdout.expected" # Apply output filters. if [ -e "$input.stdout.expected.filter" ]; then @@ -93,11 +69,7 @@ else fi if [ -e "$input.stderr.expected" ]; then - if [ -e "$input.$target.stderr.expected" ]; then - stderr_expected="$input.$target.stderr.expected" - else - stderr_expected="$input.stderr.expected" - fi + stderr_expected="$input.stderr.expected" # Apply output filters. if [ -e "$input.stderr.expected.filter" ]; then @@ -117,6 +89,6 @@ fi # If there are any differences, diff will return a non-zero exit status, and # since this script uses "set -e", it will return a non-zero exit status too. -diff -u "$stderr_expected" "$stderr_observed" -diff -u "$stdout_expected" "$stdout_observed" -diff -u "$exit_status_expected" "$exit_status_observed" +diff --ignore-space-change -u "$stderr_expected" "$stderr_observed" +diff --ignore-space-change -u "$stdout_expected" "$stdout_observed" +diff --ignore-space-change -u "$exit_status_expected" "$exit_status_observed" diff --git a/wasi-sdk-p2.cmake b/wasi-sdk-p2.cmake index 681f71dcd..f00d7732b 100644 --- a/wasi-sdk-p2.cmake +++ b/wasi-sdk-p2.cmake @@ -1,7 +1,7 @@ # Cmake toolchain description file for the Makefile # This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) +cmake_minimum_required(VERSION 3.5.0) # Until Platform/WASI.cmake is upstream we need to inject the path to it # into CMAKE_MODULE_PATH. diff --git a/wasi-sdk.cmake b/wasi-sdk.cmake index f6a0059c8..430e98f04 100644 --- a/wasi-sdk.cmake +++ b/wasi-sdk.cmake @@ -1,7 +1,7 @@ # Cmake toolchain description file for the Makefile # This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) +cmake_minimum_required(VERSION 3.5.0) # Until Platform/WASI.cmake is upstream we need to inject the path to it # into CMAKE_MODULE_PATH. diff --git a/wasi-sdk.control b/wasi-sdk.control index 0586f37ba..e3859e7eb 100644 --- a/wasi-sdk.control +++ b/wasi-sdk.control @@ -1,6 +1,6 @@ Package: wasi-sdk Version: VERSION -Architecture: amd64 +Architecture: ARCH Priority: optional Description: Clang toolchain with wasm32-wasi default target, and the wasi sysroot -Maintainer: Pat Hickey +Maintainer: WASI SDK Maintainers From 3adc86a271077c4aeb61bae3456902b93508abcd Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Fri, 12 Jul 2024 23:17:09 +0900 Subject: [PATCH 067/165] README.md: update build instructions (#437) --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index fd26ddb4e..72e01f980 100644 --- a/README.md +++ b/README.md @@ -53,11 +53,12 @@ To build the full package: ```shell script cd wasi-sdk -NINJA_FLAGS=-v make package +./ci/build.sh ``` -The built package can be found into `dist` directory. For releasing a new -version of the package on GitHub, see [RELEASING.md](RELEASING.md). +The built package can be found into `build/dist` directory. +For releasing a new version of the package on GitHub, +see [RELEASING.md](RELEASING.md). ## Install From 2e7bb65533b2a8d038a41bd1caca5d2212396492 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 12 Jul 2024 10:38:50 -0500 Subject: [PATCH 068/165] Expand README build instructions (#438) Updating for the rewrite in #429 --- README.md | 66 +++++++++++++++++++++++++++++++++++++++++-- ci/merge-artifacts.sh | 13 ++++++--- 2 files changed, 73 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 72e01f980..9632be985 100644 --- a/README.md +++ b/README.md @@ -49,10 +49,72 @@ Please refer to your OS documentation to install those packages. ## Build -To build the full package: +Building `wasi-sdk` uses CMake and is split into two halves. First you can build +the toolchain itself: + +```shell script +cmake -G Ninja -B build/toolchain -S . -DWASI_SDK_BUILD_TOOLCHAIN=ON -DCMAKE_INSTALL_PREFIX=build/install +cmake --build build/toolchain --target install +``` + +When you're developing locally you may also wish to pass +`-DCMAKE_CXX_COMPILER_LAUNCHER=ccache` to assist with rebuilds. Other supported +CMake flags are: + +* `-DLLVM_CMAKE_FLAGS` - extra flags to pass to `cmake` when building + LLVM/Clang. +* `-DRUST_TARGET` - the specific Rust target triple to build `wasm-component-ld` + for, useful for cross-compiles. + +The `clang` compiler should now be located at `build/install/bin/clang` but it's +just a compiler, the sysroot isn't built yet. Next the second step of the build +is to build the sysroot: + +```shell script +cmake -G Ninja -B build/sysroot -S . \ + -DCMAKE_INSTALL_PREFIX=build/install \ + -DCMAKE_TOOLCHAIN_FILE=build/install/share/cmake/wasi-sdk.cmake \ + -DCMAKE_C_COMPILER_WORKS=ON \ + -DCMAKE_CXX_COMPILER_WORKS=ON +cmake --build build/sysroot --target install +``` + +A full toolchain should now be present at `build/install` and is ready for use +in compiling WebAssembly code. Supported CMake flags are: + +* `-DWASI_SDK_DEBUG_PREFIX_MAKE=OFF` - disable `-fdebug-prefix-map` when + building C/C++ code to use full host paths instead. +* `-DWASI_SDK_INCLUDE_TESTS=ON` - used for building tests. +* `-DWASI_SDK_TARGETS=..` - a list of targets to build, by default all WASI + targets are compiled. + +If you'd like to build distribution artifacts you can use the `dist` target like +so: + +```shell script +cmake --build build/toolchain --target dist +cmake --build build/sysroot --target dist +``` + +Tarballs will be created under `build/toolchain/dist` and `build/sysroot/dist`. +Note that these are separate tarballs for the toolchain and sysroot. To create a +single tarball for the entire SDK you'll first want to copy all tarballs into a +new folder and then run the `./ci/merge-artifacts.sh` script: + +```shell script +mkdir dist-my-platform +cp build/toolchain/dist/* build/sysroot/dist/* dist-my-platform +./ci/merge-artifacts.sh +``` + +This will produce `dist/wasi-sdk-*.tar.gz` which is the same as the release +artifacts for this repository. + +Finally you can additionally bundle many of the above steps, minus +`merge-artifact.sh` by using the CI script to perform both the toolchain and +sysroot build: ```shell script -cd wasi-sdk ./ci/build.sh ``` diff --git a/ci/merge-artifacts.sh b/ci/merge-artifacts.sh index 0ef7e719d..e55db2d92 100755 --- a/ci/merge-artifacts.sh +++ b/ci/merge-artifacts.sh @@ -39,8 +39,6 @@ make_deb() { rm -rf dist/pkg } -compiler_rt=`ls dist-x86_64-linux/libclang_rt*` - for build in dist-*; do toolchain=`ls $build/wasi-toolchain-*` if [ -f $build/wasi-sysroot-* ]; then @@ -48,6 +46,11 @@ for build in dist-*; do else sysroot=`ls dist-x86_64-linux/wasi-sysroot-*` fi + if [ -f $build/libclang_rt* ]; then + compiler_rt=`ls $build/libclang_rt*` + else + compiler_rt=`ls dist-x86_64-linux/libclang_rt*` + fi sdk_dir=`basename $toolchain | sed 's/.tar.gz//' | sed s/toolchain/sdk/` mkdir dist/$sdk_dir @@ -75,5 +78,7 @@ done # In addition to `wasi-sdk-*` also preserve artifacts for just the sysroot # and just compiler-rt. -cp dist-x86_64-linux/wasi-sysroot-* dist -cp dist-x86_64-linux/libclang_rt* dist +if [ -d dist-x86_64-linux ]; then + cp dist-x86_64-linux/wasi-sysroot-* dist + cp dist-x86_64-linux/libclang_rt* dist +fi From 778a02bb08bdc7b60efa53f5543131a9a5e9a22e Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 12 Jul 2024 17:29:14 -0500 Subject: [PATCH 069/165] Link LLVM-based tools dynamically to LLVM (take 2) (#441) This is a revival of #397 but only applies the logic to non-Windows platforms. Windows builds still seem to not work so Windows is not improved as a result of this change. --- cmake/wasi-sdk-toolchain.cmake | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index f22861c3d..1da2ebf3f 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -55,6 +55,19 @@ set(tools c++filt llvm-config) +# By default link LLVM dynamically to all the various tools. This greatly +# reduces the binary size of all the tools through a shared library rather than +# statically linking LLVM to each individual tool. This requires a few other +# install targets as well to ensure the appropriate libraries are all installed. +# +# Also note that the `-wasi-sdk` version suffix is intended to help prevent +# these dynamic libraries from clashing with other system libraries in case the +# `lib` dir gets put on `LD_LIBRARY_PATH` or similar. +if(NOT WIN32) + list(APPEND default_cmake_args -DLLVM_LINK_LLVM_DYLIB=ON -DLLVM_VERSION_SUFFIX=-wasi-sdk) + list(APPEND tools LLVM clang-cpp) +endif() + list(TRANSFORM tools PREPEND --target= OUTPUT_VARIABLE build_targets) list(TRANSFORM tools PREPEND --target=install- OUTPUT_VARIABLE install_targets) From 297b6d02d55d1cc540d2d6d90c5b57553a233c45 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 12 Jul 2024 23:22:48 -0500 Subject: [PATCH 070/165] Automate more of the release process (#440) This commit updates the release process of `wasi-sdk` to remove most of the manual interaction and steps done. Instead now draft releases are automatically created for tags made. This means that there's only two steps necessary: (1) pushing a tag and (2) hitting publish on the generated release. This commit also removes a number of the CI scripts previously used to manage releases. --- .github/workflows/main.yml | 11 ++++- RELEASING.md | 50 ++----------------- ci/download-workflow-artifacts.sh | 81 ------------------------------- ci/draft-release.sh | 71 --------------------------- ci/get-workflows-for-tag.sh | 43 ---------------- ci/is-workflow-valid.sh | 58 ---------------------- 6 files changed, 14 insertions(+), 300 deletions(-) delete mode 100755 ci/download-workflow-artifacts.sh delete mode 100755 ci/draft-release.sh delete mode 100755 ci/get-workflows-for-tag.sh delete mode 100755 ci/is-workflow-valid.sh diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c0914e658..0b3abe693 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -1,11 +1,12 @@ name: CI on: - create: - tags: push: + tags: + - 'wasi-sdk-*' branches: - main + pull_request: workflow_dispatch: @@ -200,3 +201,9 @@ jobs: labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha cache-to: type=gha,mode=max + + - name: Publish a draft release + if: startsWith(github.ref, 'refs/tags') + run: gh release create --draft --prerelease --generate-notes ${{ github.ref_name }} ./dist/* + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/RELEASING.md b/RELEASING.md index 6bc387974..1091c8058 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,8 +1,5 @@ # Release Process -> **Note**: These instructions are out-of-date given the latest refactoring of -> the build system and should get updated before the next release. - To publish a new version of `wasi-sdk` as a GitHub release: 1. Tag a commit with an annotated tag. Note that this must be an annotated tag, @@ -17,49 +14,12 @@ To publish a new version of `wasi-sdk` as a GitHub release: git push origin $TAG ``` -2. Find a successful workflow that CI has run for the tag. That successful - workflow run will have build artifacts that need to be attached to the - release. One could search around in the GitHub [actions], but the following - script will list completed workflows for a tag (get a token [here][tokens]): - - ```shell script - ci/get-workflows-for-tag.sh $TAG $GITHUB_TOKEN - ``` - - [actions]: https://github.com/WebAssembly/wasi-sdk/actions - [tokens]: https://github.com/settings/tokens - -3. Check that the workflow built the artifacts for the given tag and that the - workflow completed successfully: - - ```shell script - ci/is-workflow-valid.sh $TAG $WORKFLOW_RUN_ID $GITHUB_TOKEN - ``` - -4. Download and unzip the workflow artifacts. Note that artifacts with `+m` or - `.m` suffixes indicate that the Git tree was modified. Expect some duplicates - since some of the same artifacts are built on multiple CI runners (e.g., - Windows, MacOS, Linux). The following script does all of this automatically: - - ```shell script - ci/download-workflow-artifacts.sh $WORKFLOW_RUN_ID $GITHUB_TOKEN - ``` - -5. Draft a new release. This could be done [manually][releases] but the - following script simplifies the uploading of all the files and auto-generates - the release description: - - ```shell script - ci/draft-release.sh $TAG $ARTIFACTS_DIR $GITHUB_TOKEN - ``` - - [releases]: https://github.com/WebAssembly/wasi-sdk/releases - -6. Publish the release; the previous step only creates a draft. Follow the link - in the previous step or navigate to the GitHub [releases] to review the - description, commit, tag, and assets before clicking "Publish." +2. Wait for the CI build of the tag to finish. This will automatically publish + a draft pre-release to [GitHub Releases](https://github.com/WebAssembly/wasi-sdk/releases). + Release notes are auto-generated and should be reviewed for accuracy. Once + everything looks good manually publish the release through the GitHub UI. -7. Remember to tag the wasi-libc repository with the new `$TAG` version. +3. Remember to tag the wasi-libc repository with the new `$TAG` version. ```shell script git submodule status -- src/wasi-libc # grab $WASI_LIBC_COMMIT from the output diff --git a/ci/download-workflow-artifacts.sh b/ci/download-workflow-artifacts.sh deleted file mode 100755 index 543f93cad..000000000 --- a/ci/download-workflow-artifacts.sh +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/env bash -set -e - -# This script downloads and unzips the artifacts produced in a workflow run. The -# script has several pre-requisites: -# - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) -# - the ID of a workflow run that has run successfully--this is where we -# retrieve the artifacts from -# - a GitHub access token, see https://github.com/settings/tokens -# -# Usage: download-workflow-artifacts.sh - -WORKFLOW_RUN_ID=${WORKFLOW_RUN_ID:-$1} -GITHUB_TOKEN=${GITHUB_TOKEN:-$2} -GITHUB_API_VERSION=${GITHUB_API_VERSION:-2022-11-28} -GITHUB_API_URL=${GITHUB_API_URL:-https://api.github.com/repos/WebAssembly/wasi-sdk} -TMP_DIR=$(mktemp -d -t wasi-sdk-artifacts.XXXXXXX) - -if [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then - >&2 echo "Missing parameter; exiting..." - >&2 echo "Usage: download-worfklow-artifacts.sh " - exit 1 -fi - -# List out the artifacts in the given workflow run. -# See https://docs.github.com/en/rest/actions/artifacts#list-workflow-run-artifacts -ARTIFACTS=$(curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/actions/runs/${WORKFLOW_RUN_ID}/artifacts" \ - | jq -r '.artifacts[] | [(.id|tostring), .name, .archive_download_url] | join(",")') - -for A in $ARTIFACTS; do - ID=$(echo $A | cut -d ',' -f 1) - NAME=$(echo $A | cut -d ',' -f 2) - URL=$(echo $A | cut -d ',' -f 3) - TO=$TMP_DIR/$NAME.zip - # Exclude dist-ubuntu-latest to prefer dist-ubuntu-bionic, which is - # compatible with wider distributions. See: - # - https://github.com/WebAssembly/wasi-sdk/pull/273#issuecomment-1373879491 - # - https://github.com/WebAssembly/wasi-sdk/issues/303 - if [ "${NAME}" = "dist-ubuntu-latest" ]; then - continue - fi - # Exclude the 32-bit Windows artifact in favor of the 64-bit one. This helps - # ensure the identically-named 32-bit tarfile doesn't overwrite the 64-bit - # one below. See: - # - https://github.com/WebAssembly/wasi-sdk/issues/326 - if [ "${NAME}" = "dist-windows-latest-x86" ]; then - continue - fi - >&2 echo "===== Downloading: ${TO} =====" - - # Download the artifacts to the temporary directory. - # See https://docs.github.com/en/rest/actions/artifacts#download-an-artifact - curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - --location --output "${TO}" \ - "${GITHUB_API_URL}/actions/artifacts/${ID}/zip" -done - -# Unzip the workflow artifacts into a `release` directory. -pushd $TMP_DIR > /dev/null -mkdir release -ls -1 *.zip | xargs -n1 unzip -q -o -d release -# Some explanation: -# -1 prints each file on a separate line -# -n1 runs the command once for each item -# -q means quietly -# -o allows unzip to overwrite existing files (e.g., multiple copies of `libclang_rt.builtins-wasm32-wasi-...`) -# -d tells unzip which directory to place things in ->&2 echo "===== Files to release: ${TMP_DIR}/release =====" ->&2 ls -1 release -popd > /dev/null - ->&2 echo ->&2 echo "Ensure the above artifacts look correct, then run \`draft-release.sh\` with the following directory:" -echo "${TMP_DIR}/release" diff --git a/ci/draft-release.sh b/ci/draft-release.sh deleted file mode 100755 index 905ecb9a7..000000000 --- a/ci/draft-release.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/usr/bin/env bash -set -e - -# This script creates a draft pre-release with the artifacts produced in a -# workflow run (see `download-workflow-artifacts.sh`). Note that the pre-release -# is not published publicly--this is kept as a manual step as a safeguard. The -# script has several pre-requisites: -# - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) -# - an already-created tag in the repository (this marks the code to release) -# - a directory containing the artifacts to attach to the release. -# - a GitHub access token, see https://github.com/settings/tokens -# -# Usage: draft-release.sh - -TAG=${TAG:-$1} -ARTIFACTS_DIR=${ARTIFACTS_DIR:-$2} -GITHUB_TOKEN=${GITHUB_TOKEN:-$3} -GITHUB_API_VERSION=${GITHUB_API_VERSION:-2022-11-28} -GITHUB_API_URL=${GITHUB_API_URL:-https://api.github.com/repos/WebAssembly/wasi-sdk} -TMP_DIR=$(mktemp -d -t release.sh.XXXXXXX) - -if [ -z "${TAG}" ] || [ -z "${ARTIFACTS_DIR}" ] || [ -z "${GITHUB_TOKEN}" ]; then - >&2 echo "Missing parameter; exiting..." - >&2 echo "Usage: draft-release.sh " - exit 1 -fi - -# Get the commit SHA for the passed tag. -# See https://docs.github.com/en/rest/commits/commits#get-a-commit -MATCHING_COMMIT=$(curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/commits/${TAG}") -COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') ->&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" - -# Create a draft pre-release for this commit. -# See https://docs.github.com/en/rest/releases/releases#create-a-release -RELEASE_JSON=$(curl \ - -X POST \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}"\ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/releases" \ - -d '{"tag_name":"'${TAG}'","target_commitish":"'${COMMIT}'","name":"'${TAG}'","draft":true,"prerelease":true,"generate_release_notes":true}') -UPLOAD_URL=$(echo $RELEASE_JSON | jq -r '.upload_url') -# Remove the "helpful" but invalid URL suffix that GitHub adds: -UPLOAD_URL=${UPLOAD_URL/\{?name,label\}} -HTML_URL=$(echo $RELEASE_JSON | jq -r '.html_url') ->&2 echo "===== Created draft release: ${HTML_URL} =====" - -# Upload the unzipped artifact files to the release. -# See https://docs.github.com/en/rest/releases/assets#upload-a-release-asset -for FILE in $(ls "${ARTIFACTS_DIR}"); do - FROM=$ARTIFACTS_DIR/$FILE - >&2 echo "===== Uploading: ${FROM} =====" - UPLOADED=$(curl \ - -X POST \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}"\ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - -H "Content-Type: application/octet-stream" \ - "${UPLOAD_URL}?name=${FILE}" \ - --data-binary "@${FROM}") -done - ->&2 echo ->&2 echo "===== Completed =====" ->&2 echo "This created a draft release, do not forget to manually publish it at:" -echo "${HTML_URL}" diff --git a/ci/get-workflows-for-tag.sh b/ci/get-workflows-for-tag.sh deleted file mode 100755 index 879204184..000000000 --- a/ci/get-workflows-for-tag.sh +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env bash -set -e - -# This script retrieves a list of GitHub workflow runs that have successfully completed for a given -# Git tag. The list is printed to stdout (all other output to stderr). It has several -# pre-requisites: -# - some standard Bash tools (curl) and one slightly more rare one (jq) -# - an already-created tag in the repository (this marks the code to release) -# - a GitHub access token, see https://github.com/settings/tokens -# -# Usage: get-workflows-for-tag.sh - -TAG=${TAG:-$1} -GITHUB_TOKEN=${GITHUB_TOKEN:-$2} -GITHUB_API_VERSION=${GITHUB_API_VERSION:-2022-11-28} -GITHUB_API_URL=${GITHUB_API_URL:-https://api.github.com/repos/WebAssembly/wasi-sdk} - -if [ -z "${TAG}" ] || [ -z "${GITHUB_TOKEN}" ]; then - >&2 echo "Missing parameter; exiting..." - >&2 echo "Usage: get-workflows-for-tag.sh " - exit 1 -fi - -# Get the commit SHA for the passed tag. -# See https://docs.github.com/en/rest/commits/commits#get-a-commit -MATCHING_COMMIT=$(curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/commits/${TAG}") -COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') ->&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" - -# Find the workflow runs matching the tag commit. -# See https://docs.github.com/en/rest/actions/workflow-runs#list-workflow-runs-for-a-repository -MATCHING_WORKFLOWS=$(curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/actions/runs?head_sha=${COMMIT}&status=success") -WORKFLOW_RUN_IDS=$(echo $MATCHING_WORKFLOWS | jq -r '.workflow_runs[].id') ->&2 echo "===== Matching workflow run IDs: =====" -echo "$WORKFLOW_RUN_IDS" diff --git a/ci/is-workflow-valid.sh b/ci/is-workflow-valid.sh deleted file mode 100755 index a7f0c80ab..000000000 --- a/ci/is-workflow-valid.sh +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env bash -set -e - -# This script checks 1) that the workflow commit corresponds to the commit for -# the given tag and 2) that the workflow has completed. This is a sanity check -# to ensure the artifacts we are about to publish are in fact built from the -# commit/tag we think. The script has several pre-requisites: -# - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) -# - an already-created tag in the repository (this marks the code to release) -# - the ID of a workflow run that has run successfully--this is where we -# retrieve the artifacts from -# - a GitHub access token, see https://github.com/settings/tokens -# -# Usage: is-workflow-valid.sh - -TAG=${TAG:-$1} -WORKFLOW_RUN_ID=${WORKFLOW_RUN_ID:-$2} -GITHUB_TOKEN=${GITHUB_TOKEN:-$3} -GITHUB_API_VERSION=${GITHUB_API_VERSION:-2022-11-28} -GITHUB_API_URL=${GITHUB_API_URL:-https://api.github.com/repos/WebAssembly/wasi-sdk} - -if [ -z "${TAG}" ] || [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then - >&2 echo "Missing parameter; exiting..." - >&2 echo "Usage: is-workflow-valid.sh " - exit 1 -fi - -# Get the commit SHA for the passed tag. -# See https://docs.github.com/en/rest/commits/commits#get-a-commit -MATCHING_COMMIT=$(curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/commits/${TAG}") -COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') ->&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" - -# Check that the commit of the workflow run matches the tag commit and that the -# workflow was successful. -# See https://docs.github.com/en/rest/actions/workflow-runs#get-a-workflow-run -WORKFLOW_RUN=$(curl \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ - "${GITHUB_API_URL}/actions/runs/${WORKFLOW_RUN_ID}") -WORKFLOW_COMMIT=$(echo $WORKFLOW_RUN | jq -r '.head_sha') -WORKFLOW_STATUS=$(echo $WORKFLOW_RUN | jq -r '.status') ->&2 echo "===== Found commit for workflow ${WORKFLOW_RUN_ID}: ${WORKFLOW_COMMIT} =====" -if [ "${COMMIT}" != "${WORKFLOW_COMMIT}" ]; then - >&2 echo "Commit at tag ${TAG} did not match the commit for workflow ${WORKFLOW_RUN_ID}, exiting...:" - >&2 echo " ${COMMIT} != ${WORKFLOW_COMMIT}" - exit 1 -fi -if [ "${WORKFLOW_STATUS}" != "completed" ]; then - >&2 echo "Workflow ${WORKFLOW_RUN_ID} did not end successfully, exiting...:" - >&2 echo " status = ${WORKFLOW_STATUS}" - exit 1 -fi From de6328701c9bead32fdbcabf8b177fc0c8c91011 Mon Sep 17 00:00:00 2001 From: Catherine Date: Mon, 15 Jul 2024 16:30:05 +0100 Subject: [PATCH 071/165] Fix determination of wasi-sdk version when built in a subdirectory. (#443) --- version.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/version.py b/version.py index a3199f12c..62f0ff9ef 100755 --- a/version.py +++ b/version.py @@ -14,13 +14,13 @@ GIT_REF_LEN = 12 -def exec(command, cwd=None): +def exec(command, cwd): result = subprocess.run(command, stdout=subprocess.PIPE, universal_newlines=True, check=True, cwd=cwd) return result.stdout.strip() -def git_commit(dir='.'): +def git_commit(dir): return exec(['git', 'rev-parse', f'--short={GIT_REF_LEN}', 'HEAD'], dir) @@ -61,7 +61,8 @@ def parse_git_version(version): def git_version(): version = exec(['git', 'describe', '--long', '--candidates=999', - '--match=wasi-sdk-*', '--dirty=+m', f'--abbrev={GIT_REF_LEN}']) + '--match=wasi-sdk-*', '--dirty=+m', f'--abbrev={GIT_REF_LEN}'], + os.path.dirname(sys.argv[0])) major, minor, git, dirty = parse_git_version(version) version = f'{major}.{minor}' if git: From 489ce6eb413a8fc05ea96951c082c9bce365aa77 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 16 Jul 2024 10:07:18 -0500 Subject: [PATCH 072/165] Update build of toolchain/sysroot to not touch installation root (#446) * Update build of toolchain/sysroot to not touch installation root This changes everything to ensure that only the `install` step actually tries to install things. Everything is staged into temporary `./install` folders inside of the build directory and then running the build system's `install` target will actually copy out everything using CMake builtins. Closes #442 * Better integrate generating a version file --- .github/workflows/main.yml | 8 +++-- ci/build.sh | 24 +++++++-------- cmake/wasi-sdk-sysroot.cmake | 27 ++++++++++------ cmake/wasi-sdk-toolchain.cmake | 56 +++++++++++++++++++++++----------- 4 files changed, 73 insertions(+), 42 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 0b3abe693..a23beb1f8 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -119,9 +119,13 @@ jobs: if: runner.os == 'Linux' # Use a shorter build directory than the default on Windows to avoid - # hitting path length and command line length limits. + # hitting path length and command line length limits. See + # WebAssembly/wasi-libc#514 - name: Build and test (Windows) - run: ./ci/build.sh C:/wasi-sdk + run: | + ./ci/build.sh C:/wasi-sdk + mkdir build + cp -r C:/wasi-sdk/dist build shell: bash if: runner.os == 'Windows' diff --git a/ci/build.sh b/ci/build.sh index f391e8ab4..6dc187c4a 100755 --- a/ci/build.sh +++ b/ci/build.sh @@ -11,19 +11,19 @@ set -ex # Optionally allow the first argument to this script to be the install # location. if [ "$1" = "" ]; then - install_dir=`pwd`/build/install + build_dir=`pwd`/build else - install_dir="$1" + build_dir="$1" fi -cmake -G Ninja -B build/toolchain -S . \ +cmake -G Ninja -B $build_dir/toolchain -S . \ -DWASI_SDK_BUILD_TOOLCHAIN=ON \ - "-DCMAKE_INSTALL_PREFIX=$install_dir" \ + "-DCMAKE_INSTALL_PREFIX=$build_dir/install" \ $WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ "-DLLVM_CMAKE_FLAGS=$WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS" -ninja -C build/toolchain install dist -v +ninja -C $build_dir/toolchain install dist -v -mv build/toolchain/dist build/dist +mv $build_dir/toolchain/dist $build_dir/dist if [ "$WASI_SDK_CI_SKIP_SYSROOT" = "1" ]; then exit 0 @@ -31,19 +31,19 @@ fi # Use the just-built toolchain and its `CMAKE_TOOLCHAIN_FILE` to build a # sysroot. -cmake -G Ninja -B build/sysroot -S . \ - "-DCMAKE_TOOLCHAIN_FILE=$install_dir/share/cmake/wasi-sdk.cmake" \ +cmake -G Ninja -B $build_dir/sysroot -S . \ + "-DCMAKE_TOOLCHAIN_FILE=$build_dir/install/share/cmake/wasi-sdk.cmake" \ -DCMAKE_C_COMPILER_WORKS=ON \ -DCMAKE_CXX_COMPILER_WORKS=ON \ -DWASI_SDK_INCLUDE_TESTS=ON \ - "-DCMAKE_INSTALL_PREFIX=$install_dir" -ninja -C build/sysroot install dist -v + "-DCMAKE_INSTALL_PREFIX=$build_dir/install" +ninja -C $build_dir/sysroot install dist -v -mv build/sysroot/dist/* build/dist +mv $build_dir/sysroot/dist/* $build_dir/dist if [ "$WASI_SDK_CI_SKIP_TESTS" = "1" ]; then exit 0 fi # Run tests to ensure that the sysroot works. -ctest --output-on-failure --parallel 10 --test-dir build/sysroot/tests +ctest --output-on-failure --parallel 10 --test-dir $build_dir/sysroot/tests diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 5c23fe610..51f01b06c 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -10,7 +10,8 @@ find_program(MAKE make REQUIRED) option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" ON) option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) -set(wasi_sysroot ${CMAKE_INSTALL_PREFIX}/share/wasi-sysroot) +set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) +set(wasi_sysroot ${wasi_tmp_install}/share/wasi-sysroot) if(WASI_SDK_DEBUG_PREFIX_MAP) add_compile_options( @@ -44,7 +45,7 @@ endif() # compiler-rt build logic # ============================================================================= -set(compiler_rt_dst ${CMAKE_INSTALL_PREFIX}/lib/clang/${clang_version}) +set(compiler_rt_dst ${wasi_tmp_install}/lib/clang/${clang_version}) ExternalProject_Add(compiler-rt-build SOURCE_DIR "${llvm_proj_dir}/compiler-rt" CMAKE_ARGS @@ -234,6 +235,10 @@ endforeach() # misc build logic # ============================================================================= +install(DIRECTORY ${wasi_tmp_install}/lib ${wasi_tmp_install}/share + USE_SOURCE_PERMISSIONS + DESTINATION ${CMAKE_INSTALL_PREFIX}) + # Add a top-level `build` target as well as `build-$target` targets. add_custom_target(build ALL) foreach(target IN LISTS WASI_SDK_TARGETS) @@ -244,14 +249,16 @@ endforeach() # Install a `VERSION` file in the output prefix with a dump of version # information. -set(version_file_tmp ${CMAKE_CURRENT_BINARY_DIR}/VERSION) execute_process( COMMAND ${PYTHON} ${version_script} dump WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} - OUTPUT_FILE ${version_file_tmp}) -install( - FILES ${version_file_tmp} - DESTINATION ${CMAKE_INSTALL_PREFIX}) + OUTPUT_VARIABLE version_dump) +set(version_file_tmp ${CMAKE_CURRENT_BINARY_DIR}/VERSION) +file(GENERATE OUTPUT ${version_file_tmp} CONTENT ${version_dump}) +add_custom_target(version-file DEPENDS ${version_file_tmp}) +add_dependencies(build version-file) +install(FILES ${version_file_tmp} + DESTINATION ${CMAKE_INSTALL_PREFIX}) if(WASI_SDK_INCLUDE_TESTS) add_subdirectory(tests) @@ -264,13 +271,13 @@ set(dist_dir ${CMAKE_CURRENT_BINARY_DIR}/dist) # Tarball with just `compiler-rt` builtins within it wasi_sdk_add_tarball(dist-compiler-rt ${dist_dir}/libclang_rt.builtins-wasm32-wasi-${wasi_sdk_version}.tar.gz - ${CMAKE_INSTALL_PREFIX}/lib/clang/${clang_version}/lib/wasi) + ${wasi_tmp_install}/lib/clang/${clang_version}/lib/wasi) add_dependencies(dist-compiler-rt compiler-rt) # Tarball with the whole sysroot wasi_sdk_add_tarball(dist-sysroot ${dist_dir}/wasi-sysroot-${wasi_sdk_version}.tar.gz - ${CMAKE_INSTALL_PREFIX}/share/wasi-sysroot) -add_dependencies(dist-sysroot build install) + ${wasi_tmp_install}/share/wasi-sysroot) +add_dependencies(dist-sysroot build) add_custom_target(dist DEPENDS dist-compiler-rt dist-sysroot) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 1da2ebf3f..92943defb 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -7,6 +7,8 @@ set(WASI_SDK_ARTIFACT "" CACHE STRING "Name of the wasi-sdk artifact being produ string(REGEX REPLACE "[ ]+" ";" llvm_cmake_flags_list "${LLVM_CMAKE_FLAGS}") +set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) + if(NOT CMAKE_BUILD_TYPE) set(CMAKE_BUILD_TYPE MinSizeRel) endif() @@ -16,7 +18,7 @@ set(default_cmake_args -DCMAKE_AR=${CMAKE_AR} -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} - -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}) + -DCMAKE_INSTALL_PREFIX=${wasi_tmp_install}) if(CMAKE_C_COMPILER_LAUNCHER) list(APPEND default_cmake_args -DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}) @@ -106,6 +108,14 @@ ExternalProject_Add(llvm-build USES_TERMINAL_INSTALL ON ) +add_custom_target(build ALL DEPENDS llvm-build) + +# Installation target for this outer project for installing the toolchain to the +# system. +install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_install}/share + USE_SOURCE_PERMISSIONS + DESTINATION ${CMAKE_INSTALL_PREFIX}) + # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) @@ -118,24 +128,34 @@ add_custom_command( COMMAND cargo install --root ${wasm_component_ld_root} ${rust_target_flag} wasm-component-ld@${wasm_component_ld_version} + COMMAND + cmake -E make_directory ${wasi_tmp_install}/bin + COMMAND + cmake -E copy ${wasm_component_ld} ${wasi_tmp_install}/bin COMMENT "Building `wasm-component-ld` ...") - -add_custom_target(wasm-component-ld ALL DEPENDS ${wasm_component_ld}) - -install( - PROGRAMS ${wasm_component_ld} - DESTINATION ${CMAKE_INSTALL_PREFIX}/bin) +add_custom_target(wasm-component-ld DEPENDS ${wasm_component_ld}) +add_dependencies(build wasm-component-ld) # Setup installation logic for CMake support files. -install( - PROGRAMS src/config/config.sub src/config/config.guess - DESTINATION ${CMAKE_INSTALL_PREFIX}/share/misc) -install( - FILES wasi-sdk.cmake wasi-sdk-pthread.cmake wasi-sdk-p2.cmake - DESTINATION ${CMAKE_INSTALL_PREFIX}/share/cmake) -install( - DIRECTORY cmake/Platform - DESTINATION ${CMAKE_INSTALL_PREFIX}/share/cmake) +add_custom_target(misc-files) +add_dependencies(build misc-files) + +function(copy_misc_file src dst_folder) + cmake_path(GET src FILENAME src_filename) + set(dst ${wasi_tmp_install}/share/${dst_folder}/${src_filename}) + add_custom_command( + OUTPUT ${dst} + COMMAND cmake -E copy ${CMAKE_CURRENT_SOURCE_DIR}/${src} ${dst}) + add_custom_target(copy-${src_filename} DEPENDS ${dst}) + add_dependencies(misc-files copy-${src_filename}) +endfunction() + +copy_misc_file(src/config/config.sub misc) +copy_misc_file(src/config/config.guess misc) +copy_misc_file(wasi-sdk.cmake cmake) +copy_misc_file(wasi-sdk-pthread.cmake cmake) +copy_misc_file(wasi-sdk-p2.cmake cmake) +copy_misc_file(cmake/Platform/WASI.cmake cmake/Platform) include(wasi-sdk-dist) @@ -156,6 +176,6 @@ endif() set(dist_dir ${CMAKE_CURRENT_BINARY_DIR}/dist) wasi_sdk_add_tarball(dist-toolchain ${dist_dir}/wasi-toolchain-${wasi_sdk_version}-${wasi_sdk_artifact}.tar.gz - ${CMAKE_INSTALL_PREFIX}) -add_dependencies(dist-toolchain llvm-build install) + ${wasi_tmp_install}) +add_dependencies(dist-toolchain build) add_custom_target(dist DEPENDS dist-toolchain) From ca4fa08fa797eab90ead937ca423053b105893de Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 16 Jul 2024 10:29:21 -0500 Subject: [PATCH 073/165] Update wasm-component-ld to 0.5.4 (#439) This updates to the latest version of the linker which has better support for standard flags like `--version` and various other LLD flags. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 92943defb..fbf738eff 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.0) +set(wasm_component_ld_version 0.5.4) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 1a3d5ee3bc5b9def501dd72f23201997679c2348 Mon Sep 17 00:00:00 2001 From: Joel Dice Date: Tue, 16 Jul 2024 12:38:27 -0600 Subject: [PATCH 074/165] update wasm-component-ld to 0.5.5 (#448) This includes support for a new `--component-type` option, which allows passing one or more component types as WIT files rather than (or in addition to) object files, which can be preferable when such files must be stored in source control. Signed-off-by: Joel Dice --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index fbf738eff..d02edbe16 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.4) +set(wasm_component_ld_version 0.5.5) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 08be2d4b29ab945305b8cc2e70375b9f52c0e3fe Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 16 Jul 2024 19:18:05 -0500 Subject: [PATCH 075/165] Don't modify the host compiler's sysroot (#445) This commit updates the building of the wasi-sdk sysroot to leverage the `-resource-dir` argument from Clang to avoid modifying the host compiler's sysroot with compiler-rt things. This should help improve the experience of building a standalone sysroot with whatever host Clang is on the system. Closes #444 --- cmake/wasi-sdk-sysroot.cmake | 32 ++++++++++++++++++++------------ tests/CMakeLists.txt | 4 ++-- 2 files changed, 22 insertions(+), 14 deletions(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 51f01b06c..1ba204a47 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -12,13 +12,18 @@ option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) set(wasi_sysroot ${wasi_tmp_install}/share/wasi-sysroot) +set(wasi_resource_dir ${wasi_tmp_install}/lib/clang/${clang_version}) + +# Force usage of the custom-built resource-dir and sysroot for the rest of the +# wasi compiles. +add_compile_options(-resource-dir ${wasi_resource_dir}) +add_compile_options(--sysroot ${wasi_sysroot}) if(WASI_SDK_DEBUG_PREFIX_MAP) add_compile_options( -fdebug-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=wasisdk://v${wasi_sdk_version}) endif() - # Default arguments for builds of cmake projects (mostly LLVM-based) to forward # along much of our own configuration into these projects. set(default_cmake_args @@ -45,7 +50,6 @@ endif() # compiler-rt build logic # ============================================================================= -set(compiler_rt_dst ${wasi_tmp_install}/lib/clang/${clang_version}) ExternalProject_Add(compiler-rt-build SOURCE_DIR "${llvm_proj_dir}/compiler-rt" CMAKE_ARGS @@ -58,7 +62,7 @@ ExternalProject_Add(compiler-rt-build -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON -DCMAKE_C_COMPILER_TARGET=wasm32-wasi -DCOMPILER_RT_OS_DIR=wasi - -DCMAKE_INSTALL_PREFIX=${compiler_rt_dst} + -DCMAKE_INSTALL_PREFIX=${wasi_resource_dir} EXCLUDE_FROM_ALL ON USES_TERMINAL_CONFIGURE ON USES_TERMINAL_BUILD ON @@ -69,26 +73,30 @@ ExternalProject_Add(compiler-rt-build # around some headers and make copies of the `wasi` directory as `wasip1` and # `wasip2` execute_process( - COMMAND ${CMAKE_C_COMPILER} -print-runtime-dir - OUTPUT_VARIABLE clang_runtime_dir + COMMAND ${CMAKE_C_COMPILER} -print-resource-dir + OUTPUT_VARIABLE clang_resource_dir OUTPUT_STRIP_TRAILING_WHITESPACE) -cmake_path(GET clang_runtime_dir PARENT_PATH clang_runtime_libdir) # chop off `wasi` -cmake_path(GET clang_runtime_libdir PARENT_PATH clang_sysroot_dir) # chop off `lib` add_custom_target(compiler-rt-post-build + # The `${wasi_resource_dir}` folder is going to get used as `-resource-dir` + # for future compiles. Copy the host compiler's own headers into this + # directory to ensure that all host-defined headers all work as well. COMMAND ${CMAKE_COMMAND} -E copy_directory - ${clang_sysroot_dir} ${compiler_rt_dst} + ${clang_resource_dir}/include ${wasi_resource_dir}/include + + # Copy the `lib/wasi` folder to `libc/wasi{p1,p2}` to ensure that those + # OS-strings also work for looking up the compiler-rt.a file. COMMAND ${CMAKE_COMMAND} -E copy_directory - ${compiler_rt_dst}/lib/wasi ${compiler_rt_dst}/lib/wasip1 + ${wasi_resource_dir}/lib/wasi ${wasi_resource_dir}/lib/wasip1 COMMAND ${CMAKE_COMMAND} -E copy_directory - ${compiler_rt_dst}/lib/wasi ${compiler_rt_dst}/lib/wasip2 - COMMAND ${CMAKE_COMMAND} -E copy_directory_if_different - ${compiler_rt_dst}/lib ${clang_runtime_libdir} + ${wasi_resource_dir}/lib/wasi ${wasi_resource_dir}/lib/wasip2 + COMMENT "finalizing compiler-rt installation" ) add_dependencies(compiler-rt-post-build compiler-rt-build) add_custom_target(compiler-rt DEPENDS compiler-rt-build compiler-rt-post-build) + # ============================================================================= # wasi-libc build logic # ============================================================================= diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 23010fa0b..fb6a03bc7 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -5,8 +5,8 @@ include(CTest) enable_testing() set(CMAKE_EXECUTABLE_SUFFIX ".wasm") -add_compile_options(--sysroot=${wasi_sysroot}) -add_link_options(--sysroot=${wasi_sysroot}) +add_compile_options(--sysroot=${wasi_sysroot} -resource-dir ${wasi_resource_dir}) +add_link_options(--sysroot=${wasi_sysroot} -resource-dir ${wasi_resource_dir}) # Sanity check setup if (NOT ${CMAKE_SYSTEM_NAME} STREQUAL WASI) From 9ee70805ec2712d1f970f08986c4382f740fb4c1 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 22 Jul 2024 12:42:35 -0500 Subject: [PATCH 076/165] Add a CI check for testing toolchains as-distributed (#449) * Add a CI check for testing toolchains as-distributed One aspect of testing lost in the CMake-based migration is the ability to test the toolchains as distributed in release artifacts. Tests use `--sysroot` and `-resource-dir` (soon) to customize how the host compiler runs but this means that it would be possible to regress the default sysroot theoretically. To rectify this situation this commit adds a new CI test which uses the release artifacts of previous steps to build a `wasi-sdk-*.tar.gz` tarball which is then extracted and tested as-is. A new flag was added to the cmake configuration to avoid depending on fresh sysroot libraries for tests and instead test the host toolchain. * Fix version.py script running * Fix artifact download * Add ninja * Update submodules in new test job * Only add extra options for libcxx build Otherwise the test directory seems like it inherits these options which isn't desired when testing the host toolchain. --- .github/workflows/main.yml | 36 ++++++++++++++++++++++++++++++++++++ README.md | 2 ++ cmake/wasi-sdk-sysroot.cmake | 17 ++++++++++------- tests/CMakeLists.txt | 19 +++++++++++++++---- 4 files changed, 63 insertions(+), 11 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index a23beb1f8..bdd237262 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -211,3 +211,39 @@ jobs: run: gh release create --draft --prerelease --generate-notes ${{ github.ref_name }} ./dist/* env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + # Test the final artifacts as-is without passing `--sysroot` or + # `-resource-dir` or any extra flags. This exercises running the compiler + # as-is from the distribution tarballs and ensuring that it can build and pass + # all tests. + test-standalone: + name: Test standalone toolchain + needs: build + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - run: git fetch --tags --force + name: Force-fetch tags to work around actions/checkout#290 + - run: git submodule update --init --depth 32 --jobs 3 + - name: Setup `wasmtime` for tests + uses: bytecodealliance/actions/wasmtime/setup@v1 + with: + version: "18.0.2" + - name: Install ninja + run: sudo apt-get install -y ninja-build + if: runner.os == 'Linux' + - uses: actions/download-artifact@v4 + with: + name: dist-x86_64-linux + path: dist-x86_64-linux + - run: ./ci/merge-artifacts.sh + - run: tar xf dist/wasi-sdk-*.tar.gz + - run: | + cmake -G Ninja -B build -S . \ + -DWASI_SDK_INCLUDE_TESTS=ON \ + -DWASI_SDK_TEST_HOST_TOOLCHAIN=ON \ + -DCMAKE_TOOLCHAIN_FILE=$(ls ./wasi-sdk-*/share/cmake/wasi-sdk.cmake) + - run: ninja -C build build-tests + - run: ctest --output-on-failure --parallel 10 --test-dir build/tests diff --git a/README.md b/README.md index 9632be985..a594f40f7 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,8 @@ in compiling WebAssembly code. Supported CMake flags are: * `-DWASI_SDK_DEBUG_PREFIX_MAKE=OFF` - disable `-fdebug-prefix-map` when building C/C++ code to use full host paths instead. * `-DWASI_SDK_INCLUDE_TESTS=ON` - used for building tests. +* `-DWASI_SDK_TEST_HOST_TOOLCHAIN=ON` - test the host toolchain's wasi-libc and + sysroot libraries, don't build or use fresh libraries for tests. * `-DWASI_SDK_TARGETS=..` - a list of targets to build, by default all WASI targets are compiled. diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 1ba204a47..eea357a38 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -14,11 +14,6 @@ set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) set(wasi_sysroot ${wasi_tmp_install}/share/wasi-sysroot) set(wasi_resource_dir ${wasi_tmp_install}/lib/clang/${clang_version}) -# Force usage of the custom-built resource-dir and sysroot for the rest of the -# wasi compiles. -add_compile_options(-resource-dir ${wasi_resource_dir}) -add_compile_options(--sysroot ${wasi_sysroot}) - if(WASI_SDK_DEBUG_PREFIX_MAP) add_compile_options( -fdebug-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=wasisdk://v${wasi_sdk_version}) @@ -167,9 +162,17 @@ function(define_libcxx target) get_property(dir_compile_opts DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) get_property(dir_link_opts DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY LINK_OPTIONS) - set(extra_cflags_list ${CMAKE_C_FLAGS} ${target_flags} --target=${target} ${dir_compile_opts} ${dir_link_opts}) + set(extra_flags + ${target_flags} + --target=${target} + ${dir_compile_opts} + ${dir_link_opts} + --sysroot ${wasi_sysroot} + -resource-dir ${wasi_resource_dir}) + + set(extra_cflags_list ${CMAKE_C_FLAGS} ${extra_flags}) list(JOIN extra_cflags_list " " extra_cflags) - set(extra_cxxflags_list ${CMAKE_CXX_FLAGS} ${target_flags} --target=${target} ${dir_compile_opts} ${dir_link_opts}) + set(extra_cxxflags_list ${CMAKE_CXX_FLAGS} ${extra_flags}) list(JOIN extra_cxxflags_list " " extra_cxxflags) ExternalProject_Add(libcxx-${target}-build diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index fb6a03bc7..c5e7edd39 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -5,8 +5,12 @@ include(CTest) enable_testing() set(CMAKE_EXECUTABLE_SUFFIX ".wasm") -add_compile_options(--sysroot=${wasi_sysroot} -resource-dir ${wasi_resource_dir}) -add_link_options(--sysroot=${wasi_sysroot} -resource-dir ${wasi_resource_dir}) +option(WASI_SDK_TEST_HOST_TOOLCHAIN "Test against the host toolchain, not a fresh sysroot" OFF) + +if(NOT WASI_SDK_TEST_HOST_TOOLCHAIN) + add_compile_options(--sysroot=${wasi_sysroot} -resource-dir ${wasi_resource_dir}) + add_link_options(--sysroot=${wasi_sysroot} -resource-dir ${wasi_resource_dir}) +endif() # Sanity check setup if (NOT ${CMAKE_SYSTEM_NAME} STREQUAL WASI) @@ -22,6 +26,8 @@ set(WASI_SDK_RUNWASI "wasmtime" CACHE STRING "Runner for tests") # Test everything at O0, O2, and O2+LTO set(opt_flags -O0 -O2 "-O2 -flto") +add_custom_target(build-tests) + # Executes a single `test` specified. # # This will compile `test` for all the various targets and with various @@ -35,6 +41,7 @@ function(add_testcase runwasi test) # Add a new test executable based on `test` add_executable(${target_name} ${test}) + add_dependencies(build-tests ${target_name}) # Configure all the compile options necessary. For example `--target` here # if the target doesn't look like it's already in the name of the compiler @@ -60,9 +67,13 @@ function(add_testcase runwasi test) # Apply language-specific options and dependencies. if(test MATCHES "cc$") target_compile_options(${target_name} PRIVATE -fno-exceptions) - add_dependencies(${target_name} libcxx-${target}) + if(NOT WASI_SDK_TEST_HOST_TOOLCHAIN) + add_dependencies(${target_name} libcxx-${target}) + endif() else() - add_dependencies(${target_name} wasi-libc-${target}) + if(NOT WASI_SDK_TEST_HOST_TOOLCHAIN) + add_dependencies(${target_name} wasi-libc-${target}) + endif() endif() # Apply target-specific options. From 9af8b0f30499ab5a2c08dbc3c8144aff80d4d24e Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 22 Jul 2024 13:41:03 -0500 Subject: [PATCH 077/165] Refactor handling env vars in CI build (#455) * Refactor handling env vars in CI build This commit updates the handling of various environment variables to centralize them in one location in `main.yml` as part of CI configuration. This removes a few custom-named variables in favor of using explicitly-listed env vars. Additionally this moves some env var management from the CI container files to the github actions config as well to keep everything in one place ideally. * Add some more comments about where configuration comes in --- .github/workflows/main.yml | 51 ++++++++++++++++++++----------- ci/docker-build.sh | 6 ++++ ci/docker/Dockerfile.arm64-linux | 24 ++++----------- ci/docker/Dockerfile.common | 6 ++++ ci/docker/Dockerfile.x86_64-linux | 7 +++-- 5 files changed, 56 insertions(+), 38 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index bdd237262..d3d542f74 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -24,26 +24,50 @@ jobs: - artifact: arm64-linux os: ubuntu-latest rust_target: aarch64-unknown-linux-gnu + env: + # Don't build a sysroot for this cross-compiled target since it + # would require a host compiler and the sysroot is otherwise + # already built on other CI builders. + WASI_SDK_CI_SKIP_SYSROOT: 1 + + WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS: >- + -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc + -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ + -DCMAKE_CROSSCOMPILING=True + -DCMAKE_CXX_FLAGS=-march=armv8-a + -DCMAKE_SYSTEM_PROCESSOR=arm64 + -DCMAKE_SYSTEM_NAME=Linux + -DLLVM_HOST_TRIPLE=aarch64-linux-gnu + -DRUST_TARGET=aarch64-unknown-linux-gnu - artifact: arm64-macos os: macos-latest - llvm_cmake_flags: -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DCMAKE_OSX_ARCHITECTURES=arm64 rust_target: aarch64-apple-darwin + env: + WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS: >- + -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 + -DCMAKE_OSX_ARCHITECTURES=arm64 - artifact: x86_64-macos os: macos-latest - llvm_cmake_flags: -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DCMAKE_OSX_ARCHITECTURES=x86_64 rust_target: x86_64-apple-darwin - skip_sysroot: 1 + env: + WASI_SDK_CI_SKIP_SYSROOT: 1 + WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS: >- + -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 + -DCMAKE_OSX_ARCHITECTURES=x86_64 - artifact: x86_64-windows os: windows-latest - # TODO: tests are pretty close to passing on Windows but need some - # final tweaks, namely testing the exit code doesn't work since - # exit codes are different on Windows and the `mmap.c` tests seems - # to have issues probably with line endings. Needs someone with a - # Windows checkout tot test further. - skip_tests: 1 + env: + # TODO: tests are pretty close to passing on Windows but need some + # final tweaks, namely testing the exit code doesn't work since + # exit codes are different on Windows and the `mmap.c` tests seems + # to have issues probably with line endings. Needs someone with a + # Windows checkout tot test further. + WASI_SDK_CI_SKIP_TESTS: 1 + + env: ${{ matrix.env || fromJSON('{}') }} steps: - uses: actions/checkout@v4 with: @@ -77,9 +101,6 @@ jobs: # Configure CMake flags for `ci/build.sh` as necessary for each # matrix entry. - - run: echo WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS=${{ matrix.llvm_cmake_flags }} >> $GITHUB_ENV - if: matrix.llvm_cmake_flags != '' - shell: bash - run: | cmake_args=-DWASI_SDK_ARTIFACT=${{ matrix.artifact }} if [ "${{ matrix.rust_target }}" != "" ]; then @@ -88,12 +109,6 @@ jobs: fi echo WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS="$cmake_args" >> $GITHUB_ENV shell: bash - - run: echo WASI_SDK_CI_SKIP_SYSROOT=1 >> $GITHUB_ENV - shell: bash - if: matrix.skip_sysroot != '' - - run: echo WASI_SDK_CI_SKIP_TESTS=1 >> $GITHUB_ENV - shell: bash - if: matrix.skip_tests != '' # Add some extra installed software on each runner as necessary. - name: Setup `wasmtime` for tests diff --git a/ci/docker-build.sh b/ci/docker-build.sh index a695d6f4a..b5b8498e0 100755 --- a/ci/docker-build.sh +++ b/ci/docker-build.sh @@ -48,6 +48,12 @@ args="$args --volume $ccache_dir:/ccache:Z --env CCACHE_DIR=/ccache" args="$args --volume `rustc --print sysroot`:/rustc:ro" args="$args --volume $(dirname $(which wasmtime)):/wasmtime:ro" +# Pass through some env vars that `build.sh` reads +args="$args --env WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS" +args="$args --env WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS" +args="$args --env WASI_SDK_CI_SKIP_SYSROOT" +args="$args --env WASI_SDK_CI_SKIP_TESTS" + # Before running `ci/build.sh` set up some rust/PATH related info to use what # was just mounted above, and then execute the build. docker run \ diff --git a/ci/docker/Dockerfile.arm64-linux b/ci/docker/Dockerfile.arm64-linux index fa9a16ea4..6be6d6ed8 100644 --- a/ci/docker/Dockerfile.arm64-linux +++ b/ci/docker/Dockerfile.arm64-linux @@ -1,23 +1,11 @@ FROM wasi-sdk-builder-base +# Install an extra C++ toolchain which can target arm64 linux. RUN apt-get install -y g++-aarch64-linux-gnu -# Don't build a sysroot for this cross-compiled target since it would require a -# host compiler and the sysroot is otherwise already built on other CI builders. -ENV WASI_SDK_CI_SKIP_SYSROOT 1 - -ENV WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS \ - -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \ - -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \ - -DCMAKE_CROSSCOMPILING=True \ - -DCMAKE_CXX_FLAGS=-march=armv8-a \ - -DCMAKE_SYSTEM_PROCESSOR=arm64 \ - -DCMAKE_SYSTEM_NAME=Linux \ - -DLLVM_HOST_TRIPLE=aarch64-linux-gnu \ - -DRUST_TARGET=aarch64-unknown-linux-gnu - -ENV WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ - -DWASI_SDK_ARTIFACT=arm64-linux \ - -DRUST_TARGET=aarch64-unknown-linux-gnu - +# Configure Rust to use this new compiler for linking Rust executables. ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER aarch64-linux-gnu-gcc + +# Note that `.github/workflows/main.yml` sets various bits and bobs of +# configuration and cmake flags that may get passed to the underlying build. For +# example LLVM is instructed to use the toolchain installed above. diff --git a/ci/docker/Dockerfile.common b/ci/docker/Dockerfile.common index 1ab98a61a..480e9b77d 100644 --- a/ci/docker/Dockerfile.common +++ b/ci/docker/Dockerfile.common @@ -4,6 +4,7 @@ FROM ubuntu:18.04 +# Various build tooling and such necessary to build LLVM and a wasi-sysroot RUN apt-get update \ && apt-get install -y --no-install-recommends \ ccache \ @@ -16,6 +17,8 @@ RUN apt-get update \ unzip \ xz-utils +# Install a more recent version of CMake than what 18.04 has since that's what +# LLVM requires. RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.29.5/cmake-3.29.5-linux-x86_64.tar.gz \ && tar xf cmake-3.29.5-linux-x86_64.tar.gz \ && rm cmake-3.29.5-linux-x86_64.tar.gz \ @@ -24,9 +27,12 @@ RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.29.5/cmake- ENV PATH /opt/cmake/bin:$PATH +# As with CMake install a later version of Ninja than waht 18.04 has. RUN curl -sSLO https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip \ && unzip ninja-linux.zip \ && rm *.zip \ && mv ninja /opt/cmake/bin +# Tell programs to cache in a location that both isn't a `--volume` mounted root +# and isn't `/root` in the container as that won't be writable during the build. ENV XDG_CACHE_HOME /tmp/cache diff --git a/ci/docker/Dockerfile.x86_64-linux b/ci/docker/Dockerfile.x86_64-linux index 47473e691..9d443e530 100644 --- a/ci/docker/Dockerfile.x86_64-linux +++ b/ci/docker/Dockerfile.x86_64-linux @@ -1,4 +1,7 @@ FROM wasi-sdk-builder-base -ENV WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ - -DWASI_SDK_ARTIFACT=x86_64-linux +# No extra configuration necessary for x86_64 over what `Dockerfile.common` +# already has. +# +# Note though that `.github/workflows/main.yml` still sets various bits and bobs +# of configuration and cmake flags that may get passed to the underlying build. From 0584df3946029e572cd48d9a462538f5a178f534 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 22 Jul 2024 13:59:29 -0500 Subject: [PATCH 078/165] Prune ccache caches in CI on each run (#457) I noticed in a [recent build] that the cache size for Windows is quite large at 500M. That might be related to switching to MSVC, I'm not sure, but something else I've realized is that as-configured wasi-sdk will continuously grow the cache over time and it won't ever get trimmed until we hit github actions limits. This is because the cache is restored from an older version, then appended to with the current build, then saved again. That theoretically means that each builder could make up to a 5G cache which is a bit too large. This commit adds an extra step that removes all objects older than 1d to ensure that older builds eventually get cleaned out of the cache. GitHub Actions should then still delete older caches pretty regularly but each individual cache should be bounded still. [recent build]: https://github.com/WebAssembly/wasi-sdk/actions/runs/10045872592/job/27764084758?pr=456 --- .github/workflows/main.yml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index d3d542f74..0ff1d36aa 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -152,6 +152,15 @@ jobs: name: ${{ format( 'dist-{0}', matrix.artifact) }} path: build/dist + # Caches are persisted across runs by restoring the latest cache which + # means that quite a lot of cruft can accumulate. Prune older entries that + # haven't been used by this run to avoid the cache continuously getting + # larger. In theory this should use `--evict-older-than $dur` where `$dur` + # is the time since the start of the run, but I'm not sure how to easily + # calculate that so pick something loose like one day instead. + - name: Prune ccache objects + run: ccache --evict-older-than 1d + # Help debug ccache issues by showing what happened. - if: always() name: Show ccache statistics From fef66e3d2319d8360825dcba1cf23061f5313c11 Mon Sep 17 00:00:00 2001 From: Matt McCormick Date: Tue, 23 Jul 2024 12:53:35 -0400 Subject: [PATCH 079/165] Remove cmake_minimum_required from toolchain files (#412) This command is generally not included in toolchain files. It is used in project configuration files that will consume the toolchain file. Also, such a low value triggers warnings, support for <3.5.0 is being removed from CMake. --- wasi-sdk-p2.cmake | 3 --- wasi-sdk-pthread.cmake | 3 --- wasi-sdk.cmake | 3 --- 3 files changed, 9 deletions(-) diff --git a/wasi-sdk-p2.cmake b/wasi-sdk-p2.cmake index f00d7732b..aeed29350 100644 --- a/wasi-sdk-p2.cmake +++ b/wasi-sdk-p2.cmake @@ -1,8 +1,5 @@ # Cmake toolchain description file for the Makefile -# This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.5.0) - # Until Platform/WASI.cmake is upstream we need to inject the path to it # into CMAKE_MODULE_PATH. list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}") diff --git a/wasi-sdk-pthread.cmake b/wasi-sdk-pthread.cmake index ba660234b..c2cd83dbd 100644 --- a/wasi-sdk-pthread.cmake +++ b/wasi-sdk-pthread.cmake @@ -1,8 +1,5 @@ # Cmake toolchain description file for the Makefile -# This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) - set(CMAKE_SYSTEM_NAME WASI) set(CMAKE_SYSTEM_VERSION 1) set(CMAKE_SYSTEM_PROCESSOR wasm32) diff --git a/wasi-sdk.cmake b/wasi-sdk.cmake index 430e98f04..2131cfcd7 100644 --- a/wasi-sdk.cmake +++ b/wasi-sdk.cmake @@ -1,8 +1,5 @@ # Cmake toolchain description file for the Makefile -# This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.5.0) - # Until Platform/WASI.cmake is upstream we need to inject the path to it # into CMAKE_MODULE_PATH. list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}") From 6387b4f7e7b12d15a2d505eadeffe4baef4779b5 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 23 Jul 2024 12:40:59 -0500 Subject: [PATCH 080/165] Use MSVC for the Windows toolchain instead of MinGW (#456) * Use MSVC for the Windows toolchain instead of MinGW Explicitly use MSVC to avoid the runtime dependencies that the default toolchain CMake is using is bringing in. Closes #454 * Update .github/workflows/main.yml Co-authored-by: Andrew Brown * Specify MinSizeRel in `build.sh` Looks like MSVC defaults to "Debug" instead of an empty string to so the default logic which works for other compilers isn't kicking in here. * Don't persist ccache stats across runs * Move zeroing --------- Co-authored-by: Andrew Brown --- .github/workflows/main.yml | 21 ++++++++++++++++++--- ci/build.sh | 1 + 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 0ff1d36aa..31095320b 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -125,6 +125,9 @@ jobs: run: sudo apt install ccache if: runner.os == 'Linux' + - name: Clear ccache statistics + run: ccache --zero-stats + - name: Build and test (macOS) run: ./ci/build.sh if: runner.os == 'macOS' @@ -133,11 +136,23 @@ jobs: run: ./ci/docker-build.sh ${{ matrix.artifact }} if: runner.os == 'Linux' - # Use a shorter build directory than the default on Windows to avoid - # hitting path length and command line length limits. See - # WebAssembly/wasi-libc#514 + # Setup the VS Developoer Prompt environment variables to explicitly use + # MSVC to compile LLVM as that avoids extra runtime dependencies + # msys/mingw might bring. + # + # As of 2024-07-22 this sha is the "v1.13.0" tag. + - uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 + if: runner.os == 'Windows' - name: Build and test (Windows) run: | + # Delete a troublesome binary as recommended here + # https://github.com/ilammy/msvc-dev-cmd?tab=readme-ov-file#name-conflicts-with-shell-bash + rm /usr/bin/link + # Use a shorter build directory than the default on Windows to avoid + # hitting path length and command line length limits. See + # WebAssembly/wasi-libc#514. Despite using a different build directory + # though still move the `dist` folder to `build/dist` so the upload + # step below doesn't need a windows-specific hook. ./ci/build.sh C:/wasi-sdk mkdir build cp -r C:/wasi-sdk/dist build diff --git a/ci/build.sh b/ci/build.sh index 6dc187c4a..09c92a35f 100755 --- a/ci/build.sh +++ b/ci/build.sh @@ -18,6 +18,7 @@ fi cmake -G Ninja -B $build_dir/toolchain -S . \ -DWASI_SDK_BUILD_TOOLCHAIN=ON \ + -DCMAKE_BUILD_TYPE=MinSizeRel \ "-DCMAKE_INSTALL_PREFIX=$build_dir/install" \ $WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ "-DLLVM_CMAKE_FLAGS=$WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS" From e4fef74fee3d3849339b6c492f2dafe229891364 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Wed, 24 Jul 2024 05:31:44 +0900 Subject: [PATCH 081/165] Retire clang_version (#451) * Retire clang_version Note: this makes the "install" target install the compiler rt into the compiler's runtime directory. IMO, it's what "install" is supposed to do. If you want to avoid modifing the runtime directory for some reasons, you can still do "dist" without "install". * wasi-sdk-sysroot.cmake: Avoid modifying things out of CMAKE_INSTALL_PREFIX * wasi-sdk-sysroot.cmake: control the resource-dir modification by a cmake option --- CMakeLists.txt | 5 ----- cmake/wasi-sdk-sysroot.cmake | 16 +++++++++++++--- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 90abca435..b39f21d0b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -24,16 +24,11 @@ find_program(PYTHON python3 python REQUIRED) # Set some variables based on the `version.py` script set(version_script ${CMAKE_CURRENT_SOURCE_DIR}/version.py) -execute_process( - COMMAND ${PYTHON} ${version_script} llvm-major --llvm-dir=${llvm_proj_dir} - OUTPUT_VARIABLE clang_version - OUTPUT_STRIP_TRAILING_WHITESPACE) execute_process( COMMAND ${PYTHON} ${version_script} OUTPUT_VARIABLE wasi_sdk_version OUTPUT_STRIP_TRAILING_WHITESPACE) -message(STATUS "wasi-sdk toolchain LLVM version is ${clang_version}") message(STATUS "wasi-sdk version is ${wasi_sdk_version}") # Only include one version of the build logic as pulling in both isn't diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index eea357a38..7e7345b0f 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -9,10 +9,11 @@ find_program(MAKE make REQUIRED) option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" ON) option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) +option(WASI_SDK_INSTALL_TO_CLANG_RESOURCE_DIR "Whether or not to modify the compiler's resource directory" OFF) set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) set(wasi_sysroot ${wasi_tmp_install}/share/wasi-sysroot) -set(wasi_resource_dir ${wasi_tmp_install}/lib/clang/${clang_version}) +set(wasi_resource_dir ${wasi_tmp_install}/wasi-resource-dir) if(WASI_SDK_DEBUG_PREFIX_MAP) add_compile_options( @@ -246,9 +247,18 @@ endforeach() # misc build logic # ============================================================================= -install(DIRECTORY ${wasi_tmp_install}/lib ${wasi_tmp_install}/share +install(DIRECTORY ${wasi_tmp_install}/share USE_SOURCE_PERMISSIONS DESTINATION ${CMAKE_INSTALL_PREFIX}) +if(WASI_SDK_INSTALL_TO_CLANG_RESOURCE_DIR) + install(DIRECTORY ${wasi_resource_dir}/lib + USE_SOURCE_PERMISSIONS + DESTINATION ${clang_resource_dir}) +else() + install(DIRECTORY ${wasi_resource_dir}/lib + USE_SOURCE_PERMISSIONS + DESTINATION ${CMAKE_INSTALL_PREFIX}/clang-resource-dir) +endif() # Add a top-level `build` target as well as `build-$target` targets. add_custom_target(build ALL) @@ -282,7 +292,7 @@ set(dist_dir ${CMAKE_CURRENT_BINARY_DIR}/dist) # Tarball with just `compiler-rt` builtins within it wasi_sdk_add_tarball(dist-compiler-rt ${dist_dir}/libclang_rt.builtins-wasm32-wasi-${wasi_sdk_version}.tar.gz - ${wasi_tmp_install}/lib/clang/${clang_version}/lib/wasi) + ${wasi_resource_dir}/lib/wasi) add_dependencies(dist-compiler-rt compiler-rt) # Tarball with the whole sysroot From 91ce489944acb033806a3534553ed7dadf25e0c1 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Thu, 25 Jul 2024 03:14:01 +0900 Subject: [PATCH 082/165] Add full LTO build of wasi-libc and libc++ (#436) * Add LTO build of wasi-libc * Add LTO build of libc++ * Update wasi-libc (to include the LTO support) * Add a comment about /llvm-lto/${llvm_version} convention * Use separate targets for wasi-libc with and without lto A downside: this makes four more copies of wasi-libc source tree. * Make LTO build optional --- cmake/wasi-sdk-sysroot.cmake | 65 +++++++++++++++++++++++++++++------- src/wasi-libc | 2 +- 2 files changed, 54 insertions(+), 13 deletions(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 7e7345b0f..6bed7d37d 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -10,6 +10,7 @@ find_program(MAKE make REQUIRED) option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" ON) option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) option(WASI_SDK_INSTALL_TO_CLANG_RESOURCE_DIR "Whether or not to modify the compiler's resource directory" OFF) +option(WASI_SDK_LTO "Whether or not to build LTO assets" ON) set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) set(wasi_sysroot ${wasi_tmp_install}/share/wasi-sysroot) @@ -97,15 +98,27 @@ add_custom_target(compiler-rt DEPENDS compiler-rt-build compiler-rt-post-build) # wasi-libc build logic # ============================================================================= -function(define_wasi_libc target) - set(build_dir ${CMAKE_CURRENT_BINARY_DIR}/wasi-libc-${target}) +function(define_wasi_libc_sub target target_suffix lto) + set(build_dir ${CMAKE_CURRENT_BINARY_DIR}/wasi-libc-${target}${target_suffix}) if(${target} MATCHES threads) - set(extra_make_flags THREAD_MODEL=posix) + if(lto) + set(extra_make_flags LTO=full THREAD_MODEL=posix) + else() + set(extra_make_flags THREAD_MODEL=posix) + endif() elseif(${target} MATCHES p2) - set(extra_make_flags WASI_SNAPSHOT=p2 default libc_so) + if(lto) + set(extra_make_flags LTO=full WASI_SNAPSHOT=p2 default) + else() + set(extra_make_flags WASI_SNAPSHOT=p2 default libc_so) + endif() else() - set(extra_make_flags default libc_so) + if(lto) + set(extra_make_flags LTO=full default) + else() + set(extra_make_flags default libc_so) + endif() endif() string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPER) @@ -114,7 +127,7 @@ function(define_wasi_libc target) "${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") list(JOIN extra_cflags_list " " extra_cflags) - ExternalProject_Add(wasi-libc-${target} + ExternalProject_Add(wasi-libc-${target}${target_suffix}-build # Currently wasi-libc doesn't support out-of-tree builds so feigh a # "download command" which copies the source tree to a different location # so out-of-tree builds are supported. @@ -129,7 +142,7 @@ function(define_wasi_libc target) NM=${CMAKE_NM} SYSROOT=${wasi_sysroot} EXTRA_CFLAGS=${extra_cflags} - TARGET_TRIPLE=${target} + TARGET_TRIPLE=${target} ${extra_make_flags} INSTALL_COMMAND "" DEPENDS compiler-rt @@ -140,6 +153,16 @@ function(define_wasi_libc target) ) endfunction() +function(define_wasi_libc target) + define_wasi_libc_sub (${target} "" OFF) + if(WASI_SDK_LTO) + define_wasi_libc_sub (${target} "-lto" ON) + endif() + + add_custom_target(wasi-libc-${target} + DEPENDS wasi-libc-${target}-build $<$:wasi-libc-${target}-lto-build>) +endfunction() + foreach(target IN LISTS WASI_SDK_TARGETS) define_wasi_libc(${target}) endforeach() @@ -148,7 +171,12 @@ endforeach() # libcxx build logic # ============================================================================= -function(define_libcxx target) +execute_process( + COMMAND ${CMAKE_C_COMPILER} -dumpversion + OUTPUT_VARIABLE llvm_version + OUTPUT_STRIP_TRAILING_WHITESPACE) + +function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_suffix) if(${target} MATCHES threads) set(threads ON) set(pic OFF) @@ -158,6 +186,10 @@ function(define_libcxx target) set(pic ON) set(target_flags "") endif() + if(${target_suffix} MATCHES lto) + set(pic OFF) + endif() + list(APPEND target_flags ${extra_target_flags}) set(runtimes "libcxx;libcxxabi") @@ -176,7 +208,7 @@ function(define_libcxx target) set(extra_cxxflags_list ${CMAKE_CXX_FLAGS} ${extra_flags}) list(JOIN extra_cxxflags_list " " extra_cxxflags) - ExternalProject_Add(libcxx-${target}-build + ExternalProject_Add(libcxx-${target}${target_suffix}-build SOURCE_DIR ${llvm_proj_dir}/runtimes CMAKE_ARGS ${default_cmake_args} @@ -214,8 +246,8 @@ function(define_libcxx target) -DUNIX:BOOL=ON -DCMAKE_C_FLAGS=${extra_cflags} -DCMAKE_CXX_FLAGS=${extra_cxxflags} - -DLIBCXX_LIBDIR_SUFFIX=/${target} - -DLIBCXXABI_LIBDIR_SUFFIX=/${target} + -DLIBCXX_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} + -DLIBCXXABI_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} # See https://www.scivision.dev/cmake-externalproject-list-arguments/ for # why this is in `CMAKE_CACHE_ARGS` instead of above @@ -229,6 +261,15 @@ function(define_libcxx target) USES_TERMINAL_BUILD ON USES_TERMINAL_INSTALL ON ) +endfunction() + +function(define_libcxx target) + define_libcxx_sub(${target} "" "" "") + if(WASI_SDK_LTO) + # Note: clang knows this /llvm-lto/${llvm_version} convention. + # https://github.com/llvm/llvm-project/blob/llvmorg-18.1.8/clang/lib/Driver/ToolChains/WebAssembly.cpp#L204-L210 + define_libcxx_sub(${target} "-lto" "-flto=full" "/llvm-lto/${llvm_version}") + endif() # As of this writing, `clang++` will ignore the target-specific include dirs # unless this one also exists: @@ -236,7 +277,7 @@ function(define_libcxx target) COMMAND ${CMAKE_COMMAND} -E make_directory ${wasi_sysroot}/include/c++/v1 COMMENT "creating libcxx-specific header file folder") add_custom_target(libcxx-${target} - DEPENDS libcxx-${target}-build libcxx-${target}-extra-dir) + DEPENDS libcxx-${target}-build $<$:libcxx-${target}-lto-build> libcxx-${target}-extra-dir) endfunction() foreach(target IN LISTS WASI_SDK_TARGETS) diff --git a/src/wasi-libc b/src/wasi-libc index 3f43ea9ab..b9e15a8af 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 3f43ea9abb24ed8d24d760989e1d87ea385f8eaa +Subproject commit b9e15a8af930603183eb13af62e193de0f9f9ee3 From 084efa01da005ba29c9a1ceadbf76ef869ed0e01 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Thu, 25 Jul 2024 23:06:32 +0900 Subject: [PATCH 083/165] Update wasi-libc (#458) To include https://github.com/WebAssembly/wasi-libc/pull/517, which is necessary to avoid modifying host compiler's resource directory. At least on my environment (macOS if it matters), without the above mentioned fix, gmake tries to create the directory for wasi compiler-rt during wasi-libc build. I'm not sure why it didn't happen for others. cf. https://github.com/WebAssembly/wasi-sdk/pull/445 --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index b9e15a8af..b9ef79d7d 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit b9e15a8af930603183eb13af62e193de0f9f9ee3 +Subproject commit b9ef79d7dbd47c6c5bafdae760823467c2f60b70 From 882577c201bc7cebb2ecfc37519b073800c48013 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 30 Jul 2024 19:16:31 -0500 Subject: [PATCH 084/165] Clone a larger depth in submodules (#464) This is an attempt to fix CI errors where the `config` submodule is now located on a commit further than 32 commits behind the main branch (presumably). --- .github/workflows/main.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 31095320b..ead42c9b8 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -79,7 +79,7 @@ jobs: # bump depth (or even better, the submodule), in case of "error: # Server does not allow request for unadvertised object" in the # future. - - run: git submodule update --init --depth 32 --jobs 3 + - run: git submodule update --init --depth 64 --jobs 3 # Persist ccache-based caches across builds. This directory is configured # via the CCACHE_DIR env var below for ccache to use. @@ -265,7 +265,7 @@ jobs: fetch-depth: 0 - run: git fetch --tags --force name: Force-fetch tags to work around actions/checkout#290 - - run: git submodule update --init --depth 32 --jobs 3 + - run: git submodule update --init --depth 64 --jobs 3 - name: Setup `wasmtime` for tests uses: bytecodealliance/actions/wasmtime/setup@v1 with: From 0de1b487fc18636cbd59367b7cf39b0c7026ccf5 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 30 Jul 2024 19:22:23 -0500 Subject: [PATCH 085/165] Fix testing the right targets on CI (#461) This commit fixes a bug from the cmake migration where tests were not actually testing the correct target. Object files were compiled with the right options but the link step was missing both `--target` and `-pthread` which caused everything to accidentally be tested as `wasm32-wasi`. When fixing this one test was needed to have its stderr updated because the component output of `wasm32-wasip2` is slightly different. A timeout was additionally added because without `-pthread` at the link step some tests infinitely ran which made debugging difficult. --- ci/build.sh | 3 ++- tests/CMakeLists.txt | 2 ++ tests/general/sigabrt.c.stderr.expected | 2 +- tests/general/sigabrt.c.stderr.expected.filter | 1 + 4 files changed, 6 insertions(+), 2 deletions(-) diff --git a/ci/build.sh b/ci/build.sh index 09c92a35f..7525aa4d4 100755 --- a/ci/build.sh +++ b/ci/build.sh @@ -47,4 +47,5 @@ if [ "$WASI_SDK_CI_SKIP_TESTS" = "1" ]; then fi # Run tests to ensure that the sysroot works. -ctest --output-on-failure --parallel 10 --test-dir $build_dir/sysroot/tests +ctest --output-on-failure --parallel 10 --test-dir $build_dir/sysroot/tests \ + --timeout 60 diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index c5e7edd39..edbb1290d 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -48,6 +48,7 @@ function(add_testcase runwasi test) # as well. if(NOT(CMAKE_C_COMPILER MATCHES ${target})) target_compile_options(${target_name} PRIVATE --target=${target}) + target_link_options(${target_name} PRIVATE --target=${target}) endif() # Apply test-specific compile options and link flags. @@ -79,6 +80,7 @@ function(add_testcase runwasi test) # Apply target-specific options. if(target MATCHES threads) target_compile_options(${target_name} PRIVATE -pthread) + target_link_options(${target_name} PRIVATE -pthread) endif() if(runwasi) diff --git a/tests/general/sigabrt.c.stderr.expected b/tests/general/sigabrt.c.stderr.expected index 72c425d0c..d702a8897 100644 --- a/tests/general/sigabrt.c.stderr.expected +++ b/tests/general/sigabrt.c.stderr.expected @@ -3,4 +3,4 @@ Program received fatal signal: Aborted Error: failed to run main module `sigabrt.c.---.wasm` Caused by: - 0: failed to invoke command default + 0: failed to invoke --- diff --git a/tests/general/sigabrt.c.stderr.expected.filter b/tests/general/sigabrt.c.stderr.expected.filter index e6b54c2b2..726f0241e 100755 --- a/tests/general/sigabrt.c.stderr.expected.filter +++ b/tests/general/sigabrt.c.stderr.expected.filter @@ -4,4 +4,5 @@ set -euo pipefail cat \ | sed -e 's/main module `.*sigabrt\.c\.wasm`/main module `sigabrt.c.---.wasm`/' \ | sed -e 's/source location: @[[:xdigit:]]*$/source location: @----/' \ + | sed -e 's/failed to invoke.*/failed to invoke ---/' \ | head -n 6 From b0075f9a8234c94a24e67a8f339c653d25f89be9 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 31 Jul 2024 11:54:37 -0500 Subject: [PATCH 086/165] Add some CMake guards when building the sysroot (#462) * Require that the compiler is Clang, for example gcc and msvc cannot compile to WebAssembly. * Require that the Clang version is above the minimum threshold. Unsure what the minimum threshold is at this time so I've set it to 18.0.0 --- cmake/wasi-sdk-sysroot.cmake | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 6bed7d37d..394b5628c 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -5,6 +5,16 @@ if(NOT CMAKE_BUILD_TYPE) set(CMAKE_BUILD_TYPE RelWithDebInfo) endif() +if(NOT CMAKE_C_COMPILER_ID MATCHES Clang) + message(FATAL_ERROR "C compiler ${CMAKE_C_COMPILER} is not `Clang`, it is ${CMAKE_C_COMPILER_ID}") +endif() + +set(minimum_clang_required 18.0.0) + +if(CMAKE_C_COMPILER_VERSION VERSION_LESS ${minimum_clang_required}) + message(FATAL_ERROR "compiler version ${CMAKE_C_COMPILER_VERSION} is less than the required version ${minimum_clang_required}") +endif() + find_program(MAKE make REQUIRED) option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" ON) From 883170df212284da68bd7a3e6a73c311cb2da56c Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Wed, 31 Jul 2024 11:48:01 -0700 Subject: [PATCH 087/165] Add `wasi-sdk-p1.cmake` (#459) Like we have for other targets, this adds a `.cmake` file for the new name of the old `wasm32-wasi` target, now called `wasm32-wasip1` in `wasi-libc`. --- wasi-sdk-p1.cmake | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 wasi-sdk-p1.cmake diff --git a/wasi-sdk-p1.cmake b/wasi-sdk-p1.cmake new file mode 100644 index 000000000..903830132 --- /dev/null +++ b/wasi-sdk-p1.cmake @@ -0,0 +1,38 @@ +# Cmake toolchain description file for the Makefile + +# Until Platform/WASI.cmake is upstream we need to inject the path to it +# into CMAKE_MODULE_PATH. +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}") + +set(CMAKE_SYSTEM_NAME WASI) +set(CMAKE_SYSTEM_VERSION 1) +set(CMAKE_SYSTEM_PROCESSOR wasm32) +set(triple wasm32-wasip1) + +if(WIN32) + set(WASI_HOST_EXE_SUFFIX ".exe") +else() + set(WASI_HOST_EXE_SUFFIX "") +endif() + +# When building from source, WASI_SDK_PREFIX represents the generated directory +if(NOT WASI_SDK_PREFIX) + set(WASI_SDK_PREFIX ${CMAKE_CURRENT_LIST_DIR}/../../) +endif() + +set(CMAKE_C_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_CXX_COMPILER ${WASI_SDK_PREFIX}/bin/clang++${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_ASM_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_AR ${WASI_SDK_PREFIX}/bin/llvm-ar${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_RANLIB ${WASI_SDK_PREFIX}/bin/llvm-ranlib${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_C_COMPILER_TARGET ${triple}) +set(CMAKE_CXX_COMPILER_TARGET ${triple}) +set(CMAKE_ASM_COMPILER_TARGET ${triple}) + +# Don't look in the sysroot for executables to run during the build +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) + +# Only look in the sysroot (not in the host paths) for the rest +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) From b2679166a9b53d09cad618bd0984899042d0d4a6 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 31 Jul 2024 14:42:34 -0500 Subject: [PATCH 088/165] Run wasm tests on Windows (#466) This commit fixes running the test suite on Windows for CI at least by fixing up a few minor edge cases to ensure that everything runs consistently. --- .github/workflows/main.yml | 7 ------- tests/general/mmap.c.dir/.gitattributes | 3 +++ tests/testcase.sh | 8 ++++++++ 3 files changed, 11 insertions(+), 7 deletions(-) create mode 100644 tests/general/mmap.c.dir/.gitattributes diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index ead42c9b8..54a7c1bce 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -59,13 +59,6 @@ jobs: - artifact: x86_64-windows os: windows-latest - env: - # TODO: tests are pretty close to passing on Windows but need some - # final tweaks, namely testing the exit code doesn't work since - # exit codes are different on Windows and the `mmap.c` tests seems - # to have issues probably with line endings. Needs someone with a - # Windows checkout tot test further. - WASI_SDK_CI_SKIP_TESTS: 1 env: ${{ matrix.env || fromJSON('{}') }} steps: diff --git a/tests/general/mmap.c.dir/.gitattributes b/tests/general/mmap.c.dir/.gitattributes new file mode 100644 index 000000000..94a9658d0 --- /dev/null +++ b/tests/general/mmap.c.dir/.gitattributes @@ -0,0 +1,3 @@ +# This input is read at runtime during testing so ensure that the same input is +# read on unix and windows by forcing just-a-newline for line endings. +*.txt text eol=lf diff --git a/tests/testcase.sh b/tests/testcase.sh index fae7443fa..551eec32d 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -53,6 +53,14 @@ exit_status=0 || exit_status=$? echo $exit_status > "$exit_status_observed" +# On Windows Wasmtime will exit with error code 3 for aborts. On Unix Wasmtime +# will exit with status 134. Paper over this difference by pretending to be Unix +# on Windows and converting exit code 3 into 134 for the purposes of asserting +# test output. +if [ "$OSTYPE" = "msys" ] && [ "$exit_status" = "3" ]; then + echo 134 > "$exit_status_observed" +fi + # Determine the reference files to compare with. if [ -e "$input.stdout.expected" ]; then stdout_expected="$input.stdout.expected" From dc74ff1cf8ab4bda8a63f440fe23fd4c88a2b25b Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 31 Jul 2024 14:48:10 -0500 Subject: [PATCH 089/165] Add a test on CI that just the sysroot can be built (#465) This commit adds a test and matrix entry to CI which asserts that the sysroot can be built to run tests using a stock Clang compiler found on the system. This fixes a few minor issues as well in developing this. This additionally refactors CI a bit to move shared steps amongst jobs into separate composite actions in this repository to avoid duplication across jobs. --- .github/actions/checkout/action.yml | 16 +++++++ .github/actions/install-deps/action.yml | 22 +++++++++ .github/workflows/main.yml | 64 +++++++++++-------------- CMakeLists.txt | 2 +- cmake/wasi-sdk-sysroot.cmake | 4 ++ 5 files changed, 72 insertions(+), 36 deletions(-) create mode 100644 .github/actions/checkout/action.yml create mode 100644 .github/actions/install-deps/action.yml diff --git a/.github/actions/checkout/action.yml b/.github/actions/checkout/action.yml new file mode 100644 index 000000000..aca1b722a --- /dev/null +++ b/.github/actions/checkout/action.yml @@ -0,0 +1,16 @@ +name: 'Prepare wasi-sdk git directory' +description: 'Prepare wasi-sdk git directory' + +runs: + using: composite + steps: + - run: git fetch --tags --force + name: Force-fetch tags to work around actions/checkout#290 + shell: bash + # We can't use `--depth 1` here sadly because the GNU config + # submodule is not pinned to a particular tag/branch. Please + # bump depth (or even better, the submodule), in case of "error: + # Server does not allow request for unadvertised object" in the + # future. + - run: git submodule update --init --depth 64 --jobs 3 + shell: bash diff --git a/.github/actions/install-deps/action.yml b/.github/actions/install-deps/action.yml new file mode 100644 index 000000000..2a75b4118 --- /dev/null +++ b/.github/actions/install-deps/action.yml @@ -0,0 +1,22 @@ +name: 'Install wasi-sdk dependencies' +description: 'Install wasi-sdk dependencies' + +runs: + using: composite + steps: + - name: Setup `wasmtime` for tests + uses: bytecodealliance/actions/wasmtime/setup@v1 + with: + version: "18.0.2" + - name: Install ccache, ninja (macOS) + run: brew install ccache ninja + if: runner.os == 'macOS' + shell: bash + - name: Install ccache, ninja (Windows) + run: choco install ccache ninja + if: runner.os == 'Windows' + shell: bash + - name: Install ccache, ninja (Linux) + run: sudo apt-get install -y ccache ninja-build + if: runner.os == 'Linux' + shell: bash diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 54a7c1bce..a10759f88 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -65,14 +65,8 @@ jobs: - uses: actions/checkout@v4 with: fetch-depth: 0 - - run: git fetch --tags --force - name: Force-fetch tags to work around actions/checkout#290 - # We can't use `--depth 1` here sadly because the GNU config - # submodule is not pinned to a particular tag/branch. Please - # bump depth (or even better, the submodule), in case of "error: - # Server does not allow request for unadvertised object" in the - # future. - - run: git submodule update --init --depth 64 --jobs 3 + - uses: ./.github/actions/checkout + - uses: ./.github/actions/install-deps # Persist ccache-based caches across builds. This directory is configured # via the CCACHE_DIR env var below for ccache to use. @@ -103,21 +97,6 @@ jobs: echo WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS="$cmake_args" >> $GITHUB_ENV shell: bash - # Add some extra installed software on each runner as necessary. - - name: Setup `wasmtime` for tests - uses: bytecodealliance/actions/wasmtime/setup@v1 - with: - version: "18.0.2" - - name: Install ccache, ninja (macOS) - run: brew install ccache ninja - if: runner.os == 'macOS' - - name: Install ccache, ninja (Windows) - run: choco install ccache ninja - if: runner.os == 'Windows' - - name: Install ccache, ninja (Linux) - run: sudo apt install ccache - if: runner.os == 'Linux' - - name: Clear ccache statistics run: ccache --zero-stats @@ -184,6 +163,30 @@ jobs: path: ${{ runner.tool_cache }}/ccache key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }} + build-only-sysroot: + name: Build only sysroot + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - uses: ./.github/actions/checkout + - uses: ./.github/actions/install-deps + - run: | + wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - + name=$(lsb_release -s -c) + sudo add-apt-repository -y "deb http://apt.llvm.org/$name/ llvm-toolchain-$name-18 main" + sudo add-apt-repository -y "deb-src http://apt.llvm.org/$name/ llvm-toolchain-$name-18 main" + sudo apt-get install -y clang-18 llvm-18 lld-18 + - run: cargo install wasm-component-ld@0.5.5 + - run: | + cmake -G Ninja -B build -S . \ + -DCMAKE_C_COMPILER=/usr/lib/llvm-18/bin/clang \ + -DCMAKE_SYSTEM_NAME=WASI \ + -DWASI_SDK_INCLUDE_TESTS=ON + - run: ninja -C build + - run: ctest --output-on-failure --parallel 10 --test-dir build/tests + # Once all of the above matrix entries have completed this job will run and # assemble the final `wasi-sdk-*` artifacts by fusing the toolchain/sysroot # artifacts. @@ -195,8 +198,7 @@ jobs: - uses: actions/checkout@v4 with: fetch-depth: 0 - - run: git fetch --tags --force - name: Force-fetch tags to work around actions/checkout#290 + - uses: ./.github/actions/checkout # Download all artifacts from all platforms in `build`, merge them into # final wasi-sdk-* artifacts, and then upload them. @@ -256,16 +258,8 @@ jobs: - uses: actions/checkout@v4 with: fetch-depth: 0 - - run: git fetch --tags --force - name: Force-fetch tags to work around actions/checkout#290 - - run: git submodule update --init --depth 64 --jobs 3 - - name: Setup `wasmtime` for tests - uses: bytecodealliance/actions/wasmtime/setup@v1 - with: - version: "18.0.2" - - name: Install ninja - run: sudo apt-get install -y ninja-build - if: runner.os == 'Linux' + - uses: ./.github/actions/checkout + - uses: ./.github/actions/install-deps - uses: actions/download-artifact@v4 with: name: dist-x86_64-linux diff --git a/CMakeLists.txt b/CMakeLists.txt index b39f21d0b..a0d3fca77 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -7,6 +7,7 @@ # the host. cmake_minimum_required(VERSION 3.26) +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake") project(wasi-sdk) include(ExternalProject) @@ -17,7 +18,6 @@ option(WASI_SDK_BUILD_TOOLCHAIN "Build a toolchain instead of the sysroot" OFF) set(llvm_proj_dir ${CMAKE_CURRENT_SOURCE_DIR}/src/llvm-project) set(wasi_libc ${CMAKE_CURRENT_SOURCE_DIR}/src/wasi-libc) -list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake") include(wasi-sdk-enable-ccache) find_program(PYTHON python3 python REQUIRED) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 394b5628c..f51a84574 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -15,6 +15,9 @@ if(CMAKE_C_COMPILER_VERSION VERSION_LESS ${minimum_clang_required}) message(FATAL_ERROR "compiler version ${CMAKE_C_COMPILER_VERSION} is less than the required version ${minimum_clang_required}") endif() +message(STATUS "Found executable for `nm`: ${CMAKE_NM}") +message(STATUS "Found executable for `ar`: ${CMAKE_AR}") + find_program(MAKE make REQUIRED) option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" ON) @@ -133,6 +136,7 @@ function(define_wasi_libc_sub target target_suffix lto) string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPER) get_property(directory_cflags DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) + list(APPEND directory_cflags -resource-dir ${wasi_resource_dir}) set(extra_cflags_list "${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") list(JOIN extra_cflags_list " " extra_cflags) From d2bea01edcc46f731156a817f710cdd9fc9c1c19 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 1 Aug 2024 16:26:00 -0500 Subject: [PATCH 090/165] Update wasm-component-ld to 0.5.6 (#467) This pulls in an update to how WIT files are processed notably around nested package forms in the text format. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index d02edbe16..619d607b9 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.5) +set(wasm_component_ld_version 0.5.6) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From fd24b79f8bb0e4f73edfc18cc26789dbef577efb Mon Sep 17 00:00:00 2001 From: SingleAccretion <62474226+SingleAccretion@users.noreply.github.com> Date: Wed, 7 Aug 2024 19:35:09 +0300 Subject: [PATCH 091/165] Add wasi-sdk-p1.cmake to the artifacts (#472) --- cmake/wasi-sdk-toolchain.cmake | 1 + 1 file changed, 1 insertion(+) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 619d607b9..dcab07347 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -154,6 +154,7 @@ copy_misc_file(src/config/config.sub misc) copy_misc_file(src/config/config.guess misc) copy_misc_file(wasi-sdk.cmake cmake) copy_misc_file(wasi-sdk-pthread.cmake cmake) +copy_misc_file(wasi-sdk-p1.cmake cmake) copy_misc_file(wasi-sdk-p2.cmake cmake) copy_misc_file(cmake/Platform/WASI.cmake cmake/Platform) From b416e2beb40beecc05de33e300203843f62b3263 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 12 Aug 2024 12:53:36 -0500 Subject: [PATCH 092/165] Add back `VERSION` file to release tarballs (#473) * Add back `VERSION` file to release tarballs Make sure it makes its way into the sysroot tarball and then copy it from the sysroot location to the top of the sdk tarball as well. Closes #471 * Fix how `VERSION` is installed --- ci/merge-artifacts.sh | 1 + cmake/wasi-sdk-sysroot.cmake | 6 ++---- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/ci/merge-artifacts.sh b/ci/merge-artifacts.sh index e55db2d92..4a69e240f 100755 --- a/ci/merge-artifacts.sh +++ b/ci/merge-artifacts.sh @@ -60,6 +60,7 @@ for build in dist-*; do tar xf $toolchain -C dist/$sdk_dir --strip-components 1 mkdir -p dist/$sdk_dir/share/wasi-sysroot tar xf $sysroot -C dist/$sdk_dir/share/wasi-sysroot --strip-components 1 + mv dist/$sdk_dir/share/wasi-sysroot/VERSION dist/$sdk_dir # Setup the compiler-rt library for wasi,wasip1,wasip2 rtlibdir=$(dirname $(find dist/$sdk_dir/lib -name include))/lib diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index f51a84574..d0c4287b9 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -329,12 +329,10 @@ execute_process( COMMAND ${PYTHON} ${version_script} dump WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} OUTPUT_VARIABLE version_dump) -set(version_file_tmp ${CMAKE_CURRENT_BINARY_DIR}/VERSION) +set(version_file_tmp ${wasi_sysroot}/VERSION) file(GENERATE OUTPUT ${version_file_tmp} CONTENT ${version_dump}) add_custom_target(version-file DEPENDS ${version_file_tmp}) add_dependencies(build version-file) -install(FILES ${version_file_tmp} - DESTINATION ${CMAKE_INSTALL_PREFIX}) if(WASI_SDK_INCLUDE_TESTS) add_subdirectory(tests) @@ -353,7 +351,7 @@ add_dependencies(dist-compiler-rt compiler-rt) # Tarball with the whole sysroot wasi_sdk_add_tarball(dist-sysroot ${dist_dir}/wasi-sysroot-${wasi_sdk_version}.tar.gz - ${wasi_tmp_install}/share/wasi-sysroot) + ${wasi_sysroot}) add_dependencies(dist-sysroot build) add_custom_target(dist DEPENDS dist-compiler-rt dist-sysroot) From 91bca73047adde2cf77c57662339dfeb22ec4095 Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Fri, 23 Aug 2024 23:37:40 +0900 Subject: [PATCH 093/165] README.md: mention WASI_SDK_INSTALL_TO_CLANG_RESOURCE_DIR (#481) --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index a594f40f7..3b506ff1a 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,9 @@ in compiling WebAssembly code. Supported CMake flags are: sysroot libraries, don't build or use fresh libraries for tests. * `-DWASI_SDK_TARGETS=..` - a list of targets to build, by default all WASI targets are compiled. +* `-DWASI_SDK_INSTALL_TO_CLANG_RESOURCE_DIR=ON` - install compiler-rt + to the compiler's resource directory. might be convenient if you want to + use the toolchain (eg. `./build/install/bin/clang`) in-place. If you'd like to build distribution artifacts you can use the `dist` target like so: From f995dbe965fc5bb3e97e84e1995e26aae99bef55 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 27 Aug 2024 18:29:06 -0500 Subject: [PATCH 094/165] Disable tests/benchmarks for libcxx (#476) These aren't needed anyway and fix a build issue discovered in #474. Additionally a typo was fixed here too. --- cmake/wasi-sdk-sysroot.cmake | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index d0c4287b9..127ca62d3 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -142,7 +142,7 @@ function(define_wasi_libc_sub target target_suffix lto) list(JOIN extra_cflags_list " " extra_cflags) ExternalProject_Add(wasi-libc-${target}${target_suffix}-build - # Currently wasi-libc doesn't support out-of-tree builds so feigh a + # Currently wasi-libc doesn't support out-of-tree builds so feign a # "download command" which copies the source tree to a different location # so out-of-tree builds are supported. DOWNLOAD_COMMAND @@ -262,6 +262,8 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DCMAKE_CXX_FLAGS=${extra_cxxflags} -DLIBCXX_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} -DLIBCXXABI_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} + -DLIBCXX_INCLUDE_TESTS=OFF + -DLIBCXX_INCLUDE_BENCHMARKS=OFF # See https://www.scivision.dev/cmake-externalproject-list-arguments/ for # why this is in `CMAKE_CACHE_ARGS` instead of above From c1b8811ac8b4fbbd510781b59efd10533dfdf75a Mon Sep 17 00:00:00 2001 From: Roman Kolesnikov Date: Wed, 28 Aug 2024 06:01:26 +0200 Subject: [PATCH 095/165] Add llvm-dwp binary to toolchain (#478) There is an option to compile debug information to separate file. It's supported in Clang via -gsplit-dwarf to generate dwo files but we need llvm-dwp tool to combine them into single dwp file supported by Chrome DWARF extension https://developer.chrome.com/blog/faster-wasm-debugging --- cmake/wasi-sdk-toolchain.cmake | 1 + 1 file changed, 1 insertion(+) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index dcab07347..fb4044c97 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -45,6 +45,7 @@ set(tools llvm-ranlib llvm-strip llvm-dwarfdump + llvm-dwp clang-resource-headers ar ranlib From 754aec3d6f5801b026fe5dfecc9c63e6a2e5968b Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 29 Aug 2024 19:10:46 -0500 Subject: [PATCH 096/165] Bump wasm-component-ld to 0.5.7 (#484) Keeping it up-to-date with its release cadence. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index fb4044c97..6c2016e82 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -120,7 +120,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.6) +set(wasm_component_ld_version 0.5.7) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From adbbf2c07fc339588d73ab483f14a4578137e50a Mon Sep 17 00:00:00 2001 From: Andrii Rublov Date: Wed, 18 Sep 2024 20:23:31 +0200 Subject: [PATCH 097/165] Fix installation and usage instructions in README.md (#488) * Add missing architecture URL part to the installation script * Remove unnecessary `export`s from installation instructions * Fix missing $ sign in export * Add `WASI_OS` as new parameter to use & install scripts in README --------- Co-authored-by: Andrew Brown --- README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 3b506ff1a..22138e5b6 100644 --- a/README.md +++ b/README.md @@ -132,10 +132,12 @@ see [RELEASING.md](RELEASING.md). A typical installation from the release binaries might look like the following: ```shell script -export WASI_VERSION=20 -export WASI_VERSION_FULL=${WASI_VERSION}.0 -wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz -tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz +WASI_OS=linux +WASI_ARCH=x86_64 +WASI_VERSION=24 +WASI_VERSION_FULL=${WASI_VERSION}.0 +wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz +tar xvf wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz ``` ## Use @@ -143,7 +145,7 @@ tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz Use the clang installed in the `wasi-sdk` directory: ```shell script -export WASI_SDK_PATH=`pwd`/wasi-sdk-${WASI_VERSION_FULL} +WASI_SDK_PATH=`pwd`/wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS} CC="${WASI_SDK_PATH}/bin/clang --sysroot=${WASI_SDK_PATH}/share/wasi-sysroot" $CC foo.c -o foo.wasm ``` From 54545d51b3eaeafccab7ab06a349e0b28f171e2a Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Wed, 25 Sep 2024 06:27:46 +0900 Subject: [PATCH 098/165] LLVM 19 + setjmp/longjmp doc (#480) * llvmorg-19.1.0-rc3 * document setjmp/longjmp support * README.md: mention SetjmpLongjmp.md * bump wasi-libc For following changes: * https://github.com/WebAssembly/wasi-libc/pull/526 * https://github.com/WebAssembly/wasi-libc/pull/529 * bump llvm to llvmorg-19.1.0-rc4 * bump llvm to llvmorg-19.1.0 --- README.md | 10 +++++-- SetjmpLongjmp.md | 75 ++++++++++++++++++++++++++++++++++++++++++++++++ src/llvm-project | 2 +- src/wasi-libc | 2 +- 4 files changed, 84 insertions(+), 5 deletions(-) create mode 100644 SetjmpLongjmp.md diff --git a/README.md b/README.md index 22138e5b6..c3c0dfa58 100644 --- a/README.md +++ b/README.md @@ -203,12 +203,16 @@ disabled in a configure step before building with WASI SDK. ## Notable Limitations This repository does not yet support __C++ exceptions__. C++ code is supported -only with -fno-exceptions for now. Similarly, there is not yet support for -setjmp/longjmp. Work on support for [exception handling] is underway at the -language level which will support both of these features. +only with -fno-exceptions for now. +Work on support for [exception handling] is underway at the +language level which will support the features. [exception handling]: https://github.com/WebAssembly/exception-handling/ +See [C setjmp/longjmp support] about setjmp/longjmp support. + +[C setjmp/longjmp support]: SetjmpLongjmp.md + This repository experimentally supports __threads__ with `--target=wasm32-wasi-threads`. It uses WebAssembly's [threads] primitives (atomics, `wait`/`notify`, shared memory) and [wasi-threads] for spawning diff --git a/SetjmpLongjmp.md b/SetjmpLongjmp.md new file mode 100644 index 000000000..df8c4af13 --- /dev/null +++ b/SetjmpLongjmp.md @@ -0,0 +1,75 @@ +# C setjmp/longjmp support + +## Build an application + +To build an application using setjmp/longjmp, you need two things: + +* Enable the necessary LLVM translation (`-mllvm -wasm-enable-sjlj`) + +* Link the setjmp library (`-lsetjmp`) + +### Example without LTO + +```shell +clang -Os -mllvm -wasm-enable-sjlj -o your_app.wasm your_app.c -lsetjmp +``` + +### Example with LTO + +```shell +clang -Os -flto=full -mllvm -wasm-enable-sjlj -Wl,-mllvm,-wasm-enable-sjlj -o your_app.wasm your_app.c -lsetjmp +``` + +## Run an application + +To run the application built as in the previous section, +you need to use a runtime with [exception handling proposal] support. + +Unfortunately, there are two incompatible versions of +[exception handling proposal], which is commonly implemented by runtimes. + +* The latest version with `exnref` + +* The [phase3] version + +### Example with the latest exception handling proposal + +Because the current version of WASI-SDK produces an old version +of [exception handling proposal] instructions, if your runtime +implements the latest version of the proposal, you need to convert +your module to the latest version. + +[toywasm] is an example of such runtimes. + +You can use binaryen `wasm-opt` command for the conversion. + +```shell +wasm-opt --translate-to-exnref -all -o your_app.exnref.wasm your_app.wasm +``` + +Then you can run it with a runtime supporting the latest version of +[exception handling proposal]. + +```shell +toywasm --wasi your_app.exnref.wasm +``` +(You may need to enable the support with `-D TOYWASM_ENABLE_WASM_EXCEPTION_HANDLING=ON`.) + +### Example with the phase3 exception handling proposal (a bit older version) + +If your runtime supports the [phase3] version of +[exception handling proposal], which is the same version as what WASI-SDK +currently produces, you can run the produced module as it is. + +For example, the classic interpreter of [wasm-micro-runtime] is +one of such runtimes. + +```shell +iwasm your_app.wasm +``` +(You may need to enable the support with `-D WAMR_BUILD_EXCE_HANDLING=1 -D WAMR_BUILD_FAST_INTERP=0`.) + +[exception handling proposal]: https://github.com/WebAssembly/exception-handling/ +[phase3]: https://github.com/WebAssembly/exception-handling/tree/main/proposals/exception-handling/legacy +[toywasm]: https://github.com/yamt/toywasm +[wasm-micro-runtime]: https://github.com/bytecodealliance/wasm-micro-runtime diff --git a/src/llvm-project b/src/llvm-project index 26a1d6601..a4bf6cd7c 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 26a1d6601d727a96f4301d0d8647b5a42760ae0c +Subproject commit a4bf6cd7cfb1a1421ba92bca9d017b49936c55e4 diff --git a/src/wasi-libc b/src/wasi-libc index b9ef79d7d..1b19fc65a 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit b9ef79d7dbd47c6c5bafdae760823467c2f60b70 +Subproject commit 1b19fc65ad84b223876c50dd4fcd7d5a08c311dc From cec5cf4f6cf37deefa790ac435e4c1edd318e12e Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 1 Oct 2024 11:27:34 -0500 Subject: [PATCH 099/165] Update wasm-component-ld to 0.5.9 (#491) Pulling in some bug fixes and minor updates for the tool. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 6c2016e82..d929cd741 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -120,7 +120,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.7) +set(wasm_component_ld_version 0.5.9) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 5e04cd81eb749edb5642537d150ab1ab7aedabe9 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 11 Oct 2024 10:15:53 -0500 Subject: [PATCH 100/165] Update wasm-component-ld to 0.5.10 (#493) Pulling in a fix for bytecodealliance/wasm-tools#1850. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index d929cd741..8756cee15 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -120,7 +120,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.9) +set(wasm_component_ld_version 0.5.10) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 3866d2943f2c39a3626a9d07d41b14a7e8991ddf Mon Sep 17 00:00:00 2001 From: Yuta Saito Date: Wed, 30 Oct 2024 01:48:50 +0900 Subject: [PATCH 101/165] Update to the latest wasi-libc (2nd try) (#496) * Update to the latest wasi-libc. This contains two significant changes: - Implement a stub pthreads library for `THREAD_MODEL=single` (#518) - Initial FTS support (#522) * Update to the latest wasi-libc with build fix This update includes a build fix for shared library builds. --------- Co-authored-by: Dan Gohman --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index 1b19fc65a..98897e29f 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 1b19fc65ad84b223876c50dd4fcd7d5a08c311dc +Subproject commit 98897e29fcfc81e2b12e487e4154ac99188330c4 From 073618692588cf56494f486acb3a3c82e3a6e87c Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 3 Dec 2024 13:58:53 -0700 Subject: [PATCH 102/165] Update wasm-component-ld to 0.5.11 (#501) --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 8756cee15..4c45abf9a 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -120,7 +120,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.10) +set(wasm_component_ld_version 0.5.11) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From ccdf52e17eec09577e7e25acef96c4d630bfb8d7 Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Wed, 11 Dec 2024 16:11:32 -0800 Subject: [PATCH 103/165] Update to the latest LLVM 19 and wasi-libc. (#502) Update to LLVM 19.1.5, and wasi-libc 574b88da. This notably picks up: - [WebAssembly] Fix rethrow's index calculation (llvm/llvm-project#114693) - [WebAssembly] Fix feature coalescing (llvm/llvm-project#110647) --- src/llvm-project | 2 +- src/wasi-libc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/llvm-project b/src/llvm-project index a4bf6cd7c..ab4b5a2db 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit a4bf6cd7cfb1a1421ba92bca9d017b49936c55e4 +Subproject commit ab4b5a2db582958af1ee308a790cfdb42bd24720 diff --git a/src/wasi-libc b/src/wasi-libc index 98897e29f..574b88da4 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 98897e29fcfc81e2b12e487e4154ac99188330c4 +Subproject commit 574b88da481569b65a237cb80daf9a2d5aeaf82d From a4d918fa119c9beb712bf08bbb8fa9996aab0a71 Mon Sep 17 00:00:00 2001 From: Carlo Cabrera Date: Fri, 13 Dec 2024 03:42:56 +0800 Subject: [PATCH 104/165] Replace `DEFAULT_SYSROOT` usage with Clang config files (#503) Upstream want to deprecate `DEFAULT_SYSROOT`[^1][^2][^3], but one blocker is wasi-sdk's usage of it. Let's try to help that along by switching to using config files[^4] instead. This should result in no user-facing changes in functionality. (If it does, then that's an LLVM bug that should be fixed there.) [^1]: https://reviews.llvm.org/D158218 [^2]: https://github.com/llvm/llvm-project/issues/94284 [^3]: https://github.com/llvm/llvm-project/pull/77537 [^4]: https://clang.llvm.org/docs/UsersManual.html#configuration-files --- clang.cfg | 1 + cmake/wasi-sdk-toolchain.cmake | 13 ++++++++++++- 2 files changed, 13 insertions(+), 1 deletion(-) create mode 100644 clang.cfg diff --git a/clang.cfg b/clang.cfg new file mode 100644 index 000000000..38ddf5439 --- /dev/null +++ b/clang.cfg @@ -0,0 +1 @@ +--sysroot=/../share/wasi-sysroot diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 4c45abf9a..cba9c63ee 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -90,7 +90,6 @@ ExternalProject_Add(llvm-build -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi -DLLVM_INSTALL_BINUTILS_SYMLINKS=TRUE -DLLVM_ENABLE_LIBXML2=OFF - -DDEFAULT_SYSROOT=../share/wasi-sysroot # Pass `-s` to strip symbols by default and shrink the size of the # distribution -DCMAKE_EXE_LINKER_FLAGS=-s @@ -159,6 +158,18 @@ copy_misc_file(wasi-sdk-p1.cmake cmake) copy_misc_file(wasi-sdk-p2.cmake cmake) copy_misc_file(cmake/Platform/WASI.cmake cmake/Platform) +function(copy_cfg_file compiler) + set(dst ${wasi_tmp_install}/bin/${compiler}.cfg) + add_custom_command( + OUTPUT ${dst} + COMMAND cmake -E copy ${CMAKE_CURRENT_SOURCE_DIR}/clang.cfg ${dst}) + add_custom_target(copy-${compiler} DEPENDS ${dst}) + add_dependencies(misc-files copy-${compiler}) +endfunction() + +copy_cfg_file(clang) +copy_cfg_file(clang++) + include(wasi-sdk-dist) # Figure out the name of the artifact which is either explicitly specified or From 67283cc6e91c851e6a418a4b274e608e84a07131 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 22 Jan 2025 14:43:40 -0600 Subject: [PATCH 105/165] Update wasm-component-ld (#506) * Update wasm-component-ld Pulls in a fix for bytecodealliance/wasmtime#10058 in the adapters that are used by default. * Update wasmtime installed in CI * Downgrade the build-only-sysroot check step Looks like this is failing on Ubuntu 24.04, the new default of `ubuntu-latest`, so downgrade it to have it get fixed in a separate PR. * Try downgrading Wasmtime version again * Update base Linux images to Ubuntu 20.04 * Update Wasmtime back to 29 --- .github/actions/install-deps/action.yml | 2 +- .github/workflows/main.yml | 2 +- ci/docker/Dockerfile.common | 4 ++-- cmake/wasi-sdk-toolchain.cmake | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/actions/install-deps/action.yml b/.github/actions/install-deps/action.yml index 2a75b4118..8b81d5677 100644 --- a/.github/actions/install-deps/action.yml +++ b/.github/actions/install-deps/action.yml @@ -7,7 +7,7 @@ runs: - name: Setup `wasmtime` for tests uses: bytecodealliance/actions/wasmtime/setup@v1 with: - version: "18.0.2" + version: "29.0.1" - name: Install ccache, ninja (macOS) run: brew install ccache ninja if: runner.os == 'macOS' diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index a10759f88..520856aae 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -165,7 +165,7 @@ jobs: build-only-sysroot: name: Build only sysroot - runs-on: ubuntu-latest + runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v4 with: diff --git a/ci/docker/Dockerfile.common b/ci/docker/Dockerfile.common index 480e9b77d..96013b745 100644 --- a/ci/docker/Dockerfile.common +++ b/ci/docker/Dockerfile.common @@ -1,8 +1,8 @@ # Use a relatively old/stable distro here to maximize the supported platforms # and avoid depending on more recent version of, say, libc. -# Here we choose Bionic 18.04. +# Here we choose Ubuntu 20.04. -FROM ubuntu:18.04 +FROM ubuntu:20.04 # Various build tooling and such necessary to build LLVM and a wasi-sysroot RUN apt-get update \ diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index cba9c63ee..646e113b9 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.11) +set(wasm_component_ld_version 0.5.12) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From e6a833783ae7d864e89f06c41ee300680ad46719 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 27 Jan 2025 09:30:56 -0600 Subject: [PATCH 106/165] Pin CI runner images to explicit versions (#507) * Pin CI runner images to explicit versions CI is run infrequently enough on this repository that there's a nontrivial chance that GitHub updates the definition of `ubuntu-latest` for example between two CI runs. This can be confusing for contributors as PRs seemingly break CI when in reality they have nothing to do with the breakage and it's due to the image changing. This PR pins all images to the definition of `*-latest` at this time. The one exception is that one builder was pinned to 22.04 and I'm going to update it to 24.04 here and see if I can't fix CI issues that come up. This'll require explicit PRs to update these images in the future, but hopefully that's only once every few ~years so not too much of a burden. * Try not custom-installing clang 18 * Try to fix CI issue * Fix CI again (attempt) * Further try to fix CI * Another fix for CI Surely if I keep making commits that say "another fix" eventually I'll run out of fixes. Surely, right? I'd put this in limerick form to be more amusing if someone comes to read this but I'm not clever enough to do that, so instead I'll just abruptly --- .github/workflows/main.yml | 28 ++++++++++++---------------- cmake/wasi-sdk-sysroot.cmake | 8 +++++++- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 520856aae..b1e3c4d3b 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -19,10 +19,10 @@ jobs: matrix: include: - artifact: x86_64-linux - os: ubuntu-latest + os: ubuntu-24.04 - artifact: arm64-linux - os: ubuntu-latest + os: ubuntu-24.04 rust_target: aarch64-unknown-linux-gnu env: # Don't build a sysroot for this cross-compiled target since it @@ -41,7 +41,7 @@ jobs: -DRUST_TARGET=aarch64-unknown-linux-gnu - artifact: arm64-macos - os: macos-latest + os: macos-14 rust_target: aarch64-apple-darwin env: WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS: >- @@ -49,7 +49,7 @@ jobs: -DCMAKE_OSX_ARCHITECTURES=arm64 - artifact: x86_64-macos - os: macos-latest + os: macos-14 rust_target: x86_64-apple-darwin env: WASI_SDK_CI_SKIP_SYSROOT: 1 @@ -58,7 +58,7 @@ jobs: -DCMAKE_OSX_ARCHITECTURES=x86_64 - artifact: x86_64-windows - os: windows-latest + os: windows-2022 env: ${{ matrix.env || fromJSON('{}') }} steps: @@ -165,25 +165,21 @@ jobs: build-only-sysroot: name: Build only sysroot - runs-on: ubuntu-22.04 + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: ./.github/actions/checkout - uses: ./.github/actions/install-deps - - run: | - wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - - name=$(lsb_release -s -c) - sudo add-apt-repository -y "deb http://apt.llvm.org/$name/ llvm-toolchain-$name-18 main" - sudo add-apt-repository -y "deb-src http://apt.llvm.org/$name/ llvm-toolchain-$name-18 main" - sudo apt-get install -y clang-18 llvm-18 lld-18 - - run: cargo install wasm-component-ld@0.5.5 + - run: cargo install wasm-component-ld@0.5.12 - run: | cmake -G Ninja -B build -S . \ -DCMAKE_C_COMPILER=/usr/lib/llvm-18/bin/clang \ -DCMAKE_SYSTEM_NAME=WASI \ - -DWASI_SDK_INCLUDE_TESTS=ON + -DWASI_SDK_INCLUDE_TESTS=ON \ + -DCMAKE_C_LINKER_DEPFILE_SUPPORTED=OFF \ + -DCMAKE_CXX_LINKER_DEPFILE_SUPPORTED=OFF - run: ninja -C build - run: ctest --output-on-failure --parallel 10 --test-dir build/tests @@ -193,7 +189,7 @@ jobs: finalize: name: Finalize wasi-sdk artifacts needs: build - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 with: @@ -253,7 +249,7 @@ jobs: test-standalone: name: Test standalone toolchain needs: build - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 with: diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 127ca62d3..0dd475fff 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -47,7 +47,13 @@ set(default_cmake_args -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_SYSROOT=${wasi_sysroot} - -DCMAKE_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}/cmake) + -DCMAKE_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}/cmake + # CMake detects this based on `CMAKE_C_COMPILER` alone and when that compiler + # is just a bare "clang" installation then it can mistakenly deduce that this + # feature is supported when it's not actually supported for WASI targets. + # Currently `wasm-ld` does not support the linker flag for this. + -DCMAKE_C_LINKER_DEPFILE_SUPPORTED=OFF + -DCMAKE_CXX_LINKER_DEPFILE_SUPPORTED=OFF) if(CMAKE_C_COMPILER_LAUNCHER) list(APPEND default_cmake_args -DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}) From 62c3d46dd735e271f2f5f351c149ab1478dbe183 Mon Sep 17 00:00:00 2001 From: mcbarton Date: Mon, 3 Feb 2025 15:06:33 +0000 Subject: [PATCH 107/165] Update Dockerfile to Ubuntu 24.04 (#512) --- docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index d3b87f239..901aeb576 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -6,7 +6,7 @@ # pre-built versions of this container are pushed as a package to the repository # as well. -FROM ubuntu:22.04 +FROM ubuntu:24.04 RUN apt-get update && \ apt-get install -y cmake ninja-build make autoconf autogen automake libtool && \ From 30a64fd513378293dc004728fd6e0e45929a4969 Mon Sep 17 00:00:00 2001 From: mcbarton Date: Tue, 4 Feb 2025 16:19:37 +0000 Subject: [PATCH 108/165] [ci] Cancel outstanding workflows when new changes are added to PRs (#513) --- .github/workflows/main.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index b1e3c4d3b..28439f9b3 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -10,6 +10,10 @@ on: pull_request: workflow_dispatch: +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + cancel-in-progress: true + jobs: build: name: Build ${{ matrix.artifact }} From 10214e10aff283531ee2c5e1fbc951dd357b9e33 Mon Sep 17 00:00:00 2001 From: mcbarton Date: Sat, 8 Feb 2025 01:06:54 +0000 Subject: [PATCH 109/165] [ci] Use native arm runner instead of cross compiling (#509) With Ubuntu arm Github runners now available for general availability (see Github blogpost here https://github.blog/changelog/2025-01-16-linux-arm64-hosted-runners-now-available-for-free-in-public-repositories-public-preview/ ) there is no longer a need to use the Ubuntu x86 runner and cross compile in the ci. This means you no longer need to skip building the sysroot on the ci for this platform. --- .github/workflows/main.yml | 18 +----------------- ci/docker-build.sh | 16 +++------------- ci/docker/{Dockerfile.common => Dockerfile} | 15 +++++++++------ ci/docker/Dockerfile.arm64-linux | 11 ----------- ci/docker/Dockerfile.x86_64-linux | 7 ------- 5 files changed, 13 insertions(+), 54 deletions(-) rename ci/docker/{Dockerfile.common => Dockerfile} (64%) delete mode 100644 ci/docker/Dockerfile.arm64-linux delete mode 100644 ci/docker/Dockerfile.x86_64-linux diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 28439f9b3..e30e11bfe 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -26,23 +26,7 @@ jobs: os: ubuntu-24.04 - artifact: arm64-linux - os: ubuntu-24.04 - rust_target: aarch64-unknown-linux-gnu - env: - # Don't build a sysroot for this cross-compiled target since it - # would require a host compiler and the sysroot is otherwise - # already built on other CI builders. - WASI_SDK_CI_SKIP_SYSROOT: 1 - - WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS: >- - -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc - -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ - -DCMAKE_CROSSCOMPILING=True - -DCMAKE_CXX_FLAGS=-march=armv8-a - -DCMAKE_SYSTEM_PROCESSOR=arm64 - -DCMAKE_SYSTEM_NAME=Linux - -DLLVM_HOST_TRIPLE=aarch64-linux-gnu - -DRUST_TARGET=aarch64-unknown-linux-gnu + os: ubuntu-22.04-arm - artifact: arm64-macos os: macos-14 diff --git a/ci/docker-build.sh b/ci/docker-build.sh index b5b8498e0..7773f7980 100755 --- a/ci/docker-build.sh +++ b/ci/docker-build.sh @@ -1,8 +1,7 @@ #!/bin/sh # This is a helper script invoked from CI which will execute the `ci/build.sh` -# script within a docker container. This builds `ci/docker/Dockerfile.common` -# along with the specified `ci/docker/Dockerfile.$x` from the command line. +# script within a docker container. This contain is built using the Dockerfile located at `ci/docker/Dockerfile` # This container is then used to execute `ci/build.sh`. set -e @@ -16,17 +15,8 @@ fi set -x -# Build the base image which the image below can used. -docker build \ - --file ci/docker/Dockerfile.common \ - --tag wasi-sdk-builder-base \ - ci/docker - -# Build the container that is going to be used -docker build \ - --file ci/docker/Dockerfile.$1 \ - --tag wasi-sdk-builder \ - ci/docker +# Build the Docker imager +docker build --tag wasi-sdk-builder ci/docker # Perform the build in `/src`. The current directory is mounted read-write at # this location as well. To ensure that container-created files are reasonable diff --git a/ci/docker/Dockerfile.common b/ci/docker/Dockerfile similarity index 64% rename from ci/docker/Dockerfile.common rename to ci/docker/Dockerfile index 96013b745..9188fc2f4 100644 --- a/ci/docker/Dockerfile.common +++ b/ci/docker/Dockerfile @@ -19,17 +19,20 @@ RUN apt-get update \ # Install a more recent version of CMake than what 18.04 has since that's what # LLVM requires. -RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.29.5/cmake-3.29.5-linux-x86_64.tar.gz \ - && tar xf cmake-3.29.5-linux-x86_64.tar.gz \ - && rm cmake-3.29.5-linux-x86_64.tar.gz \ +RUN ARCH=$(uname -m) \ + && curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.29.5/cmake-3.29.5-linux-${ARCH}.tar.gz \ + && tar xf cmake-3.29.5-linux-${ARCH}.tar.gz \ + && rm cmake-3.29.5-linux-${ARCH}.tar.gz \ && mkdir -p /opt \ - && mv cmake-3.29.5-linux-x86_64 /opt/cmake + && mv cmake-3.29.5-linux-${ARCH} /opt/cmake ENV PATH /opt/cmake/bin:$PATH # As with CMake install a later version of Ninja than waht 18.04 has. -RUN curl -sSLO https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip \ - && unzip ninja-linux.zip \ +RUN ARCH=$(uname -m) \ + && if [ "$ARCH" = "aarch64" ]; then SUFFIX=-aarch64; fi \ + && curl -sSL -o ninja.zip https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux${SUFFIX}.zip \ + && unzip ninja.zip \ && rm *.zip \ && mv ninja /opt/cmake/bin diff --git a/ci/docker/Dockerfile.arm64-linux b/ci/docker/Dockerfile.arm64-linux deleted file mode 100644 index 6be6d6ed8..000000000 --- a/ci/docker/Dockerfile.arm64-linux +++ /dev/null @@ -1,11 +0,0 @@ -FROM wasi-sdk-builder-base - -# Install an extra C++ toolchain which can target arm64 linux. -RUN apt-get install -y g++-aarch64-linux-gnu - -# Configure Rust to use this new compiler for linking Rust executables. -ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER aarch64-linux-gnu-gcc - -# Note that `.github/workflows/main.yml` sets various bits and bobs of -# configuration and cmake flags that may get passed to the underlying build. For -# example LLVM is instructed to use the toolchain installed above. diff --git a/ci/docker/Dockerfile.x86_64-linux b/ci/docker/Dockerfile.x86_64-linux deleted file mode 100644 index 9d443e530..000000000 --- a/ci/docker/Dockerfile.x86_64-linux +++ /dev/null @@ -1,7 +0,0 @@ -FROM wasi-sdk-builder-base - -# No extra configuration necessary for x86_64 over what `Dockerfile.common` -# already has. -# -# Note though that `.github/workflows/main.yml` still sets various bits and bobs -# of configuration and cmake flags that may get passed to the underlying build. From 5a3567a56a47c1ea3722f8fa238d377d4801d654 Mon Sep 17 00:00:00 2001 From: Masashi Yoshimura Date: Mon, 17 Feb 2025 23:53:16 +0900 Subject: [PATCH 110/165] Add comment for ARM64 package installation to README.md (#515) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hi! I added `WASI_ARCH=arm64` command for the arm64 package version to the installation command’s comment. I thought this would make the command more clear. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c3c0dfa58..a274301d0 100644 --- a/README.md +++ b/README.md @@ -133,7 +133,7 @@ A typical installation from the release binaries might look like the following: ```shell script WASI_OS=linux -WASI_ARCH=x86_64 +WASI_ARCH=x86_64 # or 'arm64' if running on arm64 host WASI_VERSION=24 WASI_VERSION_FULL=${WASI_VERSION}.0 wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz From d94a133384eb7a805ad7185195db96435d2ecdac Mon Sep 17 00:00:00 2001 From: Am K <20879950+guangrei@users.noreply.github.com> Date: Fri, 28 Feb 2025 22:53:42 +0700 Subject: [PATCH 111/165] Fix 5 warnings found (use docker --debug to expand) (#519) This PR can fix: 5 warnings found (use docker --debug to expand): - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 23) - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 24) - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 25) - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 26) - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 27) --- docker/Dockerfile | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index 901aeb576..ba1c6f956 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -20,8 +20,8 @@ RUN case `dpkg --print-architecture` in \ esac && \ rm wasi-sdk-*.deb -ENV CC /opt/wasi-sdk/bin/clang -ENV CXX /opt/wasi-sdk/bin/clang++ -ENV LD /opt/wasi-sdk/bin/wasm-ld -ENV AR /opt/wasi-sdk/bin/llvm-ar -ENV RANLIB /opt/wasi-sdk/bin/llvm-ranlib +ENV CC="/opt/wasi-sdk/bin/clang" +ENV CXX="/opt/wasi-sdk/bin/clang++" +ENV LD="/opt/wasi-sdk/bin/wasm-ld" +ENV AR="/opt/wasi-sdk/bin/llvm-ar" +ENV RANLIB="/opt/wasi-sdk/bin/llvm-ranlib" From 53551e59438641b25e63bf304869ab4da6d512d9 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 19 Mar 2025 14:53:31 -0500 Subject: [PATCH 112/165] Update LLVM to 20.1.1 (#508) The LLVM 20 release is now official. --- cmake/wasi-sdk-sysroot.cmake | 9 ++++++++- cmake/wasi-sdk-toolchain.cmake | 1 - src/llvm-project | 2 +- src/wasi-libc | 2 +- 4 files changed, 10 insertions(+), 4 deletions(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 0dd475fff..21c8c4f75 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -74,8 +74,15 @@ ExternalProject_Add(compiler-rt-build -DCOMPILER_RT_BUILD_XRAY=OFF -DCOMPILER_RT_INCLUDE_TESTS=OFF -DCOMPILER_RT_HAS_FPIC_FLAG=OFF - -DCOMPILER_RT_ENABLE_IOS=OFF -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON + -DCOMPILER_RT_BUILD_SANITIZERS=OFF + -DCOMPILER_RT_BUILD_XRAY=OFF + -DCOMPILER_RT_BUILD_LIBFUZZER=OFF + -DCOMPILER_RT_BUILD_PROFILE=OFF + -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF + -DCOMPILER_RT_BUILD_MEMPROF=OFF + -DCOMPILER_RT_BUILD_ORC=OFF + -DCOMPILER_RT_BUILD_GWP_ASAN=OFF -DCMAKE_C_COMPILER_TARGET=wasm32-wasi -DCOMPILER_RT_OS_DIR=wasi -DCMAKE_INSTALL_PREFIX=${wasi_resource_dir} diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 646e113b9..70b8e8bce 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -78,7 +78,6 @@ ExternalProject_Add(llvm-build SOURCE_DIR "${llvm_proj_dir}/llvm" CMAKE_ARGS ${default_cmake_args} - -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_ENABLE_ZLIB=OFF -DLLVM_ENABLE_ZSTD=OFF -DLLVM_STATIC_LINK_CXX_STDLIB=ON diff --git a/src/llvm-project b/src/llvm-project index ab4b5a2db..424c2d9b7 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit ab4b5a2db582958af1ee308a790cfdb42bd24720 +Subproject commit 424c2d9b7e4de40d0804dd374721e6411c27d1d1 diff --git a/src/wasi-libc b/src/wasi-libc index 574b88da4..640c0cfc1 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 574b88da481569b65a237cb80daf9a2d5aeaf82d +Subproject commit 640c0cfc19a96b099e0791824be5ef0105ce2084 From 5e4756e449b29a17a516a779a5931703204cb877 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 23 Apr 2025 19:26:43 -0500 Subject: [PATCH 113/165] Update to wasm-component-ld 0.5.13 (#526) Pulling in the current support for component model async support plus some miscellaneous bug fixes and flags added to the binary in the meantime. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 70b8e8bce..42dd00826 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -118,7 +118,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.12) +set(wasm_component_ld_version 0.5.13) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From a995a62b4b0f24d389311e27e821a8f6c8f1cf51 Mon Sep 17 00:00:00 2001 From: Timothy McCallum Date: Fri, 2 May 2025 00:43:48 +1000 Subject: [PATCH 114/165] Update README's installation instructions to latest release number. (#528) Just bumped the version number in the Install's shell script code block (point to the latest release). --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a274301d0..b275e8a36 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,7 @@ A typical installation from the release binaries might look like the following: ```shell script WASI_OS=linux WASI_ARCH=x86_64 # or 'arm64' if running on arm64 host -WASI_VERSION=24 +WASI_VERSION=25 WASI_VERSION_FULL=${WASI_VERSION}.0 wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz tar xvf wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz From 97506a79c26df1abf175d6727a086a4be3f290f7 Mon Sep 17 00:00:00 2001 From: mcbarton Date: Fri, 23 May 2025 14:33:13 +0100 Subject: [PATCH 115/165] Add Windows 11 arm support to ci (#524) This PR should hopefully fix https://github.com/WebAssembly/wasi-sdk/issues/522 . According to https://github.blog/changelog/2025-04-14-windows-arm64-hosted-runners-now-available-in-public-preview/ Windows arm Github runners are now available for free for public repositories. Therefore I have added them to wasi-sdks ci. --- .github/actions/install-deps/action.yml | 8 +++++++- .github/workflows/main.yml | 7 +++++-- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/.github/actions/install-deps/action.yml b/.github/actions/install-deps/action.yml index 8b81d5677..2e4967a09 100644 --- a/.github/actions/install-deps/action.yml +++ b/.github/actions/install-deps/action.yml @@ -14,7 +14,13 @@ runs: shell: bash - name: Install ccache, ninja (Windows) run: choco install ccache ninja - if: runner.os == 'Windows' + if: startsWith(matrix.os, 'windows') + shell: bash + # Windows arm runners don't come with rust by default (see https://github.com/actions/partner-runner-images/blob/main/images/arm-windows-11-image.md) + # but the x86 ones do (see https://github.com/actions/runner-images/blob/main/images/windows/Windows2025-Readme.md) + - name: Install cargo (Windows-arm) + run: choco install rust + if: matrix.os == 'windows-11-arm' shell: bash - name: Install ccache, ninja (Linux) run: sudo apt-get install -y ccache ninja-build diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e30e11bfe..4efdcee99 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -48,6 +48,9 @@ jobs: - artifact: x86_64-windows os: windows-2022 + - artifact: arm64-windows + os: windows-11-arm + env: ${{ matrix.env || fromJSON('{}') }} steps: - uses: actions/checkout@v4 @@ -102,7 +105,7 @@ jobs: # # As of 2024-07-22 this sha is the "v1.13.0" tag. - uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 - if: runner.os == 'Windows' + if: startsWith(matrix.os, 'windows') - name: Build and test (Windows) run: | # Delete a troublesome binary as recommended here @@ -117,7 +120,7 @@ jobs: mkdir build cp -r C:/wasi-sdk/dist build shell: bash - if: runner.os == 'Windows' + if: startsWith(matrix.os, 'windows') # Upload the `dist` folder from the build as the artifacts for this # runner. From 38c97448e00facd18826bc73b5bb6c69219ffb4d Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Fri, 6 Jun 2025 08:42:36 +0900 Subject: [PATCH 116/165] SetjmpLongjmp.md: update for LLVM 20 (#523) --- SetjmpLongjmp.md | 41 ++++++++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/SetjmpLongjmp.md b/SetjmpLongjmp.md index df8c4af13..9034bd2ad 100644 --- a/SetjmpLongjmp.md +++ b/SetjmpLongjmp.md @@ -1,5 +1,10 @@ # C setjmp/longjmp support +WASI-SDK provides basic setjmp/longjmp support. + +Note that it's still under active development and may change in +future versions. + ## Build an application To build an application using setjmp/longjmp, you need two things: @@ -11,13 +16,13 @@ To build an application using setjmp/longjmp, you need two things: ### Example without LTO ```shell -clang -Os -mllvm -wasm-enable-sjlj -o your_app.wasm your_app.c -lsetjmp +clang -Os -mllvm -wasm-enable-sjlj -o your_app.legacy.wasm your_app.c -lsetjmp ``` ### Example with LTO ```shell -clang -Os -flto=full -mllvm -wasm-enable-sjlj -Wl,-mllvm,-wasm-enable-sjlj -o your_app.wasm your_app.c -lsetjmp +clang -Os -flto=full -mllvm -wasm-enable-sjlj -Wl,-mllvm,-wasm-enable-sjlj -o your_app.legacy.wasm your_app.c -lsetjmp ``` ## Run an application @@ -30,42 +35,44 @@ Unfortunately, there are two incompatible versions of * The latest version with `exnref` -* The [phase3] version +* The legacy [phase3] version ### Example with the latest exception handling proposal -Because the current version of WASI-SDK produces an old version -of [exception handling proposal] instructions, if your runtime -implements the latest version of the proposal, you need to convert -your module to the latest version. +By default, the current version of WASI-SDK produces the legacy +"phase3" version of [exception handling proposal] instructions. -[toywasm] is an example of such runtimes. +You can tell the llvm to produce the latest version of proposal by +specifying `-mllvm -wasm-use-legacy-eh=false`. This is expected +to be the default in a future version. -You can use binaryen `wasm-opt` command for the conversion. +Alternatively, you can use binaryen `wasm-opt` command to convert +existing modules from the legacy "phase3" version to the "exnref" version. ```shell -wasm-opt --translate-to-exnref -all -o your_app.exnref.wasm your_app.wasm +wasm-opt --translate-to-exnref -all -o your_app.wasm your_app.legacy.wasm ``` -Then you can run it with a runtime supporting the latest version of -[exception handling proposal]. +Then you can run it with a runtime supporting the "exnref" version of +the proposal. +[toywasm] is an example of such runtimes. ```shell -toywasm --wasi your_app.exnref.wasm +toywasm --wasi your_app.wasm ``` (You may need to enable the support with `-D TOYWASM_ENABLE_WASM_EXCEPTION_HANDLING=ON`.) -### Example with the phase3 exception handling proposal (a bit older version) +### Example with the legacy phase3 exception handling proposal -If your runtime supports the [phase3] version of +If your runtime supports the legacy [phase3] version of [exception handling proposal], which is the same version as what WASI-SDK -currently produces, you can run the produced module as it is. +currently produces by default, you can run the produced module as it is. For example, the classic interpreter of [wasm-micro-runtime] is one of such runtimes. ```shell -iwasm your_app.wasm +iwasm your_app.legacy.wasm ``` (You may need to enable the support with `-D WAMR_BUILD_EXCE_HANDLING=1 -D WAMR_BUILD_FAST_INTERP=0`.) From fcc3b7532e36631f15bb4a020051c70cc0f394fc Mon Sep 17 00:00:00 2001 From: Yuta Saito Date: Fri, 13 Jun 2025 06:57:11 +0900 Subject: [PATCH 117/165] Enable per-target runtime directory for compiler-rt (#533) This change allows us to have per-target libclang_rt.* libraries instead of per-OS. This is particularly required for w/threads and wo/threads targets, whose rt libraries could be incompatible even with the same OS string. This change starts building `compiler-rt` for w/threads and wo/threads separately, so that we can enable thread-related features for p1-threads target for building upcoming ASan clang_rt libraries. Also `COMPILER_RT_OS_DIR` is not set anymore as it's unused when `LLVM_ENABLE_PER_TARGET_RUNTIME_DIR` is enabled. --- README.md | 4 +- ci/merge-artifacts.sh | 8 ++-- cmake/wasi-sdk-sysroot.cmake | 72 ++++++++++++++++++++---------------- 3 files changed, 46 insertions(+), 38 deletions(-) diff --git a/README.md b/README.md index b275e8a36..19b477071 100644 --- a/README.md +++ b/README.md @@ -24,9 +24,9 @@ convenience. One could also use a standard Clang installation, build a sysroot from the sources mentioned above, and compile with `--target=wasm32-wasi --sysroot=/path/to/sysroot`. In this scenario, one would also need the -`libclang_rt.builtins-wasm32.a` objects available separately in the [release +`libclang_rt.*.a` objects available separately in the [release downloads][releases] which must be extracted into -`$CLANG_INSTALL_DIR/$CLANG_VERSION/lib/wasi/`. +`$CLANG_INSTALL_DIR/$CLANG_VERSION/lib/`. ## Clone diff --git a/ci/merge-artifacts.sh b/ci/merge-artifacts.sh index 4a69e240f..e64b68e85 100755 --- a/ci/merge-artifacts.sh +++ b/ci/merge-artifacts.sh @@ -62,12 +62,10 @@ for build in dist-*; do tar xf $sysroot -C dist/$sdk_dir/share/wasi-sysroot --strip-components 1 mv dist/$sdk_dir/share/wasi-sysroot/VERSION dist/$sdk_dir - # Setup the compiler-rt library for wasi,wasip1,wasip2 + # Setup the compiler-rt library for all targets. rtlibdir=$(dirname $(find dist/$sdk_dir/lib -name include))/lib - mkdir -p $rtlibdir/wasi - tar xf $compiler_rt -C $rtlibdir/wasi --strip-components 1 - cp -r $rtlibdir/wasi $rtlibdir/wasip1 - cp -r $rtlibdir/wasi $rtlibdir/wasip2 + mkdir -p $rtlibdir + tar xf $compiler_rt -C $rtlibdir --strip-components 1 tar czf dist/$sdk_dir.tar.gz -C dist $sdk_dir diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 21c8c4f75..37fa118cc 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -66,31 +66,38 @@ endif() # compiler-rt build logic # ============================================================================= -ExternalProject_Add(compiler-rt-build - SOURCE_DIR "${llvm_proj_dir}/compiler-rt" - CMAKE_ARGS - ${default_cmake_args} - -DCOMPILER_RT_BAREMETAL_BUILD=ON - -DCOMPILER_RT_BUILD_XRAY=OFF - -DCOMPILER_RT_INCLUDE_TESTS=OFF - -DCOMPILER_RT_HAS_FPIC_FLAG=OFF - -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON - -DCOMPILER_RT_BUILD_SANITIZERS=OFF - -DCOMPILER_RT_BUILD_XRAY=OFF - -DCOMPILER_RT_BUILD_LIBFUZZER=OFF - -DCOMPILER_RT_BUILD_PROFILE=OFF - -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF - -DCOMPILER_RT_BUILD_MEMPROF=OFF - -DCOMPILER_RT_BUILD_ORC=OFF - -DCOMPILER_RT_BUILD_GWP_ASAN=OFF - -DCMAKE_C_COMPILER_TARGET=wasm32-wasi - -DCOMPILER_RT_OS_DIR=wasi - -DCMAKE_INSTALL_PREFIX=${wasi_resource_dir} - EXCLUDE_FROM_ALL ON - USES_TERMINAL_CONFIGURE ON - USES_TERMINAL_BUILD ON - USES_TERMINAL_INSTALL ON -) +add_custom_target(compiler-rt-build) +function(define_compiler_rt target) + ExternalProject_Add(compiler-rt-build-${target} + SOURCE_DIR "${llvm_proj_dir}/compiler-rt" + CMAKE_ARGS + ${default_cmake_args} + -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON + -DCOMPILER_RT_BAREMETAL_BUILD=ON + -DCOMPILER_RT_BUILD_XRAY=OFF + -DCOMPILER_RT_INCLUDE_TESTS=OFF + -DCOMPILER_RT_HAS_FPIC_FLAG=OFF + -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON + -DCOMPILER_RT_BUILD_SANITIZERS=OFF + -DCOMPILER_RT_BUILD_XRAY=OFF + -DCOMPILER_RT_BUILD_LIBFUZZER=OFF + -DCOMPILER_RT_BUILD_PROFILE=OFF + -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF + -DCOMPILER_RT_BUILD_MEMPROF=OFF + -DCOMPILER_RT_BUILD_ORC=OFF + -DCOMPILER_RT_BUILD_GWP_ASAN=OFF + -DCMAKE_C_COMPILER_TARGET=${target} + -DCMAKE_INSTALL_PREFIX=${wasi_resource_dir} + EXCLUDE_FROM_ALL ON + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON + ) + add_dependencies(compiler-rt-build compiler-rt-build-${target}) +endfunction() + +define_compiler_rt(wasm32-wasi) +define_compiler_rt(wasm32-wasip1-threads) # In addition to the default installation of `compiler-rt` itself also copy # around some headers and make copies of the `wasi` directory as `wasip1` and @@ -106,12 +113,15 @@ add_custom_target(compiler-rt-post-build COMMAND ${CMAKE_COMMAND} -E copy_directory ${clang_resource_dir}/include ${wasi_resource_dir}/include - # Copy the `lib/wasi` folder to `libc/wasi{p1,p2}` to ensure that those + # Copy the `lib/wasm32-unknown-wasi` folder to `lib/wasm32-unknown-wasi{p1,p2}` to ensure that those # OS-strings also work for looking up the compiler-rt.a file. COMMAND ${CMAKE_COMMAND} -E copy_directory - ${wasi_resource_dir}/lib/wasi ${wasi_resource_dir}/lib/wasip1 + ${wasi_resource_dir}/lib/wasm32-unknown-wasi ${wasi_resource_dir}/lib/wasm32-unknown-wasip1 + COMMAND ${CMAKE_COMMAND} -E copy_directory + ${wasi_resource_dir}/lib/wasm32-unknown-wasi ${wasi_resource_dir}/lib/wasm32-unknown-wasip2 + # Copy the `lib/wasm32-unknown-wasip1-threads` folder to `lib/wasm32-unknown-wasi-threads` COMMAND ${CMAKE_COMMAND} -E copy_directory - ${wasi_resource_dir}/lib/wasi ${wasi_resource_dir}/lib/wasip2 + ${wasi_resource_dir}/lib/wasm32-unknown-wasip1-threads ${wasi_resource_dir}/lib/wasm32-unknown-wasi-threads COMMENT "finalizing compiler-rt installation" ) @@ -357,10 +367,10 @@ include(wasi-sdk-dist) set(dist_dir ${CMAKE_CURRENT_BINARY_DIR}/dist) -# Tarball with just `compiler-rt` builtins within it +# Tarball with just `compiler-rt` libraries within it wasi_sdk_add_tarball(dist-compiler-rt - ${dist_dir}/libclang_rt.builtins-wasm32-wasi-${wasi_sdk_version}.tar.gz - ${wasi_resource_dir}/lib/wasi) + ${dist_dir}/libclang_rt-${wasi_sdk_version}.tar.gz + ${wasi_resource_dir}/lib) add_dependencies(dist-compiler-rt compiler-rt) # Tarball with the whole sysroot From c7c2d1b38d66bd643cbd83c2c51756a6ffd8a112 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 25 Jun 2025 17:56:42 +0200 Subject: [PATCH 118/165] Bump wasm-component-ld to 0.5.14 (#537) Pulls in upcoming support for component model async APIs --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 42dd00826..0e1cf4e56 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -118,7 +118,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.13) +set(wasm_component_ld_version 0.5.14) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From e4972a7004972af3070397807dddb74f19ba5721 Mon Sep 17 00:00:00 2001 From: Andrew Brown Date: Tue, 1 Jul 2025 11:59:34 -0700 Subject: [PATCH 119/165] Document a three-month release cadence (#539) `wasi-sdk` releases have always been a bit ad hoc, but after discussing #536 we all agreed some regularity would be helpful; let's see what we think of a 3-month cadence. --- RELEASING.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/RELEASING.md b/RELEASING.md index 1091c8058..72989dd17 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,5 +1,11 @@ # Release Process +We (maintainers) plan to release a new version of wasi-sdk every three months, +coinciding with our monthly meeting to discuss latest issues and pull requests. +This provides regularity to the release cadence, though we also reserve the +right to publish at any intervening time if there is a pressing need (i.e., open +an issue to discuss). + To publish a new version of `wasi-sdk` as a GitHub release: 1. Tag a commit with an annotated tag. Note that this must be an annotated tag, From b0d6366cfd9aa81e268a2f1764028e371567be8f Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Tue, 8 Jul 2025 13:23:02 -0700 Subject: [PATCH 120/165] Update to wasi-libc main. (#538) Co-authored-by: Alex Crichton --- cmake/wasi-sdk-sysroot.cmake | 7 +++++++ src/wasi-libc | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 37fa118cc..d7e64b94b 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -164,6 +164,12 @@ function(define_wasi_libc_sub target target_suffix lto) "${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") list(JOIN extra_cflags_list " " extra_cflags) + if(${target} MATCHES threads) + set(libcompiler_rt_a ${wasi_resource_dir}/lib/wasm32-unknown-wasip1-threads/libclang_rt.builtins.a) + else() + set(libcompiler_rt_a ${wasi_resource_dir}/lib/wasm32-unknown-wasip1/libclang_rt.builtins.a) + endif() + ExternalProject_Add(wasi-libc-${target}${target_suffix}-build # Currently wasi-libc doesn't support out-of-tree builds so feign a # "download command" which copies the source tree to a different location @@ -180,6 +186,7 @@ function(define_wasi_libc_sub target target_suffix lto) SYSROOT=${wasi_sysroot} EXTRA_CFLAGS=${extra_cflags} TARGET_TRIPLE=${target} + BUILTINS_LIB=${libcompiler_rt_a} ${extra_make_flags} INSTALL_COMMAND "" DEPENDS compiler-rt diff --git a/src/wasi-libc b/src/wasi-libc index 640c0cfc1..50ae11904 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 640c0cfc19a96b099e0791824be5ef0105ce2084 +Subproject commit 50ae11904df674fecaa311537967fe138c21fcc7 From 36e12fd3032ed4402fa5fef00919cc561fe8b9bb Mon Sep 17 00:00:00 2001 From: Dan Gohman Date: Fri, 11 Jul 2025 21:02:19 -0700 Subject: [PATCH 121/165] Update to LLVM 20.1.8. (#545) Update from LLVM 20.1.1 to 20.1.8. This release contains various bug fixes and notably for wasi-sdk includes llvm/llvm-project#146574, which fixes inline asm vector operands. --- src/llvm-project | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/llvm-project b/src/llvm-project index 424c2d9b7..87f0227cb 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 424c2d9b7e4de40d0804dd374721e6411c27d1d1 +Subproject commit 87f0227cb60147a26a1eeb4fb06e3b505e9c7261 From dadbd94e10c8f1d41e8c205c307f4b1b323d811d Mon Sep 17 00:00:00 2001 From: Catherine Date: Sun, 27 Jul 2025 03:42:08 +0100 Subject: [PATCH 122/165] Enable pthreads in libc++ for `THREAD_MODEL=single` (#548) This commit enables thread support in libc++ for all thread models, enabling C++ applications that use threading APIs like `` but do not spawn threads (e.g. Clang) to be built with minimal changes. Fixes #546. Depends on WebAssembly/wasi-libc#602. --- README.md | 13 ++++++++++++- cmake/wasi-sdk-sysroot.cmake | 10 ++++------ src/wasi-libc | 2 +- 3 files changed, 17 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 19b477071..6e1321bed 100644 --- a/README.md +++ b/README.md @@ -214,10 +214,21 @@ See [C setjmp/longjmp support] about setjmp/longjmp support. [C setjmp/longjmp support]: SetjmpLongjmp.md This repository experimentally supports __threads__ with -`--target=wasm32-wasi-threads`. It uses WebAssembly's [threads] primitives +`--target=wasm32-wasip1-threads`. It uses WebAssembly's [threads] primitives (atomics, `wait`/`notify`, shared memory) and [wasi-threads] for spawning threads. Note: this is experimental — do not expect long-term stability! +Note that the `pthread_*` family of functions, as well as C++ threading primitives +such as ``, ``, and `` are available on all targets. +Any attempt to spawn a thread will fail on `--target=wasm32-wasip1` or +`--target=wasm32-wasip2`, but other functionality, such as locks, still works. +This makes it easier to port C++ codebases, as only a fraction of code needs +to be modified to build for the single-threaded targets. + +Defining a macro `_WASI_STRICT_PTHREAD` will make `pthread_create`, +`pthread_detach`, `pthread_join`, `pthread_tryjoin_np`, and `pthread_timedjoin_np` +fail with a compile time error when building for single-threaded targets. + [threads]: https://github.com/WebAssembly/threads [wasi-threads]: https://github.com/WebAssembly/wasi-threads diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index d7e64b94b..f53fda91c 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -222,11 +222,9 @@ execute_process( function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_suffix) if(${target} MATCHES threads) - set(threads ON) set(pic OFF) set(target_flags -pthread) else() - set(threads OFF) set(pic ON) set(target_flags "") endif() @@ -262,8 +260,8 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DCMAKE_STAGING_PREFIX=${wasi_sysroot} -DCMAKE_POSITION_INDEPENDENT_CODE=${pic} -DCXX_SUPPORTS_CXX11=ON - -DLIBCXX_ENABLE_THREADS:BOOL=${threads} - -DLIBCXX_HAS_PTHREAD_API:BOOL=${threads} + -DLIBCXX_ENABLE_THREADS:BOOL=ON + -DLIBCXX_HAS_PTHREAD_API:BOOL=ON -DLIBCXX_HAS_EXTERNAL_THREAD_API:BOOL=OFF -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF @@ -280,8 +278,8 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF -DLIBCXXABI_ENABLE_SHARED:BOOL=${pic} -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON - -DLIBCXXABI_ENABLE_THREADS:BOOL=${threads} - -DLIBCXXABI_HAS_PTHREAD_API:BOOL=${threads} + -DLIBCXXABI_ENABLE_THREADS:BOOL=ON + -DLIBCXXABI_HAS_PTHREAD_API:BOOL=ON -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF diff --git a/src/wasi-libc b/src/wasi-libc index 50ae11904..3f7eb4c7d 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 50ae11904df674fecaa311537967fe138c21fcc7 +Subproject commit 3f7eb4c7d6ede4dde3c4bffa6ed14e8d656fe93f From fbdec30656d4099da5c7fd7e8275e4af777ca97e Mon Sep 17 00:00:00 2001 From: YAMAMOTO Takashi Date: Mon, 28 Jul 2025 02:46:55 +0900 Subject: [PATCH 123/165] Add an option to specify CFLAGS for wasm features and make lime1 the default (#527) Add a cmake option WASI_SDK_CPU_CFLAGS to specify CFLAGS to control wasm features to enable/disable. Make the default `-mcpu=lime1`. Comparing to the default "generic" target of the recent llvm, this effectively * disables reference-types and bulk-memory (enable their partial counterparts instead) * enables extended-const Note: as of writing this, a few popular runtimes, including wasm3 and wasm-micro-runtime, don't support extented-const yet. Note: regardless of this, wasi-libc enables necessary features for certain files. (bulk-memory and exception-handling) cf. https://github.com/WebAssembly/wasi-sdk/issues/525 References: https://github.com/WebAssembly/tool-conventions/blob/main/Lime.md https://github.com/bytecodealliance/wasm-micro-runtime/issues/4272 --- .github/workflows/main.yml | 1 + README.md | 2 ++ cmake/wasi-sdk-sysroot.cmake | 5 ++++- 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 4efdcee99..4586bab42 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -169,6 +169,7 @@ jobs: -DCMAKE_C_COMPILER=/usr/lib/llvm-18/bin/clang \ -DCMAKE_SYSTEM_NAME=WASI \ -DWASI_SDK_INCLUDE_TESTS=ON \ + -DWASI_SDK_CPU_CFLAGS="" \ -DCMAKE_C_LINKER_DEPFILE_SUPPORTED=OFF \ -DCMAKE_CXX_LINKER_DEPFILE_SUPPORTED=OFF - run: ninja -C build diff --git a/README.md b/README.md index 6e1321bed..89dadb917 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,8 @@ in compiling WebAssembly code. Supported CMake flags are: * `-DWASI_SDK_DEBUG_PREFIX_MAKE=OFF` - disable `-fdebug-prefix-map` when building C/C++ code to use full host paths instead. * `-DWASI_SDK_INCLUDE_TESTS=ON` - used for building tests. +* `-DWASI_SDK_CPU_CFLAGS=..` - used to specify CFLAGS to tweak wasm features + to enable/disable. The default is `-mcpu=lime1`. * `-DWASI_SDK_TEST_HOST_TOOLCHAIN=ON` - test the host toolchain's wasi-libc and sysroot libraries, don't build or use fresh libraries for tests. * `-DWASI_SDK_TARGETS=..` - a list of targets to build, by default all WASI diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index f53fda91c..25d583736 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -24,6 +24,7 @@ option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) option(WASI_SDK_INSTALL_TO_CLANG_RESOURCE_DIR "Whether or not to modify the compiler's resource directory" OFF) option(WASI_SDK_LTO "Whether or not to build LTO assets" ON) +set(WASI_SDK_CPU_CFLAGS "-mcpu=lime1" CACHE STRING "CFLAGS to specify wasm features to enable") set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) set(wasi_sysroot ${wasi_tmp_install}/share/wasi-sysroot) @@ -87,6 +88,7 @@ function(define_compiler_rt target) -DCOMPILER_RT_BUILD_ORC=OFF -DCOMPILER_RT_BUILD_GWP_ASAN=OFF -DCMAKE_C_COMPILER_TARGET=${target} + -DCMAKE_C_FLAGS=${WASI_SDK_CPU_CFLAGS} -DCMAKE_INSTALL_PREFIX=${wasi_resource_dir} EXCLUDE_FROM_ALL ON USES_TERMINAL_CONFIGURE ON @@ -161,7 +163,7 @@ function(define_wasi_libc_sub target target_suffix lto) get_property(directory_cflags DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) list(APPEND directory_cflags -resource-dir ${wasi_resource_dir}) set(extra_cflags_list - "${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") + "${WASI_SDK_CPU_CFLAGS} ${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") list(JOIN extra_cflags_list " " extra_cflags) if(${target} MATCHES threads) @@ -238,6 +240,7 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ get_property(dir_compile_opts DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) get_property(dir_link_opts DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY LINK_OPTIONS) set(extra_flags + ${WASI_SDK_CPU_CFLAGS} ${target_flags} --target=${target} ${dir_compile_opts} From d05b57d2a21229fbc1d492d204057cf4751de697 Mon Sep 17 00:00:00 2001 From: Catherine Date: Mon, 28 Jul 2025 14:53:18 +0100 Subject: [PATCH 124/165] Ship the `llvm-addr2line` tool (#550) Fixes #549. --- cmake/wasi-sdk-toolchain.cmake | 1 + 1 file changed, 1 insertion(+) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 0e1cf4e56..1eeb52295 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -41,6 +41,7 @@ set(tools clang-tidy clang-apply-replacements lld + llvm-addr2line llvm-mc llvm-ranlib llvm-strip From ac998c2aa3a4729b23350dd3b7b298bf538d57bf Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 13 Aug 2025 18:24:17 -0500 Subject: [PATCH 125/165] Update github CI concurrency key (#555) This bit me while releasing wasi-sdk-26/27 because while I tried to get both builds going at once they overlapped in their key meaning that they had to progress one-at-a-time. This uses `github.ref` instead of the PR number which I think should be more robust in situations like this. --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 4586bab42..cad0c108d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -11,7 +11,7 @@ on: workflow_dispatch: concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: From 400dd95ec421d789c47c1fd08b4dfceb5c36746f Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 25 Aug 2025 20:10:56 -0500 Subject: [PATCH 126/165] Bump wasm-component-ld to 0.5.16 (#557) Keeping it up-to-date. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 1eeb52295..66e14ee96 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.14) +set(wasm_component_ld_version 0.5.16) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 144b335ece5b98c954e74a05ee665c15fa2c821b Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 5 Sep 2025 17:06:43 -0500 Subject: [PATCH 127/165] Update to LLVM 21.1.0 (#559) Keeping up-to-date with the latest release --- .github/workflows/main.yml | 3 ++- src/llvm-project | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index cad0c108d..0e2df4b34 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -164,9 +164,10 @@ jobs: - uses: ./.github/actions/checkout - uses: ./.github/actions/install-deps - run: cargo install wasm-component-ld@0.5.12 + - run: sudo apt-get update -y && sudo apt-get install -y clang-20 lld-20 - run: | cmake -G Ninja -B build -S . \ - -DCMAKE_C_COMPILER=/usr/lib/llvm-18/bin/clang \ + -DCMAKE_C_COMPILER=/usr/lib/llvm-20/bin/clang \ -DCMAKE_SYSTEM_NAME=WASI \ -DWASI_SDK_INCLUDE_TESTS=ON \ -DWASI_SDK_CPU_CFLAGS="" \ diff --git a/src/llvm-project b/src/llvm-project index 87f0227cb..3623fe661 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 87f0227cb60147a26a1eeb4fb06e3b505e9c7261 +Subproject commit 3623fe661ae35c6c80ac221f14d85be76aa870f1 From 9da4551862cd87a54b3ce78b492471f650d85d78 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 5 Sep 2025 17:18:11 -0500 Subject: [PATCH 128/165] Update wasi-libc to latest (#560) Pull in recent changes e.g. around wasip2 bits --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index 3f7eb4c7d..bbb0c01b1 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 3f7eb4c7d6ede4dde3c4bffa6ed14e8d656fe93f +Subproject commit bbb0c01b112f8c3aaa1974407c5302a26b88babb From d43a9360539163a9e103709b135089dc126d4a58 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 9 Sep 2025 15:32:58 -0500 Subject: [PATCH 129/165] Update wasm-component-ld to 0.5.17 (#561) --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 66e14ee96..56949feab 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.16) +set(wasm_component_ld_version 0.5.17) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From d90d7de10e2208905b1cdf29817f89c3ed68cbcd Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 11 Sep 2025 21:04:04 -0500 Subject: [PATCH 130/165] Rewrite setjmp/longjmp docs (#562) * Provide flags for emitting the standard exception-handling instructions. * More further explain what flags are doing. * Show a sample program that can be compiled both ways. * Delegate running in a runtime to various runtimes, so remove the "run the program" documentation here. * Remove `wasm-opt` documentation since it's now documented how to emit the standard instructions directly with LLVM. --- SetjmpLongjmp.md | 131 ++++++++++++++++++++++++++++++----------------- 1 file changed, 83 insertions(+), 48 deletions(-) diff --git a/SetjmpLongjmp.md b/SetjmpLongjmp.md index 9034bd2ad..364a06181 100644 --- a/SetjmpLongjmp.md +++ b/SetjmpLongjmp.md @@ -3,80 +3,115 @@ WASI-SDK provides basic setjmp/longjmp support. Note that it's still under active development and may change in -future versions. +future versions. The tl;dr; version of this document is to pass these flags to +the C compiler: -## Build an application - -To build an application using setjmp/longjmp, you need two things: +``` +-mllvm -wasm-enable-sjlj -lsetjmp -mllvm -wasm-use-legacy-eh=false +``` -* Enable the necessary LLVM translation (`-mllvm -wasm-enable-sjlj`) +## Implementation Primitives + +Support for `setjmp` and `longjmp` is built on top of the +[exception-handling](https://github.com/WebAssembly/exception-handling) +WebAssembly proposal. This proposal is now [phase +5](https://github.com/WebAssembly/proposals) and becoming part of the official +specification. Note, however, that the exception-handling proposal has a long +history and has a "legacy" version which shipped in browsers as well. This means +that there are two different, but similar, sets of instructions that can be +emitted to support `setjmp` and `longjmp`. Clang 20 and later (wasi-sdk-26 and +later) is capable of emitting both at this time via `-mllvm +-wasm-use-legacy-eh={false,true}` compiler flags. + +Another important point is that exception-handling only provides structured +control flow primitives for exceptions. This means it is not possible to purely +define `setjmp` in C as otherwise it must be a function that returns twice. This +means that support for `setjmp` and `longjmp` in WebAssembly relies on a +compiler pass to transform invocations of `setjmp` at a compiler IR level. This +means that the `setjmp` symbol is not defined in wasi-libc, for example, but +instead primitives used to implement `setjmp`, in conjunction with LLVM, are +found in wasi-libc. -* Link the setjmp library (`-lsetjmp`) +## Build an application -### Example without LTO +To build an application using setjmp/longjmp, you need three sets of compiler +flags: -```shell -clang -Os -mllvm -wasm-enable-sjlj -o your_app.legacy.wasm your_app.c -lsetjmp -``` +1. `-mllvm -wasm-enable-sjlj`: Enable LLVM compiler pass which replaces calls to + `setjmp` and `longjmp` with a different implementation that wasi-libc + implements and hooks into. +2. `-lsetjmp`: Link the setjmp library that wasi-libc provides which contains + these hooks that LLVM uses. +2. `-mllvm -wasm-use-legacy-eh=false`: Specify which version of the + exception-handling instructions will be emitted. Note that if this is omitted + it currently defaults to `true` meaning that the legacy instructions are + emitted, not the standard instructions. -### Example with LTO +In short, these flags are required to use `setjmp`/`longjmp` -```shell -clang -Os -flto=full -mllvm -wasm-enable-sjlj -Wl,-mllvm,-wasm-enable-sjlj -o your_app.legacy.wasm your_app.c -lsetjmp +``` +-mllvm -wasm-enable-sjlj -lsetjmp -mllvm -wasm-use-legacy-eh=false ``` -## Run an application +### Examples -To run the application built as in the previous section, -you need to use a runtime with [exception handling proposal] support. +This source code: -Unfortunately, there are two incompatible versions of -[exception handling proposal], which is commonly implemented by runtimes. +```c +#include +#include +#include +#include -* The latest version with `exnref` +static jmp_buf env; -* The legacy [phase3] version +static bool test_if_longjmp(void(*f)(void)) { + if (setjmp(env)) + return true; + f(); + return false; +} -### Example with the latest exception handling proposal +static void do_not_longjmp() { +} -By default, the current version of WASI-SDK produces the legacy -"phase3" version of [exception handling proposal] instructions. +static void do_longjmp() { + longjmp(env, 1); +} -You can tell the llvm to produce the latest version of proposal by -specifying `-mllvm -wasm-use-legacy-eh=false`. This is expected -to be the default in a future version. +int main() { + bool longjmped = test_if_longjmp(do_not_longjmp); + assert(!longjmped); + longjmped = test_if_longjmp(do_longjmp); + assert(longjmped); + return 0; +} +``` -Alternatively, you can use binaryen `wasm-opt` command to convert -existing modules from the legacy "phase3" version to the "exnref" version. +can be compiled using the standard set of instructions as: ```shell -wasm-opt --translate-to-exnref -all -o your_app.wasm your_app.legacy.wasm +clang -Os -o test.wasm test.c \ + -mllvm -wasm-enable-sjlj -lsetjmp -mllvm -wasm-use-legacy-eh=false ``` -Then you can run it with a runtime supporting the "exnref" version of -the proposal. -[toywasm] is an example of such runtimes. +and then `test.wasm` can be executed in a WebAssembly runtime supporting WASI. + +You can also compile for the legacy exceptions proposal with: ```shell -toywasm --wasi your_app.wasm +clang -Os -o test.wasm test.c \ + -mllvm -wasm-enable-sjlj -lsetjmp -mllvm -wasm-use-legacy-eh=true ``` -(You may need to enable the support with `-D TOYWASM_ENABLE_WASM_EXCEPTION_HANDLING=ON`.) - -### Example with the legacy phase3 exception handling proposal -If your runtime supports the legacy [phase3] version of -[exception handling proposal], which is the same version as what WASI-SDK -currently produces by default, you can run the produced module as it is. +and then `test.wasm` can be executed in a WebAssembly runtime supporting the +legacy WebAssembly instructions. -For example, the classic interpreter of [wasm-micro-runtime] is -one of such runtimes. +Note that when compiling with LTO you'll need to pass `-mllvm` flags to the +linker in addition to Clang itself, such as: ```shell -iwasm your_app.legacy.wasm +clang -Os -flto=full -o test.wasm test.c \ + -mllvm -wasm-enable-sjlj -lsetjmp -mllvm -wasm-use-legacy-eh=false \ + -Wl,-mllvm,-wasm-enable-sjlj,-mllvm,-wasm-use-legacy-eh=false ``` -(You may need to enable the support with `-D WAMR_BUILD_EXCE_HANDLING=1 -D WAMR_BUILD_FAST_INTERP=0`.) - -[exception handling proposal]: https://github.com/WebAssembly/exception-handling/ -[phase3]: https://github.com/WebAssembly/exception-handling/tree/main/proposals/exception-handling/legacy -[toywasm]: https://github.com/yamt/toywasm -[wasm-micro-runtime]: https://github.com/bytecodealliance/wasm-micro-runtime From 90f77f9a258418e5d7a60c826f0d16fe2a504a07 Mon Sep 17 00:00:00 2001 From: Tim Chevalier Date: Wed, 8 Oct 2025 17:09:34 -0700 Subject: [PATCH 131/165] In Install instructions in README, bump version to 27 (#567) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 89dadb917..932437ecc 100644 --- a/README.md +++ b/README.md @@ -136,7 +136,7 @@ A typical installation from the release binaries might look like the following: ```shell script WASI_OS=linux WASI_ARCH=x86_64 # or 'arm64' if running on arm64 host -WASI_VERSION=25 +WASI_VERSION=27 WASI_VERSION_FULL=${WASI_VERSION}.0 wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz tar xvf wasi-sdk-${WASI_VERSION_FULL}-${WASI_ARCH}-${WASI_OS}.tar.gz From 169e79f00a1c3823ce3b0ec5db58bab892fbffdf Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 9 Oct 2025 11:45:27 -0500 Subject: [PATCH 132/165] Bump wasm-component-ld to 0.5.18 (#568) Keeping it up-to-date --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 56949feab..7476a1032 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.17) +set(wasm_component_ld_version 0.5.18) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 89d0ba6c69a097a304e8888a04d3bdba4e23e36b Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 10 Oct 2025 09:41:25 -0500 Subject: [PATCH 133/165] Always build wasm32-wasip2 with `-fPIC` (#564) This commit updates the build of the `wasm32-wasip2` target to always pass `-fPIC` to compilation of wasi-libc. This notably enables using `libc.a` in dynamic linking situations instead of being forced to use `libc.so`. While many applications likely want to use `libc.so` I've found it more flexible if objects are compatible with as many builds as possible. This will enable, for example, building shared libraries in Rust by default since Rust only ships `libc.a`, not `libc.so`, in the pre-built sysroot. This behavior additionally matches the Rust `wasm32-wasip2` target where `-fPIC` is enabled by default there. --- cmake/wasi-sdk-sysroot.cmake | 41 ++++++++++++++++++------------------ src/wasi-libc | 2 +- 2 files changed, 22 insertions(+), 21 deletions(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 25d583736..c1837c9fd 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -139,31 +139,32 @@ add_custom_target(compiler-rt DEPENDS compiler-rt-build compiler-rt-post-build) function(define_wasi_libc_sub target target_suffix lto) set(build_dir ${CMAKE_CURRENT_BINARY_DIR}/wasi-libc-${target}${target_suffix}) - if(${target} MATCHES threads) - if(lto) - set(extra_make_flags LTO=full THREAD_MODEL=posix) - else() - set(extra_make_flags THREAD_MODEL=posix) - endif() - elseif(${target} MATCHES p2) - if(lto) - set(extra_make_flags LTO=full WASI_SNAPSHOT=p2 default) - else() - set(extra_make_flags WASI_SNAPSHOT=p2 default libc_so) - endif() - else() - if(lto) - set(extra_make_flags LTO=full default) - else() - set(extra_make_flags default libc_so) - endif() - endif() - string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPER) get_property(directory_cflags DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) list(APPEND directory_cflags -resource-dir ${wasi_resource_dir}) set(extra_cflags_list "${WASI_SDK_CPU_CFLAGS} ${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") + + set(extra_make_flags default) + + # If LTO is enabled then pass that on to wasi-libc, and otherwise make sure to + # build a `libc.so` dynamic library where possible (not compatible with + # threads though) + if(lto) + list(APPEND extra_make_flags LTO=full) + elseif(NOT ${target} MATCHES threads) + list(APPEND extra_make_flags libc_so) + endif() + + if(${target} MATCHES threads) + list(APPEND extra_make_flags THREAD_MODEL=posix) + elseif(${target} MATCHES p2) + list(APPEND extra_make_flags WASI_SNAPSHOT=p2) + # Always enable `-fPIC` for the `wasm32-wasip2` target. This makes `libc.a` + # more flexible and usable in dynamic linking situations. + list(APPEND extra_cflags_list -fPIC) + endif() + list(JOIN extra_cflags_list " " extra_cflags) if(${target} MATCHES threads) diff --git a/src/wasi-libc b/src/wasi-libc index bbb0c01b1..6b45da5b0 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit bbb0c01b112f8c3aaa1974407c5302a26b88babb +Subproject commit 6b45da5b05bc0edda355a6de46101d4b21631985 From aa246fe579a02db33fd680f7fd5cefa1e944e7ff Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 3 Nov 2025 11:41:30 -0600 Subject: [PATCH 134/165] Update submodules (#572) * Update LLVM to 21.1.4 * Update wasi-libc to main --- src/llvm-project | 2 +- src/wasi-libc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/llvm-project b/src/llvm-project index 3623fe661..222fc11f2 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 3623fe661ae35c6c80ac221f14d85be76aa870f1 +Subproject commit 222fc11f2b8f25f6a0f4976272ef1bb7bf49521d diff --git a/src/wasi-libc b/src/wasi-libc index 6b45da5b0..fcc3813da 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 6b45da5b05bc0edda355a6de46101d4b21631985 +Subproject commit fcc3813da80f097830b247e0fe0daf827a18cccc From 0599c9f3fb9e502456149f291c944eb990280649 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 5 Nov 2025 16:19:37 -0600 Subject: [PATCH 135/165] Update wasi-libc to main (#574) Pull in WebAssembly/wasi-libc#630 --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index fcc3813da..d181bdd42 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit fcc3813da80f097830b247e0fe0daf827a18cccc +Subproject commit d181bdd429f0067c92d817909b01e4970ad7d70a From a1c93a043ae3913a86375bf68eafdab064025515 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 10 Nov 2025 12:03:31 -0600 Subject: [PATCH 136/165] Lower linux glibc requirement from 2.31 to 2.28 (#575) In componentize-py Joel's building custom wasi-sdk binaries for more Linux compat which is what maturin, the build tool use there, desires. I don't think there's a concrete reason to have a higher Linux version here, so let's lower it in wasi-sdk itself. This commit switches the build container from Ubuntu 20.04 (glibc 2.31) to AlmaLinux 8 (glibc 2.28). The AlmaLinux container is what Wasmtime uses, for example, and has supported security updates to 2029 and is I believe intended to be used for purposes such as this. --- ci/docker/Dockerfile | 27 +++++++++------------------ ci/docker/install-ccache.sh | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 18 deletions(-) create mode 100755 ci/docker/install-ccache.sh diff --git a/ci/docker/Dockerfile b/ci/docker/Dockerfile index 9188fc2f4..a5da2c839 100644 --- a/ci/docker/Dockerfile +++ b/ci/docker/Dockerfile @@ -1,40 +1,31 @@ # Use a relatively old/stable distro here to maximize the supported platforms # and avoid depending on more recent version of, say, libc. -# Here we choose Ubuntu 20.04. +# Here we choose AlmaLinux 8 -FROM ubuntu:20.04 +FROM almalinux:8 # Various build tooling and such necessary to build LLVM and a wasi-sysroot -RUN apt-get update \ - && apt-get install -y --no-install-recommends \ - ccache \ +RUN dnf install -y \ curl \ ca-certificates \ - build-essential \ clang \ python3 \ git \ unzip \ - xz-utils + cmake -# Install a more recent version of CMake than what 18.04 has since that's what -# LLVM requires. -RUN ARCH=$(uname -m) \ - && curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.29.5/cmake-3.29.5-linux-${ARCH}.tar.gz \ - && tar xf cmake-3.29.5-linux-${ARCH}.tar.gz \ - && rm cmake-3.29.5-linux-${ARCH}.tar.gz \ - && mkdir -p /opt \ - && mv cmake-3.29.5-linux-${ARCH} /opt/cmake +COPY ./install-ccache.sh . +RUN ./install-ccache.sh -ENV PATH /opt/cmake/bin:$PATH +ENV PATH /opt/ccache/bin:$PATH -# As with CMake install a later version of Ninja than waht 18.04 has. +# AlmaLinux 8 doesn't seem to have ninja, so install it manually. RUN ARCH=$(uname -m) \ && if [ "$ARCH" = "aarch64" ]; then SUFFIX=-aarch64; fi \ && curl -sSL -o ninja.zip https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux${SUFFIX}.zip \ && unzip ninja.zip \ && rm *.zip \ - && mv ninja /opt/cmake/bin + && mv ninja /opt/ccache/bin # Tell programs to cache in a location that both isn't a `--volume` mounted root # and isn't `/root` in the container as that won't be writable during the build. diff --git a/ci/docker/install-ccache.sh b/ci/docker/install-ccache.sh new file mode 100755 index 000000000..e70fb6482 --- /dev/null +++ b/ci/docker/install-ccache.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +# AlmaLinux 8, the container this script runs in, does not have ccache in its +# package repositories. The ccache project publishes both x86_64 and aarch64 +# binaries, however. The x86_64 binaries for ccache are themselves built in +# AlmaLinux 8 so they're compatible, but the aarch64 binaries are built in a +# newer container and don't run on AlmaLinux 8. +# +# Thus this script downloads precompiled binaries for x86_64 but builds from +# source on aarch64. + +ARCH=$(uname -m) +ver=4.12.1 + +if [ "x$ARCH" = "x86_64" ]; then + curl -sSLO https://github.com/ccache/ccache/releases/download/v${ver}/ccache-${ver}-linux-${ARCH}.tar.xz + tar -xf ccache-${ver}-linux-${ARCH}.tar.xz + rm ccache-${ver}-linux-${ARCH}.tar.xz + mv ccache-${ver}-linux-${ARCH} /opt/ccache/bin +else + curl -sSLO https://github.com/ccache/ccache/releases/download/v${ver}/ccache-${ver}.tar.xz + tar -xf ccache-${ver}.tar.xz + + cd ccache-${ver} + mkdir build + cd build + cmake .. \ + -DCMAKE_INSTALL_PREFIX=/opt/ccache \ + -DCMAKE_BUILD_TYPE=Release \ + -DHTTP_STORAGE_BACKEND=OFF \ + -DENABLE_TESTING=OFF \ + -DREDIS_STORAGE_BACKEND=OFF + make -j$(nproc) + make install +fi From 85269a848bed2df7383ab729c3ea9f8cfe37706d Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 14 Nov 2025 15:58:21 -0600 Subject: [PATCH 137/165] Update wasi-libc (#576) Pulls in a number of fixes for the wasm32-wasip2 target. --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index d181bdd42..ac020b86f 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit d181bdd429f0067c92d817909b01e4970ad7d70a +Subproject commit ac020b86fd44bafe60aa4fa12f407d16e3731329 From 338aec5def0debda9de7e47838c9883e4e0d923f Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 14 Nov 2025 16:18:17 -0600 Subject: [PATCH 138/165] Update wasm-component-ld to 0.5.19 (#577) Pulling in wasm-tools-related changes. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 7476a1032..61481e390 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.18) +set(wasm_component_ld_version 0.5.19) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 2f17883108833af90264404a3749cffdd3d9121b Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 5 Jan 2026 16:03:25 -0500 Subject: [PATCH 139/165] Update wasm-component-ld (#581) Keeping it up-to-date --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 61481e390..fa972b9de 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.19) +set(wasm_component_ld_version 0.5.20) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 3891d0fbb14e0f57133c21594056104d19ee7370 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 9 Jan 2026 13:37:54 -0600 Subject: [PATCH 140/165] Update wasi-libc to cmake-enabled build (#580) Updating to handle https://github.com/WebAssembly/wasi-libc/commit/688a685d1c5037ef8d718b9d3f91a4063c3ace77 from wasi-libc --- cmake/wasi-sdk-sysroot.cmake | 58 +++++++++++++----------------------- src/wasi-libc | 2 +- 2 files changed, 21 insertions(+), 39 deletions(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index c1837c9fd..e1e26ca41 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -137,29 +137,11 @@ add_custom_target(compiler-rt DEPENDS compiler-rt-build compiler-rt-post-build) # ============================================================================= function(define_wasi_libc_sub target target_suffix lto) - set(build_dir ${CMAKE_CURRENT_BINARY_DIR}/wasi-libc-${target}${target_suffix}) - string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPER) get_property(directory_cflags DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) - list(APPEND directory_cflags -resource-dir ${wasi_resource_dir}) - set(extra_cflags_list - "${WASI_SDK_CPU_CFLAGS} ${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") - - set(extra_make_flags default) - - # If LTO is enabled then pass that on to wasi-libc, and otherwise make sure to - # build a `libc.so` dynamic library where possible (not compatible with - # threads though) - if(lto) - list(APPEND extra_make_flags LTO=full) - elseif(NOT ${target} MATCHES threads) - list(APPEND extra_make_flags libc_so) - endif() + set(extra_cflags_list "${WASI_SDK_CPU_CFLAGS} ${CMAKE_C_FLAGS} ${directory_cflags}") - if(${target} MATCHES threads) - list(APPEND extra_make_flags THREAD_MODEL=posix) - elseif(${target} MATCHES p2) - list(APPEND extra_make_flags WASI_SNAPSHOT=p2) + if(${target} MATCHES p2) # Always enable `-fPIC` for the `wasm32-wasip2` target. This makes `libc.a` # more flexible and usable in dynamic linking situations. list(APPEND extra_cflags_list -fPIC) @@ -173,25 +155,25 @@ function(define_wasi_libc_sub target target_suffix lto) set(libcompiler_rt_a ${wasi_resource_dir}/lib/wasm32-unknown-wasip1/libclang_rt.builtins.a) endif() + set(extra_cmake_args) + + # Configure LTO in wasi libc if it's enabled. Be sure to disable shared + # libraries as well since that's not currently supported. + if (lto) + list(APPEND extra_cmake_args -DLTO=full -DBUILD_SHARED=OFF) + endif() + ExternalProject_Add(wasi-libc-${target}${target_suffix}-build - # Currently wasi-libc doesn't support out-of-tree builds so feign a - # "download command" which copies the source tree to a different location - # so out-of-tree builds are supported. - DOWNLOAD_COMMAND - ${CMAKE_COMMAND} -E copy_directory ${wasi_libc} ${build_dir} - SOURCE_DIR "${build_dir}" - CONFIGURE_COMMAND "" - BUILD_COMMAND - ${MAKE} -j8 -C ${build_dir} - CC=${CMAKE_C_COMPILER} - AR=${CMAKE_AR} - NM=${CMAKE_NM} - SYSROOT=${wasi_sysroot} - EXTRA_CFLAGS=${extra_cflags} - TARGET_TRIPLE=${target} - BUILTINS_LIB=${libcompiler_rt_a} - ${extra_make_flags} - INSTALL_COMMAND "" + SOURCE_DIR ${wasi_libc} + CMAKE_ARGS + ${default_cmake_args} + ${extra_cmake_args} + -DTARGET_TRIPLE=${target} + -DCMAKE_INSTALL_PREFIX=${wasi_sysroot} + -DCMAKE_C_FLAGS=${extra_cflags} + -DBUILTINS_LIB=${libcompiler_rt_a} + -DUSE_WASM_COMPONENT_LD=OFF + -DBINDINGS_TARGET=OFF DEPENDS compiler-rt EXCLUDE_FROM_ALL ON USES_TERMINAL_CONFIGURE ON diff --git a/src/wasi-libc b/src/wasi-libc index ac020b86f..06bef3c91 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit ac020b86fd44bafe60aa4fa12f407d16e3731329 +Subproject commit 06bef3c91a541b67edf88ff44e0e3bc0a928ad16 From 5bfe919641505129c1bac84efb8646445b68ef74 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 9 Jan 2026 17:15:09 -0600 Subject: [PATCH 141/165] Update wasi-libc with sdk version information (#583) This updates to include WebAssembly/wasi-libc#709 and then additionally configures wasi-libc with `-DWASI_SDK_VERSION=...` to ensure that wasi-sdk builds will now have `__wasi_sdk_major__` defined along with `__wasi_sdk_version__`. Closes WebAssembly/wasi-libc#688 --- cmake/wasi-sdk-sysroot.cmake | 1 + src/wasi-libc | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index e1e26ca41..5bd04bb36 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -174,6 +174,7 @@ function(define_wasi_libc_sub target target_suffix lto) -DBUILTINS_LIB=${libcompiler_rt_a} -DUSE_WASM_COMPONENT_LD=OFF -DBINDINGS_TARGET=OFF + -DWASI_SDK_VERSION=${wasi_sdk_version} DEPENDS compiler-rt EXCLUDE_FROM_ALL ON USES_TERMINAL_CONFIGURE ON diff --git a/src/wasi-libc b/src/wasi-libc index 06bef3c91..6036a9f3e 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 06bef3c91a541b67edf88ff44e0e3bc0a928ad16 +Subproject commit 6036a9f3e349315faba2732a32508d5dc9d6d928 From 1e896871f60255dbbe8925485b733acf3b45111b Mon Sep 17 00:00:00 2001 From: Nis Meinert Date: Wed, 21 Jan 2026 23:45:18 +0100 Subject: [PATCH 142/165] doc: add cargo to list of requirements (#586) During the build step, cargo is needed but this was not listed under requirements. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 932437ecc..014016dee 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,7 @@ The Wasm-sdk's build process needs some packages : * `clang` * `ninja` * `python3` +* `cargo` Please refer to your OS documentation to install those packages. From 3d4ea12dbee832e92a4734c938fa980395f59da3 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 29 Jan 2026 16:47:40 -0500 Subject: [PATCH 143/165] Update wasi-libc (#584) Removes the `_start` export for the `wasm32-wasip2` target, purging the use of the adapter. --- cmake/wasi-sdk-sysroot.cmake | 1 - src/wasi-libc | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 5bd04bb36..72088d979 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -173,7 +173,6 @@ function(define_wasi_libc_sub target target_suffix lto) -DCMAKE_C_FLAGS=${extra_cflags} -DBUILTINS_LIB=${libcompiler_rt_a} -DUSE_WASM_COMPONENT_LD=OFF - -DBINDINGS_TARGET=OFF -DWASI_SDK_VERSION=${wasi_sdk_version} DEPENDS compiler-rt EXCLUDE_FROM_ALL ON diff --git a/src/wasi-libc b/src/wasi-libc index 6036a9f3e..ec0effd76 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 6036a9f3e349315faba2732a32508d5dc9d6d928 +Subproject commit ec0effd769df5f05b647216578bcf5d3b5449725 From 372ba0d2222bcc7fb5e8f032afa36ad896c9d94f Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 12 Feb 2026 14:45:31 -0600 Subject: [PATCH 144/165] Add experimental support for C++ exceptions (#590) > **Note**: this PR is me weaving together the work of many others. For example @yerzham's and @cpetig's work on #565 pretty much shaped this PR. Thank you! This commit adds initial configuration support for building wasi-sdk with support for C++ exceptions in WebAssembly. A small test is included which is exercised in CI at this time, but otherwise this does not update CI to actually ship sdk builds with C++ exceptions enabled. Instead the intention here is to make it easier to test builds with support for C++ exceptions and centralize planning/development around this. The goal here is to get things compiling to the point that applications can be compiled. I haven't thoroughly tested C++ exceptions and as evident in this PR it still requires changes in LLVM. Some small logic is added here to apply a `*.patch` file to LLVM to avoid needing a new submodule fork or waiting for upstream changes. The intention is that this `*.patch` is short-lived once the changes are officially merged into LLVM itself. The `*.patch` included here contains llvm/llvm-project#168449 as well as another minor edit I found was necessary to get things compiling locally. Given the discussion on that PR it looks like once LLVM is updated with that merged the extra part of the `*.patch` won't be necessary. This PR notably does not enable shared libraries when exceptions are enabled. I don't know enough about how things are supposed to work to be able to fully diagnose the compilation errors I'm seeing if shared libraries are enabled. This is something I'd hope would be fixed before actually shipping exceptions support. This PR then additionally folds in [this gist][gist] for various bits of build logic to this repository itself. Finally, this PR includes some documentation about the current status of exceptions and links to various tracking issues too. [gist]: https://gist.github.com/yerzham/302efcec6a2e82c1e8de4aed576ea29d --- .github/actions/install-deps/action.yml | 2 +- .github/workflows/main.yml | 11 ++++- CppExceptions.md | 60 +++++++++++++++++++++++++ README.md | 4 ++ cmake/wasi-sdk-sysroot.cmake | 29 ++++++++++-- src/llvm-pr-168449.patch | 28 ++++++++++++ tests/CMakeLists.txt | 18 +++++++- tests/general/exceptions.cc | 17 +++++++ tests/testcase.sh | 2 +- 9 files changed, 162 insertions(+), 9 deletions(-) create mode 100644 CppExceptions.md create mode 100644 src/llvm-pr-168449.patch create mode 100644 tests/general/exceptions.cc diff --git a/.github/actions/install-deps/action.yml b/.github/actions/install-deps/action.yml index 2e4967a09..e3d00f823 100644 --- a/.github/actions/install-deps/action.yml +++ b/.github/actions/install-deps/action.yml @@ -7,7 +7,7 @@ runs: - name: Setup `wasmtime` for tests uses: bytecodealliance/actions/wasmtime/setup@v1 with: - version: "29.0.1" + version: "41.0.3" - name: Install ccache, ninja (macOS) run: brew install ccache ninja if: runner.os == 'macOS' diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 0e2df4b34..88115754c 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -155,8 +155,14 @@ jobs: key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }} build-only-sysroot: - name: Build only sysroot + name: Build only sysroot - ${{ matrix.name }} runs-on: ubuntu-24.04 + strategy: + matrix: + include: + - name: default + - name: exceptions + defines: -DWASI_SDK_EXCEPTIONS=ON steps: - uses: actions/checkout@v4 with: @@ -172,7 +178,8 @@ jobs: -DWASI_SDK_INCLUDE_TESTS=ON \ -DWASI_SDK_CPU_CFLAGS="" \ -DCMAKE_C_LINKER_DEPFILE_SUPPORTED=OFF \ - -DCMAKE_CXX_LINKER_DEPFILE_SUPPORTED=OFF + -DCMAKE_CXX_LINKER_DEPFILE_SUPPORTED=OFF \ + ${{ matrix.defines }} - run: ninja -C build - run: ctest --output-on-failure --parallel 10 --test-dir build/tests diff --git a/CppExceptions.md b/CppExceptions.md new file mode 100644 index 000000000..7ea1e2d8c --- /dev/null +++ b/CppExceptions.md @@ -0,0 +1,60 @@ +# Support for C++ Exceptions + +The released artifacts for wasi-sdk at this time do not support C++ exceptions. +LLVM and Clang, however, have support for C++ exceptions in WebAssembly and this +is intended to serve as documentation of the current state of affairs of using +C++ exceptions. It should be noted though that the current status of C++ +exceptions support is not intended to be the final state of support, and this is +all continuing to be iterated on over time. + +## Building wasi-sdk with exceptions + +When building the sysroot with wasi-sdk you can pass `-DWASI_SDK_EXCEPTIONS=ON` +to enable support for C++ exceptions. For example: + +```shell script +$ cmake -G Ninja -B build/sysroot -S . \ + -DCMAKE_TOOLCHAIN_FILE=$path/to/wasi-sdk-p1.cmake \ + -DWASI_SDK_EXCEPTIONS=ON +``` + +The C++ standard library will be compiled with support for exceptions for the +desired targets and the resulting sysroot supports using exceptions. + +## Compiling code with C++ exceptions + +Currently extra compilation flags are required to fully support C++ exceptions. +Without these flags programs using C++ exceptions will not work correctly: + +* `-fwasm-exceptions` - needed to enable the WebAssembly exception-handling + proposal. +* `-mllvm -wasm-use-legacy-eh=false` - indicates that the standard WebAssembly + exception-handling instructions should be used. +* `-lunwind` - links in support for unwinding which C++ exceptions requires. + +This can be specified for example with: + +```shell script +$ export CFLAGS="-fwasm-exceptions -mllvm -wasm-use-legacy-eh=false" +$ export LDFLAGS="-lunwind" +``` + +## Limitations + +Currently C++ exceptions support in wasi-sdk does not support shared libraries. +Fixing this will require resolving some miscellaneous build issues in this +repository itself. + +## Future Plans + +There are a few tracking issues with historical discussion about C++ exceptions +support in wasi-sdk such as [#334](https://github.com/WebAssembly/wasi-sdk/issues/334) +and [#565](https://github.com/WebAssembly/wasi-sdk/issues/565). The major +remaining items are: + +* Figure out support for shared libraries. +* Determine how to ship a sysroot that supports both with-and-without + exceptions. +* Figure out how to avoid the need for extra compiler flags when using + exceptions. +* Figure out if a new wasm target is warranted. diff --git a/README.md b/README.md index 014016dee..97bb4f559 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,10 @@ in compiling WebAssembly code. Supported CMake flags are: * `-DWASI_SDK_INCLUDE_TESTS=ON` - used for building tests. * `-DWASI_SDK_CPU_CFLAGS=..` - used to specify CFLAGS to tweak wasm features to enable/disable. The default is `-mcpu=lime1`. +* `-DWASI_SDK_LTO=ON` - whether to enable/disable builds of LTO-capable + libraries as part of the build. +* `-DWASI_SDK_EXCEPTIONS=ON` - whether to enable/disable support for C++ + exceptions, see [CppExceptions.md](./CppExceptions.md) for more information. * `-DWASI_SDK_TEST_HOST_TOOLCHAIN=ON` - test the host toolchain's wasi-libc and sysroot libraries, don't build or use fresh libraries for tests. * `-DWASI_SDK_TARGETS=..` - a list of targets to build, by default all WASI diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 72088d979..7be2a18ec 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -24,6 +24,7 @@ option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) option(WASI_SDK_INSTALL_TO_CLANG_RESOURCE_DIR "Whether or not to modify the compiler's resource directory" OFF) option(WASI_SDK_LTO "Whether or not to build LTO assets" ON) +option(WASI_SDK_EXCEPTIONS "Whether or not C++ exceptions are enabled" OFF) set(WASI_SDK_CPU_CFLAGS "-mcpu=lime1" CACHE STRING "CFLAGS to specify wasm features to enable") set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) @@ -89,6 +90,8 @@ function(define_compiler_rt target) -DCOMPILER_RT_BUILD_GWP_ASAN=OFF -DCMAKE_C_COMPILER_TARGET=${target} -DCMAKE_C_FLAGS=${WASI_SDK_CPU_CFLAGS} + -DCMAKE_CXX_FLAGS=${WASI_SDK_CPU_CFLAGS} + -DCMAKE_ASM_FLAGS=${WASI_SDK_CPU_CFLAGS} -DCMAKE_INSTALL_PREFIX=${wasi_resource_dir} EXCLUDE_FROM_ALL ON USES_TERMINAL_CONFIGURE ON @@ -231,6 +234,17 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ --sysroot ${wasi_sysroot} -resource-dir ${wasi_resource_dir}) + if (WASI_SDK_EXCEPTIONS) + # TODO: lots of builds fail with shared libraries and `-fPIC`. Looks like + # things are maybe changing in llvm/llvm-project#159143 but otherwise I'm at + # least not really sure what the state of shared libraries and exceptions + # are. For now shared libraries are disabled and supporting them is left for + # a future endeavor. + set(pic OFF) + set(runtimes "libunwind;${runtimes}") + list(APPEND extra_flags -fwasm-exceptions -mllvm -wasm-use-legacy-eh=false) + endif() + set(extra_cflags_list ${CMAKE_C_FLAGS} ${extra_flags}) list(JOIN extra_cflags_list " " extra_cflags) set(extra_cxxflags_list ${CMAKE_CXX_FLAGS} ${extra_flags}) @@ -254,14 +268,14 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DLLVM_COMPILER_CHECKED=ON -DLIBCXX_ENABLE_SHARED:BOOL=${pic} -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF - -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF + -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=${WASI_SDK_EXCEPTIONS} -DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF -DLIBCXX_CXX_ABI=libcxxabi -DLIBCXX_CXX_ABI_INCLUDE_PATHS=${llvm_proj_dir}/libcxxabi/include -DLIBCXX_HAS_MUSL_LIBC:BOOL=ON -DLIBCXX_ABI_VERSION=2 - -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF + -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=${WASI_SDK_EXCEPTIONS} -DLIBCXXABI_ENABLE_SHARED:BOOL=${pic} -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON -DLIBCXXABI_ENABLE_THREADS:BOOL=ON @@ -270,12 +284,18 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF -DLIBCXXABI_ENABLE_PIC:BOOL=${pic} - -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=OFF + -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=${WASI_SDK_EXCEPTIONS} + -DLIBUNWIND_ENABLE_SHARED:BOOL=${pic} + -DLIBUNWIND_ENABLE_THREADS:BOOL=ON + -DLIBUNWIND_USE_COMPILER_RT:BOOL=ON + -DLIBUNWIND_INCLUDE_TESTS:BOOL=OFF -DUNIX:BOOL=ON -DCMAKE_C_FLAGS=${extra_cflags} + -DCMAKE_ASM_FLAGS=${extra_cflags} -DCMAKE_CXX_FLAGS=${extra_cxxflags} -DLIBCXX_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} -DLIBCXXABI_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} + -DLIBUNWIND_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} -DLIBCXX_INCLUDE_TESTS=OFF -DLIBCXX_INCLUDE_BENCHMARKS=OFF @@ -290,6 +310,9 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ USES_TERMINAL_CONFIGURE ON USES_TERMINAL_BUILD ON USES_TERMINAL_INSTALL ON + PATCH_COMMAND + ${CMAKE_COMMAND} -E chdir .. bash -c + "git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-168449.patch || git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-168449.patch -R --check" ) endfunction() diff --git a/src/llvm-pr-168449.patch b/src/llvm-pr-168449.patch new file mode 100644 index 000000000..a0da1d8d4 --- /dev/null +++ b/src/llvm-pr-168449.patch @@ -0,0 +1,28 @@ +diff --git a/libunwind/src/assembly.h b/libunwind/src/assembly.h +index f8e83e138eff..c5097d25b0c6 100644 +--- a/libunwind/src/assembly.h ++++ b/libunwind/src/assembly.h +@@ -249,6 +249,9 @@ aliasname: \ + #define WEAK_ALIAS(name, aliasname) + #define NO_EXEC_STACK_DIRECTIVE + ++#elif defined(__wasm__) ++#define NO_EXEC_STACK_DIRECTIVE ++ + // clang-format on + #else + +diff --git a/libunwind/src/config.h b/libunwind/src/config.h +index deb5a4d4d73d..23c9f012cbcf 100644 +--- a/libunwind/src/config.h ++++ b/libunwind/src/config.h +@@ -66,7 +66,8 @@ + #define _LIBUNWIND_EXPORT + #define _LIBUNWIND_HIDDEN + #else +- #if !defined(__ELF__) && !defined(__MACH__) && !defined(_AIX) ++ #if !defined(__ELF__) && !defined(__MACH__) && !defined(_AIX) && \ ++ !defined(__wasm__) + #define _LIBUNWIND_EXPORT __declspec(dllexport) + #define _LIBUNWIND_HIDDEN + #else diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index edbb1290d..a2133923b 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -67,7 +67,12 @@ function(add_testcase runwasi test) # Apply language-specific options and dependencies. if(test MATCHES "cc$") - target_compile_options(${target_name} PRIVATE -fno-exceptions) + if(WASI_SDK_EXCEPTIONS) + target_compile_options(${target_name} PRIVATE -fwasm-exceptions -mllvm -wasm-use-legacy-eh=false) + target_link_options(${target_name} PRIVATE -lunwind) + else() + target_compile_options(${target_name} PRIVATE -fno-exceptions) + endif() if(NOT WASI_SDK_TEST_HOST_TOOLCHAIN) add_dependencies(${target_name} libcxx-${target}) endif() @@ -84,11 +89,20 @@ function(add_testcase runwasi test) endif() if(runwasi) + set(runner ${runwasi}) + if(${runner} MATCHES wasmtime) + if(target MATCHES threads) + set(runner "${runner} -Wshared-memory") + endif() + if(WASI_SDK_EXCEPTIONS) + set(runner "${runner} -Wexceptions") + endif() + endif() add_test( NAME test-${target_name} COMMAND bash ../testcase.sh - "${runwasi}" + ${runner} ${test} $ WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}) diff --git a/tests/general/exceptions.cc b/tests/general/exceptions.cc new file mode 100644 index 000000000..3daed98ce --- /dev/null +++ b/tests/general/exceptions.cc @@ -0,0 +1,17 @@ +#include +#include + +int main() { +#ifdef __wasm_exception_handling__ + try { + throw std::runtime_error("An error occurred"); + abort(); + } catch (const std::runtime_error& e) { + // .. + return 0; + } + abort(); +#else + return 0; +#endif +} diff --git a/tests/testcase.sh b/tests/testcase.sh index 551eec32d..d2c5a8c34 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -46,7 +46,7 @@ fi # Run the test, capturing stdout, stderr, and the exit status. exit_status=0 -"$runwasi" $env $dir "$wasm" $dirarg \ +$runwasi $env $dir "$wasm" $dirarg \ < "$stdin" \ > "$stdout_observed" \ 2> "$stderr_observed" \ From 1033443e5c3696389c66157877d0e1a491bd43fc Mon Sep 17 00:00:00 2001 From: Roman Volosatovs Date: Fri, 13 Feb 2026 16:11:24 +0100 Subject: [PATCH 145/165] Update `wasm-component-ld` in CI (#592) Use latest `wasm-component-ld` in CI Signed-off-by: Roman Volosatovs --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 88115754c..f42012144 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -169,7 +169,7 @@ jobs: fetch-depth: 0 - uses: ./.github/actions/checkout - uses: ./.github/actions/install-deps - - run: cargo install wasm-component-ld@0.5.12 + - run: cargo install wasm-component-ld@0.5.20 - run: sudo apt-get update -y && sudo apt-get install -y clang-20 lld-20 - run: | cmake -G Ninja -B build -S . \ From a64d51d14378e306add001fad180d5733df1491c Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 23 Feb 2026 12:11:22 -0600 Subject: [PATCH 146/165] Update wasm-component-ld (#594) Keeping it up-to-date --- .github/workflows/main.yml | 2 +- cmake/wasi-sdk-toolchain.cmake | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index f42012144..9b39e65b8 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -169,7 +169,7 @@ jobs: fetch-depth: 0 - uses: ./.github/actions/checkout - uses: ./.github/actions/install-deps - - run: cargo install wasm-component-ld@0.5.20 + - run: cargo install wasm-component-ld@0.5.21 - run: sudo apt-get update -y && sudo apt-get install -y clang-20 lld-20 - run: | cmake -G Ninja -B build -S . \ diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index fa972b9de..c02560d6b 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -119,7 +119,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.20) +set(wasm_component_ld_version 0.5.21) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From b7d8add13bc2703a3f892b7670b1e12bd2b16d26 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 3 Mar 2026 09:23:05 -0600 Subject: [PATCH 147/165] Update to LLVM 22.1.0 (#585) Keeping LLVM up-to-date. Closes #595 --- cmake/wasi-sdk-sysroot.cmake | 29 ++++++++++++++++++----------- src/llvm-project | 2 +- tests/CMakeLists.txt | 5 +++++ 3 files changed, 24 insertions(+), 12 deletions(-) diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 7be2a18ec..c19d0c60f 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -101,7 +101,7 @@ function(define_compiler_rt target) add_dependencies(compiler-rt-build compiler-rt-build-${target}) endfunction() -define_compiler_rt(wasm32-wasi) +define_compiler_rt(wasm32-wasip1) define_compiler_rt(wasm32-wasip1-threads) # In addition to the default installation of `compiler-rt` itself also copy @@ -118,12 +118,12 @@ add_custom_target(compiler-rt-post-build COMMAND ${CMAKE_COMMAND} -E copy_directory ${clang_resource_dir}/include ${wasi_resource_dir}/include - # Copy the `lib/wasm32-unknown-wasi` folder to `lib/wasm32-unknown-wasi{p1,p2}` to ensure that those + # Copy the `lib/wasm32-unknown-wasip1` folder to `lib/wasm32-unknown-wasi{,p2}` to ensure that those # OS-strings also work for looking up the compiler-rt.a file. COMMAND ${CMAKE_COMMAND} -E copy_directory - ${wasi_resource_dir}/lib/wasm32-unknown-wasi ${wasi_resource_dir}/lib/wasm32-unknown-wasip1 + ${wasi_resource_dir}/lib/wasm32-unknown-wasip1 ${wasi_resource_dir}/lib/wasm32-unknown-wasi COMMAND ${CMAKE_COMMAND} -E copy_directory - ${wasi_resource_dir}/lib/wasm32-unknown-wasi ${wasi_resource_dir}/lib/wasm32-unknown-wasip2 + ${wasi_resource_dir}/lib/wasm32-unknown-wasip1 ${wasi_resource_dir}/lib/wasm32-unknown-wasip2 # Copy the `lib/wasm32-unknown-wasip1-threads` folder to `lib/wasm32-unknown-wasi-threads` COMMAND ${CMAKE_COMMAND} -E copy_directory ${wasi_resource_dir}/lib/wasm32-unknown-wasip1-threads ${wasi_resource_dir}/lib/wasm32-unknown-wasi-threads @@ -150,6 +150,12 @@ function(define_wasi_libc_sub target target_suffix lto) list(APPEND extra_cflags_list -fPIC) endif() + # The `wasm32-wasi` target is deprecated in clang, so ignore the deprecation + # warnings for now. + if(${target} STREQUAL wasm32-wasi OR ${target} STREQUAL wasm32-wasi-threads) + list(APPEND extra_cflags_list -Wno-deprecated) + endif() + list(JOIN extra_cflags_list " " extra_cflags) if(${target} MATCHES threads) @@ -174,6 +180,7 @@ function(define_wasi_libc_sub target target_suffix lto) -DTARGET_TRIPLE=${target} -DCMAKE_INSTALL_PREFIX=${wasi_sysroot} -DCMAKE_C_FLAGS=${extra_cflags} + -DCMAKE_ASM_FLAGS=${extra_cflags} -DBUILTINS_LIB=${libcompiler_rt_a} -DUSE_WASM_COMPONENT_LD=OFF -DWASI_SDK_VERSION=${wasi_sdk_version} @@ -245,6 +252,12 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ list(APPEND extra_flags -fwasm-exceptions -mllvm -wasm-use-legacy-eh=false) endif() + # The `wasm32-wasi` target is deprecated in clang, so ignore the deprecation + # warnings for now. + if(${target} STREQUAL wasm32-wasi OR ${target} STREQUAL wasm32-wasi-threads) + list(APPEND extra_flags -Wno-deprecated) + endif() + set(extra_cflags_list ${CMAKE_C_FLAGS} ${extra_flags}) list(JOIN extra_cflags_list " " extra_cflags) set(extra_cxxflags_list ${CMAKE_CXX_FLAGS} ${extra_flags}) @@ -259,21 +272,17 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DCMAKE_INSTALL_INCLUDEDIR=${wasi_sysroot}/include/${target} -DCMAKE_STAGING_PREFIX=${wasi_sysroot} -DCMAKE_POSITION_INDEPENDENT_CODE=${pic} - -DCXX_SUPPORTS_CXX11=ON -DLIBCXX_ENABLE_THREADS:BOOL=ON -DLIBCXX_HAS_PTHREAD_API:BOOL=ON -DLIBCXX_HAS_EXTERNAL_THREAD_API:BOOL=OFF - -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF -DLLVM_COMPILER_CHECKED=ON -DLIBCXX_ENABLE_SHARED:BOOL=${pic} - -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=${WASI_SDK_EXCEPTIONS} -DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF -DLIBCXX_CXX_ABI=libcxxabi - -DLIBCXX_CXX_ABI_INCLUDE_PATHS=${llvm_proj_dir}/libcxxabi/include - -DLIBCXX_HAS_MUSL_LIBC:BOOL=ON + -DLIBCXX_HAS_MUSL_LIBC:BOOL=OFF -DLIBCXX_ABI_VERSION=2 -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=${WASI_SDK_EXCEPTIONS} -DLIBCXXABI_ENABLE_SHARED:BOOL=${pic} @@ -281,9 +290,7 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DLIBCXXABI_ENABLE_THREADS:BOOL=ON -DLIBCXXABI_HAS_PTHREAD_API:BOOL=ON -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF - -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF - -DLIBCXXABI_ENABLE_PIC:BOOL=${pic} -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=${WASI_SDK_EXCEPTIONS} -DLIBUNWIND_ENABLE_SHARED:BOOL=${pic} -DLIBUNWIND_ENABLE_THREADS:BOOL=ON diff --git a/src/llvm-project b/src/llvm-project index 222fc11f2..4434dabb6 160000 --- a/src/llvm-project +++ b/src/llvm-project @@ -1 +1 @@ -Subproject commit 222fc11f2b8f25f6a0f4976272ef1bb7bf49521d +Subproject commit 4434dabb69916856b824f68a64b029c67175e532 diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index a2133923b..3ed477658 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -88,6 +88,11 @@ function(add_testcase runwasi test) target_link_options(${target_name} PRIVATE -pthread) endif() + if(target STREQUAL wasm32-wasi OR target STREQUAL wasm32-wasi-threads) + target_compile_options(${target_name} PRIVATE -Wno-deprecated) + target_link_options(${target_name} PRIVATE -Wno-deprecated) + endif() + if(runwasi) set(runner ${runwasi}) if(${runner} MATCHES wasmtime) From 3f0e04a053bc79cc4c71e0e872a3079fd340576f Mon Sep 17 00:00:00 2001 From: Roman Volosatovs Date: Tue, 3 Mar 2026 23:40:51 +0100 Subject: [PATCH 148/165] Build `wasip3` target (#593) Build `wasip3` target ~Requires #592~ There's a lot of failing tests currently, so we'll probably want to rebase this PR as we pull in more recent versions of `wasi-libc` and merge once tests pass --------- Signed-off-by: Roman Volosatovs Co-authored-by: Alex Crichton --- cmake/wasi-sdk-sysroot.cmake | 20 +++++++++++++----- cmake/wasi-sdk-toolchain.cmake | 1 + tests/CMakeLists.txt | 3 +++ wasi-sdk-p3.cmake | 37 ++++++++++++++++++++++++++++++++++ 4 files changed, 56 insertions(+), 5 deletions(-) create mode 100644 wasi-sdk-p3.cmake diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index c19d0c60f..38f804c73 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -101,12 +101,23 @@ function(define_compiler_rt target) add_dependencies(compiler-rt-build compiler-rt-build-${target}) endfunction() +# The `compiler-rt` for `wasm32-wasip1` will be reused for `wasm32-wasip2` and +# `wasm32-wasi`. The version for `wasm32-wasip1-threads` will be reused for +# `wasm32-wasi-threads`. Different builds are needed for different codegen flags +# and such across the threaded/not target. define_compiler_rt(wasm32-wasip1) define_compiler_rt(wasm32-wasip1-threads) +# If a p3 target is requested, also build compiler-rt for that target. WASIp3 +# will eventually have a different ABI than wasm32-wasip2, so this separate +# build is needed. +if(WASI_SDK_TARGETS MATCHES p3) + define_compiler_rt(wasm32-wasip3) +endif() + # In addition to the default installation of `compiler-rt` itself also copy # around some headers and make copies of the `wasi` directory as `wasip1` and -# `wasip2` +# `wasip2` and `wasip3` execute_process( COMMAND ${CMAKE_C_COMPILER} -print-resource-dir OUTPUT_VARIABLE clang_resource_dir @@ -134,7 +145,6 @@ add_dependencies(compiler-rt-post-build compiler-rt-build) add_custom_target(compiler-rt DEPENDS compiler-rt-build compiler-rt-post-build) - # ============================================================================= # wasi-libc build logic # ============================================================================= @@ -144,9 +154,9 @@ function(define_wasi_libc_sub target target_suffix lto) get_property(directory_cflags DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) set(extra_cflags_list "${WASI_SDK_CPU_CFLAGS} ${CMAKE_C_FLAGS} ${directory_cflags}") - if(${target} MATCHES p2) - # Always enable `-fPIC` for the `wasm32-wasip2` target. This makes `libc.a` - # more flexible and usable in dynamic linking situations. + if(${target} MATCHES "p[23]") + # Always enable `-fPIC` for the `wasm32-wasip2` and `wasm32-wasip3` targets. + # This makes `libc.a` more flexible and usable in dynamic linking situations. list(APPEND extra_cflags_list -fPIC) endif() diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index c02560d6b..848bd264f 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -156,6 +156,7 @@ copy_misc_file(wasi-sdk.cmake cmake) copy_misc_file(wasi-sdk-pthread.cmake cmake) copy_misc_file(wasi-sdk-p1.cmake cmake) copy_misc_file(wasi-sdk-p2.cmake cmake) +copy_misc_file(wasi-sdk-p3.cmake cmake) copy_misc_file(cmake/Platform/WASI.cmake cmake/Platform) function(copy_cfg_file compiler) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 3ed477658..c573aea65 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -102,6 +102,9 @@ function(add_testcase runwasi test) if(WASI_SDK_EXCEPTIONS) set(runner "${runner} -Wexceptions") endif() + if(target MATCHES "wasip3") + set(runner "${runner} -Wcomponent-model-async -Sp3") + endif() endif() add_test( NAME test-${target_name} diff --git a/wasi-sdk-p3.cmake b/wasi-sdk-p3.cmake new file mode 100644 index 000000000..f73756225 --- /dev/null +++ b/wasi-sdk-p3.cmake @@ -0,0 +1,37 @@ +# Cmake toolchain description file for the Makefile + +# Until Platform/WASI.cmake is upstream we need to inject the path to it +# into CMAKE_MODULE_PATH. +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}") + +set(CMAKE_SYSTEM_NAME WASI) +set(CMAKE_SYSTEM_VERSION 1) +set(CMAKE_SYSTEM_PROCESSOR wasm32) +set(triple wasm32-wasip3) + +if(WIN32) + set(WASI_HOST_EXE_SUFFIX ".exe") +else() + set(WASI_HOST_EXE_SUFFIX "") +endif() + +# When building from source, WASI_SDK_PREFIX represents the generated directory +if(NOT WASI_SDK_PREFIX) + set(WASI_SDK_PREFIX ${CMAKE_CURRENT_LIST_DIR}/../../) +endif() + +set(CMAKE_C_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_CXX_COMPILER ${WASI_SDK_PREFIX}/bin/clang++${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_ASM_COMPILER ${WASI_SDK_PREFIX}/bin/clang${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_AR ${WASI_SDK_PREFIX}/bin/llvm-ar${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_RANLIB ${WASI_SDK_PREFIX}/bin/llvm-ranlib${WASI_HOST_EXE_SUFFIX}) +set(CMAKE_C_COMPILER_TARGET ${triple}) +set(CMAKE_CXX_COMPILER_TARGET ${triple}) +set(CMAKE_ASM_COMPILER_TARGET ${triple}) + +# Don't look in the sysroot for executables to run during the build +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) +# Only look in the sysroot (not in the host paths) for the rest +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) From 9713459bcf52ca2b9bf8cecdef1eff73c332ba6b Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 3 Mar 2026 17:24:00 -0600 Subject: [PATCH 149/165] Include `libclang.so` in wasi-sdk (#598) This includes a new library, `libclang.so`, which is Clang's public C API. This is used by tooling such as Rust's `bindgen` library to generate bindings and can be useful when bindings are generated using a wasi-sdk distribution. --- cmake/wasi-sdk-toolchain.cmake | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 848bd264f..1cc373dfa 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -57,7 +57,8 @@ set(tools objdump objcopy c++filt - llvm-config) + llvm-config + libclang) # By default link LLVM dynamically to all the various tools. This greatly # reduces the binary size of all the tools through a shared library rather than From eacaeb6be38eb91806983e8494bf3854766f899b Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 3 Mar 2026 18:58:44 -0600 Subject: [PATCH 150/165] Update wasi-libc submodule (#599) Notably pull in some fixes for the wasip2 target related to networking. --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index ec0effd76..58fa18a96 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit ec0effd769df5f05b647216578bcf5d3b5449725 +Subproject commit 58fa18a968de915bea7d962485d9838693779bf9 From 4ec071867cec6aaee9a1010885b9ea06d322c732 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 4 Mar 2026 19:04:20 -0600 Subject: [PATCH 151/165] Build LLDB as part of wasi-sdk (#596) This is intended to provide a known-good build of LLDB with wasm plugin support which can exist alongside the rest of the LLVM distribution provided by wasi-sdk. --- ci/docker/Dockerfile | 3 +- cmake/wasi-sdk-toolchain.cmake | 76 ++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 1 deletion(-) diff --git a/ci/docker/Dockerfile b/ci/docker/Dockerfile index a5da2c839..71f4a1d61 100644 --- a/ci/docker/Dockerfile +++ b/ci/docker/Dockerfile @@ -12,7 +12,8 @@ RUN dnf install -y \ python3 \ git \ unzip \ - cmake + cmake \ + ncurses-devel COPY ./install-ccache.sh . RUN ./install-ccache.sh diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 1cc373dfa..04c0a5d12 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -5,6 +5,21 @@ set(LLVM_CMAKE_FLAGS "" CACHE STRING "Extra cmake flags to pass to LLVM's build" set(RUST_TARGET "" CACHE STRING "Target to build Rust code for, if not the host") set(WASI_SDK_ARTIFACT "" CACHE STRING "Name of the wasi-sdk artifact being produced") +option(WASI_SDK_LLDB "Include a build of LLDB" ON) + +set(LIBEDIT_DEFAULT ON) +# I don't want to deal with running a `./configure` script on Windows, disable +# it by default. +if(WIN32) + set(LIBEDIT_DEFAULT OFF) +endif() +# I don't know how to resolve build failures when building libedit for x86_64 +# from arm64 on macos, so disable it for now. +if(CMAKE_SYSTEM_PROCESSOR MATCHES "arm64" AND LLVM_CMAKE_FLAGS MATCHES "x86_64") + set(LIBEDIT_DEFAULT OFF) +endif() +option(WASI_SDK_LIBEDIT "Whether or not to build libedit for LLDB" ${LIBEDIT_DEFAULT}) + string(REGEX REPLACE "[ ]+" ";" llvm_cmake_flags_list "${LLVM_CMAKE_FLAGS}") set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) @@ -73,6 +88,66 @@ if(NOT WIN32) list(APPEND tools LLVM clang-cpp) endif() +# Configure/add LLDB if requested. +# +# Note that LLDB depends on `libedit` which is more-or-less required to get a +# reasonable command-line experience, so this is built custom here to ensure +# that it's available for LLDB. +if(WASI_SDK_LLDB) + list(APPEND projects lldb) + list(APPEND tools lldb liblldb) + list(APPEND default_cmake_args + -DLLDB_INCLUDE_TESTS=OFF + -DLLDB_INCLUDE_UNITTESTS=OFF + -DLLDB_ENABLE_SWIG=OFF + -DLLDB_ENABLE_CURSES=OFF + -DLLDB_ENABLE_LZMA=OFF + -DLLDB_ENABLE_LUA=OFF + -DLLDB_ENABLE_PYTHON=OFF + -DLLDB_ENABLE_LIBXML2=OFF + -DLLDB_ENABLE_FBSDVMCORE=OFF + -DLLDB_ENABLE_LINUXPTY=OFF + ) + + if (WASI_SDK_LIBEDIT) + include(ProcessorCount) + ProcessorCount(nproc) + find_program(MAKE_EXECUTABLE make REQUIRED) + ExternalProject_Add(libedit + URL https://thrysoee.dk/editline/libedit-20251016-3.1.tar.gz + URL_HASH SHA256=21362b00653bbfc1c71f71a7578da66b5b5203559d43134d2dd7719e313ce041 + + # Without this the build system tries to find and use `aclocal-1.18` where + # with this it doesn't so turn this on. + DOWNLOAD_EXTRACT_TIMESTAMP ON + + CONFIGURE_COMMAND + /configure + --prefix=${wasi_tmp_install} + --enable-pic + --disable-examples + CC=${CMAKE_C_COMPILER} + CFLAGS=${libedit_cflags} + LDFLAGS=${libedit_cflags} + BUILD_COMMAND + ${MAKE_EXECUTABLE} -j${nproc} V=1 + + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON + ) + list(APPEND default_cmake_args + -DLLDB_ENABLE_LIBEDIT=ON + -DLibEdit_ROOT=${wasi_tmp_install} + ) + else() + list(APPEND default_cmake_args -DLLDB_ENABLE_LIBEDIT=OFF) + add_custom_target(libedit) + endif() +else() + add_custom_target(libedit) +endif() + list(TRANSFORM tools PREPEND --target= OUTPUT_VARIABLE build_targets) list(TRANSFORM tools PREPEND --target=install- OUTPUT_VARIABLE install_targets) @@ -110,6 +185,7 @@ ExternalProject_Add(llvm-build ) add_custom_target(build ALL DEPENDS llvm-build) +ExternalProject_Add_StepDependencies(llvm-build configure libedit) # Installation target for this outer project for installing the toolchain to the # system. From ab2be6e0dfa96160921e6b2f68e67a3e26ba32cb Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 5 Mar 2026 09:43:29 -0600 Subject: [PATCH 152/165] Adjust documentation of limitations in README (#591) * Update for C++ Exceptions being supported. * Update the threading-related bits slightly. * Indicate that dynamic linking works but may not be as fully tested. * Networking works, just not on WASIp1. --- README.md | 59 ++++++++++++++++++++++--------------------------------- 1 file changed, 24 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index 97bb4f559..7d28f6957 100644 --- a/README.md +++ b/README.md @@ -209,41 +209,30 @@ disabled in a configure step before building with WASI SDK. ## Notable Limitations -This repository does not yet support __C++ exceptions__. C++ code is supported -only with -fno-exceptions for now. -Work on support for [exception handling] is underway at the -language level which will support the features. - -[exception handling]: https://github.com/WebAssembly/exception-handling/ - -See [C setjmp/longjmp support] about setjmp/longjmp support. - -[C setjmp/longjmp support]: SetjmpLongjmp.md - -This repository experimentally supports __threads__ with -`--target=wasm32-wasip1-threads`. It uses WebAssembly's [threads] primitives -(atomics, `wait`/`notify`, shared memory) and [wasi-threads] for spawning -threads. Note: this is experimental — do not expect long-term stability! - -Note that the `pthread_*` family of functions, as well as C++ threading primitives -such as ``, ``, and `` are available on all targets. -Any attempt to spawn a thread will fail on `--target=wasm32-wasip1` or -`--target=wasm32-wasip2`, but other functionality, such as locks, still works. -This makes it easier to port C++ codebases, as only a fraction of code needs -to be modified to build for the single-threaded targets. - -Defining a macro `_WASI_STRICT_PTHREAD` will make `pthread_create`, -`pthread_detach`, `pthread_join`, `pthread_tryjoin_np`, and `pthread_timedjoin_np` -fail with a compile time error when building for single-threaded targets. +* C++ exceptions are disabled by default. For more information see + [CppExceptions.md]. +* C `setjmp`/`longjmp` require some extra configuration to get working, see + [SetjmpLongjmp.md]. +* Most targets do not support spawning a thread. Experimental support for + spawning threads is available with the `wasm32-wasip1-threads` target which + uses [wasi-threads]. Note that the `pthread_*` family of functions, as well as + C++ threading primitives such as ``, ``, and `` are + available on all targets. Defining a macro `_WASI_STRICT_PTHREAD` will make + `pthread_create`, `pthread_detach`, `pthread_join`, `pthread_tryjoin_np`, and + `pthread_timedjoin_np` fail with a compile time error when building for + single-threaded targets. +* Dynamic linking [is supported][dylink] but not as fully baked as static + linking. There might be obscure bugs in some situations related to dynamic + linking. +* The WASIp1 targets do not support networking, but WASIp2/WASIp3 support + networking. +* 64-bit linear memories (a "wasm64" target) are not supported at this time. + Supporting this will require resolving [WebAssembly/component-model#22] first + at which point it will be possible to add a `wasm64-wasip2` target. There are + no plans to add support for `wasm64-wasi{,-threads,p1,p1-threads}` at this + time. [threads]: https://github.com/WebAssembly/threads [wasi-threads]: https://github.com/WebAssembly/wasi-threads - -This repository does not yet support __dynamic libraries__. While there are -[some efforts] to design a system for dynamic libraries in wasm, it is still in -development and not yet generally usable. - -[some efforts]: https://github.com/WebAssembly/tool-conventions/blob/master/DynamicLinking.md - -There is no support for __networking__. It is a goal of WASI to support -networking in the future though. +[dylink]: https://github.com/WebAssembly/tool-conventions/blob/master/DynamicLinking.md +[WebAssembly/component-model#22]: https://github.com/WebAssembly/component-model/issues/22 From 4d474b1b7d22c28d1bb2217ad6fb89b2fea84a47 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 5 Mar 2026 11:02:26 -0600 Subject: [PATCH 153/165] Change Clang's default target to `wasm32-wasip1` (#600) No longer use `wasm32-wasi` since Clang issues a deprecation warning about that. This fixes the `clang` binary, by default, emitting a warning for example in the final build. --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 04c0a5d12..21cdb3907 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -163,7 +163,7 @@ ExternalProject_Add(llvm-build -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_TARGETS_TO_BUILD=WebAssembly - -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi + -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasip1 -DLLVM_INSTALL_BINUTILS_SYMLINKS=TRUE -DLLVM_ENABLE_LIBXML2=OFF # Pass `-s` to strip symbols by default and shrink the size of the From 77553563c8a2672d97fdb97c04923da3f89e8017 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 10 Mar 2026 18:03:42 -0500 Subject: [PATCH 154/165] Update wasi-libc (#602) Once more before wasi-sdk-31 --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index 58fa18a96..724d86644 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 58fa18a968de915bea7d962485d9838693779bf9 +Subproject commit 724d86644e1972fe99fcb598c5f2f9a179cfec10 From 2e6b578a6986005e29e01c10b94843fdb2a76f0a Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 11 Mar 2026 12:40:53 -0500 Subject: [PATCH 155/165] Update wasi-libc one more time (#604) Fix an issue with setjmp/longjmp when combined with LLVM 22. --- src/wasi-libc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/wasi-libc b/src/wasi-libc index 724d86644..2fc32bc81 160000 --- a/src/wasi-libc +++ b/src/wasi-libc @@ -1 +1 @@ -Subproject commit 724d86644e1972fe99fcb598c5f2f9a179cfec10 +Subproject commit 2fc32bc81b9f07f8d9525edea59bfbaf760c06d6 From 015c849728814e668669b6e1f74c782d0f6b150e Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 11 Mar 2026 17:18:33 -0500 Subject: [PATCH 156/165] Update github actions workflows (#605) Handle warnings cropping up in CI --- .github/workflows/main.yml | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 9b39e65b8..14d51b5b2 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -53,7 +53,7 @@ jobs: env: ${{ matrix.env || fromJSON('{}') }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 - uses: ./.github/actions/checkout @@ -65,7 +65,7 @@ jobs: # Bump the prefix number to evict all previous caches and enforce a clean # build, in the unlikely case that some weird build error occur and ccache # becomes a potential suspect. - - uses: actions/cache@v4 + - uses: actions/cache@v5 id: cache-restore with: path: ${{ runner.tool_cache }}/ccache @@ -125,7 +125,7 @@ jobs: # Upload the `dist` folder from the build as the artifacts for this # runner. - name: Upload artifacts - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v6 with: name: ${{ format( 'dist-{0}', matrix.artifact) }} path: build/dist @@ -149,7 +149,7 @@ jobs: # time instead of having to recreate everything each time a failure # happens. - if: always() && steps.cache-restore.outputs.cache-hit != 'true' - uses: actions/cache/save@v4 + uses: actions/cache/save@v5 with: path: ${{ runner.tool_cache }}/ccache key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }} @@ -164,7 +164,7 @@ jobs: - name: exceptions defines: -DWASI_SDK_EXCEPTIONS=ON steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 - uses: ./.github/actions/checkout @@ -191,16 +191,16 @@ jobs: needs: build runs-on: ubuntu-24.04 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 - uses: ./.github/actions/checkout # Download all artifacts from all platforms in `build`, merge them into # final wasi-sdk-* artifacts, and then upload them. - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@v8 - run: ./ci/merge-artifacts.sh - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@v6 with: name: release-artifacts path: dist @@ -251,12 +251,12 @@ jobs: needs: build runs-on: ubuntu-24.04 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 - uses: ./.github/actions/checkout - uses: ./.github/actions/install-deps - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@v8 with: name: dist-x86_64-linux path: dist-x86_64-linux From 5d5c6c0d1898912df6ed16e08eef5c667c46f151 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 11 Mar 2026 19:18:15 -0500 Subject: [PATCH 157/165] Attempt to fix finding libedit on macOS (#609) First pass at trying to fix this... --- cmake/wasi-sdk-toolchain.cmake | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 21cdb3907..56920f784 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -113,6 +113,9 @@ if(WASI_SDK_LLDB) include(ProcessorCount) ProcessorCount(nproc) find_program(MAKE_EXECUTABLE make REQUIRED) + if(CMAKE_SYSTEM_NAME STREQUAL Darwin) + set(libedit_ldflags -Wl,-install_name,@rpath/libedit.0.dylib) + endif() ExternalProject_Add(libedit URL https://thrysoee.dk/editline/libedit-20251016-3.1.tar.gz URL_HASH SHA256=21362b00653bbfc1c71f71a7578da66b5b5203559d43134d2dd7719e313ce041 @@ -127,8 +130,7 @@ if(WASI_SDK_LLDB) --enable-pic --disable-examples CC=${CMAKE_C_COMPILER} - CFLAGS=${libedit_cflags} - LDFLAGS=${libedit_cflags} + LDFLAGS=${libedit_ldflags} BUILD_COMMAND ${MAKE_EXECUTABLE} -j${nproc} V=1 @@ -169,6 +171,9 @@ ExternalProject_Add(llvm-build # Pass `-s` to strip symbols by default and shrink the size of the # distribution -DCMAKE_EXE_LINKER_FLAGS=-s + # Looks to be required on macOS for, at build time, the dynamic linker to + # find `libedit.dylib` when that's enabled. + -DCMAKE_BUILD_RPATH=${wasi_tmp_install}/lib ${llvm_cmake_flags_list} # See https://www.scivision.dev/cmake-externalproject-list-arguments/ for # why this is in `CMAKE_CACHE_ARGS` instead of above From 0991cf438b7064f87cbe1deb2d5fb6b7f64df150 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 12 Mar 2026 14:28:14 -0500 Subject: [PATCH 158/165] Fix C++ thread locals, add a test (#611) Pulls in llvm/llvm-project#186054 and adds a regression test which previously failed. Closes #610 --- cmake/wasi-sdk-sysroot.cmake | 3 ++ src/llvm-pr-186054.patch | 48 +++++++++++++++++++++++++++++++ tests/general/CMakeLists.txt | 4 +-- tests/general/cpp_thread_local.cc | 3 ++ 4 files changed, 56 insertions(+), 2 deletions(-) create mode 100644 src/llvm-pr-186054.patch create mode 100644 tests/general/cpp_thread_local.cc diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 38f804c73..38ed42636 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -330,6 +330,9 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ PATCH_COMMAND ${CMAKE_COMMAND} -E chdir .. bash -c "git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-168449.patch || git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-168449.patch -R --check" + COMMAND + ${CMAKE_COMMAND} -E chdir .. bash -c + "git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-186054.patch || git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-186054.patch -R --check" ) endfunction() diff --git a/src/llvm-pr-186054.patch b/src/llvm-pr-186054.patch new file mode 100644 index 000000000..c7756d77f --- /dev/null +++ b/src/llvm-pr-186054.patch @@ -0,0 +1,48 @@ +From f71fdfcbd6fcc7b521c74b5856ebeacdd6cf55d9 Mon Sep 17 00:00:00 2001 +From: Catherine +Date: Thu, 12 Mar 2026 08:19:49 +0000 +Subject: [PATCH] [libc++abi] Revert gating of `__cxa_thread_atexit` on + Linux||Fuchsia + +This was done in the commit 3c100d5d548d with the description +"Enable -Wmissing-prototypes" which seems incongruent to me. + +Since then it's made its way into a release and broke the use of +`thread_local` variables with destructors on Wasm/WASI: + +```cc +// repro.cc +struct c { ~c() {} }; +thread_local c v; +int main() { (void)v; } +``` + +```console +$ ./wasi-sdk-31.0-x86_64-linux/bin/clang++ repro.cc +wasm-ld: error: /tmp/repro-dd1ad7.o: undefined symbol: __cxa_thread_atexit +clang++: error: linker command failed with exit code 1 (use -v to see invocation) +``` +--- + libcxxabi/src/cxa_thread_atexit.cpp | 4 +--- + 1 file changed, 1 insertion(+), 3 deletions(-) + +diff --git a/libcxxabi/src/cxa_thread_atexit.cpp b/libcxxabi/src/cxa_thread_atexit.cpp +index 402a52c741012..1bdcb4ef192b4 100644 +--- a/libcxxabi/src/cxa_thread_atexit.cpp ++++ b/libcxxabi/src/cxa_thread_atexit.cpp +@@ -106,7 +106,6 @@ namespace { + + #endif // HAVE___CXA_THREAD_ATEXIT_IMPL + +-#if defined(__linux__) || defined(__Fuchsia__) + extern "C" { + + _LIBCXXABI_FUNC_VIS int __cxa_thread_atexit(Dtor dtor, void* obj, void* dso_symbol) throw() { +@@ -141,6 +140,5 @@ extern "C" { + } + #endif // HAVE___CXA_THREAD_ATEXIT_IMPL + } +-} // extern "C" +-#endif // defined(__linux__) || defined(__Fuchsia__) ++ } // extern "C" + } // namespace __cxxabiv1 diff --git a/tests/general/CMakeLists.txt b/tests/general/CMakeLists.txt index ef0a5528d..2954fd197 100644 --- a/tests/general/CMakeLists.txt +++ b/tests/general/CMakeLists.txt @@ -1,5 +1,5 @@ -file(GLOB c_general_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.c") -file(GLOB cxx_general_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.cc") +file(GLOB c_general_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} CONFIGURE_DEPENDS "*.c") +file(GLOB cxx_general_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} CONFIGURE_DEPENDS "*.cc") set(general_tests ${c_general_tests} ${cxx_general_tests}) diff --git a/tests/general/cpp_thread_local.cc b/tests/general/cpp_thread_local.cc new file mode 100644 index 000000000..30a151846 --- /dev/null +++ b/tests/general/cpp_thread_local.cc @@ -0,0 +1,3 @@ +struct c { ~c() {} }; +thread_local c v; +int main() { (void)v; } From d5030e3d63839ca618442b5e97d1d5cff3be0ab0 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Fri, 13 Mar 2026 17:42:33 -0500 Subject: [PATCH 159/165] Build libxml2 for LLDB (#614) Attempting to fix #612 --- cmake/wasi-sdk-toolchain.cmake | 39 ++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 56920f784..ef977c31f 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -19,6 +19,7 @@ if(CMAKE_SYSTEM_PROCESSOR MATCHES "arm64" AND LLVM_CMAKE_FLAGS MATCHES "x86_64") set(LIBEDIT_DEFAULT OFF) endif() option(WASI_SDK_LIBEDIT "Whether or not to build libedit for LLDB" ${LIBEDIT_DEFAULT}) +option(WASI_SDK_LIBXML2 "Whether or not to build libxml2 for LLDB" ON) string(REGEX REPLACE "[ ]+" ";" llvm_cmake_flags_list "${LLVM_CMAKE_FLAGS}") @@ -104,7 +105,6 @@ if(WASI_SDK_LLDB) -DLLDB_ENABLE_LZMA=OFF -DLLDB_ENABLE_LUA=OFF -DLLDB_ENABLE_PYTHON=OFF - -DLLDB_ENABLE_LIBXML2=OFF -DLLDB_ENABLE_FBSDVMCORE=OFF -DLLDB_ENABLE_LINUXPTY=OFF ) @@ -146,8 +146,43 @@ if(WASI_SDK_LLDB) list(APPEND default_cmake_args -DLLDB_ENABLE_LIBEDIT=OFF) add_custom_target(libedit) endif() + + set(libxml_cmake_args) + + # Windows doesn't have iconv by default, so disable it for now. + if (CMAKE_HOST_SYSTEM_NAME STREQUAL "Windows") + list(APPEND libxml_cmake_args -DLIBXML2_WITH_ICONV=OFF) + endif() + + if (WASI_SDK_LIBXML2) + ExternalProject_Add(libxml2 + URL https://download.gnome.org/sources/libxml2/2.15/libxml2-2.15.2.tar.xz + URL_HASH SHA256=c8b9bc81f8b590c33af8cc6c336dbff2f53409973588a351c95f1c621b13d09d + + CMAKE_ARGS + ${default_cmake_args} + -DLIBXML2_WITH_PROGRAMS=OFF + -DLIBXML2_WITH_DEBUG=OFF + -DLIBXML2_WITH_DOCS=OFF + -DLIBXML2_WITH_TESTS=OFF + ${libxml_cmake_args} + ${llvm_cmake_flags_list} + + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON + ) + list(APPEND default_cmake_args + -DLLDB_ENABLE_LIBXML2=ON + -DLibXml2_ROOT=${wasi_tmp_install} + ) + else() + list(APPEND default_cmake_args -DLLDB_ENABLE_LIBXML2=OFF) + add_custom_target(libxml2) + endif() else() add_custom_target(libedit) + add_custom_target(libxml2) endif() list(TRANSFORM tools PREPEND --target= OUTPUT_VARIABLE build_targets) @@ -190,7 +225,7 @@ ExternalProject_Add(llvm-build ) add_custom_target(build ALL DEPENDS llvm-build) -ExternalProject_Add_StepDependencies(llvm-build configure libedit) +ExternalProject_Add_StepDependencies(llvm-build configure libedit libxml2) # Installation target for this outer project for installing the toolchain to the # system. From cc5c4c98d7e5945a74759a1f40c26239bd7a4b1a Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 16 Mar 2026 11:16:00 -0500 Subject: [PATCH 160/165] Try to disable ncurses dep for libedit (#615) I'm shamelessly copying what [AlmaLinux does](https://git.almalinux.org/rpms/libedit/src/commit/3f0893c4cd8e0cbb2f556d2fad48326c9c037a6c/SPECS/libedit.spec#L44-L48), so let's try that... Closes #613 --- cmake/wasi-sdk-toolchain.cmake | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index ef977c31f..d433b9c00 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -109,6 +109,22 @@ if(WASI_SDK_LLDB) -DLLDB_ENABLE_LINUXPTY=OFF ) + set(extra_configure_commands) + if(CMAKE_SYSTEM_NAME STREQUAL Linux) + set(extra_configure_commands + # By default it looks like `libedit` tries to link to `libncurses.so` and + # such on Linux. This is problematic as systems may not have that + # installed. Turns out though at least for AlmaLinux [1] they just edit + # makefile and pkg-config info and it works out. Who knew! I thought + # one of the millions of lines in `./configure` would take care of this + # but apparently we're still resorting to editing things raw... + # + # [1]: https://git.almalinux.org/rpms/libedit/src/commit/3f0893c4cd8e0cbb2f556d2fad48326c9c037a6c/SPECS/libedit.spec#L44-L48 + COMMAND sed -i "s/lncurses/ltinfo/" src/Makefile + COMMAND sed -i "s/ -lncurses//" libedit.pc + ) + endif() + if (WASI_SDK_LIBEDIT) include(ProcessorCount) ProcessorCount(nproc) @@ -129,10 +145,13 @@ if(WASI_SDK_LLDB) --prefix=${wasi_tmp_install} --enable-pic --disable-examples + --disable-static + --disable-silent-rules CC=${CMAKE_C_COMPILER} LDFLAGS=${libedit_ldflags} + ${extra_configure_commands} BUILD_COMMAND - ${MAKE_EXECUTABLE} -j${nproc} V=1 + ${MAKE_EXECUTABLE} -j${nproc} USES_TERMINAL_CONFIGURE ON USES_TERMINAL_BUILD ON From 249054e5427023344c1906afde0e9073714d1930 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 16 Mar 2026 12:41:12 -0500 Subject: [PATCH 161/165] Don't use `lib64` dir for libxml2 (#617) Currently LLDB binaries as-is don't work, and this should fix them. --- cmake/wasi-sdk-toolchain.cmake | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index d433b9c00..5468d930a 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -173,6 +173,12 @@ if(WASI_SDK_LLDB) list(APPEND libxml_cmake_args -DLIBXML2_WITH_ICONV=OFF) endif() + # Our AlmaLinux:8 container ends up using `lib64` instead of `lib` by default + # which doesn't match LLVM, so specifically use the same dir as LLVM. + if (CMAKE_HOST_SYSTEM_NAME STREQUAL "Linux") + list(APPEND libxml_cmake_args -DCMAKE_INSTALL_LIBDIR=lib) + endif() + if (WASI_SDK_LIBXML2) ExternalProject_Add(libxml2 URL https://download.gnome.org/sources/libxml2/2.15/libxml2-2.15.2.tar.xz From 62a1a88c3dc2ced6e848b63ebe7fcf9928e6bf87 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Tue, 7 Apr 2026 12:40:27 -0500 Subject: [PATCH 162/165] Update wasm-component-ld (#623) Keeping it up-to-date --- cmake/wasi-sdk-toolchain.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 5468d930a..57a4b4b2c 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -261,7 +261,7 @@ install(DIRECTORY ${wasi_tmp_install}/bin ${wasi_tmp_install}/lib ${wasi_tmp_ins # Build logic for `wasm-component-ld` installed from Rust code. set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) -set(wasm_component_ld_version 0.5.21) +set(wasm_component_ld_version 0.5.22) if(RUST_TARGET) set(rust_target_flag --target=${RUST_TARGET}) endif() From 42b7263534b273dcb9b43310cb34ef5ef347a767 Mon Sep 17 00:00:00 2001 From: Bruno Verachten Date: Wed, 8 Apr 2026 16:42:04 +0200 Subject: [PATCH 163/165] ci: add riscv64-linux build via cross-compilation on ubuntu-24.04 (#621) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rework of the riscv64-linux CI build to use CMake cross-compilation on a standard `ubuntu-24.04` runner, rather than a native RISE runner. ## What changed **`ci/docker/Dockerfile.riscv64-linux`** (new): - Ubuntu 24.04 base — has `crossbuild-essential-riscv64` in its package repos - Sets `CC=riscv64-linux-gnu-gcc` / `CXX=riscv64-linux-gnu-g++` so CMake detects cross-compilation and causes LLVM to build a native `llvm-tblgen` first, then cross-compile the rest of the toolchain - Sets `CARGO_TARGET_RISCV64_UNKNOWN_LINUX_GNU_LINKER` for Rust cross-builds - `XDG_CACHE_HOME=/tmp/cache` avoids write permission issues in the container **`ci/docker-build.sh`**: - Select `ci/docker/Dockerfile.` if it exists, fall back to the default `ci/docker/Dockerfile` - Make the wasmtime volume mount conditional on `WASI_SDK_CI_SKIP_SYSROOT != 1` **`.github/workflows/main.yml`**: - New `riscv64-linux` matrix entry: `os: ubuntu-24.04`, `rust_target: riscv64-unknown-linux-gnu` - `cross_cmake_args: -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=riscv64 -DWASI_SDK_LLDB=OFF` - `WASI_SDK_CI_SKIP_SYSROOT: 1` - Handle `cross_cmake_args` in the cmake flags step ## Why WASI_SDK_CI_SKIP_SYSROOT The cross-compiled clang runs on riscv64, not on the x86_64 build host, so the wasm sysroot step is skipped. ## Why WASI_SDK_LLDB=OFF Avoids cross-compiling libedit and libxml2 in this first iteration; can be re-enabled as a follow-up. Closes #607 --------- Signed-off-by: Bruno Verachten --- .github/workflows/main.yml | 18 ++++++++++++++++++ ci/docker-build.sh | 15 +++++++++++---- ci/docker/Dockerfile.riscv64-linux | 28 ++++++++++++++++++++++++++++ ci/merge-artifacts.sh | 5 +++-- 4 files changed, 60 insertions(+), 6 deletions(-) create mode 100644 ci/docker/Dockerfile.riscv64-linux diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 14d51b5b2..0262f5fec 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -28,6 +28,21 @@ jobs: - artifact: arm64-linux os: ubuntu-22.04-arm + - artifact: riscv64-linux + os: ubuntu-24.04 + rust_target: riscv64gc-unknown-linux-gnu + cross_cmake_args: >- + -DCMAKE_SYSTEM_NAME=Linux + -DCMAKE_SYSTEM_PROCESSOR=riscv64 + -DCMAKE_C_COMPILER=riscv64-linux-gnu-gcc + -DCMAKE_CXX_COMPILER=riscv64-linux-gnu-g++ + -DWASI_SDK_LLDB=OFF + env: + WASI_SDK_CI_SKIP_SYSROOT: 1 + WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS: >- + -DCMAKE_SYSTEM_NAME=Linux + -DCMAKE_SYSTEM_PROCESSOR=riscv64 + - artifact: arm64-macos os: macos-14 rust_target: aarch64-apple-darwin @@ -85,6 +100,9 @@ jobs: rustup target add ${{ matrix.rust_target }} cmake_args="$cmake_args -DRUST_TARGET=${{ matrix.rust_target }}" fi + if [ "${{ matrix.cross_cmake_args }}" != "" ]; then + cmake_args="$cmake_args ${{ matrix.cross_cmake_args }}" + fi echo WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS="$cmake_args" >> $GITHUB_ENV shell: bash diff --git a/ci/docker-build.sh b/ci/docker-build.sh index 7773f7980..2915e286c 100755 --- a/ci/docker-build.sh +++ b/ci/docker-build.sh @@ -15,8 +15,13 @@ fi set -x -# Build the Docker imager -docker build --tag wasi-sdk-builder ci/docker +# Build the Docker image. Use an artifact-specific Dockerfile if one exists +# (e.g. ci/docker/Dockerfile.riscv64-linux), otherwise use the default. +dockerfile=ci/docker/Dockerfile +if [ -f "ci/docker/Dockerfile.$1" ]; then + dockerfile="ci/docker/Dockerfile.$1" +fi +docker build --tag wasi-sdk-builder --file "$dockerfile" ci/docker # Perform the build in `/src`. The current directory is mounted read-write at # this location as well. To ensure that container-created files are reasonable @@ -34,9 +39,11 @@ args="$args --volume $ccache_dir:/ccache:Z --env CCACHE_DIR=/ccache" # Inherit some tools from the host into this container. This ensures that the # decision made on CI of what versions to use is the canonical source of truth -# for theset ools +# for these tools. args="$args --volume `rustc --print sysroot`:/rustc:ro" -args="$args --volume $(dirname $(which wasmtime)):/wasmtime:ro" +if [ "${WASI_SDK_CI_SKIP_SYSROOT:-}" != "1" ]; then + args="$args --volume $(dirname $(command -v wasmtime)):/wasmtime:ro" +fi # Pass through some env vars that `build.sh` reads args="$args --env WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS" diff --git a/ci/docker/Dockerfile.riscv64-linux b/ci/docker/Dockerfile.riscv64-linux new file mode 100644 index 000000000..f4456ecf7 --- /dev/null +++ b/ci/docker/Dockerfile.riscv64-linux @@ -0,0 +1,28 @@ +# Ubuntu 24.04 is used here (rather than AlmaLinux 8) because it has +# riscv64 cross-compilation packages in its repositories. +FROM ubuntu:24.04 + +RUN apt-get update && apt-get install -y --no-install-recommends \ + curl \ + ca-certificates \ + crossbuild-essential-riscv64 \ + clang \ + lld \ + python3 \ + git \ + unzip \ + cmake \ + ninja-build \ + ccache \ + && rm -rf /var/lib/apt/lists/* + +# Cargo needs an explicit linker when cross-compiling for riscv64. +# The C/C++ cross-compiler is passed via CMAKE_C/CXX_COMPILER cmake flags +# rather than CC/CXX env vars so that LLVM's native tblgen sub-build can +# still find the host compiler (cmake cache vars are not inherited by +# subprocess cmake invocations, but env vars are). +ENV CARGO_TARGET_RISCV64GC_UNKNOWN_LINUX_GNU_LINKER=riscv64-linux-gnu-gcc + +# Tell programs to cache in a location that both isn't a `--volume` mounted root +# and isn't `/root` in the container as that won't be writable during the build. +ENV XDG_CACHE_HOME=/tmp/cache diff --git a/ci/merge-artifacts.sh b/ci/merge-artifacts.sh index e64b68e85..9a1e68386 100755 --- a/ci/merge-artifacts.sh +++ b/ci/merge-artifacts.sh @@ -21,8 +21,9 @@ make_deb() { fi case $build in - dist-x86_64-linux) deb_arch=amd64 ;; - dist-arm64-linux) deb_arch=arm64 ;; + dist-x86_64-linux) deb_arch=amd64 ;; + dist-arm64-linux) deb_arch=arm64 ;; + dist-riscv64-linux) deb_arch=riscv64 ;; *) echo "unknown build $build" exit 1 From 003cf14969ecca789c1922f9047e9a31872e9b52 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 8 Apr 2026 17:45:31 -0500 Subject: [PATCH 164/165] Build a sysroot that supports C++ exceptions by default (#606) This commit collects together some LLVM PRs, some changes in the build configuration here, and some thoughts from #565 and related issues. Specifically the changes here are: * The patch for llvm/llvm-project#168449 is updated to its upstream (unlanded) form. * Patches for the (landed) llvm/llvm-project#185770 and llvm/llvm-project#185775 are added. * The `WASI_SDK_EXCEPTIONS` configuration is now either `ON`, `OFF`, or `DUAL`. The default depends on the version of Clang in use, where 23.0.0+ (which isn't released officially yet) will be `DUAL` and otherwise it's `OFF`. CI for our custom-built patched toolchain defaults to `DUAL`. * In `DUAL` mode libcxx is built twice into two different directories, once with exceptions and once without. This is supported by LLVM patches and means that Clang will select the right set of libraries based on compiler flags. The end result here is that the produced toolchain from this repository, by default, supports C++ exceptions. Additionally if exceptions-related flags are not passed then the final binary will not use C++ exceptions nor require the wasm exception-handling proposal. There's still follow-up work from #565, such as: * Subjectively it feels wordy to pass `-fwasm-exceptions` vs `-fexceptions`. * Personally I think `-mllvm -wasm-use-legacy-eh=false` should become the default upstream. * Subjectively I don't think that `-lunwind` should be necessary and it should be injected automatically with `-fwasm-exceptions` (or `-fexceptions`). * Shared libraries for exceptions remain disabled due to build errors I do not personally know how to resolve. I'll file follow-up issues for these once this has landed since they're more minor compared to the main body of "anything works". Closes #334 Closes #565 --------- Co-authored-by: Joel Dice --- .github/workflows/main.yml | 10 ++- CppExceptions.md | 78 +++++++++-------- README.md | 4 +- ci/build.sh | 1 + cmake/wasi-sdk-sysroot.cmake | 79 +++++++++++++---- cmake/wasi-sdk-toolchain.cmake | 3 + src/llvm-pr-168449.patch | 27 +++--- src/llvm-pr-185770.patch | 105 +++++++++++++++++++++++ src/llvm-pr-185775.patch | 152 +++++++++++++++++++++++++++++++++ tests/CMakeLists.txt | 4 +- 10 files changed, 391 insertions(+), 72 deletions(-) create mode 100644 src/llvm-pr-185770.patch create mode 100644 src/llvm-pr-185775.patch diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 0262f5fec..2644c8f2f 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -188,10 +188,16 @@ jobs: - uses: ./.github/actions/checkout - uses: ./.github/actions/install-deps - run: cargo install wasm-component-ld@0.5.21 - - run: sudo apt-get update -y && sudo apt-get install -y clang-20 lld-20 + - name: Install LLVM 22 + run: | + v=22 + rel=$(lsb_release -cs) + wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/keyrings/llvm.asc + echo "deb [signed-by=/etc/apt/keyrings/llvm.asc] http://apt.llvm.org/$rel/ llvm-toolchain-$rel-$v main" | sudo tee /etc/apt/sources.list.d/llvm-$v.list + sudo apt-get update -y && sudo apt-get install -y clang-$v lld-$v - run: | cmake -G Ninja -B build -S . \ - -DCMAKE_C_COMPILER=/usr/lib/llvm-20/bin/clang \ + -DCMAKE_C_COMPILER=/usr/lib/llvm-22/bin/clang \ -DCMAKE_SYSTEM_NAME=WASI \ -DWASI_SDK_INCLUDE_TESTS=ON \ -DWASI_SDK_CPU_CFLAGS="" \ diff --git a/CppExceptions.md b/CppExceptions.md index 7ea1e2d8c..1ca79bf3c 100644 --- a/CppExceptions.md +++ b/CppExceptions.md @@ -1,25 +1,17 @@ # Support for C++ Exceptions -The released artifacts for wasi-sdk at this time do not support C++ exceptions. -LLVM and Clang, however, have support for C++ exceptions in WebAssembly and this -is intended to serve as documentation of the current state of affairs of using -C++ exceptions. It should be noted though that the current status of C++ -exceptions support is not intended to be the final state of support, and this is -all continuing to be iterated on over time. +> **Note**: this documentation does not cover wasi-sdk-31, the latest version +> of wasi-sdk at this time. -## Building wasi-sdk with exceptions +From wasi-sdk-33 and onwards the artifacts produced by this repository support +compiling C++ code both with and without exceptions. The sysroot for wasm +targets contains two copies of the C++ standard library and headers -- one with +exceptions enabled and one with exceptions disabled. These are automatically +selected based on compilation flags. This means that wasi-sdk-produced binaries +can avoid using wasm exceptions entirely by disabling C++ exceptions, or C++ +exceptions can be enabled in which case wasm exceptions will be used. -When building the sysroot with wasi-sdk you can pass `-DWASI_SDK_EXCEPTIONS=ON` -to enable support for C++ exceptions. For example: - -```shell script -$ cmake -G Ninja -B build/sysroot -S . \ - -DCMAKE_TOOLCHAIN_FILE=$path/to/wasi-sdk-p1.cmake \ - -DWASI_SDK_EXCEPTIONS=ON -``` - -The C++ standard library will be compiled with support for exceptions for the -desired targets and the resulting sysroot supports using exceptions. +Currently the default is for C++ exceptions to be disabled. ## Compiling code with C++ exceptions @@ -36,25 +28,43 @@ This can be specified for example with: ```shell script $ export CFLAGS="-fwasm-exceptions -mllvm -wasm-use-legacy-eh=false" -$ export LDFLAGS="-lunwind" +$ export LDFLAGS="-fwasm-exceptions -lunwind" ``` -## Limitations +Note that `-fwasm-exceptions` must be present when linking to select the +correct C++ standard library to link. + +## Building wasi-sdk with exceptions -Currently C++ exceptions support in wasi-sdk does not support shared libraries. -Fixing this will require resolving some miscellaneous build issues in this -repository itself. +When building the sysroot with wasi-sdk you can pass `-DWASI_SDK_EXCEPTIONS=ON` +to enable support for C++ exceptions. For example: -## Future Plans +```shell script +$ cmake -G Ninja -B build/sysroot -S . \ + -DCMAKE_TOOLCHAIN_FILE=$path/to/wasi-sdk-p1.cmake \ + -DWASI_SDK_EXCEPTIONS=ON +``` + +The C++ standard library will be compiled with support for exceptions for the +desired targets and the resulting sysroot supports using exceptions. Note that +enabling C++ exceptions requires LLVM 22 or later. + +C++ exceptions are disabled by default for local builds. With a future release +of LLVM 23 the dual-sysroot nature will be on-by-default. + +## Limitations -There are a few tracking issues with historical discussion about C++ exceptions -support in wasi-sdk such as [#334](https://github.com/WebAssembly/wasi-sdk/issues/334) -and [#565](https://github.com/WebAssembly/wasi-sdk/issues/565). The major -remaining items are: +There are a few known limitations/bugs/todos around exceptions support in +wasi-sdk at this time: -* Figure out support for shared libraries. -* Determine how to ship a sysroot that supports both with-and-without - exceptions. -* Figure out how to avoid the need for extra compiler flags when using - exceptions. -* Figure out if a new wasm target is warranted. +* Currently C++ exceptions support in wasi-sdk does not support shared + libraries. Fixing this will require resolving some miscellaneous build + issues in this repository itself as well as [resolving some upstream + issues](https://github.com/llvm/llvm-project/issues/188077). +* Currently `-fwasm-exceptions` is a required flag to enable C++ exceptions. + It's unclear whether `-fexceptions` should also be supported as a substitute. +* Currently LLVM defaults to using the legacy exception-handling proposal and + this will likely change in the future. Precompiled libraries for wasi-sdk are + all built with the standard exception-handling proposal. +* Currently `-lunwind` is required when linking, but this may become automatic + in the future. diff --git a/README.md b/README.md index 7d28f6957..7893c9639 100644 --- a/README.md +++ b/README.md @@ -209,8 +209,8 @@ disabled in a configure step before building with WASI SDK. ## Notable Limitations -* C++ exceptions are disabled by default. For more information see - [CppExceptions.md]. +* C++ exceptions are disabled by default and require extra configuration to get + working, see [CppExceptions.md]. * C `setjmp`/`longjmp` require some extra configuration to get working, see [SetjmpLongjmp.md]. * Most targets do not support spawning a thread. Experimental support for diff --git a/ci/build.sh b/ci/build.sh index 7525aa4d4..7c02fff8f 100755 --- a/ci/build.sh +++ b/ci/build.sh @@ -37,6 +37,7 @@ cmake -G Ninja -B $build_dir/sysroot -S . \ -DCMAKE_C_COMPILER_WORKS=ON \ -DCMAKE_CXX_COMPILER_WORKS=ON \ -DWASI_SDK_INCLUDE_TESTS=ON \ + -DWASI_SDK_EXCEPTIONS=DUAL \ "-DCMAKE_INSTALL_PREFIX=$build_dir/install" ninja -C $build_dir/sysroot install dist -v diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake index 38ed42636..491180ee4 100644 --- a/cmake/wasi-sdk-sysroot.cmake +++ b/cmake/wasi-sdk-sysroot.cmake @@ -20,13 +20,28 @@ message(STATUS "Found executable for `ar`: ${CMAKE_AR}") find_program(MAKE make REQUIRED) +set(EXCEPTIONS_DEFAULT "OFF") +if(CMAKE_C_COMPILER_VERSION VERSION_GREATER_EQUAL 23.0.0) + set(EXCEPTIONS_DEFAULT "DUAL") +endif() + option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" ON) option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) option(WASI_SDK_INSTALL_TO_CLANG_RESOURCE_DIR "Whether or not to modify the compiler's resource directory" OFF) option(WASI_SDK_LTO "Whether or not to build LTO assets" ON) -option(WASI_SDK_EXCEPTIONS "Whether or not C++ exceptions are enabled" OFF) +set(WASI_SDK_EXCEPTIONS "${EXCEPTIONS_DEFAULT}" CACHE STRING "Whether or not C++ exceptions are enabled") set(WASI_SDK_CPU_CFLAGS "-mcpu=lime1" CACHE STRING "CFLAGS to specify wasm features to enable") +if ((WASI_SDK_EXCEPTIONS STREQUAL "DUAL") OR (WASI_SDK_EXCEPTIONS STREQUAL "ON")) + if(CMAKE_C_COMPILER_VERSION VERSION_LESS 22.0.0) + message(FATAL_ERROR "enabling C++ exceptions requires Clang 22 or later") + endif() +elseif(WASI_SDK_EXCEPTIONS STREQUAL "OFF") + # No extra validation needed +else() + message(FATAL_ERROR "unknown WASI_SDK_EXCEPTIONS value ${WASI_SDK_EXCEPTIONS}, expected one of: OFF, ON, DUAL") +endif() + set(wasi_tmp_install ${CMAKE_CURRENT_BINARY_DIR}/install) set(wasi_sysroot ${wasi_tmp_install}/share/wasi-sysroot) set(wasi_resource_dir ${wasi_tmp_install}/wasi-resource-dir) @@ -225,7 +240,7 @@ execute_process( OUTPUT_VARIABLE llvm_version OUTPUT_STRIP_TRAILING_WHITESPACE) -function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_suffix) +function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_suffix exceptions) if(${target} MATCHES threads) set(pic OFF) set(target_flags -pthread) @@ -251,7 +266,9 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ --sysroot ${wasi_sysroot} -resource-dir ${wasi_resource_dir}) - if (WASI_SDK_EXCEPTIONS) + set(exnsuffix "") + + if (exceptions) # TODO: lots of builds fail with shared libraries and `-fPIC`. Looks like # things are maybe changing in llvm/llvm-project#159143 but otherwise I'm at # least not really sure what the state of shared libraries and exceptions @@ -260,6 +277,13 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ set(pic OFF) set(runtimes "libunwind;${runtimes}") list(APPEND extra_flags -fwasm-exceptions -mllvm -wasm-use-legacy-eh=false) + if (WASI_SDK_EXCEPTIONS STREQUAL "DUAL") + set(exnsuffix "/eh") + endif() + else() + if (WASI_SDK_EXCEPTIONS STREQUAL "DUAL") + set(exnsuffix "/noeh") + endif() endif() # The `wasm32-wasi` target is deprecated in clang, so ignore the deprecation @@ -279,7 +303,7 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ ${default_cmake_args} # Ensure headers are installed in a target-specific path instead of a # target-generic path. - -DCMAKE_INSTALL_INCLUDEDIR=${wasi_sysroot}/include/${target} + -DCMAKE_INSTALL_INCLUDEDIR=${wasi_sysroot}/include/${target}${exnsuffix} -DCMAKE_STAGING_PREFIX=${wasi_sysroot} -DCMAKE_POSITION_INDEPENDENT_CODE=${pic} -DLIBCXX_ENABLE_THREADS:BOOL=ON @@ -288,20 +312,20 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF -DLLVM_COMPILER_CHECKED=ON -DLIBCXX_ENABLE_SHARED:BOOL=${pic} - -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=${WASI_SDK_EXCEPTIONS} + -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=${exceptions} -DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF -DLIBCXX_CXX_ABI=libcxxabi -DLIBCXX_HAS_MUSL_LIBC:BOOL=OFF -DLIBCXX_ABI_VERSION=2 - -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=${WASI_SDK_EXCEPTIONS} + -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=${exceptions} -DLIBCXXABI_ENABLE_SHARED:BOOL=${pic} -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON -DLIBCXXABI_ENABLE_THREADS:BOOL=ON -DLIBCXXABI_HAS_PTHREAD_API:BOOL=ON -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF - -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=${WASI_SDK_EXCEPTIONS} + -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=${exceptions} -DLIBUNWIND_ENABLE_SHARED:BOOL=${pic} -DLIBUNWIND_ENABLE_THREADS:BOOL=ON -DLIBUNWIND_USE_COMPILER_RT:BOOL=ON @@ -310,9 +334,9 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ -DCMAKE_C_FLAGS=${extra_cflags} -DCMAKE_ASM_FLAGS=${extra_cflags} -DCMAKE_CXX_FLAGS=${extra_cxxflags} - -DLIBCXX_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} - -DLIBCXXABI_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} - -DLIBUNWIND_LIBDIR_SUFFIX=/${target}${extra_libdir_suffix} + -DLIBCXX_LIBDIR_SUFFIX=/${target}${exnsuffix}${extra_libdir_suffix} + -DLIBCXXABI_LIBDIR_SUFFIX=/${target}${exnsuffix}${extra_libdir_suffix} + -DLIBUNWIND_LIBDIR_SUFFIX=/${target}${exnsuffix}${extra_libdir_suffix} -DLIBCXX_INCLUDE_TESTS=OFF -DLIBCXX_INCLUDE_BENCHMARKS=OFF @@ -327,21 +351,45 @@ function(define_libcxx_sub target target_suffix extra_target_flags extra_libdir_ USES_TERMINAL_CONFIGURE ON USES_TERMINAL_BUILD ON USES_TERMINAL_INSTALL ON + USES_TERMINAL_PATCH ON PATCH_COMMAND ${CMAKE_COMMAND} -E chdir .. bash -c "git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-168449.patch || git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-168449.patch -R --check" COMMAND ${CMAKE_COMMAND} -E chdir .. bash -c "git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-186054.patch || git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-186054.patch -R --check" + COMMAND + ${CMAKE_COMMAND} -E chdir .. bash -c + "git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-185770.patch || git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-185770.patch -R --check" ) + add_dependencies(libcxx-${target} libcxx-${target}${target_suffix}-build) endfunction() -function(define_libcxx target) - define_libcxx_sub(${target} "" "" "") - if(WASI_SDK_LTO) +function(define_libcxx_and_lto target target_suffix exceptions) + define_libcxx_sub(${target} "${target_suffix}" "" "" ${exceptions}) + if (WASI_SDK_LTO) # Note: clang knows this /llvm-lto/${llvm_version} convention. # https://github.com/llvm/llvm-project/blob/llvmorg-18.1.8/clang/lib/Driver/ToolChains/WebAssembly.cpp#L204-L210 - define_libcxx_sub(${target} "-lto" "-flto=full" "/llvm-lto/${llvm_version}") + define_libcxx_sub(${target} ${target_suffix}-lto "-flto=full" "/llvm-lto/${llvm_version}" ${exceptions}) + endif() +endfunction() + +function(define_libcxx target) + add_custom_target(libcxx-${target}) + + # For dual-mode exceptions-and-not there are two versions of libcxx which are + # compiled and placed into the sysroot. They're named slightly differently to + # have unique CMake rules. + # + # Otherwise there's only one build of libcxx and it's either got exceptions or + # it doesn't depending on configuration. + if (WASI_SDK_EXCEPTIONS STREQUAL "DUAL") + define_libcxx_and_lto(${target} "" OFF) + define_libcxx_and_lto(${target} "-exn" ON) + elseif(WASI_SDK_EXCEPTIONS STREQUAL "ON") + define_libcxx_and_lto(${target} "" ON) + else() + define_libcxx_and_lto(${target} "" OFF) endif() # As of this writing, `clang++` will ignore the target-specific include dirs @@ -349,8 +397,7 @@ function(define_libcxx target) add_custom_target(libcxx-${target}-extra-dir COMMAND ${CMAKE_COMMAND} -E make_directory ${wasi_sysroot}/include/c++/v1 COMMENT "creating libcxx-specific header file folder") - add_custom_target(libcxx-${target} - DEPENDS libcxx-${target}-build $<$:libcxx-${target}-lto-build> libcxx-${target}-extra-dir) + add_dependencies(libcxx-${target} libcxx-${target}-extra-dir) endfunction() foreach(target IN LISTS WASI_SDK_TARGETS) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 57a4b4b2c..7b05a986f 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -247,6 +247,9 @@ ExternalProject_Add(llvm-build USES_TERMINAL_CONFIGURE ON USES_TERMINAL_BUILD ON USES_TERMINAL_INSTALL ON + PATCH_COMMAND + ${CMAKE_COMMAND} -E chdir .. bash -c + "git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-185775.patch || git apply ${CMAKE_SOURCE_DIR}/src/llvm-pr-185775.patch -R --check" ) add_custom_target(build ALL DEPENDS llvm-build) diff --git a/src/llvm-pr-168449.patch b/src/llvm-pr-168449.patch index a0da1d8d4..f5b873844 100644 --- a/src/llvm-pr-168449.patch +++ b/src/llvm-pr-168449.patch @@ -1,22 +1,17 @@ -diff --git a/libunwind/src/assembly.h b/libunwind/src/assembly.h -index f8e83e138eff..c5097d25b0c6 100644 ---- a/libunwind/src/assembly.h -+++ b/libunwind/src/assembly.h -@@ -249,6 +249,9 @@ aliasname: \ - #define WEAK_ALIAS(name, aliasname) - #define NO_EXEC_STACK_DIRECTIVE - -+#elif defined(__wasm__) -+#define NO_EXEC_STACK_DIRECTIVE -+ - // clang-format on - #else - +From 852c8a2ebc0fdb1e781591e3e6e08d3a539bcfc3 Mon Sep 17 00:00:00 2001 +From: Yerzhan Zhamashev +Date: Wed, 21 Jan 2026 16:50:41 +0200 +Subject: [PATCH] libunwind: exclude __declspec from wasm build + +--- + libunwind/src/config.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + diff --git a/libunwind/src/config.h b/libunwind/src/config.h -index deb5a4d4d73d..23c9f012cbcf 100644 +index f017403fa2234..6014a37e27212 100644 --- a/libunwind/src/config.h +++ b/libunwind/src/config.h -@@ -66,7 +66,8 @@ +@@ -75,7 +75,8 @@ #define _LIBUNWIND_EXPORT #define _LIBUNWIND_HIDDEN #else diff --git a/src/llvm-pr-185770.patch b/src/llvm-pr-185770.patch new file mode 100644 index 000000000..2d47923ef --- /dev/null +++ b/src/llvm-pr-185770.patch @@ -0,0 +1,105 @@ +From d702761d9135ebbb83590d4dd1323be433701ebd Mon Sep 17 00:00:00 2001 +From: Alex Crichton +Date: Tue, 10 Mar 2026 15:49:55 -0700 +Subject: [PATCH] [WebAssembly] Move __cpp_exception to libunwind + +The `__cpp_exception` symbol is now defined in libunwind instead of +compiler-rt. This is moved for a few reasons, but the primary reason is +that compiler-rt is linked duplicate-ly into all shared objects meaning +that it's not suitable for define-once symbols such as +`__cpp_exception`. By moving the definition to the user of the symbol, +libunwind itself, that guarantees that the symbol should be defined +exactly once and only when appropriate. A secondary reason for this +movement is that it avoids the need to compile compiler-rt twice: once +with exception and once without, and instead the same build can be used +for both exceptions-and-not. +--- + compiler-rt/lib/builtins/CMakeLists.txt | 1 - + .../lib/builtins/wasm/__cpp_exception.S | 26 ------------------- + libunwind/src/Unwind-wasm.c | 15 +++++++++++ + .../compiler-rt/lib/builtins/sources.gni | 1 - + 4 files changed, 15 insertions(+), 28 deletions(-) + delete mode 100644 compiler-rt/lib/builtins/wasm/__cpp_exception.S + +diff --git a/compiler-rt/lib/builtins/CMakeLists.txt b/compiler-rt/lib/builtins/CMakeLists.txt +index 6c27f6d4d529e..f0570a9092f40 100644 +--- a/compiler-rt/lib/builtins/CMakeLists.txt ++++ b/compiler-rt/lib/builtins/CMakeLists.txt +@@ -891,7 +891,6 @@ set(s390x_SOURCES + + set(wasm_SOURCES + wasm/__c_longjmp.S +- wasm/__cpp_exception.S + ${GENERIC_TF_SOURCES} + ${GENERIC_SOURCES} + ) +diff --git a/compiler-rt/lib/builtins/wasm/__cpp_exception.S b/compiler-rt/lib/builtins/wasm/__cpp_exception.S +deleted file mode 100644 +index 0496e1dbf6158..0000000000000 +--- a/compiler-rt/lib/builtins/wasm/__cpp_exception.S ++++ /dev/null +@@ -1,26 +0,0 @@ +-//===-- __cpp_exception.S - Implement __cpp_exception ---------------------===// +-// +-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +-// See https://llvm.org/LICENSE.txt for license information. +-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +-// +-//===----------------------------------------------------------------------===// +-// +-// This file implements __cpp_exception which LLVM uses to implement exception +-// handling when Wasm EH is enabled. +-// +-//===----------------------------------------------------------------------===// +- +-#ifdef __wasm_exception_handling__ +- +-#ifdef __wasm64__ +-#define PTR i64 +-#else +-#define PTR i32 +-#endif +- +-.globl __cpp_exception +-.tagtype __cpp_exception PTR +-__cpp_exception: +- +-#endif // !__wasm_exception_handling__ +diff --git a/libunwind/src/Unwind-wasm.c b/libunwind/src/Unwind-wasm.c +index 2f4498c3f3989..c0ca9b775d244 100644 +--- a/libunwind/src/Unwind-wasm.c ++++ b/libunwind/src/Unwind-wasm.c +@@ -69,6 +69,21 @@ _Unwind_RaiseException(_Unwind_Exception *exception_object) { + __builtin_wasm_throw(0, exception_object); + } + ++// Define the `__cpp_exception` symbol which `__builtin_wasm_throw` above will ++// reference. This is defined here in `libunwind` as the single canonical ++// definition for this API and it's required for users to ensure that there's ++// only one copy of `libunwind` within a wasm module to ensure this is only ++// defined once and exactly once. ++__asm__(".globl __cpp_exception\n" ++#if defined(__wasm32__) ++ ".tagtype __cpp_exception i32\n" ++#elif defined(__wasm64__) ++ ".tagtype __cpp_exception i64\n" ++#else ++#error "Unsupported Wasm architecture" ++#endif ++ "__cpp_exception:\n"); ++ + /// Called by __cxa_end_catch. + _LIBUNWIND_EXPORT void + _Unwind_DeleteException(_Unwind_Exception *exception_object) { +diff --git a/llvm/utils/gn/secondary/compiler-rt/lib/builtins/sources.gni b/llvm/utils/gn/secondary/compiler-rt/lib/builtins/sources.gni +index 2ac71aa8e8367..c9eeede16e3eb 100644 +--- a/llvm/utils/gn/secondary/compiler-rt/lib/builtins/sources.gni ++++ b/llvm/utils/gn/secondary/compiler-rt/lib/builtins/sources.gni +@@ -539,7 +539,6 @@ if (current_cpu == "ve") { + if (current_cpu == "wasm") { + builtins_sources += [ + "wasm/__c_longjmp.S", +- "wasm/__cpp_exception.S", + ] + } + diff --git a/src/llvm-pr-185775.patch b/src/llvm-pr-185775.patch new file mode 100644 index 000000000..f1fdb6a57 --- /dev/null +++ b/src/llvm-pr-185775.patch @@ -0,0 +1,152 @@ +From 0e36e8f304cd5f3997916f5d85201bb17e340337 Mon Sep 17 00:00:00 2001 +From: Alex Crichton +Date: Tue, 10 Mar 2026 16:14:36 -0700 +Subject: [PATCH] [WebAssembly] Clang support for exception-based lookup paths + +This commit is an attempt to make progress on WebAssembly/wasi-sdk#565 +where with wasi-sdk I'd like to ship a single toolchain which is +capable of building binaries both with C++ exceptions and without. This +means that there can't be a single set of precompiled libraries that are +used because one set of libraries is wrong for the other mode. The +support added here is to use `-fwasm-exceptions` to automatically select +a lookup path in the sysroot. The intention is then that wasi-sdk will +ship both a "eh" set of C++ libraries as well as a "noeh" set of C++ +libraries too. Clang will automatically select the correct one based on +compilation flags which means that the final distribution will be able +to build both binaries with exceptions and without. +--- + clang/lib/Driver/ToolChains/WebAssembly.cpp | 51 ++++++++++++++------- + clang/test/Driver/wasm-toolchain.cpp | 35 ++++++++++++++ + 2 files changed, 70 insertions(+), 16 deletions(-) + +diff --git a/clang/lib/Driver/ToolChains/WebAssembly.cpp b/clang/lib/Driver/ToolChains/WebAssembly.cpp +index b5fa5760a46a0..e532ef0743cc2 100644 +--- a/clang/lib/Driver/ToolChains/WebAssembly.cpp ++++ b/clang/lib/Driver/ToolChains/WebAssembly.cpp +@@ -34,6 +34,15 @@ std::string WebAssembly::getMultiarchTriple(const Driver &D, + TargetTriple.getOSAndEnvironmentName()).str(); + } + ++/// Returns a directory name in which separate objects compile with/without ++/// exceptions may lie. This is used both for `#include` paths as well as lib ++/// paths. ++static std::string GetCXXExceptionsDir(const ArgList &DriverArgs) { ++ if (DriverArgs.getLastArg(options::OPT_fwasm_exceptions)) ++ return "eh"; ++ return "noeh"; ++} ++ + std::string wasm::Linker::getLinkerPath(const ArgList &Args) const { + const ToolChain &ToolChain = getToolChain(); + if (const Arg* A = Args.getLastArg(options::OPT_fuse_ld_EQ)) { +@@ -230,12 +239,16 @@ void wasm::Linker::ConstructJob(Compilation &C, const JobAction &JA, + } + } + +-/// Given a base library directory, append path components to form the +-/// LTO directory. +-static std::string AppendLTOLibDir(const std::string &Dir) { ++/// Append `Dir` to `Paths`, but also include the LTO directories before that if ++/// LTO is eanbled. ++static void AppendLibDirAndLTODir(ToolChain::path_list &Paths, const Driver &D, ++ const std::string &Dir) { ++ if (D.isUsingLTO()) { + // The version allows the path to be keyed to the specific version of + // LLVM in used, as the bitcode format is not stable. +- return Dir + "/llvm-lto/" LLVM_VERSION_STRING; ++ Paths.push_back(Dir + "/llvm-lto/" LLVM_VERSION_STRING); ++ } ++ Paths.push_back(Dir); + } + + WebAssembly::WebAssembly(const Driver &D, const llvm::Triple &Triple, +@@ -256,14 +269,15 @@ WebAssembly::WebAssembly(const Driver &D, const llvm::Triple &Triple, + } else { + const std::string MultiarchTriple = + getMultiarchTriple(getDriver(), Triple, SysRoot); +- if (D.isUsingLTO()) { +- // For LTO, enable use of lto-enabled sysroot libraries too, if available. +- // Note that the directory is keyed to the LLVM revision, as LLVM's +- // bitcode format is not stable. +- auto Dir = AppendLTOLibDir(SysRoot + "/lib/" + MultiarchTriple); +- getFilePaths().push_back(Dir); +- } +- getFilePaths().push_back(SysRoot + "/lib/" + MultiarchTriple); ++ std::string TripleLibDir = SysRoot + "/lib/" + MultiarchTriple; ++ // Allow sysroots to segregate objects based on whether exceptions are ++ // enabled or not. This is intended to assist with distribution of pre-built ++ // sysroots that contain libraries that are capable of producing binaries ++ // entirely without exception-handling instructions but also with if ++ // exceptions are enabled, for example. ++ AppendLibDirAndLTODir(getFilePaths(), D, ++ TripleLibDir + "/" + GetCXXExceptionsDir(Args)); ++ AppendLibDirAndLTODir(getFilePaths(), D, TripleLibDir); + } + + if (getTriple().getOS() == llvm::Triple::WASI) { +@@ -580,13 +594,18 @@ void WebAssembly::addLibCxxIncludePaths( + if (Version.empty()) + return; + +- // First add the per-target include path if the OS is known. ++ // First add the per-target-per-exception-handling include path if the ++ // OS is known, then second add the per-target include path. + if (IsKnownOs) { +- std::string TargetDir = LibPath + "/" + MultiarchTriple + "/c++/" + Version; +- addSystemInclude(DriverArgs, CC1Args, TargetDir); ++ std::string TargetDir = LibPath + "/" + MultiarchTriple; ++ std::string Suffix = "/c++/" + Version; ++ addSystemInclude(DriverArgs, CC1Args, ++ TargetDir + "/" + GetCXXExceptionsDir(DriverArgs) + ++ Suffix); ++ addSystemInclude(DriverArgs, CC1Args, TargetDir + Suffix); + } + +- // Second add the generic one. ++ // Third add the generic one. + addSystemInclude(DriverArgs, CC1Args, LibPath + "/c++/" + Version); + } + +diff --git a/clang/test/Driver/wasm-toolchain.cpp b/clang/test/Driver/wasm-toolchain.cpp +index d7ff76cedfd10..30a2f9397e3f4 100644 +--- a/clang/test/Driver/wasm-toolchain.cpp ++++ b/clang/test/Driver/wasm-toolchain.cpp +@@ -111,3 +111,38 @@ + // COMPILE_WALI_STDCXX: "-internal-isystem" "[[RESOURCE_DIR]]{{(/|\\\\)}}include" + // COMPILE_WALI_STDCXX: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-linux-muslwali" + // COMPILE_WALI_STDCXX: "-internal-isystem" "[[SYSROOT:[^"]+]]/include" ++ ++// With a known OS "eh" and "noeh" directories are added to enable segregating ++// object built with/without exception-handling ++ ++// RUN: %clangxx -### --target=wasm32-wasi --stdlib=libc++ %s 2>&1 \ ++// RUN: --sysroot=%S/Inputs/basic_linux_libcxx_tree/usr \ ++// RUN: | FileCheck -check-prefix=EH_OFF %s ++// EH_OFF: "-cc1" ++// EH_OFF: "-isysroot" "[[SYSROOT:[^"]+]]" ++// EH_OFF: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-wasi/noeh/c++/v1" ++// EH_OFF-NOT: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-wasi/eh/c++/v1" ++// EH_OFF: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-wasi/c++/v1" ++// EH_OFF: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/c++/v1" ++// EH_OFF: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-wasi" ++// EH_OFF: "-internal-isystem" "[[SYSROOT:[^"]+]]/include" ++ ++// RUN: %clangxx -### --target=wasm32-wasi -fwasm-exceptions --stdlib=libc++ %s 2>&1 \ ++// RUN: --sysroot=%S/Inputs/basic_linux_libcxx_tree/usr \ ++// RUN: | FileCheck -check-prefix=EH_ON %s ++// EH_ON: "-cc1" ++// EH_ON: "-isysroot" "[[SYSROOT:[^"]+]]" ++// EH_ON: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-wasi/eh/c++/v1" ++// EH_ON-NOT: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-wasi/noeh/c++/v1" ++// EH_ON: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-wasi/c++/v1" ++// EH_ON: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/c++/v1" ++// EH_ON: "-internal-isystem" "[[SYSROOT:[^"]+]]/include/wasm32-wasi" ++// EH_ON: "-internal-isystem" "[[SYSROOT:[^"]+]]/include" ++// ++// RUN: %clangxx -### --target=wasm32-wasi --sysroot=/foo --stdlib=libc++ %s 2>&1 \ ++// RUN: | FileCheck -check-prefix=EH_OFF_LINK %s ++// EH_OFF_LINK: wasm-ld{{.*}}" "-L/foo/lib/wasm32-wasi/noeh" "-L/foo/lib/wasm32-wasi" ++// ++// RUN: %clangxx -### --target=wasm32-wasi -fwasm-exceptions --sysroot=/foo --stdlib=libc++ %s 2>&1 \ ++// RUN: | FileCheck -check-prefix=EH_ON_LINK %s ++// EH_ON_LINK: wasm-ld{{.*}}" "-L/foo/lib/wasm32-wasi/eh" "-L/foo/lib/wasm32-wasi" diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index c573aea65..9bb5fc2eb 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -67,9 +67,9 @@ function(add_testcase runwasi test) # Apply language-specific options and dependencies. if(test MATCHES "cc$") - if(WASI_SDK_EXCEPTIONS) + if(NOT (WASI_SDK_EXCEPTIONS STREQUAL "OFF")) target_compile_options(${target_name} PRIVATE -fwasm-exceptions -mllvm -wasm-use-legacy-eh=false) - target_link_options(${target_name} PRIVATE -lunwind) + target_link_options(${target_name} PRIVATE -fwasm-exceptions -lunwind) else() target_compile_options(${target_name} PRIVATE -fno-exceptions) endif() From 37ea0a6f25b6a7abe20d508667f58d552cdd297e Mon Sep 17 00:00:00 2001 From: Daniel Nicoletti Date: Fri, 10 Apr 2026 16:30:25 -0300 Subject: [PATCH 165/165] Add clang-scan-deps to the toolchain (#624) I tried to built it locally but failed, hopefully CI can do it. --- cmake/wasi-sdk-toolchain.cmake | 1 + 1 file changed, 1 insertion(+) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake index 7b05a986f..1d023c879 100644 --- a/cmake/wasi-sdk-toolchain.cmake +++ b/cmake/wasi-sdk-toolchain.cmake @@ -56,6 +56,7 @@ set(tools clang-format clang-tidy clang-apply-replacements + clang-scan-deps lld llvm-addr2line llvm-mc