From 3bd22b1e2ba1cd9bdfe9152ff71e1a0f79e672ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Swen=20Sch=C3=A4ferjohann?= <42959314+SwenSchaeferjohann@users.noreply.github.com> Date: Thu, 10 Apr 2025 02:24:43 +0100 Subject: [PATCH 01/26] chore: backport breaking api changes to v1 js SDKs (#1661) * stateless.js add treeinfos * wip * add getTokenPoolInfos * wip * wip * wip - storageoptions * wip * wip * wip * wip - known bug in rpc-interop.test.ts if using random trees * debugged test-rpc in ctoken * all ctoken tests working * rm logs * clean * ctxs to infos, removed redundant getMintProgramId calls * rm deadcode, storageoptions * fix cli getMindProgramId use * fix tokenpool * fix test stateless.js add treeinfos wip add getTokenPoolInfos wip wip wip - storageoptions wip wip wip wip - known bug in rpc-interop.test.ts if using random trees debugged test-rpc in ctoken all ctoken tests working rm logs clean ctxs to infos, removed redundant getMintProgramId calls rm deadcode, storageoptions fix cli getMindProgramId use fix tokenpool fix test update CHANGELOG.md files update changelog update CHANGELOG.md export get-token-pool-infos.ts wip clean wip refactor merklecontext refactor StateTreeInfo wip wip debug test-rpc getCompressedTokenAccountsByOwner fix unit test wip wip wip debug .readUIntLE wip wip wip fix buffer conversion at test-rpc decode wip tests working compressedProof -> validityProof update changelog wip refactor: do not allow output trees for decompress, transfer fmt wip rename getCachedStateTreeInfos upd changelog use with getCachedStateTreeInfos getStateTreeInfos wip wip delegate test wip added transfer-delegated.test.ts added transfer-delegated test cases. all green add tests for decompress-delegated fix rm logs from program update changelog update err msg fix state-tree-luts wip getActiveStateTreeInfos -> getAllStateTreeInfos wip fix sigs for nullifyStateTree wip wip cleanup add getCachedActiveStateTreeInfos mock revert to compressedProof add tokenpools tests update comment update CHANGELOG.md update changelog 0.21.0 update changelog fix fix event parsing post rebase stateless js refactor dedupe types link to computebudgetprogram wip fix DX for instructions: add docstrings to call signatures wip dont break mergeTokenAccounts wip wip add v2 trees to test-rpc v1 mergeable test-rpc tests working rpc-interop tests working with v2 all stateless.js tests working with v2 compressed-token tests working rebase to main fixup cargo lock wip wip (#1794) wip (#1781) chore: backport breaking api changes to v1 js SDKs (#1661) * stateless.js add treeinfos * wip * add getTokenPoolInfos * wip * wip * wip - storageoptions * wip * wip * wip * wip - known bug in rpc-interop.test.ts if using random trees * debugged test-rpc in ctoken * all ctoken tests working * rm logs * clean * ctxs to infos, removed redundant getMintProgramId calls * rm deadcode, storageoptions * fix cli getMindProgramId use * fix tokenpool * fix test stateless.js add treeinfos wip add getTokenPoolInfos wip wip wip - storageoptions wip wip wip wip - known bug in rpc-interop.test.ts if using random trees debugged test-rpc in ctoken all ctoken tests working rm logs clean ctxs to infos, removed redundant getMintProgramId calls rm deadcode, storageoptions fix cli getMindProgramId use fix tokenpool fix test update CHANGELOG.md files update changelog update CHANGELOG.md export get-token-pool-infos.ts wip clean wip refactor merklecontext refactor StateTreeInfo wip wip debug test-rpc getCompressedTokenAccountsByOwner fix unit test wip wip wip debug .readUIntLE wip wip wip fix buffer conversion at test-rpc decode wip tests working compressedProof -> validityProof update changelog wip refactor: do not allow output trees for decompress, transfer fmt wip rename getCachedStateTreeInfos upd changelog use with getCachedStateTreeInfos getStateTreeInfos wip wip delegate test wip added transfer-delegated.test.ts added transfer-delegated test cases. all green add tests for decompress-delegated fix rm logs from program update changelog update err msg fix state-tree-luts wip getActiveStateTreeInfos -> getAllStateTreeInfos wip fix sigs for nullifyStateTree wip wip cleanup add getCachedActiveStateTreeInfos mock revert to compressedProof add tokenpools tests update comment update CHANGELOG.md update changelog 0.21.0 update changelog fix fix event parsing post rebase stateless js refactor dedupe types link to computebudgetprogram wip fix DX for instructions: add docstrings to call signatures wip dont break mergeTokenAccounts wip wip add v2 trees to test-rpc v1 mergeable test-rpc tests working rpc-interop tests working with v2 all stateless.js tests working with v2 compressed-token tests working rebase to main fixup cargo lock wip chore: backport breaking api changes to v1 js SDKs (#1661) * stateless.js add treeinfos * wip * add getTokenPoolInfos * wip * wip * wip - storageoptions * wip * wip * wip * wip - known bug in rpc-interop.test.ts if using random trees * debugged test-rpc in ctoken * all ctoken tests working * rm logs * clean * ctxs to infos, removed redundant getMintProgramId calls * rm deadcode, storageoptions * fix cli getMindProgramId use * fix tokenpool * fix test stateless.js add treeinfos wip add getTokenPoolInfos wip wip wip - storageoptions wip wip wip wip - known bug in rpc-interop.test.ts if using random trees debugged test-rpc in ctoken all ctoken tests working rm logs clean ctxs to infos, removed redundant getMintProgramId calls rm deadcode, storageoptions fix cli getMindProgramId use fix tokenpool fix test update CHANGELOG.md files update changelog update CHANGELOG.md export get-token-pool-infos.ts wip clean wip refactor merklecontext refactor StateTreeInfo wip wip debug test-rpc getCompressedTokenAccountsByOwner fix unit test wip wip wip debug .readUIntLE wip wip wip fix buffer conversion at test-rpc decode wip tests working compressedProof -> validityProof update changelog wip refactor: do not allow output trees for decompress, transfer fmt wip rename getCachedStateTreeInfos upd changelog use with getCachedStateTreeInfos getStateTreeInfos wip wip delegate test wip added transfer-delegated.test.ts added transfer-delegated test cases. all green add tests for decompress-delegated fix rm logs from program update changelog update err msg fix state-tree-luts wip getActiveStateTreeInfos -> getAllStateTreeInfos wip fix sigs for nullifyStateTree wip wip cleanup add getCachedActiveStateTreeInfos mock revert to compressedProof add tokenpools tests update comment update CHANGELOG.md update changelog 0.21.0 update changelog fix fix event parsing post rebase stateless js refactor dedupe types link to computebudgetprogram wip fix DX for instructions: add docstrings to call signatures wip dont break mergeTokenAccounts wip wip add v2 trees to test-rpc v1 mergeable test-rpc tests working rpc-interop tests working with v2 all stateless.js tests working with v2 compressed-token tests working rebase to main fixup cargo lock wip wip rename statetreeinfo -> treeinfo wip wip wip cli test works wip wip wip wip wip wip wip wip wip wip wip wip rm logs fix lint clean upd comment chore: backport breaking api changes to v1 js SDKs (#1661) (#1790) * stateless.js add treeinfos * wip * add getTokenPoolInfos * wip * wip * wip - storageoptions * wip * wip * wip * wip - known bug in rpc-interop.test.ts if using random trees * debugged test-rpc in ctoken * all ctoken tests working * rm logs * clean * ctxs to infos, removed redundant getMintProgramId calls * rm deadcode, storageoptions * fix cli getMindProgramId use * fix tokenpool * fix test stateless.js add treeinfos wip add getTokenPoolInfos wip wip wip - storageoptions wip wip wip wip - known bug in rpc-interop.test.ts if using random trees debugged test-rpc in ctoken all ctoken tests working rm logs clean ctxs to infos, removed redundant getMintProgramId calls rm deadcode, storageoptions fix cli getMindProgramId use fix tokenpool fix test update CHANGELOG.md files update changelog update CHANGELOG.md export get-token-pool-infos.ts wip clean wip refactor merklecontext refactor StateTreeInfo wip wip debug test-rpc getCompressedTokenAccountsByOwner fix unit test wip wip wip debug .readUIntLE wip wip wip fix buffer conversion at test-rpc decode wip tests working compressedProof -> validityProof update changelog wip refactor: do not allow output trees for decompress, transfer fmt wip rename getCachedStateTreeInfos upd changelog use with getCachedStateTreeInfos getStateTreeInfos wip wip delegate test wip added transfer-delegated.test.ts added transfer-delegated test cases. all green add tests for decompress-delegated fix rm logs from program update changelog update err msg fix state-tree-luts wip getActiveStateTreeInfos -> getAllStateTreeInfos wip fix sigs for nullifyStateTree wip wip cleanup add getCachedActiveStateTreeInfos mock revert to compressedProof add tokenpools tests update comment update CHANGELOG.md update changelog 0.21.0 update changelog fix fix event parsing post rebase stateless js refactor dedupe types link to computebudgetprogram wip fix DX for instructions: add docstrings to call signatures wip dont break mergeTokenAccounts wip wip add v2 trees to test-rpc v1 mergeable test-rpc tests working rpc-interop tests working with v2 all stateless.js tests working with v2 compressed-token tests working rebase to main fixup cargo lock wip chore: backport breaking api changes to v1 js SDKs (#1661) * stateless.js add treeinfos * wip * add getTokenPoolInfos * wip * wip * wip - storageoptions * wip * wip * wip * wip - known bug in rpc-interop.test.ts if using random trees * debugged test-rpc in ctoken * all ctoken tests working * rm logs * clean * ctxs to infos, removed redundant getMintProgramId calls * rm deadcode, storageoptions * fix cli getMindProgramId use * fix tokenpool * fix test stateless.js add treeinfos wip add getTokenPoolInfos wip wip wip - storageoptions wip wip wip wip - known bug in rpc-interop.test.ts if using random trees debugged test-rpc in ctoken all ctoken tests working rm logs clean ctxs to infos, removed redundant getMintProgramId calls rm deadcode, storageoptions fix cli getMindProgramId use fix tokenpool fix test update CHANGELOG.md files update changelog update CHANGELOG.md export get-token-pool-infos.ts wip clean wip refactor merklecontext refactor StateTreeInfo wip wip debug test-rpc getCompressedTokenAccountsByOwner fix unit test wip wip wip debug .readUIntLE wip wip wip fix buffer conversion at test-rpc decode wip tests working compressedProof -> validityProof update changelog wip refactor: do not allow output trees for decompress, transfer fmt wip rename getCachedStateTreeInfos upd changelog use with getCachedStateTreeInfos getStateTreeInfos wip wip delegate test wip added transfer-delegated.test.ts added transfer-delegated test cases. all green add tests for decompress-delegated fix rm logs from program update changelog update err msg fix state-tree-luts wip getActiveStateTreeInfos -> getAllStateTreeInfos wip fix sigs for nullifyStateTree wip wip cleanup add getCachedActiveStateTreeInfos mock revert to compressedProof add tokenpools tests update comment update CHANGELOG.md update changelog 0.21.0 update changelog fix fix event parsing post rebase stateless js refactor dedupe types link to computebudgetprogram wip fix DX for instructions: add docstrings to call signatures wip dont break mergeTokenAccounts wip wip add v2 trees to test-rpc v1 mergeable test-rpc tests working rpc-interop tests working with v2 all stateless.js tests working with v2 compressed-token tests working rebase to main fixup cargo lock wip wip rename statetreeinfo -> treeinfo wip wip wip cli test works wip wip wip wip wip wip wip wip wip wip wip wip rm logs fix lint clean upd comment wip stateless.js tests working ctoken tests working lint address comments selective installs fmt v1 and v2 builds and tests working ctoken v1,v2 working rm logs ci --- .github/actions/setup-and-build/action.yml | 12 ++- .github/workflows/js-v2.yml | 83 +++++++++++++++++++ .github/workflows/js.yml | 55 +++++++----- .github/workflows/release.yml | 2 + Cargo.lock | 3 + js/compressed-token/package.json | 8 ++ js/compressed-token/rollup.config.js | 7 ++ js/compressed-token/src/types.ts | 1 - .../src/utils/version-check.ts | 40 +++++++++ .../tests/setup/version-check.ts | 16 ++++ .../tests/unit/version.test.ts | 62 ++++++++++++++ js/compressed-token/vitest.config.ts | 1 + js/stateless.js/BUILD.md | 16 ++++ js/stateless.js/package.json | 8 +- js/stateless.js/rollup.config.js | 9 +- js/stateless.js/src/constants.ts | 21 ++++- js/stateless.js/src/globals.d.ts | 1 + js/stateless.js/src/rpc.ts | 6 +- js/stateless.js/test-version.js | 17 ++++ .../tests/e2e/browser/rpc.browser.spec.ts | 1 + js/stateless.js/tests/e2e/rpc-interop.test.ts | 2 +- js/stateless.js/tests/unit/version.test.ts | 27 ++++++ scripts/INSTALL.md | 40 +++++++++ scripts/devenv.sh | 1 + scripts/install.sh | 40 ++++++--- 25 files changed, 438 insertions(+), 41 deletions(-) create mode 100644 .github/workflows/js-v2.yml create mode 100644 js/compressed-token/src/utils/version-check.ts create mode 100644 js/compressed-token/tests/setup/version-check.ts create mode 100644 js/compressed-token/tests/unit/version.test.ts create mode 100644 js/stateless.js/BUILD.md create mode 100644 js/stateless.js/src/globals.d.ts create mode 100644 js/stateless.js/test-version.js create mode 100644 js/stateless.js/tests/unit/version.test.ts create mode 100644 scripts/INSTALL.md diff --git a/.github/actions/setup-and-build/action.yml b/.github/actions/setup-and-build/action.yml index 460362f5fc..1c4fd06593 100644 --- a/.github/actions/setup-and-build/action.yml +++ b/.github/actions/setup-and-build/action.yml @@ -1,6 +1,12 @@ name: Setup and build description: Checkout sources, install dependencies, build and prepare for tests +inputs: + components: + description: 'Comma-separated list of components to install (e.g., "node,pnpm,solana,anchor,jq,dependencies"). If not specified, all components are installed.' + required: false + default: "" + runs: using: "composite" steps: @@ -34,7 +40,11 @@ runs: if: steps.restore-local-cache.outputs.cache-hit != 'true' shell: bash run: | - ./scripts/install.sh + if [ -n "${{ inputs.components }}" ]; then + ./scripts/install.sh --components "${{ inputs.components }}" + else + ./scripts/install.sh + fi - name: Set local environment shell: bash diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml new file mode 100644 index 0000000000..e7c3fdd2fc --- /dev/null +++ b/.github/workflows/js-v2.yml @@ -0,0 +1,83 @@ +on: + push: + branches: + - main + pull_request: + branches: + - "*" + types: + - opened + - synchronize + - reopened + - ready_for_review + +name: js-tests-v2 + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + stateless-js-v2: + name: stateless-js-v2 + if: github.event.pull_request.draft == false + runs-on: ubuntu-latest + + env: + LIGHT_PROTOCOL_VERSION: V2 + + # services: + # redis: + # image: redis:8.0.1 + # ports: + # - 6379:6379 + # options: >- + # --health-cmd "redis-cli ping" + # --health-interval 10s + # --health-timeout 5s + # --health-retries 5 + + # env: + # REDIS_URL: redis://localhost:6379 + + steps: + - name: Checkout sources + uses: actions/checkout@v4 + + - name: Setup and build + uses: ./.github/actions/setup-and-build + with: + components: "node,pnpm,solana,anchor,jq,keys,dependencies" + + - name: Build stateless.js with V2 + run: | + source ./scripts/devenv.sh + cd js/stateless.js + pnpm build:v2 + + - name: Build compressed-token with V2 + run: | + source ./scripts/devenv.sh + cd js/compressed-token + pnpm build:v2 + + - name: Build CLI + run: | + source ./scripts/devenv.sh + npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache + + # Comment for breaking changes to Photon + - name: Run CLI tests + run: | + source ./scripts/devenv.sh + npx nx test @lightprotocol/zk-compression-cli + + - name: Run stateless.js tests with V2 + run: | + source ./scripts/devenv.sh + LIGHT_PROTOCOL_VERSION=V2 npx nx test @lightprotocol/stateless.js + + - name: Run compressed-token tests with V2 + run: | + source ./scripts/devenv.sh + LIGHT_PROTOCOL_VERSION=V2 npx nx test @lightprotocol/compressed-token diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index f8da21ae13..87dd171baa 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -11,31 +11,34 @@ on: - reopened - ready_for_review -name: js-tests +name: js-tests-v1 concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: - stateless-js: - name: stateless-js + stateless-js-v1: + name: stateless-js-v1 if: github.event.pull_request.draft == false runs-on: ubuntu-latest - services: - redis: - image: redis:8.0.1 - ports: - - 6379:6379 - options: >- - --health-cmd "redis-cli ping" - --health-interval 10s - --health-timeout 5s - --health-retries 5 - env: - REDIS_URL: redis://localhost:6379 + LIGHT_PROTOCOL_VERSION: V1 + + # services: + # redis: + # image: redis:8.0.1 + # ports: + # - 6379:6379 + # options: >- + # --health-cmd "redis-cli ping" + # --health-interval 10s + # --health-timeout 5s + # --health-retries 5 + + # env: + # REDIS_URL: redis://localhost:6379 steps: - name: Checkout sources @@ -43,6 +46,20 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + components: "node,pnpm,solana,anchor,jq,keys,dependencies" + + - name: Build stateless.js with V1 + run: | + source ./scripts/devenv.sh + cd js/stateless.js + pnpm build:v1 + + - name: Build compressed-token with V1 + run: | + source ./scripts/devenv.sh + cd js/compressed-token + pnpm build:v1 - name: Build CLI run: | @@ -55,12 +72,12 @@ jobs: source ./scripts/devenv.sh npx nx test @lightprotocol/zk-compression-cli - - name: Run stateless.js tests + - name: Run stateless.js tests with V1 run: | source ./scripts/devenv.sh - npx nx test @lightprotocol/stateless.js + LIGHT_PROTOCOL_VERSION=V1 npx nx test @lightprotocol/stateless.js - - name: Run compressed-token tests + - name: Run compressed-token tests with V1 run: | source ./scripts/devenv.sh - npx nx test @lightprotocol/compressed-token + LIGHT_PROTOCOL_VERSION=V1 npx nx test @lightprotocol/compressed-token diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index eec2218324..9f1bf76a3c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -14,6 +14,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + components: "rust,node,pnpm,solana,anchor,jq,keys,dependencies" - name: Install cargo-workspaces run: | diff --git a/Cargo.lock b/Cargo.lock index d7eb03566e..1ff5a95726 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3642,6 +3642,7 @@ dependencies = [ "quote", "solana-pubkey", "syn 2.0.101", +<<<<<<< HEAD ] [[package]] @@ -3674,6 +3675,8 @@ dependencies = [ "light-zero-copy", "solana-pubkey", "thiserror 2.0.12", +======= +>>>>>>> ad722943f (chore: backport breaking api changes to v1 js SDKs (#1661)) ] [[package]] diff --git a/js/compressed-token/package.json b/js/compressed-token/package.json index 9b0fab25b5..c9fdc922ba 100644 --- a/js/compressed-token/package.json +++ b/js/compressed-token/package.json @@ -78,8 +78,12 @@ }, "scripts": { "test": "pnpm test:e2e:all", + "test:v1": "LIGHT_PROTOCOL_VERSION=V1 pnpm test", + "test:v2": "LIGHT_PROTOCOL_VERSION=V2 pnpm test", "test-all": "vitest run", "test:unit:all": "EXCLUDE_E2E=true vitest run", + "test:unit:all:v1": "LIGHT_PROTOCOL_VERSION=V1 vitest run tests/unit --reporter=verbose", + "test:unit:all:v2": "LIGHT_PROTOCOL_VERSION=V2 vitest run tests/unit --reporter=verbose", "test-all:verbose": "vitest run --reporter=verbose", "test-validator": "./../../cli/test_bin/run test-validator --prover-run-mode rpc", "test-validator-skip-prover": "./../../cli/test_bin/run test-validator --skip-prover", @@ -104,6 +108,10 @@ "pull-idl": "../../scripts/push-compressed-token-idl.sh", "build": "rimraf dist && pnpm build:bundle", "build:bundle": "rollup -c", + "build:v1": "BUILD_VERSION=V1 pnpm build:stateless:v1 && BUILD_VERSION=V1 pnpm build", + "build:v2": "BUILD_VERSION=V2 pnpm build:stateless:v2 && BUILD_VERSION=V2 pnpm build", + "build:stateless:v1": "cd ../stateless.js && pnpm build:v1", + "build:stateless:v2": "cd ../stateless.js && pnpm build:v2", "format": "prettier --write .", "lint": "eslint ." }, diff --git a/js/compressed-token/rollup.config.js b/js/compressed-token/rollup.config.js index 4fac8ae5ef..e4e9ef8d6d 100644 --- a/js/compressed-token/rollup.config.js +++ b/js/compressed-token/rollup.config.js @@ -6,6 +6,7 @@ import commonjs from '@rollup/plugin-commonjs'; import alias from '@rollup/plugin-alias'; import json from '@rollup/plugin-json'; import terser from '@rollup/plugin-terser'; +import replace from '@rollup/plugin-replace'; const rolls = (fmt, env) => ({ input: 'src/index.ts', @@ -22,6 +23,12 @@ const rolls = (fmt, env) => ({ '@lightprotocol/stateless.js', ], plugins: [ + replace({ + preventAssignment: true, + values: { + '__BUILD_VERSION__': JSON.stringify(process.env.BUILD_VERSION || 'V1'), + }, + }), json(), typescript({ target: fmt === 'es' ? 'ES2022' : 'ES2017', diff --git a/js/compressed-token/src/types.ts b/js/compressed-token/src/types.ts index 65aa9fd6b1..09ff12f1cd 100644 --- a/js/compressed-token/src/types.ts +++ b/js/compressed-token/src/types.ts @@ -73,7 +73,6 @@ export type BatchCompressInstructionData = { bump: number; }; - export type MintToInstructionData = { recipients: PublicKey[]; amounts: BN[]; diff --git a/js/compressed-token/src/utils/version-check.ts b/js/compressed-token/src/utils/version-check.ts new file mode 100644 index 0000000000..41b0bc4f83 --- /dev/null +++ b/js/compressed-token/src/utils/version-check.ts @@ -0,0 +1,40 @@ +import { featureFlags, VERSION } from '@lightprotocol/stateless.js'; + +/** + * Validates that the built version of stateless.js matches the expected version. + * Throws an error if there's a mismatch. + * + * @param expectedVersion - The version expected (defaults to LIGHT_PROTOCOL_VERSION env var or V1) + * @throws Error if the versions don't match + */ +export function validateVersionConsistency(expectedVersion?: string): void { + const expected = + expectedVersion || process.env.LIGHT_PROTOCOL_VERSION || VERSION.V1; + const actual = featureFlags.version.replace(/['"]/g, ''); + + if (actual !== expected) { + throw new Error( + `Version mismatch detected!\n` + + `Expected: ${expected} (from ${expectedVersion ? 'parameter' : 'LIGHT_PROTOCOL_VERSION env var'})\n` + + `Actual: ${actual} (from built stateless.js)\n\n` + + `This means stateless.js was built with ${actual} but you're trying to use ${expected}.\n` + + `Please rebuild both packages with the same version:\n` + + ` pnpm build:${expected.toLowerCase()}\n` + + `This command will automatically build both stateless.js and compressed-token with ${expected}.`, + ); + } +} + +/** + * Gets the current version from stateless.js + */ +export function getCurrentVersion(): string { + return featureFlags.version.replace(/['"]/g, ''); +} + +/** + * Checks if the current version is V2 + */ +export function isV2(): boolean { + return featureFlags.isV2(); +} diff --git a/js/compressed-token/tests/setup/version-check.ts b/js/compressed-token/tests/setup/version-check.ts new file mode 100644 index 0000000000..9c1fdaedba --- /dev/null +++ b/js/compressed-token/tests/setup/version-check.ts @@ -0,0 +1,16 @@ +import { validateVersionConsistency } from '../../src/utils/version-check'; + +// Only used in tests. +export default function setup() { + console.log('Checking version consistency...'); + + try { + validateVersionConsistency(); + const expectedVersion = process.env.LIGHT_PROTOCOL_VERSION || 'V1'; + console.log(`✅ Version check passed: Using ${expectedVersion}`); + } catch (error) { + console.error('❌ Version check failed:'); + console.error(error.message); + process.exit(1); + } +} diff --git a/js/compressed-token/tests/unit/version.test.ts b/js/compressed-token/tests/unit/version.test.ts new file mode 100644 index 0000000000..fd6b369116 --- /dev/null +++ b/js/compressed-token/tests/unit/version.test.ts @@ -0,0 +1,62 @@ +import { describe, it, expect, beforeAll } from 'vitest'; +import { featureFlags, VERSION } from '@lightprotocol/stateless.js'; + +describe('Versioning', () => { + beforeAll(() => { + const expectedVersion = + process.env.LIGHT_PROTOCOL_VERSION || VERSION.V1; + const actualVersion = featureFlags.version.replace(/['"]/g, ''); + + if (actualVersion !== expectedVersion) { + throw new Error( + `Version mismatch detected!\n` + + `Expected: ${expectedVersion} (from LIGHT_PROTOCOL_VERSION env var)\n` + + `Actual: ${actualVersion} (from built stateless.js)\n\n` + + `This means stateless.js was built with ${actualVersion} but you're trying to test with ${expectedVersion}.\n` + + `Please rebuild stateless.js with the correct version:\n` + + ` cd ../stateless.js && pnpm build:${expectedVersion.toLowerCase()}\n` + + `Or use the compressed-token build command that handles this automatically:\n` + + ` pnpm build:${expectedVersion.toLowerCase()}`, + ); + } + }); + + it('should use version from stateless.js', () => { + console.log('Current version from stateless.js:', featureFlags.version); + console.log('isV2() from stateless.js:', featureFlags.isV2()); + console.log( + 'Environment variable:', + process.env.LIGHT_PROTOCOL_VERSION, + ); + + expect(featureFlags.version).toBeDefined(); + const actualVersion = featureFlags.version.replace(/['"]/g, ''); + expect([VERSION.V1, VERSION.V2]).toContain(actualVersion); + }); + + it('should respect LIGHT_PROTOCOL_VERSION environment variable', () => { + const expectedVersion = + process.env.LIGHT_PROTOCOL_VERSION || VERSION.V1; + const actualVersion = featureFlags.version.replace(/['"]/g, ''); + expect(actualVersion).toBe(expectedVersion); + }); + + it('isV2() should return correct value', () => { + const actualVersion = featureFlags.version.replace(/['"]/g, ''); + const expectedIsV2 = actualVersion === VERSION.V2; + expect(featureFlags.isV2()).toBe(expectedIsV2); + }); + + it('compressed-token should use the same version as stateless.js', () => { + const actualVersion = featureFlags.version.replace(/['"]/g, ''); + const isV2 = featureFlags.isV2(); + + if (process.env.LIGHT_PROTOCOL_VERSION === 'V2') { + expect(actualVersion).toBe(VERSION.V2); + expect(isV2).toBe(true); + } else { + expect(actualVersion).toBe(VERSION.V1); + expect(isV2).toBe(false); + } + }); +}); diff --git a/js/compressed-token/vitest.config.ts b/js/compressed-token/vitest.config.ts index 26a728edbe..3bbbf1ad7c 100644 --- a/js/compressed-token/vitest.config.ts +++ b/js/compressed-token/vitest.config.ts @@ -12,6 +12,7 @@ export default defineConfig({ testTimeout: 350000, hookTimeout: 100000, reporters: ['verbose'], + globalSetup: './tests/setup/version-check.ts', }, define: { 'import.meta.vitest': false, diff --git a/js/stateless.js/BUILD.md b/js/stateless.js/BUILD.md new file mode 100644 index 0000000000..5d1aa6f59c --- /dev/null +++ b/js/stateless.js/BUILD.md @@ -0,0 +1,16 @@ +Stateless.js compiles with V1 API by default. To switch over to V2 endpoints (with backward compatibility for V1 state), run: + +```bash +pnpm build:v2 +# or +BUILD_VERSION=V2 pnpm build +``` + +## Usage in Code + +```typescript +// From rpc.ts +const endpoint = featureFlags.isV2() + ? versionedEndpoint('getCompressedAccountV2') + : versionedEndpoint('getCompressedAccount'); +``` diff --git a/js/stateless.js/package.json b/js/stateless.js/package.json index f5430e0bb9..79876f60e9 100644 --- a/js/stateless.js/package.json +++ b/js/stateless.js/package.json @@ -87,7 +87,11 @@ "scripts": { "test": "pnpm test:unit:all && pnpm test:e2e:all", "test-all": "vitest run", + "test:v1": "LIGHT_PROTOCOL_VERSION=V1 pnpm test", + "test:v2": "LIGHT_PROTOCOL_VERSION=V2 pnpm test", "test:unit:all": "vitest run tests/unit --reporter=verbose", + "test:unit:all:v1": "LIGHT_PROTOCOL_VERSION=V1 vitest run tests/unit --reporter=verbose", + "test:unit:all:v2": "LIGHT_PROTOCOL_VERSION=V2 vitest run tests/unit --reporter=verbose", "test:unit:tree-info": "vitest run tests/unit/utils/tree-info.test.ts --reporter=verbose", "test:conversions": "vitest run tests/unit/utils/conversion.test.ts --reporter=verbose", "test-validator": "./../../cli/test_bin/run test-validator --prover-run-mode rpc", @@ -96,7 +100,7 @@ "test:e2e:compress": "pnpm test-validator && vitest run tests/e2e/compress.test.ts --reporter=verbose", "test:e2e:test-rpc": "pnpm test-validator && vitest run tests/e2e/test-rpc.test.ts --reporter=verbose --bail=1", "test:e2e:rpc-interop": "pnpm test-validator && vitest run tests/e2e/rpc-interop.test.ts --reporter=verbose --bail=1", - "test:e2e:rpc-multi-trees": "pnpm test-validator && vitest run tests/e2e/rpc-multi-trees.test.ts", + "test:e2e:rpc-multi-trees": "pnpm test-validator && vitest run tests/e2e/rpc-multi-trees.test.ts --reporter=verbose --bail=1", "test:e2e:browser": "pnpm playwright test", "test:e2e:all": "pnpm test-validator && vitest run tests/e2e/test-rpc.test.ts && vitest run tests/e2e/compress.test.ts && vitest run tests/e2e/transfer.test.ts && vitest run tests/e2e/rpc-interop.test.ts && pnpm test-validator-skip-prover && vitest run tests/e2e/rpc-multi-trees.test.ts && vitest run tests/e2e/layout.test.ts && vitest run tests/e2e/safe-conversion.test.ts", "test:index": "vitest run tests/e2e/program.test.ts", @@ -107,6 +111,8 @@ "pull-idls": "../../scripts/push-stateless-js-idls.sh && ../../scripts/push-compressed-token-idl.sh", "build": "rimraf dist && pnpm build:bundle", "build:bundle": "rollup -c", + "build:v1": "BUILD_VERSION=V1 pnpm build", + "build:v2": "BUILD_VERSION=V2 pnpm build", "format": "prettier --write .", "lint": "eslint ." }, diff --git a/js/stateless.js/rollup.config.js b/js/stateless.js/rollup.config.js index c78a4a1ea1..8e559bc459 100644 --- a/js/stateless.js/rollup.config.js +++ b/js/stateless.js/rollup.config.js @@ -4,8 +4,9 @@ import dts from 'rollup-plugin-dts'; import resolve from '@rollup/plugin-node-resolve'; import commonjs from '@rollup/plugin-commonjs'; import terser from '@rollup/plugin-terser'; - +import replace from '@rollup/plugin-replace'; import json from '@rollup/plugin-json'; + const rolls = (fmt, env) => ({ input: 'src/index.ts', output: { @@ -16,6 +17,12 @@ const rolls = (fmt, env) => ({ }, external: ['@solana/web3.js'], plugins: [ + replace({ + preventAssignment: true, + values: { + '__BUILD_VERSION__': JSON.stringify(process.env.BUILD_VERSION || 'V1'), + }, + }), typescript({ target: fmt === 'es' ? 'ES2022' : 'ES2017', outDir: `dist/${fmt}/${env}`, diff --git a/js/stateless.js/src/constants.ts b/js/stateless.js/src/constants.ts index af9041e6f4..3b4de323e1 100644 --- a/js/stateless.js/src/constants.ts +++ b/js/stateless.js/src/constants.ts @@ -14,8 +14,23 @@ export enum VERSION { * Feature flags. Only use if you know what you are doing. */ export const featureFlags = { - version: VERSION.V1, - isV2: () => featureFlags.version.toUpperCase() === 'V2', + version: ((): VERSION => { + // Check if we're in a build environment (replaced by rollup) + if ('__BUILD_VERSION__' !== '__BUILD_' + 'VERSION__') { + return '__BUILD_VERSION__' as VERSION; + } + // Otherwise, check runtime environment variable (for tests) + if ( + typeof process !== 'undefined' && + process.env?.LIGHT_PROTOCOL_VERSION + ) { + return process.env.LIGHT_PROTOCOL_VERSION as VERSION; + } + // Default to V1 + return VERSION.V1; + })(), + isV2: () => + featureFlags.version.replace(/['"]/g, '').toUpperCase() === 'V2', }; /** @@ -24,7 +39,7 @@ export const featureFlags = { * or 'getCompressedAccountV2' (V2) */ export const versionedEndpoint = (base: string) => - featureFlags.version.toUpperCase() === 'V1' ? base : `${base}V2`; + featureFlags.isV2() ? `${base}V2` : base; export const FIELD_SIZE = new BN( '21888242871839275222246405745257275088548364400416034343698204186575808495617', diff --git a/js/stateless.js/src/globals.d.ts b/js/stateless.js/src/globals.d.ts new file mode 100644 index 0000000000..b2ba4b3072 --- /dev/null +++ b/js/stateless.js/src/globals.d.ts @@ -0,0 +1 @@ +declare const __BUILD_VERSION__: 'V1' | 'V2'; diff --git a/js/stateless.js/src/rpc.ts b/js/stateless.js/src/rpc.ts index 9d57305700..9bc3660e5f 100644 --- a/js/stateless.js/src/rpc.ts +++ b/js/stateless.js/src/rpc.ts @@ -339,7 +339,7 @@ export const proverRequest = async ( method: 'inclusion' | 'new-address' | 'combined', params: any = [], log = false, - publicInputHash: BN | undefined = undefined, + _publicInputHash: BN | undefined = undefined, // Not supported. ): Promise => { let logMsg: string = ''; @@ -1065,7 +1065,7 @@ export class Rpc extends Connection implements CompressionApiInterface { stateTreeInfo, bn(item.hash.toArray('be', 32)), item.leafIndex, - false, + featureFlags.isV2() ? item.proveByIndex : false, ), item.owner, bn(item.lamports), @@ -1892,7 +1892,7 @@ export class Rpc extends Connection implements CompressionApiInterface { .concat(value.addresses.map((r: any) => r.rootIndex)), proveByIndices: value.accounts .map((r: any) => r.rootIndex.proveByIndex) - .concat(value.addresses.map((r: any) => false)), + .concat(value.addresses.map((r: any) => false)), // addresses.proveByIndex is always false. treeInfos: value.accounts .map((r: any) => r.merkleContext) .concat( diff --git a/js/stateless.js/test-version.js b/js/stateless.js/test-version.js new file mode 100644 index 0000000000..b6a3b7f712 --- /dev/null +++ b/js/stateless.js/test-version.js @@ -0,0 +1,17 @@ +#!/usr/bin/env node + +// Test script to verify the compiled version is correctly set +const { featureFlags } = require('./dist/cjs/node/index.cjs'); + +console.log('Testing build version...'); +console.log('featureFlags.version:', featureFlags.version); +console.log('featureFlags.isV2():', featureFlags.isV2()); + +const expectedVersion = process.env.EXPECTED_VERSION || 'V1'; +if (featureFlags.version === expectedVersion) { + console.log(`✅ Success: Version is correctly set to ${expectedVersion}`); + process.exit(0); +} else { + console.error(`❌ Error: Expected version ${expectedVersion} but got ${featureFlags.version}`); + process.exit(1); +} \ No newline at end of file diff --git a/js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts b/js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts index ca19488726..e106c634ee 100644 --- a/js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts +++ b/js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts @@ -11,6 +11,7 @@ import { test.describe('RPC in browser', () => { const { merkleTree } = defaultTestStateTreeAccounts(); + test.beforeAll(async ({ page }) => { try { diff --git a/js/stateless.js/tests/e2e/rpc-interop.test.ts b/js/stateless.js/tests/e2e/rpc-interop.test.ts index b175435633..93e491053c 100644 --- a/js/stateless.js/tests/e2e/rpc-interop.test.ts +++ b/js/stateless.js/tests/e2e/rpc-interop.test.ts @@ -69,7 +69,7 @@ describe('rpc-interop', () => { const transferAmount = 1e4; const numberOfTransfers = 15; - it('getCompressedAccountsByOwner [noforester] filter should work', async () => { + it.only('getCompressedAccountsByOwner [noforester] filter should work', async () => { let accs = await rpc.getCompressedAccountsByOwner(payer.publicKey, { filters: [ { diff --git a/js/stateless.js/tests/unit/version.test.ts b/js/stateless.js/tests/unit/version.test.ts new file mode 100644 index 0000000000..76285d96fd --- /dev/null +++ b/js/stateless.js/tests/unit/version.test.ts @@ -0,0 +1,27 @@ +import { describe, it, expect } from 'vitest'; +import { featureFlags, VERSION } from '../../src/constants'; + +describe('Version System', () => { + it('should have version set', () => { + console.log('Current version:', featureFlags.version); + console.log('isV2():', featureFlags.isV2()); + console.log( + 'Environment variable:', + process.env.LIGHT_PROTOCOL_VERSION, + ); + + expect(featureFlags.version).toBeDefined(); + expect([VERSION.V1, VERSION.V2]).toContain(featureFlags.version); + }); + + it('should respect LIGHT_PROTOCOL_VERSION environment variable', () => { + const expectedVersion = + process.env.LIGHT_PROTOCOL_VERSION || VERSION.V1; + expect(featureFlags.version).toBe(expectedVersion); + }); + + it('isV2() should return correct value', () => { + const expectedIsV2 = featureFlags.version === VERSION.V2; + expect(featureFlags.isV2()).toBe(expectedIsV2); + }); +}); diff --git a/scripts/INSTALL.md b/scripts/INSTALL.md new file mode 100644 index 0000000000..525626e8b9 --- /dev/null +++ b/scripts/INSTALL.md @@ -0,0 +1,40 @@ +# Component-Based Installation + +The `install.sh` script now supports selective component installation to reduce CI/CD times. + +## Usage + +```bash +# Install all components (default) +./scripts/install.sh + +# Install specific components only +./scripts/install.sh --components "node,pnpm,dependencies" + +# Install with full keys +./scripts/install.sh --full-keys --components "rust,solana,anchor" +``` + +## Available Components + +- `go` - Golang +- `rust` - Rust toolchain +- `node` - Node.js runtime +- `pnpm` - Package manager +- `solana` - Solana CLI tools +- `anchor` - Anchor +- `jq` - JSON processor +- `keys` - Gnark proving keys +- `dependencies` - all PNPM deps +- `redis` - Redis server (not needed for some tests) + +## GitHub Actions Usage + +In workflow files, specify components via the `setup-and-build` action: + +```yaml +- name: Setup and build + uses: ./.github/actions/setup-and-build + with: + components: "node,pnpm,solana,anchor,jq,keys,dependencies" +``` diff --git a/scripts/devenv.sh b/scripts/devenv.sh index 66c3867751..c164e39e98 100755 --- a/scripts/devenv.sh +++ b/scripts/devenv.sh @@ -66,6 +66,7 @@ PATH="${CARGO_HOME}/bin:${PATH}" # Export the modified PATH export PATH +# Comment to start prover without redis export REDIS_URL="redis://localhost:6379" # Enable small_ix feature by default in devenv diff --git a/scripts/install.sh b/scripts/install.sh index bfed6da5a3..5dbe444988 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -337,6 +337,9 @@ main() { # Parse command line arguments local key_type="light" local reset_log=true + local components="" + local install_all=true + while [[ $# -gt 0 ]]; do case $1 in --full-keys) @@ -347,6 +350,11 @@ main() { reset_log=false shift ;; + --components) + components="$2" + install_all=false + shift 2 + ;; *) echo "Unknown option: $1" exit 1 @@ -358,18 +366,28 @@ main() { rm -f "$INSTALL_LOG" fi - install_go - install_rust - install_node - install_pnpm - install_solana - install_anchor - install_jq - download_gnark_keys "$key_type" - install_dependencies - install_redis + # Helper function to check if component should be installed + should_install() { + local component=$1 + if $install_all; then + return 0 + fi + [[ ",$components," == *",$component,"* ]] + } + + # Install components based on selection + should_install "go" && install_go + should_install "rust" && install_rust + should_install "node" && install_node + should_install "pnpm" && install_pnpm + should_install "solana" && install_solana + should_install "anchor" && install_anchor + should_install "jq" && install_jq + should_install "keys" && download_gnark_keys "$key_type" + should_install "dependencies" && install_dependencies + should_install "redis" && install_redis echo "✨ Light Protocol development dependencies installed" } -main +main "$@" From 4f52bde26065af2a4f508da1d16ec02aed86b358 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 14:12:52 -0400 Subject: [PATCH 02/26] rm conflict in lockfile --- Cargo.lock | 3 --- js/compressed-token/rollup.config.js | 4 +++- js/stateless.js/rollup.config.js | 4 +++- js/stateless.js/test-version.js | 6 ++++-- js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts | 1 - 5 files changed, 10 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 1ff5a95726..d7eb03566e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3642,7 +3642,6 @@ dependencies = [ "quote", "solana-pubkey", "syn 2.0.101", -<<<<<<< HEAD ] [[package]] @@ -3675,8 +3674,6 @@ dependencies = [ "light-zero-copy", "solana-pubkey", "thiserror 2.0.12", -======= ->>>>>>> ad722943f (chore: backport breaking api changes to v1 js SDKs (#1661)) ] [[package]] diff --git a/js/compressed-token/rollup.config.js b/js/compressed-token/rollup.config.js index e4e9ef8d6d..120bf06cf7 100644 --- a/js/compressed-token/rollup.config.js +++ b/js/compressed-token/rollup.config.js @@ -26,7 +26,9 @@ const rolls = (fmt, env) => ({ replace({ preventAssignment: true, values: { - '__BUILD_VERSION__': JSON.stringify(process.env.BUILD_VERSION || 'V1'), + __BUILD_VERSION__: JSON.stringify( + process.env.BUILD_VERSION || 'V1', + ), }, }), json(), diff --git a/js/stateless.js/rollup.config.js b/js/stateless.js/rollup.config.js index 8e559bc459..a8bdce4a2f 100644 --- a/js/stateless.js/rollup.config.js +++ b/js/stateless.js/rollup.config.js @@ -20,7 +20,9 @@ const rolls = (fmt, env) => ({ replace({ preventAssignment: true, values: { - '__BUILD_VERSION__': JSON.stringify(process.env.BUILD_VERSION || 'V1'), + __BUILD_VERSION__: JSON.stringify( + process.env.BUILD_VERSION || 'V1', + ), }, }), typescript({ diff --git a/js/stateless.js/test-version.js b/js/stateless.js/test-version.js index b6a3b7f712..82f685d30f 100644 --- a/js/stateless.js/test-version.js +++ b/js/stateless.js/test-version.js @@ -12,6 +12,8 @@ if (featureFlags.version === expectedVersion) { console.log(`✅ Success: Version is correctly set to ${expectedVersion}`); process.exit(0); } else { - console.error(`❌ Error: Expected version ${expectedVersion} but got ${featureFlags.version}`); + console.error( + `❌ Error: Expected version ${expectedVersion} but got ${featureFlags.version}`, + ); process.exit(1); -} \ No newline at end of file +} diff --git a/js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts b/js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts index e106c634ee..ca19488726 100644 --- a/js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts +++ b/js/stateless.js/tests/e2e/browser/rpc.browser.spec.ts @@ -11,7 +11,6 @@ import { test.describe('RPC in browser', () => { const { merkleTree } = defaultTestStateTreeAccounts(); - test.beforeAll(async ({ page }) => { try { From 5b5cbdbf5192d6f2e51243a45938a26953543b9e Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 14:21:43 -0400 Subject: [PATCH 03/26] fix lint --- js/compressed-token/rollup.config.js | 1 + js/stateless.js/.eslintignore | 1 + js/stateless.js/rollup.config.js | 1 + js/stateless.js/src/constants.ts | 1 + 4 files changed, 4 insertions(+) diff --git a/js/compressed-token/rollup.config.js b/js/compressed-token/rollup.config.js index 120bf06cf7..7b02152c7e 100644 --- a/js/compressed-token/rollup.config.js +++ b/js/compressed-token/rollup.config.js @@ -1,3 +1,4 @@ +/* global process */ import typescript from '@rollup/plugin-typescript'; import nodePolyfills from 'rollup-plugin-polyfill-node'; import dts from 'rollup-plugin-dts'; diff --git a/js/stateless.js/.eslintignore b/js/stateless.js/.eslintignore index d2939734e4..26caad229d 100644 --- a/js/stateless.js/.eslintignore +++ b/js/stateless.js/.eslintignore @@ -1,3 +1,4 @@ node_modules lib dist +test-version.js diff --git a/js/stateless.js/rollup.config.js b/js/stateless.js/rollup.config.js index a8bdce4a2f..8b3abaa371 100644 --- a/js/stateless.js/rollup.config.js +++ b/js/stateless.js/rollup.config.js @@ -1,3 +1,4 @@ +/* global process */ import typescript from '@rollup/plugin-typescript'; import nodePolyfills from 'rollup-plugin-polyfill-node'; import dts from 'rollup-plugin-dts'; diff --git a/js/stateless.js/src/constants.ts b/js/stateless.js/src/constants.ts index 3b4de323e1..b34cfaaff5 100644 --- a/js/stateless.js/src/constants.ts +++ b/js/stateless.js/src/constants.ts @@ -16,6 +16,7 @@ export enum VERSION { export const featureFlags = { version: ((): VERSION => { // Check if we're in a build environment (replaced by rollup) + // eslint-disable-next-line no-constant-condition if ('__BUILD_VERSION__' !== '__BUILD_' + 'VERSION__') { return '__BUILD_VERSION__' as VERSION; } From acff4770552a7ebf5749ff91acfe9692af160f58 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 14:32:40 -0400 Subject: [PATCH 04/26] wip --- .github/workflows/js-v2.yml | 30 ++++++++++++++---------------- .github/workflows/js.yml | 30 ++++++++++++++---------------- 2 files changed, 28 insertions(+), 32 deletions(-) diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index e7c3fdd2fc..660f2a1dd3 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -23,22 +23,20 @@ jobs: if: github.event.pull_request.draft == false runs-on: ubuntu-latest + services: + redis: + image: redis:8.0.1 + ports: + - 6379:6379 + options: >- + --health-cmd "redis-cli ping" + --health-interval 10s + --health-timeout 5s + --health-retries 5 + env: LIGHT_PROTOCOL_VERSION: V2 - - # services: - # redis: - # image: redis:8.0.1 - # ports: - # - 6379:6379 - # options: >- - # --health-cmd "redis-cli ping" - # --health-interval 10s - # --health-timeout 5s - # --health-retries 5 - - # env: - # REDIS_URL: redis://localhost:6379 + REDIS_URL: redis://localhost:6379 steps: - name: Checkout sources @@ -46,8 +44,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build - with: - components: "node,pnpm,solana,anchor,jq,keys,dependencies" + # with: + # components: "node,pnpm,solana,anchor,jq,keys,dependencies" - name: Build stateless.js with V2 run: | diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 87dd171baa..1624efdb55 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -23,22 +23,20 @@ jobs: if: github.event.pull_request.draft == false runs-on: ubuntu-latest + services: + redis: + image: redis:8.0.1 + ports: + - 6379:6379 + options: >- + --health-cmd "redis-cli ping" + --health-interval 10s + --health-timeout 5s + --health-retries 5 + env: LIGHT_PROTOCOL_VERSION: V1 - - # services: - # redis: - # image: redis:8.0.1 - # ports: - # - 6379:6379 - # options: >- - # --health-cmd "redis-cli ping" - # --health-interval 10s - # --health-timeout 5s - # --health-retries 5 - - # env: - # REDIS_URL: redis://localhost:6379 + REDIS_URL: redis://localhost:6379 steps: - name: Checkout sources @@ -46,8 +44,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build - with: - components: "node,pnpm,solana,anchor,jq,keys,dependencies" + # with: + # components: "node,pnpm,solana,anchor,jq,keys,dependencies" - name: Build stateless.js with V1 run: | From 0926e6f313b52c62e6a2b2ec0d9e7aba5afb8dcd Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 16:32:34 -0400 Subject: [PATCH 05/26] wip --- js/stateless.js/tests/e2e/rpc-interop.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/js/stateless.js/tests/e2e/rpc-interop.test.ts b/js/stateless.js/tests/e2e/rpc-interop.test.ts index 93e491053c..b175435633 100644 --- a/js/stateless.js/tests/e2e/rpc-interop.test.ts +++ b/js/stateless.js/tests/e2e/rpc-interop.test.ts @@ -69,7 +69,7 @@ describe('rpc-interop', () => { const transferAmount = 1e4; const numberOfTransfers = 15; - it.only('getCompressedAccountsByOwner [noforester] filter should work', async () => { + it('getCompressedAccountsByOwner [noforester] filter should work', async () => { let accs = await rpc.getCompressedAccountsByOwner(payer.publicKey, { filters: [ { From c14b05cdd1028cc7a14e0986cd01689b6139328c Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 17:45:57 -0400 Subject: [PATCH 06/26] simplify build for ci --- .github/workflows/js-v2.yml | 14 +------------- .github/workflows/js.yml | 14 +------------- js/compressed-token/package.json | 8 ++++---- js/compressed-token/rollup.config.js | 2 +- js/stateless.js/BUILD.md | 2 +- js/stateless.js/package.json | 8 ++++---- js/stateless.js/rollup.config.js | 2 +- js/stateless.js/test-version.js | 7 ++++--- 8 files changed, 17 insertions(+), 40 deletions(-) diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index 660f2a1dd3..59d5e08312 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -47,22 +47,10 @@ jobs: # with: # components: "node,pnpm,solana,anchor,jq,keys,dependencies" - - name: Build stateless.js with V2 - run: | - source ./scripts/devenv.sh - cd js/stateless.js - pnpm build:v2 - - - name: Build compressed-token with V2 - run: | - source ./scripts/devenv.sh - cd js/compressed-token - pnpm build:v2 - - name: Build CLI run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache + npx nx build @lightprotocol/zk-compression-cli # Comment for breaking changes to Photon - name: Run CLI tests diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 1624efdb55..cac0148615 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -47,22 +47,10 @@ jobs: # with: # components: "node,pnpm,solana,anchor,jq,keys,dependencies" - - name: Build stateless.js with V1 - run: | - source ./scripts/devenv.sh - cd js/stateless.js - pnpm build:v1 - - - name: Build compressed-token with V1 - run: | - source ./scripts/devenv.sh - cd js/compressed-token - pnpm build:v1 - - name: Build CLI run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache + npx nx build @lightprotocol/zk-compression-cli # Comment for breaking changes to Photon - name: Run CLI tests diff --git a/js/compressed-token/package.json b/js/compressed-token/package.json index c9fdc922ba..f8fc8bad4c 100644 --- a/js/compressed-token/package.json +++ b/js/compressed-token/package.json @@ -106,10 +106,10 @@ "test:e2e:multi-pool": "pnpm test-validator && vitest run tests/e2e/multi-pool.test.ts --reporter=verbose", "test:e2e:all": "pnpm test-validator && vitest run tests/e2e/create-mint.test.ts && vitest run tests/e2e/mint-to.test.ts && vitest run tests/e2e/transfer.test.ts && vitest run tests/e2e/delegate.test.ts && vitest run tests/e2e/transfer-delegated.test.ts && vitest run tests/e2e/multi-pool.test.ts && vitest run tests/e2e/decompress-delegated.test.ts && pnpm test-validator-skip-prover && vitest run tests/e2e/compress.test.ts && vitest run tests/e2e/compress-spl-token-account.test.ts && vitest run tests/e2e/decompress.test.ts && vitest run tests/e2e/create-token-pool.test.ts && vitest run tests/e2e/approve-and-mint-to.test.ts && vitest run tests/e2e/rpc-token-interop.test.ts && vitest run tests/e2e/rpc-multi-trees.test.ts && vitest run tests/e2e/layout.test.ts && vitest run tests/e2e/select-accounts.test.ts", "pull-idl": "../../scripts/push-compressed-token-idl.sh", - "build": "rimraf dist && pnpm build:bundle", - "build:bundle": "rollup -c", - "build:v1": "BUILD_VERSION=V1 pnpm build:stateless:v1 && BUILD_VERSION=V1 pnpm build", - "build:v2": "BUILD_VERSION=V2 pnpm build:stateless:v2 && BUILD_VERSION=V2 pnpm build", + "build": "if [ \"$LIGHT_PROTOCOL_VERSION\" = \"V2\" ]; then LIGHT_PROTOCOL_VERSION=V2 pnpm build:bundle; else LIGHT_PROTOCOL_VERSION=V1 pnpm build:bundle; fi", + "build:bundle": "rimraf dist && rollup -c", + "build:v1": "LIGHT_PROTOCOL_VERSION=V1 pnpm build:stateless:v1 && LIGHT_PROTOCOL_VERSION=V1 pnpm build:bundle", + "build:v2": "LIGHT_PROTOCOL_VERSION=V2 pnpm build:stateless:v2 && LIGHT_PROTOCOL_VERSION=V2 pnpm build:bundle", "build:stateless:v1": "cd ../stateless.js && pnpm build:v1", "build:stateless:v2": "cd ../stateless.js && pnpm build:v2", "format": "prettier --write .", diff --git a/js/compressed-token/rollup.config.js b/js/compressed-token/rollup.config.js index 7b02152c7e..f19a4b3c29 100644 --- a/js/compressed-token/rollup.config.js +++ b/js/compressed-token/rollup.config.js @@ -28,7 +28,7 @@ const rolls = (fmt, env) => ({ preventAssignment: true, values: { __BUILD_VERSION__: JSON.stringify( - process.env.BUILD_VERSION || 'V1', + process.env.LIGHT_PROTOCOL_VERSION || 'V1', ), }, }), diff --git a/js/stateless.js/BUILD.md b/js/stateless.js/BUILD.md index 5d1aa6f59c..a3a6a37a08 100644 --- a/js/stateless.js/BUILD.md +++ b/js/stateless.js/BUILD.md @@ -3,7 +3,7 @@ Stateless.js compiles with V1 API by default. To switch over to V2 endpoints (wi ```bash pnpm build:v2 # or -BUILD_VERSION=V2 pnpm build +LIGHT_PROTOCOL_VERSION=V2 pnpm build ``` ## Usage in Code diff --git a/js/stateless.js/package.json b/js/stateless.js/package.json index 79876f60e9..b8a18226a2 100644 --- a/js/stateless.js/package.json +++ b/js/stateless.js/package.json @@ -109,10 +109,10 @@ "test:verbose": "vitest run --reporter=verbose", "test:testnet": "vitest run tests/e2e/testnet.test.ts --reporter=verbose", "pull-idls": "../../scripts/push-stateless-js-idls.sh && ../../scripts/push-compressed-token-idl.sh", - "build": "rimraf dist && pnpm build:bundle", - "build:bundle": "rollup -c", - "build:v1": "BUILD_VERSION=V1 pnpm build", - "build:v2": "BUILD_VERSION=V2 pnpm build", + "build": "if [ \"$LIGHT_PROTOCOL_VERSION\" = \"V2\" ]; then LIGHT_PROTOCOL_VERSION=V2 pnpm build:bundle; else LIGHT_PROTOCOL_VERSION=V1 pnpm build:bundle; fi", + "build:bundle": "rimraf dist && rollup -c", + "build:v1": "LIGHT_PROTOCOL_VERSION=V1 pnpm build:bundle", + "build:v2": "LIGHT_PROTOCOL_VERSION=V2 pnpm build:bundle", "format": "prettier --write .", "lint": "eslint ." }, diff --git a/js/stateless.js/rollup.config.js b/js/stateless.js/rollup.config.js index 8b3abaa371..c42b978ea9 100644 --- a/js/stateless.js/rollup.config.js +++ b/js/stateless.js/rollup.config.js @@ -22,7 +22,7 @@ const rolls = (fmt, env) => ({ preventAssignment: true, values: { __BUILD_VERSION__: JSON.stringify( - process.env.BUILD_VERSION || 'V1', + process.env.LIGHT_PROTOCOL_VERSION || 'V1', ), }, }), diff --git a/js/stateless.js/test-version.js b/js/stateless.js/test-version.js index 82f685d30f..9f76abe3ec 100644 --- a/js/stateless.js/test-version.js +++ b/js/stateless.js/test-version.js @@ -1,19 +1,20 @@ #!/usr/bin/env node // Test script to verify the compiled version is correctly set -const { featureFlags } = require('./dist/cjs/node/index.cjs'); +import { featureFlags } from './dist/cjs/node/index.cjs'; console.log('Testing build version...'); console.log('featureFlags.version:', featureFlags.version); console.log('featureFlags.isV2():', featureFlags.isV2()); const expectedVersion = process.env.EXPECTED_VERSION || 'V1'; -if (featureFlags.version === expectedVersion) { +const actualVersion = featureFlags.version.replace(/['"]/g, ''); +if (actualVersion === expectedVersion) { console.log(`✅ Success: Version is correctly set to ${expectedVersion}`); process.exit(0); } else { console.error( - `❌ Error: Expected version ${expectedVersion} but got ${featureFlags.version}`, + `❌ Error: Expected version ${expectedVersion} but got ${actualVersion}`, ); process.exit(1); } From 6d2b6f4b243bcb26952ea5bcaad6479a3e4f3fcd Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 17:52:48 -0400 Subject: [PATCH 07/26] wip --- .github/workflows/cli-v1.yml | 44 ++++++++++++++++++++++++++++++++++++ .github/workflows/cli-v2.yml | 44 ++++++++++++++++++++++++++++++++++++ .github/workflows/js-v2.yml | 15 ++---------- .github/workflows/js.yml | 15 ++---------- 4 files changed, 92 insertions(+), 26 deletions(-) create mode 100644 .github/workflows/cli-v1.yml create mode 100644 .github/workflows/cli-v2.yml diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml new file mode 100644 index 0000000000..381bbf4f00 --- /dev/null +++ b/.github/workflows/cli-v1.yml @@ -0,0 +1,44 @@ +on: + push: + branches: + - main + pull_request: + branches: + - "*" + types: + - opened + - synchronize + - reopened + - ready_for_review + +name: cli-tests-v1 + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + cli-v1: + name: cli-v1 + if: github.event.pull_request.draft == false + runs-on: ubuntu-latest + + env: + LIGHT_PROTOCOL_VERSION: V1 + + steps: + - name: Checkout sources + uses: actions/checkout@v4 + + - name: Setup and build + uses: ./.github/actions/setup-and-build + + - name: Build CLI with V1 + run: | + source ./scripts/devenv.sh + npx nx build @lightprotocol/zk-compression-cli + + - name: Run CLI tests with V1 + run: | + source ./scripts/devenv.sh + npx nx test @lightprotocol/zk-compression-cli diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml new file mode 100644 index 0000000000..3a85cc4331 --- /dev/null +++ b/.github/workflows/cli-v2.yml @@ -0,0 +1,44 @@ +on: + push: + branches: + - main + pull_request: + branches: + - "*" + types: + - opened + - synchronize + - reopened + - ready_for_review + +name: cli-tests-v2 + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + cli-v2: + name: cli-v2 + if: github.event.pull_request.draft == false + runs-on: ubuntu-latest + + env: + LIGHT_PROTOCOL_VERSION: V2 + + steps: + - name: Checkout sources + uses: actions/checkout@v4 + + - name: Setup and build + uses: ./.github/actions/setup-and-build + + - name: Build CLI with V2 + run: | + source ./scripts/devenv.sh + npx nx build @lightprotocol/zk-compression-cli + + - name: Run CLI tests with V2 + run: | + source ./scripts/devenv.sh + npx nx test @lightprotocol/zk-compression-cli diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index 59d5e08312..1857db779e 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -47,23 +47,12 @@ jobs: # with: # components: "node,pnpm,solana,anchor,jq,keys,dependencies" - - name: Build CLI - run: | - source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli - - # Comment for breaking changes to Photon - - name: Run CLI tests - run: | - source ./scripts/devenv.sh - npx nx test @lightprotocol/zk-compression-cli - - name: Run stateless.js tests with V2 run: | source ./scripts/devenv.sh - LIGHT_PROTOCOL_VERSION=V2 npx nx test @lightprotocol/stateless.js + npx nx test @lightprotocol/stateless.js - name: Run compressed-token tests with V2 run: | source ./scripts/devenv.sh - LIGHT_PROTOCOL_VERSION=V2 npx nx test @lightprotocol/compressed-token + npx nx test @lightprotocol/compressed-token diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index cac0148615..8c4cef615c 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -47,23 +47,12 @@ jobs: # with: # components: "node,pnpm,solana,anchor,jq,keys,dependencies" - - name: Build CLI - run: | - source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli - - # Comment for breaking changes to Photon - - name: Run CLI tests - run: | - source ./scripts/devenv.sh - npx nx test @lightprotocol/zk-compression-cli - - name: Run stateless.js tests with V1 run: | source ./scripts/devenv.sh - LIGHT_PROTOCOL_VERSION=V1 npx nx test @lightprotocol/stateless.js + npx nx test @lightprotocol/stateless.js - name: Run compressed-token tests with V1 run: | source ./scripts/devenv.sh - LIGHT_PROTOCOL_VERSION=V1 npx nx test @lightprotocol/compressed-token + npx nx test @lightprotocol/compressed-token From 95251d5c3d966e9f3cbdda1ce912135ea16012be Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 19:33:11 -0400 Subject: [PATCH 08/26] wip --- .github/workflows/js-v2.yml | 19 +++++++++++++++++-- .github/workflows/js.yml | 19 +++++++++++++++++-- 2 files changed, 34 insertions(+), 4 deletions(-) diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index 1857db779e..577a7a5bc0 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -44,8 +44,23 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build - # with: - # components: "node,pnpm,solana,anchor,jq,keys,dependencies" + + - name: Build stateless.js with V2 + run: | + source ./scripts/devenv.sh + cd js/stateless.js + pnpm build:v2 + + - name: Build compressed-token with V2 + run: | + source ./scripts/devenv.sh + cd js/compressed-token + pnpm build:v2 + + - name: Build CLI + run: | + source ./scripts/devenv.sh + npx nx build @lightprotocol/zk-compression-cli - name: Run stateless.js tests with V2 run: | diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 8c4cef615c..a871fa7e04 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -44,8 +44,23 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build - # with: - # components: "node,pnpm,solana,anchor,jq,keys,dependencies" + + - name: Build stateless.js with V1 + run: | + source ./scripts/devenv.sh + cd js/stateless.js + pnpm build:v1 + + - name: Build compressed-token with V1 + run: | + source ./scripts/devenv.sh + cd js/compressed-token + pnpm build:v1 + + - name: Build CLI + run: | + source ./scripts/devenv.sh + npx nx build @lightprotocol/zk-compression-cli - name: Run stateless.js tests with V1 run: | From 3de8d4bef18a33c8c4612d11e43c9b3aaff35664 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 19:38:08 -0400 Subject: [PATCH 09/26] wip --- .github/workflows/cli-v1.yml | 12 ++++++++++++ .github/workflows/cli-v2.yml | 12 ++++++++++++ 2 files changed, 24 insertions(+) diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index 381bbf4f00..038b48e9a5 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -33,6 +33,18 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + - name: Build stateless.js with V1 + run: | + source ./scripts/devenv.sh + cd js/stateless.js + pnpm build:v1 + + - name: Build compressed-token with V1 + run: | + source ./scripts/devenv.sh + cd js/compressed-token + pnpm build:v1 + - name: Build CLI with V1 run: | source ./scripts/devenv.sh diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index 3a85cc4331..79e3c79156 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -33,6 +33,18 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + - name: Build stateless.js with V2 + run: | + source ./scripts/devenv.sh + cd js/stateless.js + pnpm build:v2 + + - name: Build compressed-token with V2 + run: | + source ./scripts/devenv.sh + cd js/compressed-token + pnpm build:v2 + - name: Build CLI with V2 run: | source ./scripts/devenv.sh From 65321141570ff3c32f15ecb607a5d2d9f193373d Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 20:17:37 -0400 Subject: [PATCH 10/26] rm nx cache again --- .github/workflows/cli-v1.yml | 15 ++++++++++++++- .github/workflows/cli-v2.yml | 15 ++++++++++++++- .github/workflows/js-v2.yml | 15 ++++++++++++++- .github/workflows/js.yml | 15 ++++++++++++++- 4 files changed, 56 insertions(+), 4 deletions(-) diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index 038b48e9a5..93464e4c71 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -30,6 +30,19 @@ jobs: - name: Checkout sources uses: actions/checkout@v4 + # - name: Cache nx + # uses: actions/cache@v4 + # with: + # path: | + # node_modules/.cache/nx + # js/stateless.js/node_modules/.cache/nx + # js/compressed-token/node_modules/.cache/nx + # cli/node_modules/.cache/nx + # key: nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + # restore-keys: | + # nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- + # nx-cli-v1-${{ runner.os }}- + - name: Setup and build uses: ./.github/actions/setup-and-build @@ -48,7 +61,7 @@ jobs: - name: Build CLI with V1 run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli + npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache - name: Run CLI tests with V1 run: | diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index 79e3c79156..98a944680a 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -30,6 +30,19 @@ jobs: - name: Checkout sources uses: actions/checkout@v4 + # - name: Cache nx + # uses: actions/cache@v4 + # with: + # path: | + # node_modules/.cache/nx + # js/stateless.js/node_modules/.cache/nx + # js/compressed-token/node_modules/.cache/nx + # cli/node_modules/.cache/nx + # key: nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + # restore-keys: | + # nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- + # nx-cli-v2-${{ runner.os }}- + - name: Setup and build uses: ./.github/actions/setup-and-build @@ -48,7 +61,7 @@ jobs: - name: Build CLI with V2 run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli + npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache - name: Run CLI tests with V2 run: | diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index 577a7a5bc0..baafd20400 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -42,6 +42,19 @@ jobs: - name: Checkout sources uses: actions/checkout@v4 + - name: Cache nx + uses: actions/cache@v4 + with: + path: | + node_modules/.cache/nx + js/stateless.js/node_modules/.cache/nx + js/compressed-token/node_modules/.cache/nx + cli/node_modules/.cache/nx + key: nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + restore-keys: | + nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- + nx-js-v2-${{ runner.os }}- + - name: Setup and build uses: ./.github/actions/setup-and-build @@ -60,7 +73,7 @@ jobs: - name: Build CLI run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli + npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache - name: Run stateless.js tests with V2 run: | diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index a871fa7e04..3f465e0d5f 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -42,6 +42,19 @@ jobs: - name: Checkout sources uses: actions/checkout@v4 + - name: Cache nx + uses: actions/cache@v4 + with: + path: | + node_modules/.cache/nx + js/stateless.js/node_modules/.cache/nx + js/compressed-token/node_modules/.cache/nx + cli/node_modules/.cache/nx + key: nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + restore-keys: | + nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- + nx-js-v1-${{ runner.os }}- + - name: Setup and build uses: ./.github/actions/setup-and-build @@ -60,7 +73,7 @@ jobs: - name: Build CLI run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli + npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache - name: Run stateless.js tests with V1 run: | From 107bcc25ef8a6344d82abe53facf436a955d80f2 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 20:47:32 -0400 Subject: [PATCH 11/26] improve logging if prover exists with code 1 --- .github/workflows/cli-v1.yml | 6 ++++ .github/workflows/cli-v2.yml | 6 ++++ .github/workflows/js-v2.yml | 6 ++++ cli/src/utils/initTestEnv.ts | 8 ++++- cli/src/utils/process.ts | 48 +++++++++++++++++++++++++++- cli/src/utils/processProverServer.ts | 14 ++++++-- 6 files changed, 83 insertions(+), 5 deletions(-) diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index 93464e4c71..35d9d051ba 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -67,3 +67,9 @@ jobs: run: | source ./scripts/devenv.sh npx nx test @lightprotocol/zk-compression-cli + + - name: Display prover logs on failure + if: failure() + run: | + echo "=== Displaying prover logs ===" + find cli/test-ledger -name "*prover*.log" -type f -exec echo "=== Contents of {} ===" \; -exec cat {} \; -exec echo "=== End of {} ===" \; || echo "No prover logs found" diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index 98a944680a..d20b2efd3a 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -67,3 +67,9 @@ jobs: run: | source ./scripts/devenv.sh npx nx test @lightprotocol/zk-compression-cli + + - name: Display prover logs on failure + if: failure() + run: | + echo "=== Displaying prover logs ===" + find . -path "*/test-ledger/*prover*.log" -type f -exec echo "=== Contents of {} ===" \; -exec cat {} \; -exec echo "=== End of {} ===" \; || echo "No prover logs found" diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index baafd20400..420f7b1b18 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -84,3 +84,9 @@ jobs: run: | source ./scripts/devenv.sh npx nx test @lightprotocol/compressed-token + + - name: Display prover logs on failure + if: failure() + run: | + echo "=== Displaying prover logs ===" + find . -path "*/test-ledger/*prover*.log" -type f -exec echo "=== Contents of {} ===" \; -exec cat {} \; -exec echo "=== End of {} ===" \; || echo "No prover logs found" diff --git a/cli/src/utils/initTestEnv.ts b/cli/src/utils/initTestEnv.ts index 47d16b6b35..3622392ae9 100644 --- a/cli/src/utils/initTestEnv.ts +++ b/cli/src/utils/initTestEnv.ts @@ -149,7 +149,13 @@ export async function initTestEnv({ const config = getConfig(); config.proverUrl = `http://127.0.0.1:${proverPort}`; setConfig(config); - await startProver(proverPort, proverRunMode, circuits); + try { + await startProver(proverPort, proverRunMode, circuits); + } catch (error) { + console.error("Failed to start prover:", error); + // Prover logs will be automatically displayed by spawnBinary in process.ts + throw error; + } } } diff --git a/cli/src/utils/process.ts b/cli/src/utils/process.ts index ec70832a36..f3055ced41 100644 --- a/cli/src/utils/process.ts +++ b/cli/src/utils/process.ts @@ -7,6 +7,47 @@ import { promisify } from "util"; import axios from "axios"; const waitOn = require("wait-on"); +const readdir = promisify(fs.readdir); +const readFile = promisify(fs.readFile); + +/** + * Logs the contents of prover log files in test-ledger dir. + */ +export async function logProverFileContents() { + const testLedgerDir = path.join(__dirname, "../..", "test-ledger"); + + try { + if (!fs.existsSync(testLedgerDir)) { + console.log("test-ledger directory does not exist"); + return; + } + + const files = await readdir(testLedgerDir); + + const proverFiles = files.filter((file) => file.includes("prover")); + + if (proverFiles.length === 0) { + console.log("No prover log files found in test-ledger directory"); + return; + } + + for (const file of proverFiles) { + const filePath = path.join(testLedgerDir, file); + console.log(`\n========== Contents of ${file} ==========`); + + try { + const contents = await readFile(filePath, "utf8"); + console.log(contents); + console.log(`========== End of ${file} ==========\n`); + } catch (error) { + console.error(`Error reading ${file}:`, error); + } + } + } catch (error) { + console.error("Error accessing test-ledger directory:", error); + } +} + export async function killProcess(processName: string) { const processList = await find("name", processName); @@ -174,8 +215,13 @@ export function spawnBinary(command: string, args: string[] = []) { detached: true, }); - spawnedProcess.on("close", (code) => { + spawnedProcess.on("close", async (code) => { console.log(`${binaryName} process exited with code ${code}`); + // Log prover file contents if prover exits with non-zero code + if (code !== 0 && binaryName.includes("prover")) { + console.error(`Prover process failed with exit code ${code}`); + await logProverFileContents(); + } }); return spawnedProcess; diff --git a/cli/src/utils/processProverServer.ts b/cli/src/utils/processProverServer.ts index bdd7f0b84c..38359c2911 100644 --- a/cli/src/utils/processProverServer.ts +++ b/cli/src/utils/processProverServer.ts @@ -131,9 +131,17 @@ export async function startProver( args.push("--redis-url", redisUrl); } - spawnBinary(getProverPathByArch(), args); - await waitForServers([{ port: proverPort, path: "/" }]); - console.log(`Prover started successfully!`); + const proverProcess = spawnBinary(getProverPathByArch(), args); + + try { + await waitForServers([{ port: proverPort, path: "/" }]); + console.log(`Prover started successfully!`); + } catch (error) { + console.error( + "Failed to start prover - prover logs will be displayed above", + ); + throw error; + } } export function getProverNameByArch(): string { From 5d406f1123700c450463448ff334617e0044617f Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sat, 14 Jun 2025 20:52:54 -0400 Subject: [PATCH 12/26] fix cli yml --- .github/workflows/cli-v1.yml | 12 ++++++++++++ .github/workflows/cli-v2.yml | 12 ++++++++++++ cli/src/utils/initTestEnv.ts | 1 + 3 files changed, 25 insertions(+) diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index 35d9d051ba..45cc994909 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -23,8 +23,20 @@ jobs: if: github.event.pull_request.draft == false runs-on: ubuntu-latest + services: + redis: + image: redis:8.0.1 + ports: + - 6379:6379 + options: >- + --health-cmd "redis-cli ping" + --health-interval 10s + --health-timeout 5s + --health-retries 5 + env: LIGHT_PROTOCOL_VERSION: V1 + REDIS_URL: redis://localhost:6379 steps: - name: Checkout sources diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index d20b2efd3a..b97b152c1b 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -23,8 +23,20 @@ jobs: if: github.event.pull_request.draft == false runs-on: ubuntu-latest + services: + redis: + image: redis:8.0.1 + ports: + - 6379:6379 + options: >- + --health-cmd "redis-cli ping" + --health-interval 10s + --health-timeout 5s + --health-retries 5 + env: LIGHT_PROTOCOL_VERSION: V2 + REDIS_URL: redis://localhost:6379 steps: - name: Checkout sources diff --git a/cli/src/utils/initTestEnv.ts b/cli/src/utils/initTestEnv.ts index 3622392ae9..a2356de889 100644 --- a/cli/src/utils/initTestEnv.ts +++ b/cli/src/utils/initTestEnv.ts @@ -150,6 +150,7 @@ export async function initTestEnv({ config.proverUrl = `http://127.0.0.1:${proverPort}`; setConfig(config); try { + // TODO: check if using redisUrl is better here. await startProver(proverPort, proverRunMode, circuits); } catch (error) { console.error("Failed to start prover:", error); From ab686f527ea58477e82ff96610107e81043a593d Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 09:26:36 -0400 Subject: [PATCH 13/26] wip --- .github/workflows/cli-v1.yml | 28 +++++++++++++++------------- .github/workflows/cli-v2.yml | 28 +++++++++++++++------------- .github/workflows/js-v2.yml | 4 +++- .github/workflows/js.yml | 4 +++- nx.json | 21 ++++++++++++++++++--- 5 files changed, 54 insertions(+), 31 deletions(-) diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index 45cc994909..eca9daff02 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -37,23 +37,25 @@ jobs: env: LIGHT_PROTOCOL_VERSION: V1 REDIS_URL: redis://localhost:6379 + CI: true steps: - name: Checkout sources uses: actions/checkout@v4 - # - name: Cache nx - # uses: actions/cache@v4 - # with: - # path: | - # node_modules/.cache/nx - # js/stateless.js/node_modules/.cache/nx - # js/compressed-token/node_modules/.cache/nx - # cli/node_modules/.cache/nx - # key: nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} - # restore-keys: | - # nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- - # nx-cli-v1-${{ runner.os }}- + - name: Cache nx + uses: actions/cache@v4 + with: + path: | + .nx/cache + node_modules/.cache/nx + js/stateless.js/node_modules/.cache/nx + js/compressed-token/node_modules/.cache/nx + cli/node_modules/.cache/nx + key: nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + restore-keys: | + nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- + nx-cli-v1-${{ runner.os }}- - name: Setup and build uses: ./.github/actions/setup-and-build @@ -73,7 +75,7 @@ jobs: - name: Build CLI with V1 run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache + npx nx build @lightprotocol/zk-compression-cli - name: Run CLI tests with V1 run: | diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index b97b152c1b..d2e4683e70 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -37,23 +37,25 @@ jobs: env: LIGHT_PROTOCOL_VERSION: V2 REDIS_URL: redis://localhost:6379 + CI: true steps: - name: Checkout sources uses: actions/checkout@v4 - # - name: Cache nx - # uses: actions/cache@v4 - # with: - # path: | - # node_modules/.cache/nx - # js/stateless.js/node_modules/.cache/nx - # js/compressed-token/node_modules/.cache/nx - # cli/node_modules/.cache/nx - # key: nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} - # restore-keys: | - # nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- - # nx-cli-v2-${{ runner.os }}- + - name: Cache nx + uses: actions/cache@v4 + with: + path: | + .nx/cache + node_modules/.cache/nx + js/stateless.js/node_modules/.cache/nx + js/compressed-token/node_modules/.cache/nx + cli/node_modules/.cache/nx + key: nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + restore-keys: | + nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- + nx-cli-v2-${{ runner.os }}- - name: Setup and build uses: ./.github/actions/setup-and-build @@ -73,7 +75,7 @@ jobs: - name: Build CLI with V2 run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache + npx nx build @lightprotocol/zk-compression-cli - name: Run CLI tests with V2 run: | diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index 420f7b1b18..f698603cac 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -37,6 +37,7 @@ jobs: env: LIGHT_PROTOCOL_VERSION: V2 REDIS_URL: redis://localhost:6379 + CI: true steps: - name: Checkout sources @@ -46,6 +47,7 @@ jobs: uses: actions/cache@v4 with: path: | + .nx/cache node_modules/.cache/nx js/stateless.js/node_modules/.cache/nx js/compressed-token/node_modules/.cache/nx @@ -73,7 +75,7 @@ jobs: - name: Build CLI run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache + npx nx build @lightprotocol/zk-compression-cli - name: Run stateless.js tests with V2 run: | diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 3f465e0d5f..85169a05fb 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -37,6 +37,7 @@ jobs: env: LIGHT_PROTOCOL_VERSION: V1 REDIS_URL: redis://localhost:6379 + CI: true steps: - name: Checkout sources @@ -46,6 +47,7 @@ jobs: uses: actions/cache@v4 with: path: | + .nx/cache node_modules/.cache/nx js/stateless.js/node_modules/.cache/nx js/compressed-token/node_modules/.cache/nx @@ -73,7 +75,7 @@ jobs: - name: Build CLI run: | source ./scripts/devenv.sh - npx nx build @lightprotocol/zk-compression-cli --skip-nx-cache + npx nx build @lightprotocol/zk-compression-cli - name: Run stateless.js tests with V1 run: | diff --git a/nx.json b/nx.json index 3c8bc1b96c..e5f23b2f79 100644 --- a/nx.json +++ b/nx.json @@ -9,7 +9,7 @@ }, "targetDefaults": { "build": { - "cache": false, + "cache": true, "inputs": [ "noMarkdown", "^noMarkdown", @@ -22,11 +22,14 @@ "outputs": [ "{workspaceRoot}/target/deploy", "{workspaceRoot}/target/idl", - "{workspaceRoot}/target/types" + "{workspaceRoot}/target/types", + "{projectRoot}/dist", + "{projectRoot}/lib", + "{projectRoot}/bin" ] }, "test": { - "cache": false, + "cache": true, "inputs": [ "noMarkdown", "^noMarkdown", @@ -65,5 +68,17 @@ "^noTestLedger" ] } + }, + "daemon": { + "enabled": false + }, + "tasksRunnerOptions": { + "default": { + "runner": "nx/tasks-runners/default", + "options": { + "cacheableOperations": ["build", "test"], + "cacheDirectory": ".nx/cache" + } + } } } From 9e313a4126bc421e79edf11b4a767389345cbecb Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 10:35:53 -0400 Subject: [PATCH 14/26] check if conditional installs should stay --- .github/workflows/cli-v1.yml | 2 ++ .github/workflows/cli-v2.yml | 2 ++ .github/workflows/js-v2.yml | 2 ++ .github/workflows/js.yml | 2 ++ scripts/install.sh | 11 ++++++++++- 5 files changed, 18 insertions(+), 1 deletion(-) diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index eca9daff02..9d629015e2 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -59,6 +59,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + components: "node,pnpm,solana,anchor,jq,dependencies" - name: Build stateless.js with V1 run: | diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index d2e4683e70..5a14db5edd 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -59,6 +59,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + components: "node,pnpm,solana,anchor,jq,dependencies" - name: Build stateless.js with V2 run: | diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index f698603cac..bc18edc306 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -59,6 +59,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + components: "node,pnpm,solana,anchor,jq,dependencies" - name: Build stateless.js with V2 run: | diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 85169a05fb..407ffd48b3 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -59,6 +59,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + components: "node,pnpm,solana,anchor,jq,dependencies" - name: Build stateless.js with V1 run: | diff --git a/scripts/install.sh b/scripts/install.sh index 5dbe444988..d83a649dad 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -172,7 +172,7 @@ download_gnark_keys() { if ! is_installed "gnark_keys"; then echo "Downloading gnark keys..." ROOT_DIR="$(git rev-parse --show-toplevel)" - "${ROOT_DIR}/prover/server/scripts/download_keys.sh" "$key_type" + "${ROOT_DIR}/prover/server/scripts/download_keys.sh" "$1" log "gnark_keys" fi } @@ -351,12 +351,18 @@ main() { shift ;; --components) + if [ -z "$2" ] || [[ "$2" == --* ]]; then + echo "Error: --components requires a value" + exit 1 + fi components="$2" install_all=false shift 2 ;; *) echo "Unknown option: $1" + echo "Usage: $0 [--full-keys] [--no-reset] [--components ]" + echo "Components: go,rust,node,pnpm,solana,anchor,jq,keys,dependencies,redis" exit 1 ;; esac @@ -388,6 +394,9 @@ main() { should_install "redis" && install_redis echo "✨ Light Protocol development dependencies installed" + if ! $install_all; then + echo " Installed components: $components" + fi } main "$@" From 6f9a1244365973f9ea563f23f709e2f1e2d0f189 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 11:07:39 -0400 Subject: [PATCH 15/26] wip --- .github/actions/setup-and-build/action.yml | 8 +-- .github/workflows/cli-v1.yml | 2 +- .github/workflows/cli-v2.yml | 2 +- .github/workflows/forester-tests.yml | 2 + .github/workflows/js-v2.yml | 2 +- .github/workflows/js.yml | 2 +- .github/workflows/light-examples-tests.yml | 2 + .../workflows/light-system-programs-tests.yml | 2 + .github/workflows/lint.yml | 2 + .github/workflows/release.yml | 2 +- .github/workflows/rust.yml | 2 + scripts/install.sh | 49 +++++++++---------- 12 files changed, 41 insertions(+), 36 deletions(-) diff --git a/.github/actions/setup-and-build/action.yml b/.github/actions/setup-and-build/action.yml index 1c4fd06593..3be5c1de56 100644 --- a/.github/actions/setup-and-build/action.yml +++ b/.github/actions/setup-and-build/action.yml @@ -2,8 +2,8 @@ name: Setup and build description: Checkout sources, install dependencies, build and prepare for tests inputs: - components: - description: 'Comma-separated list of components to install (e.g., "node,pnpm,solana,anchor,jq,dependencies"). If not specified, all components are installed.' + skip-components: + description: 'Comma-separated list of components to skip (e.g., "redis,go"). If not specified, all components are installed.' required: false default: "" @@ -40,8 +40,8 @@ runs: if: steps.restore-local-cache.outputs.cache-hit != 'true' shell: bash run: | - if [ -n "${{ inputs.components }}" ]; then - ./scripts/install.sh --components "${{ inputs.components }}" + if [ -n "${{ inputs.skip-components }}" ]; then + ./scripts/install.sh --skip-components "${{ inputs.skip-components }}" else ./scripts/install.sh fi diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index 9d629015e2..fca2bb45d1 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -60,7 +60,7 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build with: - components: "node,pnpm,solana,anchor,jq,dependencies" + skip-components: "redis" - name: Build stateless.js with V1 run: | diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index 5a14db5edd..180fc6357a 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -60,7 +60,7 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build with: - components: "node,pnpm,solana,anchor,jq,dependencies" + skip-components: "redis" - name: Build stateless.js with V2 run: | diff --git a/.github/workflows/forester-tests.yml b/.github/workflows/forester-tests.yml index 29e2c29d32..d078ea52e8 100644 --- a/.github/workflows/forester-tests.yml +++ b/.github/workflows/forester-tests.yml @@ -96,6 +96,8 @@ jobs: - uses: actions/checkout@v4 - name: Setup and build uses: ./.github/actions/setup-and-build + with: + skip-components: "redis" - name: Clean build artifacts before tests shell: bash run: | diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index bc18edc306..dff50674c2 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -60,7 +60,7 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build with: - components: "node,pnpm,solana,anchor,jq,dependencies" + skip-components: "redis" - name: Build stateless.js with V2 run: | diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 407ffd48b3..8717d7103b 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -60,7 +60,7 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build with: - components: "node,pnpm,solana,anchor,jq,dependencies" + skip-components: "redis" - name: Build stateless.js with V1 run: | diff --git a/.github/workflows/light-examples-tests.yml b/.github/workflows/light-examples-tests.yml index 2c0d1d355a..ba06e8ae12 100644 --- a/.github/workflows/light-examples-tests.yml +++ b/.github/workflows/light-examples-tests.yml @@ -64,6 +64,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + skip-components: "redis" - name: build-programs run: | diff --git a/.github/workflows/light-system-programs-tests.yml b/.github/workflows/light-system-programs-tests.yml index a0c38ad6a4..6a27e02135 100644 --- a/.github/workflows/light-system-programs-tests.yml +++ b/.github/workflows/light-system-programs-tests.yml @@ -75,6 +75,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + skip-components: "redis" - name: Build CLI run: | diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 5657f8c9f1..d0070d5b46 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -28,6 +28,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + skip-components: "redis,go,keys" - name: Run linters run: | diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 9f1bf76a3c..781b74ad77 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -15,7 +15,7 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build with: - components: "rust,node,pnpm,solana,anchor,jq,keys,dependencies" + skip-components: "redis" - name: Install cargo-workspaces run: | diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 62775c9ba6..9051398500 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -87,6 +87,8 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build + with: + skip-components: "redis" - name: Build CLI run: | diff --git a/scripts/install.sh b/scripts/install.sh index d83a649dad..f727e42c72 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -337,8 +337,7 @@ main() { # Parse command line arguments local key_type="light" local reset_log=true - local components="" - local install_all=true + local skip_components="" while [[ $# -gt 0 ]]; do case $1 in @@ -350,19 +349,18 @@ main() { reset_log=false shift ;; - --components) + --skip-components) if [ -z "$2" ] || [[ "$2" == --* ]]; then - echo "Error: --components requires a value" + echo "Error: --skip-components requires a value" exit 1 fi - components="$2" - install_all=false + skip_components="$2" shift 2 ;; *) echo "Unknown option: $1" - echo "Usage: $0 [--full-keys] [--no-reset] [--components ]" - echo "Components: go,rust,node,pnpm,solana,anchor,jq,keys,dependencies,redis" + echo "Usage: $0 [--full-keys] [--no-reset] [--skip-components ]" + echo "Components that can be skipped: go,rust,node,pnpm,solana,anchor,jq,keys,dependencies,redis" exit 1 ;; esac @@ -372,30 +370,27 @@ main() { rm -f "$INSTALL_LOG" fi - # Helper function to check if component should be installed - should_install() { + # Helper function to check if component should be skipped + should_skip() { local component=$1 - if $install_all; then - return 0 - fi - [[ ",$components," == *",$component,"* ]] + [[ ",$skip_components," == *",$component,"* ]] } - # Install components based on selection - should_install "go" && install_go - should_install "rust" && install_rust - should_install "node" && install_node - should_install "pnpm" && install_pnpm - should_install "solana" && install_solana - should_install "anchor" && install_anchor - should_install "jq" && install_jq - should_install "keys" && download_gnark_keys "$key_type" - should_install "dependencies" && install_dependencies - should_install "redis" && install_redis + # Install components unless explicitly skipped + should_skip "go" || install_go + should_skip "rust" || install_rust + should_skip "node" || install_node + should_skip "pnpm" || install_pnpm + should_skip "solana" || install_solana + should_skip "anchor" || install_anchor + should_skip "jq" || install_jq + should_skip "keys" || download_gnark_keys "$key_type" + should_skip "dependencies" || install_dependencies + should_skip "redis" || install_redis echo "✨ Light Protocol development dependencies installed" - if ! $install_all; then - echo " Installed components: $components" + if [ -n "$skip_components" ]; then + echo " Skipped components: $skip_components" fi } From 81a93b35f82e80ca87eb16348c59357b4674f0b4 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 11:09:23 -0400 Subject: [PATCH 16/26] wip --- .github/workflows/release.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 781b74ad77..eec2218324 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -14,8 +14,6 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build - with: - skip-components: "redis" - name: Install cargo-workspaces run: | From 6d140f637f094e4d64bb220bf5a1221d9cd26234 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 12:06:26 -0400 Subject: [PATCH 17/26] add INSTALL.md --- INSTALL.md | 34 ++++++++++++++++++++++++++++++++++ scripts/INSTALL.md | 40 ---------------------------------------- 2 files changed, 34 insertions(+), 40 deletions(-) create mode 100644 INSTALL.md delete mode 100644 scripts/INSTALL.md diff --git a/INSTALL.md b/INSTALL.md new file mode 100644 index 0000000000..5a92044b0b --- /dev/null +++ b/INSTALL.md @@ -0,0 +1,34 @@ +Running `install.sh` is the first thing you should do after cloning this monorepo. + +```bash +# Install all components (default) +./scripts/install.sh + +# Install with all proving keys +./scripts/install.sh --full-keys + +# Skip specific components (only use if you know what you are doing) +./scripts/install.sh --skip-components "redis,keys,go" +``` + +## Default Components + +- `go` - Golang +- `rust` - Rust toolchain +- `node` - Node.js runtime +- `pnpm` - Package manager +- `solana` - Solana CLI tools +- `anchor` - Anchor +- `jq` - JSON processor +- `keys` - Gnark proving keys +- `dependencies` - all PNPM deps +- `redis` - Redis server (not needed for some tests) + +## CI Usage + +```yaml +- name: Setup and build + uses: ./.github/actions/setup-and-build + with: + skip-components: "redis" +``` diff --git a/scripts/INSTALL.md b/scripts/INSTALL.md deleted file mode 100644 index 525626e8b9..0000000000 --- a/scripts/INSTALL.md +++ /dev/null @@ -1,40 +0,0 @@ -# Component-Based Installation - -The `install.sh` script now supports selective component installation to reduce CI/CD times. - -## Usage - -```bash -# Install all components (default) -./scripts/install.sh - -# Install specific components only -./scripts/install.sh --components "node,pnpm,dependencies" - -# Install with full keys -./scripts/install.sh --full-keys --components "rust,solana,anchor" -``` - -## Available Components - -- `go` - Golang -- `rust` - Rust toolchain -- `node` - Node.js runtime -- `pnpm` - Package manager -- `solana` - Solana CLI tools -- `anchor` - Anchor -- `jq` - JSON processor -- `keys` - Gnark proving keys -- `dependencies` - all PNPM deps -- `redis` - Redis server (not needed for some tests) - -## GitHub Actions Usage - -In workflow files, specify components via the `setup-and-build` action: - -```yaml -- name: Setup and build - uses: ./.github/actions/setup-and-build - with: - components: "node,pnpm,solana,anchor,jq,keys,dependencies" -``` From eb23ea90852a4989d9e69844d7829547e0edc425 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 13:20:40 -0400 Subject: [PATCH 18/26] fix keys --- .github/actions/setup-and-build-nocheck/action.yml | 11 +++++++++++ .github/actions/setup-and-build/action.yml | 11 +++++++++++ cli/scripts/buildProver.sh | 7 +++++++ scripts/install.sh | 7 +++++-- 4 files changed, 34 insertions(+), 2 deletions(-) diff --git a/.github/actions/setup-and-build-nocheck/action.yml b/.github/actions/setup-and-build-nocheck/action.yml index 739174ad6d..9718cb4467 100644 --- a/.github/actions/setup-and-build-nocheck/action.yml +++ b/.github/actions/setup-and-build-nocheck/action.yml @@ -28,6 +28,17 @@ runs: run: | ./scripts/install.sh + - name: Ensure proving keys exist + shell: bash + run: | + # Always check if proving keys exist, even if cache was restored + if [ ! -d "prover/server/proving-keys" ] || [ -z "$(ls -A prover/server/proving-keys 2>/dev/null)" ]; then + echo "Proving keys not found or empty, downloading..." + ./prover/server/scripts/download_keys.sh light + else + echo "Proving keys already exist" + fi + - name: Set local environment shell: bash run: | diff --git a/.github/actions/setup-and-build/action.yml b/.github/actions/setup-and-build/action.yml index 3be5c1de56..6b233ef23a 100644 --- a/.github/actions/setup-and-build/action.yml +++ b/.github/actions/setup-and-build/action.yml @@ -46,6 +46,17 @@ runs: ./scripts/install.sh fi + - name: Ensure proving keys exist + shell: bash + run: | + # Always check if proving keys exist, even if cache was restored + if [ ! -d "prover/server/proving-keys" ] || [ -z "$(ls -A prover/server/proving-keys 2>/dev/null)" ]; then + echo "Proving keys not found or empty, downloading..." + ./prover/server/scripts/download_keys.sh light + else + echo "Proving keys already exist" + fi + - name: Set local environment shell: bash run: | diff --git a/cli/scripts/buildProver.sh b/cli/scripts/buildProver.sh index 81a15019bf..d71ec31974 100755 --- a/cli/scripts/buildProver.sh +++ b/cli/scripts/buildProver.sh @@ -14,6 +14,13 @@ if [ ! -e "$out_dir" ]; then mkdir -p "$out_dir" fi +# Check if proving keys exist before copying +if [ ! -d "${gnark_dir}/proving-keys" ] || [ -z "$(ls -A "${gnark_dir}/proving-keys" 2>/dev/null)" ]; then + echo "ERROR: Proving keys not found at ${gnark_dir}/proving-keys" + echo "Please run: ./prover/server/scripts/download_keys.sh light" + exit 1 +fi + cp -r "${gnark_dir}/proving-keys" "$out_dir" cd "$gnark_dir" diff --git a/scripts/install.sh b/scripts/install.sh index f727e42c72..8fcc4ae1dd 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -169,11 +169,14 @@ install_jq() { } download_gnark_keys() { - if ! is_installed "gnark_keys"; then + ROOT_DIR="$(git rev-parse --show-toplevel)" + # Always check if keys actually exist, not just the install log + if [ ! -d "${ROOT_DIR}/prover/server/proving-keys" ] || [ -z "$(ls -A "${ROOT_DIR}/prover/server/proving-keys" 2>/dev/null)" ]; then echo "Downloading gnark keys..." - ROOT_DIR="$(git rev-parse --show-toplevel)" "${ROOT_DIR}/prover/server/scripts/download_keys.sh" "$1" log "gnark_keys" + else + echo "Gnark keys already exist, skipping download..." fi } From 15586b29f5437add7830999ada4529f2c7b0ebf7 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 16:43:30 -0400 Subject: [PATCH 19/26] update INSTALL.md --- INSTALL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 5a92044b0b..46246fbfef 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -7,11 +7,11 @@ Running `install.sh` is the first thing you should do after cloning this monorep # Install with all proving keys ./scripts/install.sh --full-keys -# Skip specific components (only use if you know what you are doing) +# Skip components (only use if you know what you are doing) ./scripts/install.sh --skip-components "redis,keys,go" ``` -## Default Components +## Components - `go` - Golang - `rust` - Rust toolchain @@ -22,7 +22,7 @@ Running `install.sh` is the first thing you should do after cloning this monorep - `jq` - JSON processor - `keys` - Gnark proving keys - `dependencies` - all PNPM deps -- `redis` - Redis server (not needed for some tests) +- `redis` - Redis server (not needed for some ci workflows) ## CI Usage From 500e225499dff34069d2237828ca7ea04e448c15 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 17:25:07 -0400 Subject: [PATCH 20/26] fix cache --- .../setup-and-build-nocheck/action.yml | 52 ++++++-- .github/actions/setup-and-build/action.yml | 100 ++++++++++++-- cli/src/utils/process.ts | 2 - cli/src/utils/processProverServer.ts | 122 +++++++++++++++--- scripts/devenv.sh | 60 +++++++-- scripts/install.sh | 45 +++++-- 6 files changed, 315 insertions(+), 66 deletions(-) diff --git a/.github/actions/setup-and-build-nocheck/action.yml b/.github/actions/setup-and-build-nocheck/action.yml index 9718cb4467..162b3f3aec 100644 --- a/.github/actions/setup-and-build-nocheck/action.yml +++ b/.github/actions/setup-and-build-nocheck/action.yml @@ -20,7 +20,10 @@ runs: path: | .local prover/server/proving-keys - key: ${{ runner.os }}-local-${{ hashFiles('scripts/install.sh') }} + key: ${{ runner.os }}-local-v3-${{ hashFiles('scripts/install.sh', 'prover/server/scripts/download_keys.sh') }} + restore-keys: | + ${{ runner.os }}-local-v3- + ${{ runner.os }}-local-v2- - name: Install dependencies if: steps.restore-local-cache.outputs.cache-hit != 'true' @@ -28,24 +31,44 @@ runs: run: | ./scripts/install.sh - - name: Ensure proving keys exist + - name: Verify and repair installation shell: bash run: | - # Always check if proving keys exist, even if cache was restored - if [ ! -d "prover/server/proving-keys" ] || [ -z "$(ls -A prover/server/proving-keys 2>/dev/null)" ]; then - echo "Proving keys not found or empty, downloading..." - ./prover/server/scripts/download_keys.sh light - else - echo "Proving keys already exist" - fi + # Even if cache was restored, run install script to check and install missing components + # The install script now checks for actual file existence, so it will only install what's missing + echo "=== Verifying installation integrity ===" + ./scripts/install.sh --no-reset + + - name: Validate environment setup + shell: bash + run: | + echo "=== Validating environment setup ===" + source ./scripts/devenv.sh + + # Quick validation of critical components + [ -f "$GOROOT/bin/go" ] || { echo "✗ Go not found"; exit 1; } + [ -f "$CARGO_HOME/bin/cargo" ] || { echo "✗ Cargo not found"; exit 1; } + [ -d "prover/server/proving-keys" ] || { echo "✗ Proving keys not found"; exit 1; } + + echo "✓ All critical components found" - name: Set local environment shell: bash run: | source ./scripts/devenv.sh + # Export critical environment variables for subsequent steps + echo "GOROOT=$GOROOT" >> $GITHUB_ENV + echo "CARGO_HOME=$CARGO_HOME" >> $GITHUB_ENV + echo "RUSTUP_HOME=$RUSTUP_HOME" >> $GITHUB_ENV + echo "PATH=$PATH" >> $GITHUB_ENV + echo "CARGO_FEATURES=$CARGO_FEATURES" >> $GITHUB_ENV + echo "REDIS_URL=$REDIS_URL" >> $GITHUB_ENV - name: Rust cache uses: swatinem/rust-cache@v2 + with: + cache-all-crates: true + cache-on-failure: true - name: Setup pnpm uses: pnpm/action-setup@v4 @@ -74,14 +97,17 @@ runs: run: | source ./scripts/devenv.sh mkdir -p /home/runner/.config/solana/ - solana-keygen new --no-bip39-passphrase -o /home/runner/.config/solana/id.json + if [ ! -f /home/runner/.config/solana/id.json ]; then + solana-keygen new --no-bip39-passphrase -o /home/runner/.config/solana/id.json + fi npx nx build @lightprotocol/programs npx nx build @lightprotocol/zk-compression-cli - - name: Cache .local directory - uses: actions/cache@v4 + - name: Save .local directory cache + if: steps.restore-local-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 with: path: | .local prover/server/proving-keys - key: ${{ runner.os }}-local-${{ hashFiles('scripts/install.sh') }} + key: ${{ runner.os }}-local-v3-${{ hashFiles('scripts/install.sh', 'prover/server/scripts/download_keys.sh') }} diff --git a/.github/actions/setup-and-build/action.yml b/.github/actions/setup-and-build/action.yml index 6b233ef23a..719e64f03a 100644 --- a/.github/actions/setup-and-build/action.yml +++ b/.github/actions/setup-and-build/action.yml @@ -28,13 +28,16 @@ runs: path: | .local prover/server/proving-keys - key: ${{ runner.os }}-local-${{ hashFiles('scripts/install.sh', 'prover/server/scripts/download_keys.sh') }} + key: ${{ runner.os }}-local-v3-${{ hashFiles('scripts/install.sh', 'prover/server/scripts/download_keys.sh') }} + restore-keys: | + ${{ runner.os }}-local-v3- + ${{ runner.os }}-local-v2- - - name: Install dependencies + - name: Install system dependencies shell: bash run: | sudo apt-get update - sudo apt-get install -y libudev-dev pkg-config + sudo apt-get install -y libudev-dev pkg-config build-essential - name: Install dependencies if: steps.restore-local-cache.outputs.cache-hit != 'true' @@ -46,24 +49,82 @@ runs: ./scripts/install.sh fi - - name: Ensure proving keys exist + - name: Verify and repair installation + shell: bash + run: | + # Even if cache was restored, run install script to check and install missing components + # The install script now checks for actual file existence, so it will only install what's missing + echo "=== Verifying installation integrity ===" + if [ -n "${{ inputs.skip-components }}" ]; then + ./scripts/install.sh --no-reset --skip-components "${{ inputs.skip-components }}" + else + ./scripts/install.sh --no-reset + fi + + - name: Validate environment setup shell: bash run: | - # Always check if proving keys exist, even if cache was restored - if [ ! -d "prover/server/proving-keys" ] || [ -z "$(ls -A prover/server/proving-keys 2>/dev/null)" ]; then - echo "Proving keys not found or empty, downloading..." - ./prover/server/scripts/download_keys.sh light + echo "=== Validating environment setup ===" + source ./scripts/devenv.sh + + # Check critical binaries exist + echo "Checking Go installation..." + if [ -f "$GOROOT/bin/go" ]; then + echo "✓ Go found at: $GOROOT/bin/go" + $GOROOT/bin/go version || echo "⚠ Go binary exists but failed to run" + else + echo "✗ Go not found at expected location: $GOROOT/bin/go" + exit 1 + fi + + echo "Checking Rust installation..." + if [ -f "$CARGO_HOME/bin/cargo" ]; then + echo "✓ Cargo found at: $CARGO_HOME/bin/cargo" + $CARGO_HOME/bin/cargo --version || echo "⚠ Cargo binary exists but failed to run" + else + echo "✗ Cargo not found at expected location: $CARGO_HOME/bin/cargo" + exit 1 + fi + + echo "Checking Node installation..." + which node && node --version || echo "⚠ Node not in PATH" + + echo "Checking pnpm installation..." + which pnpm && pnpm --version || echo "⚠ pnpm not in PATH" + + echo "Checking Solana installation..." + which solana && solana --version || echo "⚠ Solana not in PATH" + + echo "Checking Anchor installation..." + which anchor && anchor --version || echo "⚠ Anchor not in PATH" + + # Check proving keys + if [ -d "prover/server/proving-keys" ] && [ -n "$(ls -A prover/server/proving-keys 2>/dev/null)" ]; then + echo "✓ Proving keys found: $(ls prover/server/proving-keys | wc -l) files" else - echo "Proving keys already exist" + echo "✗ Proving keys directory missing or empty" + exit 1 fi + echo "=== Environment validation complete ===" + - name: Set local environment shell: bash run: | source ./scripts/devenv.sh + # Export critical environment variables for subsequent steps + echo "GOROOT=$GOROOT" >> $GITHUB_ENV + echo "CARGO_HOME=$CARGO_HOME" >> $GITHUB_ENV + echo "RUSTUP_HOME=$RUSTUP_HOME" >> $GITHUB_ENV + echo "PATH=$PATH" >> $GITHUB_ENV + echo "CARGO_FEATURES=$CARGO_FEATURES" >> $GITHUB_ENV + echo "REDIS_URL=$REDIS_URL" >> $GITHUB_ENV - name: Rust cache uses: swatinem/rust-cache@v2 + with: + cache-all-crates: true + cache-on-failure: true - name: Setup pnpm uses: pnpm/action-setup@v4 @@ -88,7 +149,9 @@ runs: run: | source ./scripts/devenv.sh mkdir -p /home/runner/.config/solana/ - solana-keygen new --no-bip39-passphrase -o /home/runner/.config/solana/id.json + if [ ! -f /home/runner/.config/solana/id.json ]; then + solana-keygen new --no-bip39-passphrase -o /home/runner/.config/solana/id.json + fi - name: Copy spl_noop.so to target/deploy shell: bash @@ -96,14 +159,25 @@ runs: mkdir -p ./target/deploy cp ./third-party/solana-program-library/spl_noop.so ./target/deploy/spl_noop.so - - name: Cache .local directory + - name: Build Rust programs + shell: bash + run: | + source ./scripts/devenv.sh + echo "Building Rust programs..." + npx nx build @lightprotocol/programs || { + echo "Failed to build programs, retrying with verbose output..." + npx nx build @lightprotocol/programs --verbose + exit 1 + } + + - name: Save .local directory cache if: steps.restore-local-cache.outputs.cache-hit != 'true' - uses: actions/cache@v4 + uses: actions/cache/save@v4 with: path: | .local prover/server/proving-keys - key: ${{ runner.os }}-local-${{ hashFiles('scripts/install.sh', 'prover/server/scripts/download_keys.sh') }} + key: ${{ runner.os }}-local-v3-${{ hashFiles('scripts/install.sh', 'prover/server/scripts/download_keys.sh') }} - name: Check for git changes shell: bash diff --git a/cli/src/utils/process.ts b/cli/src/utils/process.ts index f3055ced41..6c824fce94 100644 --- a/cli/src/utils/process.ts +++ b/cli/src/utils/process.ts @@ -63,7 +63,6 @@ export async function killProcess(processName: string) { } } - // Double-check if processes are still running const remainingProcesses = await find("name", processName); if (remainingProcesses.length > 0) { console.warn( @@ -217,7 +216,6 @@ export function spawnBinary(command: string, args: string[] = []) { spawnedProcess.on("close", async (code) => { console.log(`${binaryName} process exited with code ${code}`); - // Log prover file contents if prover exits with non-zero code if (code !== 0 && binaryName.includes("prover")) { console.error(`Prover process failed with exit code ${code}`); await logProverFileContents(); diff --git a/cli/src/utils/processProverServer.ts b/cli/src/utils/processProverServer.ts index 38359c2911..6fe708acb5 100644 --- a/cli/src/utils/processProverServer.ts +++ b/cli/src/utils/processProverServer.ts @@ -1,4 +1,5 @@ import path from "path"; +import fs from "fs"; import { killProcess, killProcessByPort, @@ -9,6 +10,7 @@ import { LIGHT_PROVER_PROCESS_NAME } from "./constants"; import find from "find-process"; const KEYS_DIR = "proving-keys/"; +const MAX_START_RETRIES = 3; export async function killProver() { await killProcess(getProverNameByArch()); @@ -94,7 +96,7 @@ export async function startProver( ) { if ( !force && - (await isProverRunningWithFlags(runMode, circuits, proverPort)) + (await isProverRunningWithFlags(runMode, circuits, proverPort, redisUrl)) ) { return; } @@ -103,7 +105,23 @@ export async function startProver( await killProver(); await killProcessByPort(proverPort); + // Verify prover binary exists + const proverPath = getProverPathByArch(); + if (!fs.existsSync(proverPath)) { + throw new Error( + `Prover binary not found at ${proverPath}. Please run: npx nx build @lightprotocol/zk-compression-cli`, + ); + } + const keysDir = path.join(__dirname, "../..", "bin", KEYS_DIR); + + // Verify proving keys exist + if (!fs.existsSync(keysDir) || fs.readdirSync(keysDir).length === 0) { + throw new Error( + `Proving keys not found at ${keysDir}. Please run: ./prover/server/scripts/download_keys.sh light`, + ); + } + const args = ["start"]; args.push("--keys-dir", keysDir); args.push("--prover-address", `0.0.0.0:${proverPort}`); @@ -131,17 +149,59 @@ export async function startProver( args.push("--redis-url", redisUrl); } - const proverProcess = spawnBinary(getProverPathByArch(), args); + let lastError: Error | undefined; + for (let attempt = 1; attempt <= MAX_START_RETRIES; attempt++) { + try { + console.log( + `Starting prover (attempt ${attempt}/${MAX_START_RETRIES})...`, + ); - try { - await waitForServers([{ port: proverPort, path: "/" }]); - console.log(`Prover started successfully!`); - } catch (error) { - console.error( - "Failed to start prover - prover logs will be displayed above", - ); - throw error; + const proverProcess = spawnBinary(proverPath, args); + + await new Promise((resolve) => setTimeout(resolve, 1000)); + + if (proverProcess.exitCode !== null) { + throw new Error( + `Prover process exited with code ${proverProcess.exitCode}`, + ); + } + + try { + await waitForServers([{ port: proverPort, path: "/" }]); + console.log(`Prover started successfully!`); + + // Perform health check + const healthy = await healthCheck(proverPort); + if (!healthy) { + console.warn( + "Prover started but health check failed - it may still be initializing", + ); + } + + return; + } catch (error) { + // Kill the process if it didn't start properly + proverProcess.kill(); + throw error; + } + } catch (error) { + lastError = error as Error; + console.error(`Failed to start prover on attempt ${attempt}:`, error); + + if (attempt < MAX_START_RETRIES) { + console.log(`Waiting 5 seconds before retry...`); + await new Promise((resolve) => setTimeout(resolve, 5000)); + + // Clean up before retry + await killProver(); + await killProcessByPort(proverPort); + } + } } + + throw new Error( + `Failed to start prover after ${MAX_START_RETRIES} attempts. Last error: ${lastError?.message}`, + ); } export function getProverNameByArch(): string { @@ -152,9 +212,18 @@ export function getProverNameByArch(): string { throw new Error("Unsupported platform or architecture"); } - let binaryName = `prover-${platform}-${arch}`; + // Map Node.js arch names to our binary naming convention + const archMap: Record = { + x64: "x64", + arm64: "arm64", + x86: "x86", + aarch64: "arm64", + }; + + const mappedArch = archMap[arch] || arch; + let binaryName = `prover-${platform}-${mappedArch}`; - if (platform.toString() === "windows") { + if (platform === "win32") { binaryName += ".exe"; } return binaryName; @@ -177,15 +246,34 @@ export async function healthCheck( const fetch = (await import("node-fetch")).default; for (let i = 0; i < retries; i++) { try { - const res = await fetch(`http://localhost:${port}/health`); + const controller = new AbortController(); + const timeoutId = setTimeout(() => controller.abort(), timeout); + + const res = await fetch(`http://localhost:${port}/health`, { + signal: controller.signal, + }); + + clearTimeout(timeoutId); + if (res.ok) { - console.log("Health check passed!"); + const data = await res.text(); + console.log("Health check passed!", data); return true; + } else { + console.error(`Health check returned status ${res.status}`); } - } catch (e) { - console.error("Health check error:", e); + } catch (e: any) { + if (e.name === "AbortError") { + console.error( + `Health check attempt ${i + 1} timed out after ${timeout}ms`, + ); + } else { + console.error(`Health check attempt ${i + 1} failed:`, e.message); + } + } + if (i < retries - 1) { + await new Promise((r) => setTimeout(r, timeout)); } - await new Promise((r) => setTimeout(r, timeout)); } console.log("Health check failed after all attempts."); return false; diff --git a/scripts/devenv.sh b/scripts/devenv.sh index c164e39e98..0349400d9c 100755 --- a/scripts/devenv.sh +++ b/scripts/devenv.sh @@ -2,7 +2,10 @@ # Command to deactivate the devenv. It sets the old environment variables. deactivate () { - redis-stop + # Only try to stop redis if the command exists + if command -v redis-stop >/dev/null 2>&1; then + redis-stop 2>/dev/null || true + fi PS1="${LIGHT_PROTOCOL_OLD_PS1}" RUSTUP_HOME="${LIGHT_PROTOCOL_OLD_RUSTUP_HOME}" @@ -24,15 +27,27 @@ deactivate () { if [ -z "${LIGHT_PROTOCOL_DEVENV:-}" ]; then LIGHT_PROTOCOL_DEVENV=1 else - return + return 2>/dev/null || exit 0 fi # The root of the git repository. -LIGHT_PROTOCOL_TOPLEVEL="`git rev-parse --show-toplevel`" +LIGHT_PROTOCOL_TOPLEVEL="$(git rev-parse --show-toplevel 2>/dev/null)" +if [ -z "$LIGHT_PROTOCOL_TOPLEVEL" ]; then + echo "Error: Not in a git repository" >&2 + return 1 2>/dev/null || exit 1 +fi + +# Verify the .local directory exists +if [ ! -d "${LIGHT_PROTOCOL_TOPLEVEL}/.local" ]; then + echo "Error: .local directory not found. Please run ./scripts/install.sh first" >&2 + return 1 2>/dev/null || exit 1 +fi -# Shell prompt. -LIGHT_PROTOCOL_OLD_PS1="${PS1:-}" -PS1="[🧢 Light Protocol devenv] ${PS1:-}" +# Shell prompt (only set if PS1 exists - not in CI) +if [ -n "${PS1:-}" ]; then + LIGHT_PROTOCOL_OLD_PS1="${PS1}" + PS1="[🧢 Light Protocol devenv] ${PS1}" +fi # Ensure that our rustup environment is used. LIGHT_PROTOCOL_OLD_RUSTUP_HOME="${RUSTUP_HOME:-}" @@ -54,8 +69,10 @@ PATH="${LIGHT_PROTOCOL_TOPLEVEL}/.local/npm-global/bin:${PATH}" # Remove the original Rust-related PATH entries PATH=$(echo "$PATH" | tr ':' '\n' | grep -vE "/.rustup/|/.cargo/" | tr '\n' ':' | sed 's/:$//') -# Define alias of `light` to use the CLI built from source. -alias light="${LIGHT_PROTOCOL_TOPLEVEL}/cli/test_bin/run" +# Define alias of `light` to use the CLI built from source (only if not in CI) +if [ -z "${CI:-}" ]; then + alias light="${LIGHT_PROTOCOL_TOPLEVEL}/cli/test_bin/run" +fi # Define GOROOT for Go. export GOROOT="${LIGHT_PROTOCOL_TOPLEVEL}/.local/go" @@ -63,16 +80,35 @@ export GOROOT="${LIGHT_PROTOCOL_TOPLEVEL}/.local/go" # Ensure Rust binaries are in PATH PATH="${CARGO_HOME}/bin:${PATH}" -# Export the modified PATH +# Export all critical environment variables export PATH +export RUSTUP_HOME +export CARGO_HOME +export NPM_CONFIG_PREFIX +export LIGHT_PROTOCOL_TOPLEVEL +export LIGHT_PROTOCOL_DEVENV -# Comment to start prover without redis -export REDIS_URL="redis://localhost:6379" +# Set Redis URL if not already set +export REDIS_URL="${REDIS_URL:-redis://localhost:6379}" # Enable small_ix feature by default in devenv -export CARGO_FEATURES="small_ix" +export CARGO_FEATURES="${CARGO_FEATURES:-small_ix}" +# macOS-specific settings if [[ "$(uname)" == "Darwin" ]]; then LIGHT_PROTOCOL_OLD_CPATH="${CPATH:-}" export CPATH="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include:${CPATH:-}" fi + +# Validate critical tools are available (only warn, don't fail) +if [ -z "${LIGHT_PROTOCOL_SKIP_VALIDATION:-}" ]; then + if ! command -v cargo >/dev/null 2>&1; then + echo "Warning: cargo not found in PATH. Run ./scripts/install.sh to install Rust." >&2 + fi + if ! command -v go >/dev/null 2>&1; then + echo "Warning: go not found in PATH. Run ./scripts/install.sh to install Go." >&2 + fi + if ! command -v node >/dev/null 2>&1; then + echo "Warning: node not found in PATH. Run ./scripts/install.sh to install Node.js." >&2 + fi +fi diff --git a/scripts/install.sh b/scripts/install.sh index 8fcc4ae1dd..08ae190633 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -81,7 +81,8 @@ download() { } install_go() { - if ! is_installed "go"; then + # Check if Go is actually installed, not just logged + if [ ! -d "${PREFIX}/go" ] || [ ! -f "${PREFIX}/go/bin/go" ]; then echo "Installing Go..." local version=$(get_version "go") local suffix=$(get_suffix "go") @@ -90,11 +91,14 @@ install_go() { tar -xzf "${PREFIX}/go.tar.gz" -C "${PREFIX}" rm "${PREFIX}/go.tar.gz" log "go" + else + echo "Go already installed, skipping..." fi } install_rust() { - if ! is_installed "rust"; then + # Check if Rust is actually installed + if [ ! -d "${PREFIX}/rustup" ] || [ ! -d "${PREFIX}/cargo" ] || [ ! -f "${PREFIX}/cargo/bin/cargo" ]; then echo "Installing Rust..." export RUSTUP_HOME="${PREFIX}/rustup" export CARGO_HOME="${PREFIX}/cargo" @@ -105,11 +109,14 @@ install_rust() { cargo install cargo-expand --locked cargo install --git https://github.com/helius-labs/photon.git --rev cf58facb4e0521843e3afd21d09d8e7e7f772140 --locked log "rust" + else + echo "Rust already installed, skipping..." fi } install_node() { - if ! is_installed "node"; then + # Check if Node is actually installed + if [ ! -f "${PREFIX}/bin/node" ] || [ ! -f "${PREFIX}/bin/npm" ]; then echo "Installing Node.js..." local version=$(get_version "node") local suffix=$(get_suffix "node") @@ -118,11 +125,14 @@ install_node() { tar -xzf "${PREFIX}/node.tar.gz" -C "${PREFIX}" --strip-components 1 rm "${PREFIX}/node.tar.gz" log "node" + else + echo "Node.js already installed, skipping..." fi } install_pnpm() { - if ! is_installed "pnpm"; then + # Check if pnpm is actually installed + if [ ! -f "${PREFIX}/bin/pnpm" ]; then echo "Installing pnpm..." local version=$(get_version "pnpm") local suffix=$(get_suffix "pnpm") @@ -130,11 +140,14 @@ install_pnpm() { download "$url" "${PREFIX}/bin/pnpm" chmod +x "${PREFIX}/bin/pnpm" log "pnpm" + else + echo "pnpm already installed, skipping..." fi } install_solana() { - if ! is_installed "solana"; then + # Check if Solana is actually installed + if [ ! -f "${PREFIX}/bin/solana" ] || [ ! -f "${PREFIX}/bin/solana-keygen" ]; then echo "Installing Solana..." local version=$(get_version "solana") local suffix=$(get_suffix "solana") @@ -143,28 +156,36 @@ install_solana() { tar -xjf "${PREFIX}/solana-release.tar.bz2" -C "${PREFIX}/bin" --strip-components 2 rm "${PREFIX}/solana-release.tar.bz2" log "solana" + else + echo "Solana already installed, skipping..." fi } install_anchor() { - if ! is_installed "anchor"; then + # Check if Anchor is actually installed + if [ ! -f "${PREFIX}/bin/anchor" ]; then echo "Installing Anchor..." local version=$(get_version "anchor") local suffix=$(get_suffix "anchor") local url="https://github.com/Lightprotocol/binaries/releases/download/${version}/anchor-${suffix}" download "$url" "${PREFIX}/bin/anchor" log "anchor" + else + echo "Anchor already installed, skipping..." fi } install_jq() { - if ! is_installed "jq"; then + # Check if jq is actually installed + if [ ! -f "${PREFIX}/bin/jq" ]; then echo "Installing jq..." local version=$(get_version "jq") local suffix=$(get_suffix "jq") local url="https://github.com/jqlang/jq/releases/download/${version}/${suffix}" download "$url" "${PREFIX}/bin/jq" log "jq" + else + echo "jq already installed, skipping..." fi } @@ -181,17 +202,21 @@ download_gnark_keys() { } install_dependencies() { - if ! is_installed "dependencies"; then + # Check if node_modules exists and has content + if [ ! -d "node_modules" ] || [ -z "$(ls -A node_modules 2>/dev/null)" ]; then echo "Installing dependencies..." export PATH="${PREFIX}/bin:${PATH}" pnpm install log "dependencies" + else + echo "Dependencies already installed, skipping..." fi } install_redis() { - if ! is_installed "redis"; then + # Check if Redis is actually installed + if [ ! -f "${PREFIX}/bin/redis-server" ] || [ ! -f "${PREFIX}/bin/redis-cli" ]; then echo "Installing Redis..." local version=$(get_version "redis") local url="http://download.redis.io/releases/redis-${version}.tar.gz" @@ -329,6 +354,8 @@ EOF chmod +x "${PREFIX}/bin/redis-start" "${PREFIX}/bin/redis-stop" "${PREFIX}/bin/redis-status" mkdir -p "${PREFIX}/etc" "${PREFIX}/var" log "redis" + else + echo "Redis already installed, skipping..." fi } From ca8fac67303f65e402b11389c10bef3b7df02eb5 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 18:00:04 -0400 Subject: [PATCH 21/26] wip --- .../setup-and-build-nocheck/action.yml | 38 ++++++++++++++--- .github/actions/setup-and-build/action.yml | 42 ++++++++++++++++--- .github/workflows/cli-v1.yml | 8 +--- .github/workflows/cli-v2.yml | 8 +--- .github/workflows/js-v2.yml | 8 +--- .github/workflows/js.yml | 8 +--- 6 files changed, 76 insertions(+), 36 deletions(-) diff --git a/.github/actions/setup-and-build-nocheck/action.yml b/.github/actions/setup-and-build-nocheck/action.yml index 162b3f3aec..66aa83fe11 100644 --- a/.github/actions/setup-and-build-nocheck/action.yml +++ b/.github/actions/setup-and-build-nocheck/action.yml @@ -69,6 +69,11 @@ runs: with: cache-all-crates: true cache-on-failure: true + # Add workspace target directory + workspaces: | + . -> target + cli -> cli/target + examples -> examples/target - name: Setup pnpm uses: pnpm/action-setup@v4 @@ -76,21 +81,42 @@ runs: run_install: false - name: Get pnpm store directory + id: pnpm-store shell: bash run: | - echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV + # Get the store path before any pnpm operations + STORE_PATH=$(pnpm store path --silent) + echo "STORE_PATH=${STORE_PATH}" >> $GITHUB_ENV + echo "path=${STORE_PATH}" >> $GITHUB_OUTPUT - - uses: actions/cache@v4 - name: Setup pnpm cache + - name: Setup pnpm cache + id: pnpm-cache + uses: actions/cache@v4 with: - path: ${{ env.STORE_PATH }} - key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }} + path: ${{ steps.pnpm-store.outputs.path }} + key: ${{ runner.os }}-pnpm-store-${{ hashFiles('pnpm-lock.yaml') }} restore-keys: | ${{ runner.os }}-pnpm-store- - name: Install dependencies shell: bash - run: pnpm install + run: | + # Install dependencies + pnpm install --frozen-lockfile + + # Validate node_modules was created + if [ ! -d "node_modules" ] || [ -z "$(ls -A node_modules 2>/dev/null)" ]; then + echo "Error: node_modules not created after pnpm install" + exit 1 + fi + + - name: Save pnpm cache + # Save cache even on failure to speed up retries + if: steps.pnpm-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: ${{ steps.pnpm-store.outputs.path }} + key: ${{ runner.os }}-pnpm-store-${{ hashFiles('pnpm-lock.yaml') }} - name: Build and prepare for tests shell: bash diff --git a/.github/actions/setup-and-build/action.yml b/.github/actions/setup-and-build/action.yml index 719e64f03a..fb5e7469da 100644 --- a/.github/actions/setup-and-build/action.yml +++ b/.github/actions/setup-and-build/action.yml @@ -125,25 +125,55 @@ runs: with: cache-all-crates: true cache-on-failure: true + # Add workspace target directory + workspaces: | + . -> target + cli -> cli/target + examples -> examples/target - name: Setup pnpm uses: pnpm/action-setup@v4 with: - run_install: true + run_install: false - name: Get pnpm store directory + id: pnpm-store shell: bash run: | - echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV + # Get the store path before any pnpm operations + STORE_PATH=$(pnpm store path --silent) + echo "STORE_PATH=${STORE_PATH}" >> $GITHUB_ENV + echo "path=${STORE_PATH}" >> $GITHUB_OUTPUT - - uses: actions/cache@v4 - name: Setup pnpm cache + - name: Setup pnpm cache + id: pnpm-cache + uses: actions/cache@v4 with: - path: ${{ env.STORE_PATH }} - key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }} + path: ${{ steps.pnpm-store.outputs.path }} + key: ${{ runner.os }}-pnpm-store-${{ hashFiles('pnpm-lock.yaml') }} restore-keys: | ${{ runner.os }}-pnpm-store- + - name: Install dependencies + shell: bash + run: | + # Install dependencies with frozen lockfile for consistency + pnpm install --frozen-lockfile + + # Validate node_modules was created + if [ ! -d "node_modules" ] || [ -z "$(ls -A node_modules 2>/dev/null)" ]; then + echo "Error: node_modules not created after pnpm install" + exit 1 + fi + + - name: Save pnpm cache + # Save cache even on failure to speed up retries + if: steps.pnpm-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: ${{ steps.pnpm-store.outputs.path }} + key: ${{ runner.os }}-pnpm-store-${{ hashFiles('pnpm-lock.yaml') }} + - name: Generate Solana keypair shell: bash run: | diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index fca2bb45d1..5f63f0193e 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -49,13 +49,9 @@ jobs: path: | .nx/cache node_modules/.cache/nx - js/stateless.js/node_modules/.cache/nx - js/compressed-token/node_modules/.cache/nx - cli/node_modules/.cache/nx - key: nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + key: nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}-${{ github.sha }} restore-keys: | - nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- - nx-cli-v1-${{ runner.os }}- + nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}- - name: Setup and build uses: ./.github/actions/setup-and-build diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index 180fc6357a..fb1d9a69cb 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -49,13 +49,9 @@ jobs: path: | .nx/cache node_modules/.cache/nx - js/stateless.js/node_modules/.cache/nx - js/compressed-token/node_modules/.cache/nx - cli/node_modules/.cache/nx - key: nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + key: nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}-${{ github.sha }} restore-keys: | - nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- - nx-cli-v2-${{ runner.os }}- + nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}- - name: Setup and build uses: ./.github/actions/setup-and-build diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index dff50674c2..ceb74357f0 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -49,13 +49,9 @@ jobs: path: | .nx/cache node_modules/.cache/nx - js/stateless.js/node_modules/.cache/nx - js/compressed-token/node_modules/.cache/nx - cli/node_modules/.cache/nx - key: nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + key: nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}-${{ github.sha }} restore-keys: | - nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- - nx-js-v2-${{ runner.os }}- + nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}- - name: Setup and build uses: ./.github/actions/setup-and-build diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 8717d7103b..5218e40d86 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -49,13 +49,9 @@ jobs: path: | .nx/cache node_modules/.cache/nx - js/stateless.js/node_modules/.cache/nx - js/compressed-token/node_modules/.cache/nx - cli/node_modules/.cache/nx - key: nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}-${{ github.sha }} + key: nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}-${{ github.sha }} restore-keys: | - nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'js/**/package.json', 'cli/package.json') }}- - nx-js-v1-${{ runner.os }}- + nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}- - name: Setup and build uses: ./.github/actions/setup-and-build From 0065237ea3de5bd5fa04e3ee4f705ea374576371 Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 18:22:01 -0400 Subject: [PATCH 22/26] wip --- .github/workflows/cli-v1.yml | 4 ++-- .github/workflows/cli-v2.yml | 4 ++-- .github/workflows/js-v2.yml | 4 ++-- .github/workflows/js.yml | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/cli-v1.yml b/.github/workflows/cli-v1.yml index 5f63f0193e..018ac91842 100644 --- a/.github/workflows/cli-v1.yml +++ b/.github/workflows/cli-v1.yml @@ -49,9 +49,9 @@ jobs: path: | .nx/cache node_modules/.cache/nx - key: nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}-${{ github.sha }} + key: nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }} restore-keys: | - nx-cli-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}- + nx-cli-v1-${{ runner.os }}- - name: Setup and build uses: ./.github/actions/setup-and-build diff --git a/.github/workflows/cli-v2.yml b/.github/workflows/cli-v2.yml index fb1d9a69cb..2f7fd728ef 100644 --- a/.github/workflows/cli-v2.yml +++ b/.github/workflows/cli-v2.yml @@ -49,9 +49,9 @@ jobs: path: | .nx/cache node_modules/.cache/nx - key: nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}-${{ github.sha }} + key: nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }} restore-keys: | - nx-cli-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}- + nx-cli-v2-${{ runner.os }}- - name: Setup and build uses: ./.github/actions/setup-and-build diff --git a/.github/workflows/js-v2.yml b/.github/workflows/js-v2.yml index ceb74357f0..e6ecbc8f89 100644 --- a/.github/workflows/js-v2.yml +++ b/.github/workflows/js-v2.yml @@ -49,9 +49,9 @@ jobs: path: | .nx/cache node_modules/.cache/nx - key: nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}-${{ github.sha }} + key: nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }} restore-keys: | - nx-js-v2-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}- + nx-js-v2-${{ runner.os }}- - name: Setup and build uses: ./.github/actions/setup-and-build diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 5218e40d86..077ddfa0d0 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -49,9 +49,9 @@ jobs: path: | .nx/cache node_modules/.cache/nx - key: nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}-${{ github.sha }} + key: nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }} restore-keys: | - nx-js-v1-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'nx.json') }}- + nx-js-v1-${{ runner.os }}- - name: Setup and build uses: ./.github/actions/setup-and-build From 488b8bc25dca848268a685a62c376324eea19e8a Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Sun, 15 Jun 2025 18:23:27 -0400 Subject: [PATCH 23/26] wip --- cli/src/utils/processProverServer.ts | 118 +++------------------------ 1 file changed, 11 insertions(+), 107 deletions(-) diff --git a/cli/src/utils/processProverServer.ts b/cli/src/utils/processProverServer.ts index 6fe708acb5..bdd7f0b84c 100644 --- a/cli/src/utils/processProverServer.ts +++ b/cli/src/utils/processProverServer.ts @@ -1,5 +1,4 @@ import path from "path"; -import fs from "fs"; import { killProcess, killProcessByPort, @@ -10,7 +9,6 @@ import { LIGHT_PROVER_PROCESS_NAME } from "./constants"; import find from "find-process"; const KEYS_DIR = "proving-keys/"; -const MAX_START_RETRIES = 3; export async function killProver() { await killProcess(getProverNameByArch()); @@ -96,7 +94,7 @@ export async function startProver( ) { if ( !force && - (await isProverRunningWithFlags(runMode, circuits, proverPort, redisUrl)) + (await isProverRunningWithFlags(runMode, circuits, proverPort)) ) { return; } @@ -105,23 +103,7 @@ export async function startProver( await killProver(); await killProcessByPort(proverPort); - // Verify prover binary exists - const proverPath = getProverPathByArch(); - if (!fs.existsSync(proverPath)) { - throw new Error( - `Prover binary not found at ${proverPath}. Please run: npx nx build @lightprotocol/zk-compression-cli`, - ); - } - const keysDir = path.join(__dirname, "../..", "bin", KEYS_DIR); - - // Verify proving keys exist - if (!fs.existsSync(keysDir) || fs.readdirSync(keysDir).length === 0) { - throw new Error( - `Proving keys not found at ${keysDir}. Please run: ./prover/server/scripts/download_keys.sh light`, - ); - } - const args = ["start"]; args.push("--keys-dir", keysDir); args.push("--prover-address", `0.0.0.0:${proverPort}`); @@ -149,59 +131,9 @@ export async function startProver( args.push("--redis-url", redisUrl); } - let lastError: Error | undefined; - for (let attempt = 1; attempt <= MAX_START_RETRIES; attempt++) { - try { - console.log( - `Starting prover (attempt ${attempt}/${MAX_START_RETRIES})...`, - ); - - const proverProcess = spawnBinary(proverPath, args); - - await new Promise((resolve) => setTimeout(resolve, 1000)); - - if (proverProcess.exitCode !== null) { - throw new Error( - `Prover process exited with code ${proverProcess.exitCode}`, - ); - } - - try { - await waitForServers([{ port: proverPort, path: "/" }]); - console.log(`Prover started successfully!`); - - // Perform health check - const healthy = await healthCheck(proverPort); - if (!healthy) { - console.warn( - "Prover started but health check failed - it may still be initializing", - ); - } - - return; - } catch (error) { - // Kill the process if it didn't start properly - proverProcess.kill(); - throw error; - } - } catch (error) { - lastError = error as Error; - console.error(`Failed to start prover on attempt ${attempt}:`, error); - - if (attempt < MAX_START_RETRIES) { - console.log(`Waiting 5 seconds before retry...`); - await new Promise((resolve) => setTimeout(resolve, 5000)); - - // Clean up before retry - await killProver(); - await killProcessByPort(proverPort); - } - } - } - - throw new Error( - `Failed to start prover after ${MAX_START_RETRIES} attempts. Last error: ${lastError?.message}`, - ); + spawnBinary(getProverPathByArch(), args); + await waitForServers([{ port: proverPort, path: "/" }]); + console.log(`Prover started successfully!`); } export function getProverNameByArch(): string { @@ -212,18 +144,9 @@ export function getProverNameByArch(): string { throw new Error("Unsupported platform or architecture"); } - // Map Node.js arch names to our binary naming convention - const archMap: Record = { - x64: "x64", - arm64: "arm64", - x86: "x86", - aarch64: "arm64", - }; - - const mappedArch = archMap[arch] || arch; - let binaryName = `prover-${platform}-${mappedArch}`; + let binaryName = `prover-${platform}-${arch}`; - if (platform === "win32") { + if (platform.toString() === "windows") { binaryName += ".exe"; } return binaryName; @@ -246,34 +169,15 @@ export async function healthCheck( const fetch = (await import("node-fetch")).default; for (let i = 0; i < retries; i++) { try { - const controller = new AbortController(); - const timeoutId = setTimeout(() => controller.abort(), timeout); - - const res = await fetch(`http://localhost:${port}/health`, { - signal: controller.signal, - }); - - clearTimeout(timeoutId); - + const res = await fetch(`http://localhost:${port}/health`); if (res.ok) { - const data = await res.text(); - console.log("Health check passed!", data); + console.log("Health check passed!"); return true; - } else { - console.error(`Health check returned status ${res.status}`); } - } catch (e: any) { - if (e.name === "AbortError") { - console.error( - `Health check attempt ${i + 1} timed out after ${timeout}ms`, - ); - } else { - console.error(`Health check attempt ${i + 1} failed:`, e.message); - } - } - if (i < retries - 1) { - await new Promise((r) => setTimeout(r, timeout)); + } catch (e) { + console.error("Health check error:", e); } + await new Promise((r) => setTimeout(r, timeout)); } console.log("Health check failed after all attempts."); return false; From 3569f09a0df78e536e1f1b1e98a24bd284335cec Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Mon, 16 Jun 2025 12:35:22 -0400 Subject: [PATCH 24/26] double-checking CI caching --- INSTALL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/INSTALL.md b/INSTALL.md index 46246fbfef..f3b779164e 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -22,7 +22,7 @@ Running `install.sh` is the first thing you should do after cloning this monorep - `jq` - JSON processor - `keys` - Gnark proving keys - `dependencies` - all PNPM deps -- `redis` - Redis server (not needed for some ci workflows) +- `redis` - Redis server ## CI Usage From 912831191780c47fb404359c2ebdabcece4f996c Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Mon, 16 Jun 2025 13:55:46 -0400 Subject: [PATCH 25/26] reduce ci times for remaining long running workflows --- .../workflows/light-system-programs-tests.yml | 8 ++++++-- .github/workflows/rust.yml | 16 +++++++++++----- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/.github/workflows/light-system-programs-tests.yml b/.github/workflows/light-system-programs-tests.yml index 6a27e02135..0be5d51a2b 100644 --- a/.github/workflows/light-system-programs-tests.yml +++ b/.github/workflows/light-system-programs-tests.yml @@ -65,8 +65,12 @@ jobs: sub-tests: '["cargo-test-sbf -p compressed-token-test"]' - program: system-cpi-test sub-tests: '["cargo-test-sbf -p system-cpi-test"]' - - program: system-cpi-test-v2 - sub-tests: '["cargo-test-sbf -p system-cpi-v2-test"]' + - program: system-cpi-test-v2-event + sub-tests: '["cargo-test-sbf -p system-cpi-v2-test -- event::parse"]' + - program: system-cpi-test-v2-functional + sub-tests: '["cargo-test-sbf -p system-cpi-v2-test -- functional_"]' + - program: system-cpi-test-v2-other + sub-tests: '["cargo-test-sbf -p system-cpi-v2-test -- --skip functional_ --skip event::parse"]' - program: random-e2e-test sub-tests: '["cargo-test-sbf -p e2e-test"]' steps: diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 9051398500..efa24bff78 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -35,17 +35,19 @@ jobs: matrix: group: - name: concurrent-merkle-tree - packages: light-concurrent-merkle-tree light-batched-merkle-tree + packages: light-concurrent-merkle-tree test_cmd: | cargo test -p light-concurrent-merkle-tree + - name: batched-merkle-tree-simulate + packages: light-batched-merkle-tree + test_cmd: | RUST_LOG=light_prover_client=debug cargo test -p light-batched-merkle-tree --features test-only -- --test test_simulate_transactions - - name: program-libs + - name: program-libs-fast packages: - aligned-sized light-bloom-filter light-hasher light-compressed-account light-account-checks \ - light-verifier light-merkle-tree-metadata light-zero-copy light-hash-set light-indexed-merkle-tree light-batched-merkle-tree + aligned-sized light-hasher light-compressed-account light-account-checks \ + light-verifier light-merkle-tree-metadata light-zero-copy light-hash-set test_cmd: | cargo test -p aligned-sized - cargo test -p light-bloom-filter --all-features cargo test -p light-hasher --all-features cargo test -p light-compressed-account --all-features cargo test -p light-account-checks --all-features @@ -53,6 +55,10 @@ jobs: cargo test -p light-merkle-tree-metadata --all-features cargo test -p light-zero-copy --features std cargo test -p light-hash-set --all-features + - name: program-libs-slow + packages: light-bloom-filter light-indexed-merkle-tree light-batched-merkle-tree + test_cmd: | + cargo test -p light-bloom-filter --all-features cargo test -p light-indexed-merkle-tree --all-features cargo test -p light-batched-merkle-tree --all-features -- --test test_e2e - name: sdk-libs From 6ced45696e1b8dfea12ccf8e73746274025f30fd Mon Sep 17 00:00:00 2001 From: Swenschaeferjohann Date: Mon, 16 Jun 2025 14:12:04 -0400 Subject: [PATCH 26/26] lint.yml all deps --- .github/workflows/lint.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index d0070d5b46..5657f8c9f1 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -28,8 +28,6 @@ jobs: - name: Setup and build uses: ./.github/actions/setup-and-build - with: - skip-components: "redis,go,keys" - name: Run linters run: |