✅ server: improve test stability and ci error handling

[cruzdanilo]

---

Summary by CodeRabbit

• Tests
  • More reliable test runs with retryable dependency installs, improved process/database lifecycle handling, and faster, concurrent wait patterns for stability.
• Chores
  • Added a placeholder changeset for versioning.
  • CI/workflow improvements: broader artifact inclusion and more robust setup/cleanup during end-to-end runs.

[changeset-bot]

🦋 Changeset detected

Latest commit: e0df745

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Walkthrough

Refactors test infra and workflows: adds a placeholder changeset, improves CI Maestro install retry and path selection, overhauls embedded Postgres and child-process lifecycle in tests with logging and AbortController-based teardown, updates tests to use concurrent/wait-for-state helpers, and makes spotlight teardown async with refined error handling.

Changes

Cohort / File(s)	Summary
Changeset \.changeset/smooth-boxes-sip.md	Adds a new placeholder changeset file with two line delimiters (---).
CI workflow .github/workflows/test.yaml	Broadened server-events path selection; added retry loop (up to 5 attempts) for Maestro install with cleanup and pauses; added set -o pipefail for better shell failure propagation.
Local EAS workflow artifacts .eas/workflows/local.yaml	Added server/node_modules/@exactly/.runtime to artifact upload paths.
Database test lifecycle server/test/database.ts	Reworks embedded Postgres and dependent service startup with readiness races, extended timeouts, AbortController-driven child process lifecycle, centralized watchProcessOutput streaming to logs, and robust teardown/cleanup paths.
Hook tests (activity & block) server/test/hooks/activity.test.ts, server/test/hooks/block.test.ts	Replace sequential wait-for-transaction patterns with concurrent Promise.all and new wait-for-state helpers (e.g., waitForWethMarket, waitForSuccessfulProposalExecutions); collect receipts from intercepted keeper calls and update transfer assertions accordingly.
Spotlight test harness server/test/spotlight.ts	Changed returned teardown from sync to async; added process.kill handling, typed ExecaError checks (isCanceled/isTerminated/signal), and conditional swallowing of termination errors.
Test utilities / logging server/test/... (implicit additions)	Introduces watchProcessOutput helper and startup log management (create/flush log files) used by test database startup and child processes.

Sequence Diagram(s)

sequenceDiagram
    participant TestRunner as Test Runner
    participant DBSetup as Database Setup
    participant PG as Embedded Postgres
    participant Firehose as Firehose Process
    participant Substreams as Substreams Process
    participant WaitHelpers as Wait Helpers

    TestRunner->>DBSetup: start()
    DBSetup->>PG: spawn Postgres
    DBSetup->>Firehose: spawn firehose (with AbortController)
    DBSetup->>Substreams: spawn substreams (with AbortController)

    par readiness checks
        DBSetup->>PG: wait for readiness file / TCP
        DBSetup->>WaitHelpers: wait for eth TCP endpoint
        DBSetup->>WaitHelpers: wait for substreams TCP endpoint
    end

    Firehose->>DBSetup: stdout/stderr streamed via watchProcessOutput
    Substreams->>DBSetup: stdout/stderr streamed via watchProcessOutput

    alt all ready
        DBSetup->>TestRunner: return client + teardown
    else timeout or error
        DBSetup->>Firehose: abort/kill
        DBSetup->>Substreams: abort/kill
        DBSetup->>PG: terminate process
        DBSetup->>TestRunner: fail with aggregated logs
    end

    TestRunner->>DBSetup: teardown()
    DBSetup->>Firehose: graceful terminate
    DBSetup->>Substreams: graceful terminate
    DBSetup->>PG: close client & terminate
    DBSetup-->>TestRunner: teardown complete

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

• ✅ server: use single postgres instance for test #610: Overlaps with embedded Postgres/test DB lifecycle changes in server/test/database.ts.
• 🥅 server: resolve orphaned proposals on nonce-too-low revert #765: Related changes to server/test/hooks/block.test.ts around proposal execution / receipt-wait logic.
• ✅ server: wait for expected transfers instead of receipt count #764: Related refactor of transfer detection/assertion logic in server/test/hooks/block.test.ts.

Suggested reviewers

• nfmelendez

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is specific and directly related to the main changes—improving test stability and CI error handling—which aligns with the core modifications across test setup, error handling, and CI workflows.
Merge Conflict Detection	✅ Passed	✅ No merge conflicts detected when merging into main

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch github

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Summary of Changes

Hello @cruzdanilo, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the stability and diagnostic capabilities of the server's integration tests. By adding comprehensive error handling, detailed logging for external processes, and refining test waiting conditions, it aims to reduce flaky test failures and provide clearer insights into the root cause of issues, especially within continuous integration environments.

Highlights

• Improved Test Stability: Enhanced the robustness of integration tests by implementing more resilient startup sequences for external services like Embedded Postgres, Firehose, and Substreams, including longer timeouts and explicit process monitoring.
• Enhanced CI Error Handling: Introduced detailed logging for external process outputs during test setup, ensuring that if a service fails to start, its logs are captured and included in the error message, significantly aiding debugging in CI environments.
• Refactored Activity Tests: Streamlined and made activity-related tests more reliable by replacing generic waitForTransactionReceipt checks with dedicated helper functions (waitForWethMarket, getWethMarket) that specifically wait for the desired state of the system.
• Graceful Process Teardown: Modified process teardown logic to correctly handle and ignore expected ExecaError instances when processes are intentionally killed, preventing false-positive test failures during cleanup.

Changelog

• .changeset/smooth-boxes-sip.md
  • Added a new changeset file.
• server/test/database.ts
  • Added imports for file system operations (createWriteStream, WriteStream, mkdir) and Execa's ResultPromise type.
  • Introduced a postgresExited promise to detect early termination of the embedded Postgres process.
  • Wrapped the database setup logic in an immediately invoked async function to centralize error handling and ensure Postgres is stopped on any setup failure.
  • Increased waitOn timeouts from 33 to 120 seconds for Postgres, RPC, and Firehose/Substreams readiness checks to improve stability.
  • Implemented logging of Firehose and Substreams process outputs to dedicated files (firehose.log, substreams.log) within a new .runtime/startup directory.
  • Added a watchProcessOutput helper function to stream subprocess stdout/stderr to a WriteStream and monitor for exit or output errors.
  • Enhanced error messages during startup failures to include content from the captured Firehose and Substreams logs.
  • Modified the teardown function to await the completion of all process exits and output flushing, ensuring proper cleanup.
• server/test/hooks/activity.test.ts
  • Replaced direct waitForTransactionReceipt checks with a Promise.all that includes a new waitForWethMarket helper function for more precise market state validation.
  • Introduced getWethMarket and waitForWethMarket helper functions to abstract away the logic for reading WETH market data and waiting for specific market conditions.
  • Adjusted the vi.waitUntil condition for waitForTransactionReceipt to consider only new settled results, preventing interference from previous transactions.
  • Modified the 'deploy account for non market asset' test to wait for the account's code to be deployed using publicClient.getCode instead of waitForTransactionReceipt.
• server/test/spotlight.ts
  • Imported ExecaError for specific error handling.
  • Updated the teardown function to explicitly catch and ignore ExecaError instances when the spotlight subprocess is intentionally killed (canceled or terminated), preventing test failures due to expected process termination.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/test.yaml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly improves test stability and CI error handling. The changes in server/test/database.ts introduce robust management of child processes with better logging and error reporting, which will be very helpful for debugging CI failures. The test logic in server/test/hooks/activity.test.ts is refactored to use more reliable waiting mechanisms, enhancing test stability. The teardown logic in server/test/spotlight.ts is also improved. Overall, these are excellent changes. I have a couple of minor suggestions to further refine the code.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5b8dfa75c1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[devin-ai-integration]

Devin Review found 1 potential issue.

View 6 additional findings in Devin Review.

[sentry]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.99%. Comparing base (0257dd4) to head (e0df745).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #778      +/-   ##
==========================================
+ Coverage   68.90%   68.99%   +0.08%     
==========================================
  Files         207      207              
  Lines        7015     7069      +54     
  Branches     2192     2218      +26     
==========================================
+ Hits         4834     4877      +43     
- Misses       2001     2003       +2     
- Partials      180      189       +9     

Flag	Coverage Δ
e2e	68.96% <ø> (+16.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coderabbitai]

Actionable comments posted: 1

[coderabbitai]

Actionable comments posted: 3