feat!: Use Thirdweb Engine in Request Node to persist transactions

[MantisClone]

Problem

Fixes #1292

• Fix "OldNonce" errors in Request Node #1292

Description of the changes

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[MantisClone]

Initial Success 🙌

I persisted a transaction via local Request Node + Thirdweb Engine. ✅

• I used the quickstart-node-js createRequest.js script.
• I ran a local IPFS node.
• I connected to the Storage Subgraph on the Staging Graph Node.

👀 Observations

All observations indicate Success! 🙌

• The Request Node initialized successfully - printing Thirdweb Engine URL and Backend Wallet Address.
• The Request Node printed the Transaction Queue ID after submitting the transaction to Thirdweb Engine.
• The Request Node logged transaction hash "0x000..." and confirmation in block 0 (dummy data) and that the transaction was confirmed immediately after submission, showing "fire and forget" behavior.
• The Request Network SDK polled the /getConfirmedTransaction endpoint until the transaction was successfully indexed by the Storage Subgraph on the Staging Graph Node.

🔍 Details

Full Request Node Logs

Full Request Node Logs

mantisclone@MantisClones-MacBook-Pro request-node % yarn run start
yarn run v1.22.19
$ ts-node src/bin.ts
lit-js-sdk:constants:errors deprecated LitErrorKind is deprecated and will be removed in a future version. Use LIT_ERROR_KIND instead. node:internal/modules/cjs/loader:1734:14
Using config:
  Ethereum network id: 11155111
  Log Level: DEBUG
  Log Mode: human
  Web3 provider url: https://indulgent-tiniest-replica.ethereum-sepolia.quiknode.pro/<api key omitted>/
  TheGraph url: https://sepolia.graph-node.stage.request.network/subgraphs/name/RequestNetwork/request-storage
  IPFS url: http://0.0.0.0:5001
  IPFS timeout: 30000
  Storage block confirmations: 2
  Lit Protocol Network: datil
  Lit Protocol RPC: https://yellowstone-rpc.litprotocol.com
  Lit Protocol Capacity Credits Uses: 30
  Lit Protocol Capacity Credits Expiration in seconds: 600
  Thirdweb Engine URL: https://f4c35a20.engine-usw2.thirdweb.com
  Thirdweb Backend Wallet: 0x29f1A1C80d711Bdf05b7a2c6265937a71b5AECe7

[09:06:10]  INFO: Checking ipfs connection (#ipfs, #sanity)
[09:06:10]  INFO: Checking ipfs network (#ipfs, #sanity)
[09:06:10]  INFO: Using Thirdweb Engine for transaction submission 
[09:06:10]  INFO: Initializing ThirdwebTransactionSubmitter for network sepolia (chainId: 11155111) 
[09:06:11]  INFO: Successfully connected to Thirdweb Engine 
[09:06:11]  INFO: Node initialization 
[09:06:11]  INFO: Listening on port 3000 
[09:06:11]  INFO: Node initialized 
[09:06:11]  INFO: Time to initialize: 0.431s (#metric, #initialization)
[13:40:01] DEBUG: Persisting Transaction: {"transactionHash":{"type":"keccak256","value":"0x35a15c07b3fbebda5c39720530b0c982b188ce12a91e0fc8b57dea73a95fdf68"},"channelId":"01c2c892147d2da04ecb7f7627ef5e262131258a1e67d8556112795aadd229096b","topics":["01f7175437dabf71350b955ddeb68e53afc2eccd775934d361a73dc08940f36422","01f7175437dabf71350b955ddeb68e53afc2eccd775934d361a73dc08940f36422","01c2c892147d2da04ecb7f7627ef5e262131258a1e67d8556112795aadd229096b"],"transactionData":{"data":"{\"data\":{\"name\":\"create\",\"parameters\":{\"currency\":{\"type\":\"ERC20\",\"value\":\"0x370DE27fdb7D1Ff1e1BaA7D11c5820a324Cf623C\",\"network\":\"sepolia\"},\"expectedAmount\":\"1000000000000000000\",\"payee\":{\"type\":\"ethereumAddress\",\"value\":\"0x7eB023BFbAeE228de6DC5B92D0BeEB1eDb1Fd567\"},\"payer\":{\"type\":\"ethereumAddress\",\"value\":\"0x7eB023BFbAeE228de6DC5B92D0BeEB1eDb1Fd567\"},\"timestamp\":1744825201,\"extensionsData\":[{\"action\":\"create\",\"id\":\"pn-erc20-fee-proxy-contract\",\"parameters\":{\"feeAddress\":\"0x0000000000000000000000000000000000000000\",\"feeAmount\":\"0\",\"paymentAddress\":\"0x7eB023BFbAeE228de6DC5B92D0BeEB1eDb1Fd567\",\"paymentNetworkName\":\"sepolia\",\"salt\":\"cdcb67abcf9de13c\"},\"version\":\"0.2.0\"},{\"action\":\"create\",\"id\":\"content-data\",\"parameters\":{\"content\":{\"reason\":\"🍕\",\"dueDate\":\"2023.06.16\"}},\"version\":\"0.1.0\"}]},\"version\":\"2.0.3\"},\"signature\":{\"method\":\"ecdsa\",\"value\":\"0x5e6ae3ef4feb942255f392960e392c05e8167ddd540366d2989be363ef925bd75c090cb1a7e721abb1c582e2a2752a3f8bcbc2fb3bcd85c356a90ffdd121090a1c\"}}"}} 
[13:40:02]  INFO: Submitting IPFS CID QmXEsS2yXTyXjxnmkYPLuqDzvuWNMzYAQikN43TjLJphNg with size 1562 via Thirdweb Engine 
[13:40:02]  INFO: Transaction submitted. Queue ID: e8d5f8ed-555a-4d35-bd0e-cf86ed773d3b 
[13:40:02] DEBUG: TX 0x0000000000000000000000000000000000000000000000000000000000000000 submitted, waiting for confirmation... 
[13:40:02] DEBUG: TX 0x0000000000000000000000000000000000000000000000000000000000000000 confirmed at block 0 
[13:40:02] DEBUG: persistTransaction successfully completed {"transactionHash":{"type":"keccak256","value":"0x35a15c07b3fbebda5c39720530b0c982b188ce12a91e0fc8b57dea73a95fdf68"},"channelId":"01c2c892147d2da04ecb7f7627ef5e262131258a1e67d8556112795aadd229096b"} (#metric, #successRate)
::1 - - [16/Apr/2025:17:40:02 +0000] "POST /persistTransaction HTTP/1.1" 200 778 "-" "node"
[13:40:02] DEBUG: persistTransaction latency: 640ms. Status: 200 (#metric, #latency)
::1 - - [16/Apr/2025:17:40:03 +0000] "GET /getTransactionsByChannelId?channelId=01c2c892147d2da04ecb7f7627ef5e262131258a1e67d8556112795aadd229096b HTTP/1.1" 200 1323 "-" "node"
[13:40:03] DEBUG: getTransactionsByChannelId latency: 674ms. Status: 200 (#metric, #latency)
::1 - - [16/Apr/2025:17:40:05 +0000] "GET /getConfirmedTransaction?transactionHash=0x35a15c07b3fbebda5c39720530b0c982b188ce12a91e0fc8b57dea73a95fdf68 HTTP/1.1" 404 - "-" "node"
[13:40:05] DEBUG: getConfirmedTransaction latency: 102ms. Status: 404 (#metric, #latency)
::1 - - [16/Apr/2025:17:40:06 +0000] "GET /getConfirmedTransaction?transactionHash=0x35a15c07b3fbebda5c39720530b0c982b188ce12a91e0fc8b57dea73a95fdf68 HTTP/1.1" 404 - "-" "node"
[13:40:06] DEBUG: getConfirmedTransaction latency: 107ms. Status: 404 (#metric, #latency)
::1 - - [16/Apr/2025:17:40:08 +0000] "GET /getConfirmedTransaction?transactionHash=0x35a15c07b3fbebda5c39720530b0c982b188ce12a91e0fc8b57dea73a95fdf68 HTTP/1.1" 404 - "-" "node"
[13:40:08] DEBUG: getConfirmedTransaction latency: 101ms. Status: 404 (#metric, #latency)
::1 - - [16/Apr/2025:17:40:13 +0000] "GET /getConfirmedTransaction?transactionHash=0x35a15c07b3fbebda5c39720530b0c982b188ce12a91e0fc8b57dea73a95fdf68 HTTP/1.1" 404 - "-" "node"
[13:40:13] DEBUG: getConfirmedTransaction latency: 266ms. Status: 404 (#metric, #latency)
::1 - - [16/Apr/2025:17:40:21 +0000] "GET /getConfirmedTransaction?transactionHash=0x35a15c07b3fbebda5c39720530b0c982b188ce12a91e0fc8b57dea73a95fdf68 HTTP/1.1" 404 - "-" "node"
[13:40:21] DEBUG: getConfirmedTransaction latency: 234ms. Status: 404 (#metric, #latency)
::1 - - [16/Apr/2025:17:40:37 +0000] "GET /getConfirmedTransaction?transactionHash=0x35a15c07b3fbebda5c39720530b0c982b188ce12a91e0fc8b57dea73a95fdf68 HTTP/1.1" 200 691 "-" "node"
[13:40:37] DEBUG: getConfirmedTransaction latency: 225ms. Status: 200 (#metric, #latency)
::1 - - [16/Apr/2025:17:40:37 +0000] "GET /getTransactionsByChannelId?channelId=01c2c892147d2da04ecb7f7627ef5e262131258a1e67d8556112795aadd229096b HTTP/1.1" 200 1683 "-" "node"
[13:40:37] DEBUG: getTransactionsByChannelId latency: 152ms. Status: 200 (#metric, #latency)

Thirdweb Engine Screenshots

Thirdweb Engine Screenshots

[MantisClone]

😓🏃 Stress tests - 👍 Some success, ❓Raised new questions, Testing ongoing 🚧

• I stress-tested the Request Node + Thirdweb Engine using a new createManyConcurrentRequests() script in the quickstart-node-js repo (PR: feat: create many concurrent requests quickstart-node-js#14)
• 3 Trials:
  • Trial 1: 100 requests, 10 workers - 100% success
  • Trial 2: 100 requests, 100 workers - 6% success (94 transactions were rate-limited by the underlying RPC and failed in the Thirdweb Engine)
  • Trial 3: 100 requests, 10 workers - 98% success (2 transactions got stuck in the "sent" state, timed out after 30 minutes)

❓ Questions

1. Why doesn't the Thirdweb Engine retry transactions rate-limited by the underlying RPC during the 2nd trial?
2. Why doesn't the Thirdweb Engine retry transactions stuck in the "sent" state during the 3rd trial?
3. Is there a way to force-retry failed or stuck transactions?
4. How long will the "Legacy" pricing be available?

⏭️ Action Items

• Send questions to Thirdweb support (expect response within 3 days)
• Run additional stress tests using the "Starter - Legacy" team with the higher 50 RPS rate limit.

💡 Learnings

• Thirdweb "Free" Team uses RPC with a 10 requests per second (RPS) rate limit.
• Thirdweb "Starter - Legacy" Team uses RPC with 50 requests per second (RPS) rate limit.

🧩 Context

• Without getting into details - we used the Thirdweb "Free" Team for these trials.

📷 Screenshots

Screenshots

• Trial 2: 94 Transactions failed due to underlying RPC rate limit
• Trial 3: 2 Transactions stuck in "sent", timed out after 30 minutes
• Thirdweb Engine "Free" Team has 10 RPS rate limit
• Thirdweb Engine "Starter - Legacy" team has 50 RPS rate limit

[MantisClone]

I queried the transactions configuration endpoint and received:

{
  "result": {
    "minTxsToProcess": 1,
    "maxTxsToProcess": 30,
    "minedTxListenerCronSchedule": "*/5 * * * * *",
    "maxTxsToUpdate": 50,
    "retryTxListenerCronSchedule": "*/15 * * * * *",
    "minEllapsedBlocksBeforeRetry": 12,
    "maxFeePerGasForRetries": "10000000000000",
    "maxPriorityFeePerGasForRetries": "10000000000000",
    "maxRetriesPerTx": 3,
    "clearCacheCronSchedule": "*/30 * * * * *"
  }
}

[MantisClone]

😓 🏃 Stress Tests (part 2) - 📊 More trials ,💡 New Learnings ,⏳ Waiting on Thirdweb Support

📊 3 more trials:

• Trial 4: 100 requests, 25 workers - 96% success, 3% stuck and timed out, 1% (first transaction) failed outright with error [RPC] invalid_argument possibly due to switch from "Free" RPC to "Starter - Legacy" RPC?
• Trial 5: 100 requests, 50 workers - 99% success, 1% stuck and timed out (91st transaction)
• Trial 6: 100 requests, 100 workers - 100% success in the Thirdweb Engine! - but only 37% success in script due to premature timeout in RN SDK.
  • RN SDK 0.54.0 times out after 130 seconds (according to 0.50.0 Release Notes)
  • Engine took about 5 minutes to finalize all transactions.

💡 Learnings

• Thirdweb Engine supports configurations for transaction processing and retries but they're undocumented. It also supports gas overrides.
• Current Engine configuration can handle 100 simultaneous requests, but it's only processing 30 at a time "maxTxsToProcess": 30,. Peak usage during Trial 6 was 40.3 RPS.
• Transactions are not retried until 12 blocks after they're sent. "minEllapsedBlocksBeforeRetry": 12,
• 🐞 Bug: "Cancel Transaction" button in Thirdweb Engine UI is visible but not clickable 😢

Thirdweb Support - ⏳ Waiting

• It's still not clear why some transactions get stuck in the "sent" state, are not retried, and time out after 30 minutes. This timeout does not seem to be configurable.

⏭️ Action Items

• Report broken "Cancel transaction" button to Thirdweb Support

Screenshots

Screenshots

• Peak usage during Trial 6 was 40.3 RPS
• Trial 6: 100% success in Thirdweb Engine! - but only 37% success in script due to premature timeout in RN SDK.
  
• Trial 6: The first and last request were queued within 1 second of each other.
• Cancel transaction button is not clickable
• Trial 4: 96% success, 3% stuck and timed out, 1% (first transaction) failed outright