From 3fd88aa37be2d9de89cfe6b50ae4899ed32986cd Mon Sep 17 00:00:00 2001 From: Dmitry Zakharov Date: Thu, 12 Jun 2025 15:12:11 +0400 Subject: [PATCH 1/3] Welcome envio@2.22.0 --- docs/HyperIndex/Advanced/loaders.md | 16 ++--- .../Advanced/multichain-indexing.mdx | 38 ++++++------ docs/HyperIndex/Guides/event-handlers.mdx | 59 ++++++++++++++++--- .../HyperIndex/Hosted_Service/self-hosting.md | 2 +- docs/HyperIndex/benchmarks.md | 10 ++-- 5 files changed, 80 insertions(+), 45 deletions(-) diff --git a/docs/HyperIndex/Advanced/loaders.md b/docs/HyperIndex/Advanced/loaders.md index 36d2d937..88f9c296 100644 --- a/docs/HyperIndex/Advanced/loaders.md +++ b/docs/HyperIndex/Advanced/loaders.md @@ -333,13 +333,9 @@ During the processing of the batch, another handler may set it before it is avai ```typescript ERC20.Transfer.handlerWithLoader({ loader: async ({ event, context }) => { - const sender = await context.Account.get(event.params.from); - // BE CAREFUL HERE // The loader will be run twice and sender may not exist on the first run - if (!sender) { - throw new Error(`Sender account not found: ${event.params.from}`); - } + const sender = await context.Account.getOrThrow(event.params.from); return { sender, @@ -353,31 +349,29 @@ ERC20.Transfer.handlerWithLoader({ }); ``` -The example above could crash unnecessarily. If you want to achieve this behaviour you should rather throw the error in the handler. +Starting from `envio@2.22.0` errors on the first loader run will be automatically caught and silently ignored, making your indexer to continue processing the batch. + +If you're using an earlier version of `envio`, the example above could crash unnecessarily. If you want to achieve the same behaviour you should rather throw the error in the handler. But better to upgrade your indexer with `pnpm install envio@latest`! ```typescript ERC20.Transfer.handlerWithLoader({ loader: async ({ event, context }) => { const sender = await context.Account.get(event.params.from); - return { sender, }; }, - handler: async ({ event, context, loaderReturn }) => { const { sender } = loaderReturn; - if (!sender) { throw new Error(`Sender account not found: ${event.params.from}`); } - // ... handler logic }, }); ``` -Now the indexer behaves as expected. The indexer will only crash if the "sender" `Account` entity was actually not set in an event preceding the one being processed. +The indexer will only crash if the `sender` entity was actually not set in an event preceding the one being processed. ## Limitations diff --git a/docs/HyperIndex/Advanced/multichain-indexing.mdx b/docs/HyperIndex/Advanced/multichain-indexing.mdx index 076f30a3..34f6895b 100644 --- a/docs/HyperIndex/Advanced/multichain-indexing.mdx +++ b/docs/HyperIndex/Advanced/multichain-indexing.mdx @@ -84,11 +84,13 @@ networks: When indexing multiple chains, you have two approaches for handling event ordering: -### Unordered Multichain Mode (Default) +### Unordered Multichain Mode -By default, the indexer processes events as soon as they're available from each chain, without waiting for other chains. This "Unordered Multichain Mode" provides better performance and lower latency. +:::note +Unordered mode is recommended for most applications. +::: -With unordered mode (the default): +The indexer processes events as soon as they're available from each chain, without waiting for other chains. This "Unordered Multichain Mode" provides better performance and lower latency. - Events will still be processed in order within each individual chain - Events across different chains may be processed out of order @@ -101,7 +103,20 @@ This mode is ideal for most applications, especially when: - Entities from different networks never interact with each other - Processing speed is more important than guaranteed cross-chain ordering -### Ordered Mode (Optional) +#### How to Enable Unordered Mode + +In your config.yaml: + +```yaml +unordered_multichain_mode: true +networks: ... +``` + +### Ordered Mode + +:::note +Ordered mode is currently the default mode. But it'll be changed to unordered mode in the future. If you don't need strict deterministic ordering of events across all chains, it's recommended to use unordered mode. +::: If your application requires strict deterministic ordering of events across all chains, you can enable "Ordered Mode". In this mode, the indexer synchronizes event processing across all chains, ensuring that events are processed in the exact same order in every indexer run, regardless of which chain they came from. @@ -120,21 +135,6 @@ Cross-chain ordering is particularly important for applications like: - **Multi-chain financial applications**: Where the sequence of transactions across chains affects accounting or risk calculations - **Data consistency systems**: Where the state must be consistent across multiple chains in a specific order -#### How to Enable Ordered Mode - -In your config.yaml: - -```yaml -unordered_multichain_mode: false -networks: ... -``` - -Or via environment variable: - -```sh -UNORDERED_MULTICHAIN_MODE=false -``` - #### Technical Details With ordered mode enabled: diff --git a/docs/HyperIndex/Guides/event-handlers.mdx b/docs/HyperIndex/Guides/event-handlers.mdx index cf12cbcc..c2c65a93 100644 --- a/docs/HyperIndex/Guides/event-handlers.mdx +++ b/docs/HyperIndex/Guides/event-handlers.mdx @@ -230,30 +230,71 @@ The handler `context` provides methods to interact with entities stored in the d ### Retrieving Entities -Retrieve entities asynchronously using `get`: +Retrieve entities from the database using `context.Entity.get` where `Entity` is the name of the entity you want to retrieve, which is defined in your [schema.graphql](/docs/HyperIndex/schema-file) file. -```javascript -await context..get(entityId); +```typescript +await context.Entity.get(entityId); +``` + +It'll return `Entity` object or `undefined` if the entity doesn't exist. + +Starting from `envio@2.22.0` you can use `context.Entity.getOrThrow` to conveniently throw an error if the entity doesn't exist: + +```typescript +const pool = await context.Pool.getOrThrow(poolId); +// Will throw: Entity 'Pool' with ID '...' is expected to exist. + +// Or you can pass a custom message as a second argument: +const pool = await context.Pool.getOrThrow( + poolId, + `Pool with ID ${poolId} is expected.` +); +``` + +Or use `context.Entity.getOrCreate` to automatically create an entity with default values if it doesn't exist: + +```typescript +const pool = await context.Pool.getOrCreate({ + id: poolId, + totalValueLockedETH: 0n, +}); + +// Which is equivalent to: +let pool = await context.Pool.get(poolId); +if (!pool) { + pool = { + id: poolId, + totalValueLockedETH: 0n, + }; + context.Pool.set(pool); +} ``` ### Modifying Entities -Use `set` to create or update an entity: +Use `context.Entity.set` to create or update an entity: -```javascript -context..set(entityObject); +```typescript +context.Entity.set({ + id: entityId, + ...otherEntityFields, +}); ``` +:::note +Both `context.Entity.set` and `context.Entity.deleteUnsafe` methods use the In-Memory Storage under the hood and don't require `await` in front of them. +::: + ### Deleting Entities (Unsafe) To delete an entity: -```javascript -context..deleteUnsafe(entityId); +```typescript +context.Entity.deleteUnsafe(entityId); ``` :::warning -The `deleteUnsafe` method is experimental and **unsafe**. Manually handle all entity references after deletion to maintain database consistency. +The `deleteUnsafe` method is experimental and **unsafe**. You need to manually handle all entity references after deletion to maintain database consistency. ::: ### Updating Specific Entity Fields diff --git a/docs/HyperIndex/Hosted_Service/self-hosting.md b/docs/HyperIndex/Hosted_Service/self-hosting.md index deb463c8..8e9881b1 100644 --- a/docs/HyperIndex/Hosted_Service/self-hosting.md +++ b/docs/HyperIndex/Hosted_Service/self-hosting.md @@ -56,7 +56,7 @@ services: - my-proxy-net graphql-engine: - image: hasura/graphql-engine:v2.23.0 + image: hasura/graphql-engine:v2.43.0 ports: - "${HASURA_EXTERNAL_PORT:-8080}:8080" user: 1001:1001 diff --git a/docs/HyperIndex/benchmarks.md b/docs/HyperIndex/benchmarks.md index 24bdfa2e..407d0735 100644 --- a/docs/HyperIndex/benchmarks.md +++ b/docs/HyperIndex/benchmarks.md @@ -19,14 +19,14 @@ The most comprehensive and up-to-date benchmarks were conducted by Sentio in Apr | Case | Description | Envio | Nearest Competitor | TheGraph | Ponder | Advantage vs. Nearest | | ------------------------------ | ------------------------------------------- | ------ | ------------------ | -------- | ------ | --------------------- | -| LBTC Token Transfers | Event handling, No RPC calls, Write-only | 2m | 8m (Sentio) | 3h9m | 1h40m | 4x faster | -| LBTC Token with RPC calls | Event handling, RPC calls, Read-after-write | 15s | 32m (Subsquid) | 18h38m | 4h38m | 128x faster | +| LBTC Token Transfers | Event handling, No RPC calls, Write-only | 3m | 8m (Sentio) | 3h9m | 1h40m | 2.6x faster | +| LBTC Token with RPC calls | Event handling, RPC calls, Read-after-write | 1m | 6m (Sentio) | 1h30m | 45m | 6x faster | | Ethereum Block Processing | 100K blocks with Metadata extraction | 7.9s | 1m (Subsquid) | 10m | 33m | 7.5x faster | -| Ethereum Transaction Gas Usage | Transaction handling, Gas calculations | 1m 26s | 5m (Subsquid) | N/A | 33m | 3.5x faster | +| Ethereum Transaction Gas Usage | Transaction handling, Gas calculations | 1m 26s | 7m (Subsquid) | N/A | 33m | 4.8x faster | | Uniswap V2 Swap Trace Analysis | Transaction trace handling, Swap decoding | 41s | 2m (Subsquid) | 8m | N/A | 3x faster | -| Uniswap V2 Factory | Event handling, Pair and swap analysis | 10s | 2m (Subsquid) | 19m | 2h24m | 12x faster | +| Uniswap V2 Factory | Event handling, Pair and swap analysis | 8s | 2m (Subsquid) | 19m | 21m | 15x faster | -The independent benchmark results demonstrate that HyperIndex consistently outperforms all competitors across every tested scenario. The most significant performance advantage was seen in real-world indexing scenarios with external RPC calls, where HyperIndex was up to 128x faster than the nearest competitor and over 4000x faster than TheGraph. +The independent benchmark results demonstrate that HyperIndex consistently outperforms all competitors across every tested scenario. This includes the most realistic real-world indexing scenario LBTC Token with RPC calls - where HyperIndex was up to 6x faster than the nearest competitor and over 90x faster than TheGraph. ## Historical Benchmarking Results From ebc0bf02059b9068e937f4477ceb7588eb613c71 Mon Sep 17 00:00:00 2001 From: Dmitry Zakharov Date: Thu, 12 Jun 2025 15:14:48 +0400 Subject: [PATCH 2/3] Fix path --- docs/HyperIndex/Guides/event-handlers.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/HyperIndex/Guides/event-handlers.mdx b/docs/HyperIndex/Guides/event-handlers.mdx index c2c65a93..c2033ace 100644 --- a/docs/HyperIndex/Guides/event-handlers.mdx +++ b/docs/HyperIndex/Guides/event-handlers.mdx @@ -230,7 +230,7 @@ The handler `context` provides methods to interact with entities stored in the d ### Retrieving Entities -Retrieve entities from the database using `context.Entity.get` where `Entity` is the name of the entity you want to retrieve, which is defined in your [schema.graphql](/docs/HyperIndex/schema-file) file. +Retrieve entities from the database using `context.Entity.get` where `Entity` is the name of the entity you want to retrieve, which is defined in your [schema.graphql](./schema-file.md) file. ```typescript await context.Entity.get(entityId); From b0d2c0818d0046c5b4623d3d6e2adea813491dd1 Mon Sep 17 00:00:00 2001 From: Dmitry Zakharov Date: Thu, 12 Jun 2025 15:43:08 +0400 Subject: [PATCH 3/3] Fix benchmark --- docs/HyperIndex/benchmarks.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/HyperIndex/benchmarks.md b/docs/HyperIndex/benchmarks.md index 407d0735..e5f0e303 100644 --- a/docs/HyperIndex/benchmarks.md +++ b/docs/HyperIndex/benchmarks.md @@ -17,16 +17,16 @@ The most comprehensive and up-to-date benchmarks were conducted by Sentio in Apr ### Key Performance Highlights -| Case | Description | Envio | Nearest Competitor | TheGraph | Ponder | Advantage vs. Nearest | -| ------------------------------ | ------------------------------------------- | ------ | ------------------ | -------- | ------ | --------------------- | -| LBTC Token Transfers | Event handling, No RPC calls, Write-only | 3m | 8m (Sentio) | 3h9m | 1h40m | 2.6x faster | -| LBTC Token with RPC calls | Event handling, RPC calls, Read-after-write | 1m | 6m (Sentio) | 1h30m | 45m | 6x faster | -| Ethereum Block Processing | 100K blocks with Metadata extraction | 7.9s | 1m (Subsquid) | 10m | 33m | 7.5x faster | -| Ethereum Transaction Gas Usage | Transaction handling, Gas calculations | 1m 26s | 7m (Subsquid) | N/A | 33m | 4.8x faster | -| Uniswap V2 Swap Trace Analysis | Transaction trace handling, Swap decoding | 41s | 2m (Subsquid) | 8m | N/A | 3x faster | -| Uniswap V2 Factory | Event handling, Pair and swap analysis | 8s | 2m (Subsquid) | 19m | 21m | 15x faster | - -The independent benchmark results demonstrate that HyperIndex consistently outperforms all competitors across every tested scenario. This includes the most realistic real-world indexing scenario LBTC Token with RPC calls - where HyperIndex was up to 6x faster than the nearest competitor and over 90x faster than TheGraph. +| Case | Description | Envio | Nearest Competitor | TheGraph | Ponder | +| ------------------------------ | ------------------------------------------- | ------ | --------------------------- | ------------------- | -------------------- | +| LBTC Token Transfers | Event handling, No RPC calls, Write-only | 3m | 8m - 2.6x slower (Sentio) | 3h9m - 3780x slower | 1h40m - 2000x slower | +| LBTC Token with RPC calls | Event handling, RPC calls, Read-after-write | 1m | 6m - 6x slower (Sentio) | 1h3m - 63x slower | 45m - 45x slower | +| Ethereum Block Processing | 100K blocks with Metadata extraction | 7.9s | 1m - 7.5x slower (Subsquid) | 10m - 75x slower | 33m - 250x slower | +| Ethereum Transaction Gas Usage | Transaction handling, Gas calculations | 1m 26s | 7m - 4.8x slower (Subsquid) | N/A | 33m - 23x slower | +| Uniswap V2 Swap Trace Analysis | Transaction trace handling, Swap decoding | 41s | 2m - 3x slower (Subsquid) | 8m - 11x slower | N/A | +| Uniswap V2 Factory | Event handling, Pair and swap analysis | 8s | 2m - 15x slower (Subsquid) | 19m - 142x slower | 21m - 157x slower | + +The independent benchmark results demonstrate that HyperIndex consistently outperforms all competitors across every tested scenario. This includes the most realistic real-world indexing scenario LBTC Token with RPC calls - where HyperIndex was up to 6x faster than the nearest competitor and over 63x faster than TheGraph. ## Historical Benchmarking Results