feat: demo command, benchmark comparison, OpenClaw integration#113
Merged
feat: demo command, benchmark comparison, OpenClaw integration#113
Conversation
…oken cost in viewer Product & ecosystem improvements surfaced from the last30days research and competitive analysis of the AI agent memory space. ## New ### agentmemory demo command Seeds 3 realistic sessions (JWT auth, N+1 query fix, rate limiting) and runs smart-search queries against them. Shows token-aware semantic search finding "N+1 query fix" when you search "database performance optimization" — proof that keyword matching can't do that. npx @agentmemory/agentmemory demo Surfaces the "show don't tell" moment in 30 seconds without requiring the user to integrate anything. ### benchmark/COMPARISON.md Head-to-head comparison vs mem0 (53K stars, 68.5% LoCoMo), Letta/MemGPT (22K stars, 83.2% LoCoMo), Khoj (34K stars), claude-mem (46K stars), and Hippo. Feature matrix across 18 dimensions. Token efficiency table. Honest caveats about LongMemEval vs LoCoMo apples-vs-oranges. Lets users compare without trusting our README. ### integrations/openclaw/ OpenClaw gateway plugin following the same pattern as the Hermes integration. README, plugin.yaml with hooks config, and plugin.mjs implementing the 4 lifecycle hooks (on_session_start, on_pre_llm_call, on_post_tool_use, on_session_end). Gets agentmemory into the OpenClaw ecosystem for distribution. ## Improved ### Viewer token savings card Now shows tokens saved AND dollar cost saved (using $0.30/1K as a reasonable mid-tier model rate) instead of just raw numbers. Makes the ROI immediately tangible. ### README additions - "Try it in 30 seconds" section featuring the new demo command - Competitor comparison link in benchmarks section - OpenClaw integration link in supported agents table
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdded OpenClaw gateway plugin and manifest; introduced a CLI Changes
Sequence DiagramssequenceDiagram
participant User
participant CLI as Demo CLI
participant Server as agentmemory Server
participant Engine as Memory Engine
User->>CLI: run `npx `@agentmemory/agentmemory` demo`
CLI->>Server: GET /status
Server-->>CLI: 200 OK
loop for each demo session (3)
CLI->>Server: POST /session/start
Server->>Engine: create session
Engine-->>Server: session created
Server-->>CLI: session ID
loop observations
CLI->>Server: POST /agentmemory/observe (tool,input,output)
Server->>Engine: store observation
Engine-->>Server: ack
Server-->>CLI: OK
end
CLI->>Server: POST /session/end
Server->>Engine: finalize session
Engine-->>Server: finalized
Server-->>CLI: OK
end
loop queries
CLI->>Server: POST /agentmemory/smart-search (query)
Server->>Engine: semantic search
Engine-->>Server: hits + metadata
Server-->>CLI: results
end
CLI-->>User: print demo summary + viewer URL
sequenceDiagram
participant Agent as OpenClaw Agent
participant Plugin as agentmemory Plugin
participant Server as agentmemory Server
participant Engine as Memory Engine
Agent->>Plugin: onSessionStart(ctx)
Plugin->>Server: POST /session/start
Server->>Engine: create session
Engine-->>Server: session context
Server-->>Plugin: context
Plugin->>Agent: injectContext
Agent->>Plugin: onPreLlmCall(ctx, userMessage)
Plugin->>Server: POST /context (sessionId, message, tokenBudget, minConfidence)
Server->>Engine: retrieve relevant context
Engine-->>Server: context
Server-->>Plugin: context
Plugin->>Agent: injectContext
Agent->>Plugin: onPostToolUse(tool, input, output)
Plugin->>Server: POST /observe (metadata, timestamp)
Server->>Engine: record observation
Engine-->>Server: ack
Server-->>Plugin: OK
Agent->>Plugin: onSessionEnd(sessionId)
Plugin->>Server: POST /session/end
Server->>Engine: finalize session
Engine-->>Server: finalized
Server-->>Plugin: OK
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- Extract postJson helper in cli.ts runDemo (4x repeated fetch boilerplate → single call site) - Break runDemo into buildDemoSessions, seedDemoSession, runDemoSearch for single-responsibility - Replace three-branch if/else command dispatch with a commands map (single error handler) - Extract typed DemoSession/DemoObservation/SearchResult interfaces - Simplify OpenClaw plugin.mjs: collapse headers() + fetchJson() into a single postJson method Each hook is now ~5 lines instead of ~10 No behavior change. 654/654 tests passing, CLI help and error paths verified.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@integrations/openclaw/plugin.mjs`:
- Around line 17-24: The constructor never records the provided enabled flag so
the plugin runs even when disabled; update the constructor to set this.enabled =
config.enabled !== false (or Boolean(config.enabled)) and then add a guard at
the start of all public action methods (the methods that perform calls/work in
this class) to short-circuit when !this.enabled — e.g., return early or throw a
clear Disabled error in those methods so the plugin respects the enabled config.
- Around line 45-49: The code returns null for all non-OK HTTP responses
(res.ok) which ignores the fail-fast flag this.fallbackOnError; change the
!res.ok branch so that if this.fallbackOnError is false it throws a new Error
(include status/statusText or response body for context) and only returns null
when this.fallbackOnError is true, ensuring thrown errors propagate when
fail-fast is enabled.
In `@integrations/openclaw/README.md`:
- Around line 74-84: Update the fenced code block that begins with "Project
profile:" so its opening fence includes a language identifier (e.g., change ```
to ```text) to satisfy markdownlint MD040; keep the block content and closing
``` unchanged and ensure the identifier is added to the opening fence only.
- Around line 55-57: Update the README bullet list to use the actual hook names
used in the implementation: replace `prefetch()` with the pre-LLM hook,
`capture()` with the post-tool hook, and `consolidate()` with the session-end
hook, and include the session hook too; ensure each bullet uses the exact
lifecycle hook names (session, pre-LLM, post-tool, session-end) and short
descriptions that match their implemented behavior (e.g., pre-LLM injects
token-budgeted memories before each LLM call, post-tool records tool
use/errors/decisions after execution, session-end compresses observations into
structured memory).
In `@src/cli.ts`:
- Around line 287-325: Replace the timestamp-based IDs (e.g.,
`demo-session-1-${Date.now()}`, `demo-session-2-${Date.now() + 1}`, etc.) with
calls to the repository helper `generateId()` to follow project conventions and
avoid collisions; update each session object's `id` property to use
`generateId()` (for example `id: generateId()`), ensure `generateId` is imported
where `src/cli.ts` defines these demo sessions, and remove the `Date.now()`
offsets (do not use `fingerprintId()` which is for content-addressable
deduplication).
- Around line 341-373: The demo flow currently swallows fetch failures for the
agentmemory endpoints; update the session lifecycle and observation calls so
critical POSTs fail fast and non-2xx responses include error details: for the
session/start and session/end POSTs (calls to
`${base}/agentmemory/session/start` and `${base}/agentmemory/session/end`)
remove the .catch(() => null), await the response into a variable (e.g.,
startRes/endRes), if !res.ok read await res.text() and throw a new Error
including the status and body so the command exits; for the observation POST in
the loop (`${base}/agentmemory/observe`) similarly stop swallowing errors —
await obsRes, and if !obsRes.ok read the body and either log or throw as
appropriate (or at minimum log a warning with status/body) while still
incrementing totalObs only when obsRes.ok; reference variables: base,
demoProject, session.id, totalObs, obs.toolName/toolInput/toolOutput when
locating the calls.
In `@src/viewer/index.html`:
- Around line 1029-1032: The token-to-dollar conversion multiplies by the dollar
rate as if it were cents, causing ~100x underreporting; update the calculation
so cents = Math.round(tokensSaved * 0.03) (or compute dollars = tokensSaved /
1000 * 0.3 then cents = Math.round(dollars * 100)), then build costStr from
cents (show cents + 'ct' for <100 or (cents/100).toFixed(2) with '$' for >=100);
adjust the code around the variables tokensSaved, costCents/costStr, and the
HTML string that uses costStr accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 0268e78b-09c6-4731-bb49-cc2c890192a1
📒 Files selected for processing (7)
README.mdbenchmark/COMPARISON.mdintegrations/openclaw/README.mdintegrations/openclaw/plugin.mjsintegrations/openclaw/plugin.yamlsrc/cli.tssrc/viewer/index.html
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (2)
integrations/openclaw/plugin.mjs (2)
41-45:⚠️ Potential issue | 🟠 MajorApply
fallback_on_errorto non-2xx HTTP responses too.Line 41 always returns
nullfor non-OK responses, sofallback_on_error: falsedoes not fail fast on 4xx/5xx.Proposed fix
- if (!res.ok) return null; + if (!res.ok) { + const err = new Error( + `agentmemory request failed: ${res.status} ${res.statusText}`, + ); + if (!this.fallbackOnError) throw err; + return null; + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@integrations/openclaw/plugin.mjs` around lines 41 - 45, The current response-handling always returns null on non-2xx responses (the "if (!res.ok) return null;" block) so fallback_on_error is ignored; change that branch to respect this.fallbackOnError: when res.ok is false, if this.fallbackOnError is true then return null, otherwise throw an Error (include status and statusText) so callers fail fast; update the same function/block that uses this.fallbackOnError and res.ok to implement this conditional throw-or-return behavior.
17-24:⚠️ Potential issue | 🟠 MajorHonor the
enabledconfig flag across lifecycle hooks.Line 17 does not persist
config.enabled, and Lines 49-83 run unconditionally. This breaks the plugin manifest contract when users setenabled: false.Proposed fix
export class AgentmemoryPlugin { constructor(config = {}) { + this.enabled = config.enabled !== false; this.baseUrl = config.base_url || DEFAULT_BASE_URL; @@ async onSessionStart(ctx) { + if (!this.enabled) return; const result = await this.postJson("/agentmemory/session/start", { @@ async onPreLlmCall(ctx) { + if (!this.enabled) return; const result = await this.postJson("/agentmemory/context", { @@ async onPostToolUse(ctx) { + if (!this.enabled) return; await this.postJson("/agentmemory/observe", { @@ async onSessionEnd(ctx) { + if (!this.enabled) return; await this.postJson("/agentmemory/session/end", { sessionId: ctx.sessionId }); } }Also applies to: 49-83
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@integrations/openclaw/plugin.mjs` around lines 17 - 24, Persist the config.enabled flag in the constructor (e.g., set this.enabled = config.enabled !== false) and then guard all plugin lifecycle hooks so they no-op when disabled: add an early return at the top of each lifecycle method in this module (the lifecycle hooks that currently run unconditionally) to check if (!this.enabled) return or throw as appropriate; update any logic that assumes enabled (fallbackOnError, timeoutMs, secret usage) to only execute when this.enabled is true.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@integrations/openclaw/plugin.mjs`:
- Around line 59-64: The payload sent to /agentmemory/context (in plugin.mjs
where postJson is called) doesn't match the server contract: replace the
tokenBudget field with budget and add the required project field; e.g. send {
sessionId: ctx.sessionId, project: <use this.project or ctx.project>, query:
ctx.userMessage || "", budget: this.tokenBudget, minConfidence:
this.minConfidence } so the API handler (expecting sessionId, project, optional
budget) receives the correct fields.
- Around line 19-23: Replace the use of || for configuration defaults with
nullish coalescing so valid falsy values (e.g., 0) are preserved: change the
assignments for tokenBudget, minConfidence, and timeoutMs to use ?? instead of
||; also set fallbackOnError using ?? true (instead of !== false) to make the
default explicit while allowing a false value from config; keep the secret
assignment as-is. Reference the properties tokenBudget, minConfidence,
fallbackOnError, and timeoutMs in your changes.
---
Duplicate comments:
In `@integrations/openclaw/plugin.mjs`:
- Around line 41-45: The current response-handling always returns null on
non-2xx responses (the "if (!res.ok) return null;" block) so fallback_on_error
is ignored; change that branch to respect this.fallbackOnError: when res.ok is
false, if this.fallbackOnError is true then return null, otherwise throw an
Error (include status and statusText) so callers fail fast; update the same
function/block that uses this.fallbackOnError and res.ok to implement this
conditional throw-or-return behavior.
- Around line 17-24: Persist the config.enabled flag in the constructor (e.g.,
set this.enabled = config.enabled !== false) and then guard all plugin lifecycle
hooks so they no-op when disabled: add an early return at the top of each
lifecycle method in this module (the lifecycle hooks that currently run
unconditionally) to check if (!this.enabled) return or throw as appropriate;
update any logic that assumes enabled (fallbackOnError, timeoutMs, secret usage)
to only execute when this.enabled is true.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d1287196-809d-4a87-a916-6ac8bd1efc11
📒 Files selected for processing (2)
integrations/openclaw/plugin.mjssrc/cli.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/cli.ts
Comment on lines
+19
to
+23
| this.tokenBudget = config.token_budget || 2000; | ||
| this.minConfidence = config.min_confidence || 0.5; | ||
| this.fallbackOnError = config.fallback_on_error !== false; | ||
| this.timeoutMs = config.timeout_ms || DEFAULT_TIMEOUT_MS; | ||
| this.secret = process.env.AGENTMEMORY_SECRET; |
There was a problem hiding this comment.
Use nullish coalescing for config defaults.
Lines 19-23 use ||, which overwrites valid falsy values like 0 (e.g., min_confidence: 0, timeout_ms: 0).
Proposed fix
- this.baseUrl = config.base_url || DEFAULT_BASE_URL;
- this.tokenBudget = config.token_budget || 2000;
- this.minConfidence = config.min_confidence || 0.5;
+ this.baseUrl = config.base_url ?? DEFAULT_BASE_URL;
+ this.tokenBudget = config.token_budget ?? 2000;
+ this.minConfidence = config.min_confidence ?? 0.5;
this.fallbackOnError = config.fallback_on_error !== false;
- this.timeoutMs = config.timeout_ms || DEFAULT_TIMEOUT_MS;
+ this.timeoutMs = config.timeout_ms ?? DEFAULT_TIMEOUT_MS;🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@integrations/openclaw/plugin.mjs` around lines 19 - 23, Replace the use of ||
for configuration defaults with nullish coalescing so valid falsy values (e.g.,
0) are preserved: change the assignments for tokenBudget, minConfidence, and
timeoutMs to use ?? instead of ||; also set fallbackOnError using ?? true
(instead of !== false) to make the default explicit while allowing a false value
from config; keep the secret assignment as-is. Reference the properties
tokenBudget, minConfidence, fallbackOnError, and timeoutMs in your changes.
Owner
Author
There was a problem hiding this comment.
Fixed in b837a51. Changed || to ?? for baseUrl, tokenBudget, minConfidence, and timeoutMs so falsy-but-valid values like 0 or empty string aren't clobbered. fallbackOnError kept as !== false because it's a boolean, not a numeric config value.
There was a problem hiding this comment.
Rate Limit Exceeded
@rohitg00 have exceeded the limit for the number of chat messages per hour. Please wait 1 minutes and 1 seconds before sending another message.
## plugin.mjs
- Honor the enabled config flag: constructor now sets this.enabled and
all four hooks (onSessionStart, onPreLlmCall, onPostToolUse,
onSessionEnd) return early when disabled
- Fail-fast on non-OK HTTP: previously !res.ok returned null regardless
of fallbackOnError. Now throws with status/statusText/body when
fallbackOnError is false
- Update header JSDoc to use the real hook names instead of the
fictional prefetch/capture/consolidate
## openclaw README.md
- Replace stale hook names (prefetch/capture/consolidate) with the
actual onSessionStart/onPreLlmCall/onPostToolUse/onSessionEnd names
matching plugin.mjs
- Add 'text' language identifier to the Project profile code fence to
satisfy markdownlint MD040
## cli.ts
- Replace ad-hoc Date.now()-based session IDs with generateId("demo")
from src/state/schema.ts, matching the rest of the codebase
(snapshot.ts, migrate.ts, consolidation-pipeline.ts)
- Add postJsonStrict helper that throws on non-OK with status + body
- seedDemoSession now uses postJsonStrict for session/start and
session/end (critical lifecycle calls that should fail fast)
- The observation POST in the loop now inlines the fetch to surface
HTTP errors via p.log.warn with status/body, still only incrementing
the stored counter on success
## viewer/index.html
- Fix token-to-dollar conversion that was 100x underreporting:
tokensSaved / 1000 * 0.3 returns DOLLARS, not cents. Now computes
dollars first, then cents = Math.round(dollars * 100)
- 100K tokens saved now correctly displays $30.00, not 30ct
All 654 tests still passing.
There was a problem hiding this comment.
♻️ Duplicate comments (1)
integrations/openclaw/plugin.mjs (1)
69-73:⚠️ Potential issue | 🟠 MajorFix
/agentmemory/contextpayload keys to match the API contract.Line 69–73 still sends
tokenBudgetand omitsproject. If the backend contract remainsproject+ optionalbudget, this call can silently degrade retrieval or fail.Proposed fix
async onPreLlmCall(ctx) { if (!this.enabled) return; const result = await this.postJson("/agentmemory/context", { sessionId: ctx.sessionId, + project: ctx.project || ctx.cwd, query: ctx.userMessage || "", - tokenBudget: this.tokenBudget, + budget: this.tokenBudget, minConfidence: this.minConfidence, }); if (result?.context) ctx.injectContext(result.context); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@integrations/openclaw/plugin.mjs` around lines 69 - 73, The payload sent to this.postJson("/agentmemory/context") uses tokenBudget and omits project, which doesn't match the backend contract; update the payload keys so tokenBudget is renamed to budget and add the project property (use this.project if the plugin holds it, or ctx.project if project is per-request), keeping sessionId, query (ctx.userMessage || ""), and minConfidence as before so the call matches the API contract.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@integrations/openclaw/plugin.mjs`:
- Around line 69-73: The payload sent to this.postJson("/agentmemory/context")
uses tokenBudget and omits project, which doesn't match the backend contract;
update the payload keys so tokenBudget is renamed to budget and add the project
property (use this.project if the plugin holds it, or ctx.project if project is
per-request), keeping sessionId, query (ctx.userMessage || ""), and
minConfidence as before so the call matches the API contract.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 778baac9-4471-48f2-870b-9469f7df1020
📒 Files selected for processing (4)
integrations/openclaw/README.mdintegrations/openclaw/plugin.mjssrc/cli.tssrc/viewer/index.html
✅ Files skipped from review due to trivial changes (1)
- integrations/openclaw/README.md
🚧 Files skipped from review as they are similar to previous changes (2)
- src/viewer/index.html
- src/cli.ts
- Use nullish coalescing (??) for config defaults so valid falsy
values like min_confidence: 0 or timeout_ms: 0 aren't overwritten
by defaults. || would clobber them.
- Fix /agentmemory/context payload to match the server contract at
src/triggers/api.ts:115. The handler expects { sessionId, project,
budget? } but the plugin was sending { sessionId, query, tokenBudget,
minConfidence }. Three issues:
1. Missing required 'project' field
2. Wrong field name: 'tokenBudget' should be 'budget'
3. 'query' and 'minConfidence' are not part of this endpoint's
contract (probably belong on /smart-search instead)
Owner
Author
All 9 CodeRabbit findings addressed
Verification
Ready for re-review. |
4 tasks
4 tasks
This was referenced Apr 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Product and ecosystem additions surfaced from the last30days research (mem0, Letta, Khoj, claude-mem, Hippo competitive analysis) and the post-security-merge growth backlog.
New
`agentmemory demo` command
Seeds 3 realistic sessions and runs smart-search queries against them — proves the "find N+1 when you search database performance" moment in 30 seconds. Shows up in `--help` and new README "Try it in 30 seconds" section.
```bash
npx @agentmemory/agentmemory demo
```
`benchmark/COMPARISON.md`
Head-to-head comparison vs mem0 (53K⭐, 68.5% LoCoMo), Letta/MemGPT (22K⭐, 83.2% LoCoMo), Khoj (34K⭐), claude-mem (46K⭐), and Hippo. 18-dimension feature matrix. Token efficiency table. Honest caveats about LongMemEval vs LoCoMo.
`integrations/openclaw/`
OpenClaw gateway plugin following the Hermes pattern. README + plugin.yaml + plugin.mjs implementing 4 lifecycle hooks. Gets agentmemory into the OpenClaw ecosystem.
Improved
Test plan
Summary by CodeRabbit
New Features
Documentation
UI