docs(nips): NIP-AE — Agent Engrams

[tlongwell-block]

What this is

A draft NIP — NIP-AE, Agent Engrams — for AI agents to keep persistent, structured memory on Nostr. The whole spec is one file: docs/nips/NIP-AE.md. No code changes.

The convention is small on purpose: a dedicated addressable kind plus a few mechanical rules. No relay changes, no new crypto, no negotiation. Two implementations following the spec from text alone converge on bit-identical events.

The model in one paragraph

Memory is scoped to a (pubkey_a, pubkey_o) pair — one agent, one owner. The agent is the sole signer; the agent and owner share a NIP-44 conversation key, so both can decrypt every record. The owner is a first-class reader by construction, not by permission. Two record types: core (one per pair, holds identity/rules/goals) and memory (zero or more, one logical entry each). Both ride a kind:30174 envelope, differing only by the slug at which they are addressed. Slugs are never visible on the wire — the d tag is HMAC-SHA256(K_c, "agent-memory/v1/d-tag" || 0x00 || slug). Tombstones are an in-band write with value: null. Cross-record references use wiki-link syntax [[mem/foo]] extracted by literal substring; a reachability graph rooted at core.profile (non-normative) gives implementations a deterministic answer to "which memories are referenced from the agent's identity surface," and anything outside it is an orphan surfaced for review.

Why a dedicated kind (30174)?

Earlier drafts rode kind:30078 (NIP-78 application-specific data) and demultiplexed via the validity gate. That worked — the spec was honest about it — but it shoved a coordination concern (sharing an address space with arbitrary other apps on the same author) into the protocol when we didn't have to.

A dedicated kind:

• Isolates this NIP's address space from any other app the agent's pubkey might also write.
• Lets observers and unknown-kind viewers identify these events from the kind alone, without attempting NIP-44 decryption as a namespace demultiplexer.
• Honestly claims a slot in the NIP-01 addressable range rather than borrowing one.

Choice of 30174 is mnemonic: 30 || 0xAE where AE is Agent Engrams — the same hex encoding the NIP indicator uses. Verified unused against the kinds table in nostr-protocol/nips and unmentioned in the broader repo.

Design discipline (what the spec actually pins)

The structural moves are what makes it small:

• HMAC-SHA256 d-tags under K_c, domain-separated by a fixed version-tagged prefix. Reveals nothing to passive observers; deterministic for both parties.
• Validity is one atomic gate of five rules (kind/pubkey/tags, signature, decrypt + JSON parse that rejects duplicate member names, slug↔d re-derivation, body shape). Applied before head selection, identical for read/write-verify/list. One predicate everywhere.
• Monotonic created_at per slug (max(now, T_head + 1)) defeats NIP-01's same-second tiebreak under NIP-44 random nonces. A head whose created_at is implausibly far ahead of wall-clock time SHOULD be treated as clock-poisoned and surfaced as a conflict rather than published; the threshold is left to implementations.
• The walk is the source of truth. Listing is defined as a result-shape — the set of head tuples produced by querying every configured relay, unioning, validating, and reducing — not a mechanism. Clients MAY keep an out-of-band {slug → {event_id, created_at}} cache to accelerate listing, but it is implementation-local and outside the wire format; an in-band index field was dropped after review because it created a concurrent-writer conflict surface for no protocol-level benefit.
• Discovery specified by result, not mechanism. The same shape-not-mechanism rule means a future relay-maintained materialized view over public d tags can slot in without a wire change.
• Listing is best-effort. Nostr has no protocol-level pagination, so any complete-set claim is provisional. Clients SHOULD surface per-relay truncation signals (limit-reached, event counts) where available so callers can detect under-reporting.
• Reachability via literal [[<slug>]] substring extraction — clean machine-extractable signal without forcing the agent to maintain a parallel structured-links field. Marked non-normative; conformance doesn't require honoring it. A companion NIP can make it normative.
• Configured relays = filtered NIP-65 write relays of pubkey_a (write or no-marker entries, valid ws:///wss:// URLs), with an out-of-band fallback when the published list is absent or yields zero usable entries. URL canonicalization is for comparison only (lowercase scheme/host, strip default port, strip empty-path trailing slash, dedupe so two parties compute the same set); connections SHOULD be made to the advertised form so relay-side path/host disambiguation is preserved.
• p tag carries its usual NIP-01 meaning. Generic NIP-65-aware clients may fan a record out to the owner's read relays; this NIP neither requires nor forbids that. Authoritative discovery is always from the agent's configured relays so owners and observers converge on the same head set.
• Write verification (recommended, not required). Implementations SHOULD recompute the head after a relay OK (optionally with a short propagation delay) and surface a conflict if the published event is not the head. Best-effort by design: writes arriving after the recompute window remain subject to the eventual-consistency semantics. Fixed-window and "MUST NOT silently retry" policies were softened to SHOULDs.

Reference test vectors (embedded, self-verifying)

The spec embeds deterministic values derived from pinned seckey_a/seckey_o/Schnorr aux/NIP-44 nonces. An implementer can re-derive every event id, signature, ciphertext, and d-value from the spec's own inputs and byte-diff against the spec's own outputs.

• Reuses NIP-44's published test scalars (sec1=…01, sec2=…02) so K_c cross-checks for free against nip44.vectors.json line 281 (c41c775356fd92eadc63ff5a0dc1da211b268cbea22316767095b2871ea1412d).
• Reuses BIP-340's test vector 0 sig prefix (e907831f80…) as an external anchor for the aux-interpretation gotcha.
• Four events exercise: single-segment slug, multi-segment slug, tombstone-supersedes-prior (same d, greater created_at), and a core event whose profile contains wiki-link references to both memory slugs — exercising wiki-link extraction from both profile and value, plus a dangling reference to a tombstoned slug.

Three documented implementation gotchas — places where independent re-derivations are most likely to diverge silently:

1. NIP-44 ECDH IKM is the raw shared_x, not its SHA-256 (some bindings' default ecdh() hashes it).
2. BIP-340 aux = 0x00…00 is not "aux omitted" — it must be passed through the BIP0340/aux tagged hash and XOR'd with the secret. Some libsecp256k1 wrappers default sign_custom to NULL extraparams and silently skip the XOR.
3. NIP-01 event-id serialization needs ensure_ascii=False — the moment a body contains a non-ASCII character, \u escapes diverge.

The vectors were originally co-produced and cross-validated against an independent implementation by Sami; outputs matched byte-for-byte across two crypto stacks. After moving to kind 30174 the event ids and signatures changed (kind is in the id pre-image); the pubkeys, conversation key, d-tags, and ciphertexts 1–3 are unchanged from those earlier verified vectors. Event 4's body changed again when core.index was dropped, so its ciphertext, id, and signature were regenerated; all four sigs re-verify under Schnorr.

What's explicitly out of scope

Provenance / trust levels, attention or working sets, structured links, owner-to-agent directives, admission control. Bodies use a strict-required-fields / permissive-unknown-fields rule so companion NIPs can add these later without breaking v1 readers.

Acknowledged caveats (in the spec, not just the PR)

• No forward secrecy. Static ECDH; compromise of either party's secret key exposes the full history. Trade-off accepted for symmetric read access.
• Compromised agent key = silent rewrite and full read. Holders of seckey_a can rewrite or tombstone any record, and can re-derive K_c against every known owner pubkey to decrypt all past and future records for those pairs. Replaceable events leave no protocol-level trace of overwrites on compliant relays. Archival relays may show that rewrites occurred but cannot identify which version is authoritative; the NIP defines no version-chaining mechanism.
• Metadata leak. The triple (pubkey_a, kind:30174, p=pubkey_o) reveals that an account uses agent memory and identifies its owner. Pseudonymous, not anonymous.
• No owner write authority. The owner can read everything but cannot author records. Any control plane is out of band.
• Memory poisoning is unsolved at the spec layer. Encryption protects confidentiality, not the truthfulness of what the agent decides to remember. Admission control is the implementer's problem.

Risk / scope of this PR

Documentation only. One file added to docs/nips/. No code paths touched.

How it got here

This NIP went through a long iteration in the #sprout-memory channel. Highlights of what the design walked back from earlier sketches:

• Dropped a parallel owner control plane (four verbs + polling + replay protection). The owner influences memory by talking to the agent, not by signing.
• Dropped append-only knowledge / ULID / supersession-link machinery in favor of two replaceable types. Audit trails are a deployment concern, not a protocol one.
• Dropped private-from-owner memory. An agent that can cryptographically hide work from its owner is misaligned by construction; we don't build that affordance.
• Dropped structured links arrays in favor of literal [[<slug>]] substring references — the agent already thinks in prose, so the references live in prose.
• Dropped core.index (was in earlier drafts of this PR). The walk over configured relays was already authoritative; an in-band cache added a concurrent-writer conflict surface for no protocol-level benefit. Clients that want one keep it locally.

Reviewed cross-NIP against NIP-01 / 09 / 44 / 65 / 78 in two parallel passes, then again head-to-head against ten upstream comparators (01, 11, 17, 22, 23, 44, 51, 65, 78, 90). Beats nine; ties NIP-44 in original orientation and loses to NIP-44 in spec-identity mode — a structurally honest result (NIP-44 is a byte-level crypto primitive with full algorithmic pinning; a coordination-layer NIP can't match its correctness density).

After PR open, five rounds of codex review against the upstream NIPs produced the structural simplifications committed as 93d2c46: dropping core.index, decoupling the p tag from a NIP-65 carve-out, softening write-verify and clock-poison from MUSTs to SHOULDs, splitting URL canonicalization (compare-only) from URL connect (advertised form), and marking the reachability graph non-normative. Codex never crossed 8/10 on the scored rubric; each pass pivoted to a new "major," which read as the rubric flooring drafts at 8 against "merge upstream as-is" rather than a genuine remaining defect. Shipped at 8.