diff --git a/AGENTS.md b/AGENTS.md index 47bf128..628bd90 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,6 +29,7 @@ | AI Agent Commits & Git Analysis Impact | ai-agent-commits-git-analysis-impact.md | 25 | medium | 2026-03-14 | | AI Commit Detection Forensics | ai-commit-detection-forensics.md | 42 | deep | 2026-03-14 | | glide-mq (Message Queue on Valkey/Redis) | glide-mq.md | 12 | medium | 2026-03-19 | +| Bee-Queue API Reference (v2.0.0) | bee-queue-api.md | npm docs | reference | 2026-03-19 | | Skills for Library/SDK - Best Practices | skills-for-library-and-sdk-best-practices.md | 16 | medium | 2026-03-19 | | RAG Skill Indexing for AI Agent Knowledge | rag-skill-indexing.md | 18 | deep | 2026-03-31 | | Codex CLI Plugin Manifest System | codex-plugin-manifest.md | 42 | deep | 2026-04-01 | @@ -382,6 +383,12 @@ Use this knowledge when user asks about: - "valkey streams queue" -> glide-mq.md - "bullmq alternative" -> glide-mq.md - "node.js job queue valkey" -> glide-mq.md +- "bee-queue API" -> bee-queue-api.md +- "bee-queue reference" -> bee-queue-api.md +- "bee-queue migration" -> bee-queue-api.md +- "bee-queue queue constructor" -> bee-queue-api.md +- "bee-queue events" -> bee-queue-api.md +- "bee-queue vs bullmq API" -> bee-queue-api.md - "SKILL.md best practices" -> skills-for-library-and-sdk-best-practices.md - "writing skills for library" -> skills-for-library-and-sdk-best-practices.md - "SDK skill design" -> skills-for-library-and-sdk-best-practices.md @@ -696,6 +703,9 @@ Use this knowledge when user asks about: | bullmq alternative, bullmq migration | glide-mq.md | | FCALL, server functions, NAPI rust | glide-mq.md | | FlowProducer, DAG workflow, fan-out | glide-mq.md | +| bee-queue API, bee-queue migration | bee-queue-api.md | +| bee-queue Queue constructor, Job methods | bee-queue-api.md | +| bee-queue stalled, delayed, retries | bee-queue-api.md | | SKILL.md, skill authoring, skill format | skills-for-library-and-sdk-best-practices.md | | skill description, skill trigger, discovery | skills-for-library-and-sdk-best-practices.md | | progressive disclosure, token budget | skills-for-library-and-sdk-best-practices.md | diff --git a/CLAUDE.md b/CLAUDE.md index 47bf128..628bd90 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,6 +29,7 @@ | AI Agent Commits & Git Analysis Impact | ai-agent-commits-git-analysis-impact.md | 25 | medium | 2026-03-14 | | AI Commit Detection Forensics | ai-commit-detection-forensics.md | 42 | deep | 2026-03-14 | | glide-mq (Message Queue on Valkey/Redis) | glide-mq.md | 12 | medium | 2026-03-19 | +| Bee-Queue API Reference (v2.0.0) | bee-queue-api.md | npm docs | reference | 2026-03-19 | | Skills for Library/SDK - Best Practices | skills-for-library-and-sdk-best-practices.md | 16 | medium | 2026-03-19 | | RAG Skill Indexing for AI Agent Knowledge | rag-skill-indexing.md | 18 | deep | 2026-03-31 | | Codex CLI Plugin Manifest System | codex-plugin-manifest.md | 42 | deep | 2026-04-01 | @@ -382,6 +383,12 @@ Use this knowledge when user asks about: - "valkey streams queue" -> glide-mq.md - "bullmq alternative" -> glide-mq.md - "node.js job queue valkey" -> glide-mq.md +- "bee-queue API" -> bee-queue-api.md +- "bee-queue reference" -> bee-queue-api.md +- "bee-queue migration" -> bee-queue-api.md +- "bee-queue queue constructor" -> bee-queue-api.md +- "bee-queue events" -> bee-queue-api.md +- "bee-queue vs bullmq API" -> bee-queue-api.md - "SKILL.md best practices" -> skills-for-library-and-sdk-best-practices.md - "writing skills for library" -> skills-for-library-and-sdk-best-practices.md - "SDK skill design" -> skills-for-library-and-sdk-best-practices.md @@ -696,6 +703,9 @@ Use this knowledge when user asks about: | bullmq alternative, bullmq migration | glide-mq.md | | FCALL, server functions, NAPI rust | glide-mq.md | | FlowProducer, DAG workflow, fan-out | glide-mq.md | +| bee-queue API, bee-queue migration | bee-queue-api.md | +| bee-queue Queue constructor, Job methods | bee-queue-api.md | +| bee-queue stalled, delayed, retries | bee-queue-api.md | | SKILL.md, skill authoring, skill format | skills-for-library-and-sdk-best-practices.md | | skill description, skill trigger, discovery | skills-for-library-and-sdk-best-practices.md | | progressive disclosure, token budget | skills-for-library-and-sdk-best-practices.md | diff --git a/README.md b/README.md index 301f1d2..242971d 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,7 @@ See `CLAUDE.md` / `AGENTS.md` for the full topic index with trigger phrases and | AI Agent Commits & Git Analysis Impact | ai-agent-commits-git-analysis-impact.md | | AI Commit Detection Forensics | ai-commit-detection-forensics.md | | glide-mq (Message Queue on Valkey/Redis) | glide-mq.md | +| Bee-Queue API Reference (v2.0.0) | bee-queue-api.md | | Skills for Library/SDK - Best Practices | skills-for-library-and-sdk-best-practices.md | | RAG Skill Indexing for AI Agent Knowledge | rag-skill-indexing.md | | Codex CLI Plugin Manifest System | codex-plugin-manifest.md | diff --git a/codex-plugin-manifest.md b/codex-plugin-manifest.md new file mode 100644 index 0000000..ab55cd4 --- /dev/null +++ b/codex-plugin-manifest.md @@ -0,0 +1,651 @@ +# Learning Guide: Codex CLI Plugin Manifest System + +**Generated**: 2026-04-01 +**Sources**: 42 resources analyzed (source code, official docs, PRs, issues, TypeScript protocol) +**Depth**: deep +**Codex CLI Version**: v0.117.0+ (plugin system introduced March 26, 2026) + +## Prerequisites + +- Familiarity with JSON manifest formats +- Understanding of Codex CLI basics (installation, config.toml) +- Basic knowledge of MCP (Model Context Protocol) concepts +- Understanding of plugin/extension architectures + +## TL;DR + +- Plugins live in `.codex-plugin/` with a required `plugin.json` manifest +- Three component types: skills (SKILL.md files), MCP servers (.mcp.json), apps (.app.json) +- All paths must be relative, starting with `./`, no `..` traversal allowed +- Plugin names must be kebab-case, ASCII alphanumeric plus hyphens/underscores only +- Marketplace distribution via `.agents/plugins/marketplace.json` at repo or home level +- Default prompts limited to 3 entries, 128 characters each +- Interface section is optional - entirely omitted if all fields are empty +- Plugin hooks are NOT supported in plugin.json despite some docs suggesting otherwise + +## Core Concepts + +### Plugin Directory Structure + +A Codex plugin is a directory containing a `.codex-plugin/plugin.json` manifest and optional component directories: + +``` +my-plugin/ + .codex-plugin/ + plugin.json # Required manifest (only file in .codex-plugin/) + skills/ + skill-name/ + SKILL.md # Skill instructions + .mcp.json # MCP server configuration + .app.json # App connector mappings + assets/ + icon.png # Visual assets (logo, screenshots, etc.) +``` + +**Critical rule**: Only `plugin.json` belongs inside `.codex-plugin/`. All other files go at the plugin root. + +The manifest path constant is `.codex-plugin/plugin.json` (defined as `PLUGIN_MANIFEST_PATH` in `codex-utils-plugins`). + +### Manifest Loading Flow + +``` +CLI startup + -> PluginsManager::plugins_for_config() + -> For each configured plugin: + -> load_plugin_manifest(plugin_root) + -> Read {plugin_root}/.codex-plugin/plugin.json + -> Deserialize as RawPluginManifest (serde_json) + -> Resolve paths (skills, mcpServers, apps) + -> Resolve interface fields (assets, prompts) + -> Return PluginManifest or None + -> Load skills from resolved skills path + -> Load MCP servers from resolved mcpServers path + -> Load app connectors from resolved apps path + -> Build PluginLoadOutcome with all plugins +``` + +## Manifest Schema (Comprehensive) + +### Top-Level Fields + +```json +{ + "name": "string (required, validated)", + "description": "string | null (optional)", + "skills": "./relative-path (optional)", + "mcpServers": "./relative-path (optional)", + "apps": "./relative-path (optional)", + "interface": { /* optional PluginManifestInterface */ } +} +``` + +**Note on field naming**: The raw deserialization uses `camelCase` (`#[serde(rename_all = "camelCase")]`). The path fields `skills`, `mcpServers`, and `apps` are top-level string fields, NOT nested under a `paths` object. The internal Rust struct `PluginManifestPaths` groups them after parsing, but the JSON is flat. + +### Name Field + +- **Required**: Yes, but if empty/whitespace, falls back to the plugin directory name +- **Validation**: Must match `validate_plugin_segment()` rules when used as PluginId: + - Cannot be empty + - Only ASCII alphanumeric characters, hyphens (`-`), and underscores (`_`) + - No path separators (`/`, `\`) + - No dot sequences (`.`, `..`) +- **Convention**: Kebab-case (e.g., `my-first-plugin`) +- **Purpose**: Used as both the plugin identifier and skill namespace prefix + +```rust +// Name fallback logic (from manifest.rs): +let name = plugin_root + .file_name() + .and_then(|entry| entry.to_str()) + .filter(|_| raw_name.trim().is_empty()) + .unwrap_or(&raw_name) + .to_string(); +``` + +When the manifest `name` is empty or whitespace-only, the directory name is used instead. + +### Description Field + +- **Optional**: Defaults to None +- **Sanitization for capability summary**: Newlines and irregular whitespace collapsed to single spaces +- **Truncation**: `MAX_CAPABILITY_SUMMARY_DESCRIPTION_LEN` (1024 characters) for the capability summary +- **Original preserved**: The manifest description is kept as-is; only the `PluginCapabilitySummary` description is sanitized + +### Component Path Fields (skills, mcpServers, apps) + +All three path fields follow identical validation rules defined in `resolve_manifest_path()`: + +1. **Must start with `./`** - Paths without this prefix are silently ignored with a warning +2. **Must not be just `./`** - Empty relative part after stripping prefix is rejected +3. **Must not contain `..`** - Parent directory traversal is blocked +4. **Must stay within plugin root** - Only `Component::Normal` path components allowed +5. **Must resolve to absolute path** - Final path is `plugin_root.join(normalized)` +6. **Empty string = None** - An empty string is treated as absent + +```rust +// Validation logic (from manifest.rs): +fn resolve_manifest_path(plugin_root: &Path, field: &'static str, path: Option<&str>) -> Option { + let path = path?; + if path.is_empty() { return None; } + + let Some(relative_path) = path.strip_prefix("./") else { + tracing::warn!("ignoring {field}: path must start with `./` relative to plugin root"); + return None; + }; + if relative_path.is_empty() { + tracing::warn!("ignoring {field}: path must not be `./`"); + return None; + } + + let mut normalized = PathBuf::new(); + for component in Path::new(relative_path).components() { + match component { + Component::Normal(component) => normalized.push(component), + Component::ParentDir => { + tracing::warn!("ignoring {field}: path must not contain '..'"); + return None; + } + _ => { + tracing::warn!("ignoring {field}: path must stay within the plugin root"); + return None; + } + } + } + + AbsolutePathBuf::try_from(plugin_root.join(normalized)).ok() +} +``` + +**Default paths when manifest fields are absent**: +- Skills: `./skills/` (directory containing skill subdirectories) +- MCP Servers: `./.mcp.json` (JSON file at plugin root) +- Apps: `./.app.json` (JSON file at plugin root) + +### Interface Section + +The `interface` object provides display metadata for the plugin marketplace/directory. It is entirely optional - if present but all fields are empty/default, the interface is omitted (set to `None`). + +```json +{ + "interface": { + "displayName": "string | null", + "shortDescription": "string | null", + "longDescription": "string | null", + "developerName": "string | null", + "category": "string | null", + "capabilities": ["array of strings"], + "websiteUrl": "string | null", + "privacyPolicyUrl": "string | null", + "termsOfServiceUrl": "string | null", + "defaultPrompt": "string | [array] | null", + "brandColor": "string | null", + "composerIcon": "./relative-path | null", + "logo": "./relative-path | null", + "screenshots": ["./screenshot-1.png", "./screenshot-2.png"] + } +} +``` + +**Field aliases**: The raw deserialization accepts both camelCase and URL-suffix variants: +- `websiteUrl` or `websiteURL` +- `privacyPolicyUrl` or `privacyPolicyURL` +- `termsOfServiceUrl` or `termsOfServiceURL` + +**Interface omission logic**: The interface is set to `None` if ALL of these are true: +- `displayName` is None +- `shortDescription` is None +- `longDescription` is None +- `developerName` is None +- `category` is None +- `capabilities` is empty +- `websiteUrl` is None +- `privacyPolicyUrl` is None +- `termsOfServiceUrl` is None +- `defaultPrompt` is None (after resolution) +- `brandColor` is None +- `composerIcon` is None +- `logo` is None +- `screenshots` is empty + +### Default Prompt Validation + +The `defaultPrompt` field supports three input shapes: + +1. **Single string**: `"defaultPrompt": "Summarize my inbox"` +2. **Array of strings**: `"defaultPrompt": ["Prompt 1", "Prompt 2", "Prompt 3"]` +3. **Invalid type**: Any other JSON type is rejected with a warning + +**Constants**: +- `MAX_DEFAULT_PROMPT_COUNT`: 3 (maximum number of prompts) +- `MAX_DEFAULT_PROMPT_LEN`: 128 (maximum characters per prompt after normalization) + +**Processing rules**: +1. Whitespace is normalized: `split_whitespace().collect::>().join(" ")` +2. Leading/trailing whitespace is stripped (via the split_whitespace normalization) +3. Empty prompts (after normalization) are rejected: "prompt must not be empty" +4. Prompts exceeding 128 characters (by `chars().count()`) are rejected +5. Non-string entries in an array (e.g., numbers, objects) are skipped with a warning +6. Only the first 3 valid entries are kept; excess entries trigger a warning +7. If a single string input is valid, it becomes a one-element array +8. If all entries are invalid, the field resolves to `None` + +**Example of normalization behavior** (from tests): +```json +// Input: +"defaultPrompt": [ + " Summarize my inbox ", // -> "Summarize my inbox" (valid) + 123, // -> skipped (not a string) + "xxx...129chars", // -> skipped (too long) + " ", // -> skipped (empty after normalization) + "Draft the reply ", // -> "Draft the reply" (valid) + "Find my next action", // -> valid + "Archive old mail" // -> skipped (max 3 reached) +] +// Output: ["Summarize my inbox", "Draft the reply", "Find my next action"] +``` + +### Interface Asset Paths (composerIcon, logo, screenshots) + +These use the same `resolve_manifest_path()` validation as component paths: +- Must start with `./` +- No `..` traversal +- Must stay within plugin root +- Resolved to absolute paths +- Invalid paths are silently ignored (the field becomes `None` or is omitted from the screenshots array) + +**TypeScript protocol type** (after resolution): +```typescript +composerIcon: AbsolutePathBuf | null; +logo: AbsolutePathBuf | null; +screenshots: Array; // never null, may be empty +``` + +## Plugin Identity System + +### PluginId Format + +Plugins are identified by a compound key: `{plugin_name}@{marketplace_name}` + +```rust +pub struct PluginId { + pub plugin_name: String, + pub marketplace_name: String, +} +``` + +**Parsing**: The `@` separator is split on the rightmost occurrence. Both segments must pass `validate_plugin_segment()`. + +**Segment validation** (`validate_plugin_segment()`): +- Cannot be empty +- Only ASCII alphanumeric characters (`a-z`, `A-Z`, `0-9`), hyphens (`-`), and underscores (`_`) +- No other characters (no dots, slashes, spaces, etc.) + +**Error format**: `"invalid plugin key 'sample'; expected @"` + +### Plugin Namespace for Skills + +Skills within a plugin are namespaced with the plugin name: +- Pattern: `{plugin_name}:{skill_name}` +- Example: `sample:sample-search` for skill `sample-search` in plugin `sample` + +The namespace is resolved by walking up directory ancestors from the skill file path until a `.codex-plugin/plugin.json` is found, then extracting the `name` field. + +## Plugin Discovery and Loading + +### Discovery Locations + +Plugins are discovered from: + +1. **User config** (`~/.codex/config.toml`): Plugin entries with explicit paths +2. **Repo marketplace** (`$REPO_ROOT/.agents/plugins/marketplace.json`) +3. **Home marketplace** (`~/.agents/plugins/marketplace.json`) +4. **OpenAI curated marketplace**: Synced via git (preferred) or HTTP fallback + +**Important**: Project-level `.codex/config.toml` plugin settings are ignored. Only user-level config is processed. + +### Plugin Cache Structure + +Installed plugins are cached at: +``` +~/.codex/plugins/cache/{marketplace_name}/{plugin_name}/{version}/ +``` + +- **Local plugins**: Version is `"local"` +- **Curated plugins**: Version is the git SHA +- **Version priority**: `"local"` always preferred; otherwise lexicographically last + +### Plugin Loading Details + +A loaded plugin (`LoadedPlugin`) contains: +- `config_name`: The key used in config (e.g., `sample@marketplace`) +- `manifest_name`: Name from plugin.json (may differ from config_name's plugin segment) +- `manifest_description`: Raw description from plugin.json +- `root`: Absolute path to plugin root +- `enabled`: Whether the plugin is active +- `skill_roots`: Directories containing skills +- `disabled_skill_paths`: Skills explicitly disabled by config +- `has_enabled_skills`: Whether any skills remain enabled +- `mcp_servers`: MCP server configurations loaded from the plugin +- `apps`: App connector IDs +- `error`: Optional error message if loading partially failed + +A plugin is considered **active** only if `enabled == true` AND `error.is_none()`. + +## Marketplace System + +### Marketplace Manifest Format + +Located at `.agents/plugins/marketplace.json`: + +```json +{ + "name": "marketplace-name", + "interface": { + "displayName": "Human-Readable Name" + }, + "plugins": [ + { + "name": "plugin-name", + "source": { + "source": "local", + "path": "./plugins/plugin-name" + }, + "policy": { + "installation": "AVAILABLE", + "authentication": "ON_INSTALL", + "products": ["CODEX", "CHATGPT", "ATLAS"] + }, + "category": "Productivity" + } + ] +} +``` + +### Marketplace Plugin Policy + +**Installation policy** (`PluginInstallPolicy`): +- `"NOT_AVAILABLE"` - Plugin cannot be installed +- `"AVAILABLE"` - Plugin can be installed by user (default) +- `"INSTALLED_BY_DEFAULT"` - Plugin is pre-installed + +**Authentication policy** (`PluginAuthPolicy`): +- `"ON_INSTALL"` - Auth required during installation (default) +- `"ON_USE"` - Auth deferred until first use + +**Product filtering** (`products` array): +- If absent/missing: Plugin available to ALL products (permissive default) +- If present but empty `[]`: Plugin available to NO products (explicit denial) +- If present with values: Plugin only available to listed products +- Known products: `"CODEX"`, `"CHATGPT"`, `"ATLAS"` + +### Marketplace Path Validation + +Source paths in marketplace.json follow the same rules as plugin manifest paths: +- Must start with `./` +- No `..` traversal +- Resolved relative to marketplace root (NOT `.agents/plugins/`) + +### Legacy Field Handling + +Legacy top-level `installPolicy` and `authPolicy` fields on plugin entries are silently ignored. Only the nested `policy` object is used. + +### Marketplace Discovery + +Marketplace files are found at `.agents/plugins/marketplace.json` relative to: +1. Codex home directory (`~/.codex/` or equivalent) +2. Git repository roots discovered from provided working directories +3. The OpenAI curated repository (`openai-curated`) + +When multiple marketplace roots point to the same git repository, only one marketplace entry is created (deduplication). + +### Duplicate Plugin Handling + +When the same plugin name appears multiple times within a single marketplace, the **first entry wins**. Later duplicates are silently ignored. + +When the same plugin name appears across different marketplaces, both entries are preserved but resolution prioritizes the repository marketplace over the home marketplace. + +## Plugin Installation + +### Installation Flow + +1. Validate source directory exists +2. Read manifest, extract plugin name +3. Confirm manifest name matches marketplace plugin name +4. Atomically copy files to cache using staging directory +5. Update config.toml with plugin entry +6. Set `enabled = true` automatically + +### Atomic Installation + +Installation uses a staging pattern: +1. Create temporary staging directory +2. Copy plugin files to staging +3. Back up existing version (if any) +4. Atomic rename of staging to final path +5. Rollback on failure (restore backup) + +### Uninstallation + +1. Remove plugin from cache directory +2. Remove plugin entry from config.toml +3. Idempotent - calling twice succeeds + +## Plugin Injection into Agent + +When plugins are mentioned (via `@plugin-name`), the system: + +1. Collects mentioned plugins from user input +2. Filters MCP servers belonging to those plugins +3. Locates enabled app connectors for those plugins +4. Renders plugin instructions as `DeveloperInstructions` +5. Injects as `ResponseItem` objects into the conversation + +### Instruction Rendering + +Plugin instructions are rendered as markdown sections: +- General `## Plugins` section listing all available plugins +- Per-plugin sections listing capabilities (skills, MCP servers, apps) +- Skills use `plugin_name:` prefix convention +- Wrapped in XML tags (`PLUGINS_INSTRUCTIONS_OPEN_TAG`/`CLOSE_TAG`) + +## Plugin Feature Gating + +The entire plugin system is behind a feature flag: +- Set in config: `[features] plugins = true` +- When disabled: All plugin operations return empty/default results +- Feature check is the first gate in most plugin operations + +## Startup Synchronization + +On CLI startup: +1. Sync OpenAI curated plugins repository (git preferred, HTTP fallback) +2. Git sync: `ls-remote` to check SHA, shallow clone (depth=1) if outdated +3. HTTP fallback: GitHub API for branch info, download zipball +4. Validate marketplace manifest after extraction +5. Remote plugin sync (additive only): reconcile local state with remote +6. Marker file prevents duplicate startup syncs +7. Wait timeout: 5 seconds for prerequisites; 30 seconds for git/HTTP operations + +### SHA-based Caching + +- SHA file stored at `.tmp/plugins.sha` +- If local SHA matches remote, sync is skipped +- Temporary directories auto-cleaned after 10 minutes + +## Known Limitations and Edge Cases + +### Hooks Not Supported in Plugins + +Despite documentation suggesting `hooks.json` as a plugin companion surface, **the runtime does NOT execute plugin-local hooks**. Only global `~/.codex/hooks.json` is loaded. The manifest parser does not process a `hooks` field. (GitHub issue #16430) + +### Startup Sync Directory Leaks + +On sync failure, `~/.codex/.tmp/plugins-clone-*` directories may be orphaned. (GitHub issue #16004) + +### Empty Interface Omission + +If the `interface` section is present in JSON but all fields are default/empty, the parsed manifest will have `interface: None`. This means validating the presence of interface fields must account for this normalization. + +### Name vs Directory Name Fallback + +When `name` is empty or whitespace, the plugin directory name becomes the plugin name. This means two different manifest states can produce the same effective name. + +### Path Normalization Differences + +The `resolve_manifest_path()` function normalizes paths by iterating components, meaning `./foo/../bar` would be rejected (contains `..`) even though it mathematically resolves within the plugin root. This is intentional for security. + +### camelCase Field Naming + +The JSON uses camelCase (`mcpServers`, `displayName`, etc.) via serde's `rename_all = "camelCase"`. The raw deserialization struct maps directly to JSON field names. + +## TypeScript Protocol Types + +The app-server-protocol defines these types for the plugin system: + +```typescript +// Plugin summary in marketplace listings +type PluginSummary = { + id: string; + name: string; + source: PluginSource; + installed: boolean; + enabled: boolean; + installPolicy: PluginInstallPolicy; + authPolicy: PluginAuthPolicy; + interface: PluginInterface | null; +}; + +// Plugin source (currently only local) +type PluginSource = { type: "local"; path: AbsolutePathBuf }; + +// Plugin detail (full view) +type PluginDetail = { + marketplaceName: string; + marketplacePath: AbsolutePathBuf; + summary: PluginSummary; + description: string | null; + skills: Array; + apps: Array; + mcpServers: Array; +}; + +// Marketplace entry +type PluginMarketplaceEntry = { + name: string; + path: AbsolutePathBuf; + interface: MarketplaceInterface | null; + plugins: Array; +}; + +// Policies +type PluginInstallPolicy = "NOT_AVAILABLE" | "AVAILABLE" | "INSTALLED_BY_DEFAULT"; +type PluginAuthPolicy = "ON_INSTALL" | "ON_USE"; +``` + +## Validation Rules Summary (for Linter Implementation) + +### Manifest-Level Rules + +| Rule | Severity | Description | +|------|----------|-------------| +| `name` required | Error | Must be non-empty (or directory name fallback) | +| `name` characters | Error | ASCII alphanumeric, hyphens, underscores only | +| `name` no path separators | Error | No `/`, `\`, or `..` | +| `name` kebab-case | Warning | Convention, not enforced by parser | +| JSON parse valid | Error | Must be valid JSON | +| Unknown fields | Info | Serde `default` allows extra fields silently | + +### Path Rules (skills, mcpServers, apps, asset paths) + +| Rule | Severity | Description | +|------|----------|-------------| +| Must start with `./` | Warning | Invalid paths silently ignored | +| Must not be just `./` | Warning | Empty relative path rejected | +| No `..` traversal | Warning | Path must not contain parent dir | +| Stay within plugin root | Warning | Only Normal path components | +| Must resolve to absolute | Warning | Plugin root must be absolute | +| Empty string = absent | Info | Empty strings treated as None | + +### Interface Rules + +| Rule | Severity | Description | +|------|----------|-------------| +| All-empty interface | Info | Entire interface is omitted | +| `defaultPrompt` max 3 | Warning | Excess entries ignored | +| `defaultPrompt` max 128 chars | Warning | Over-length entries ignored | +| `defaultPrompt` non-empty | Warning | Empty-after-normalization rejected | +| `defaultPrompt` type check | Warning | Non-string/array types rejected | +| `defaultPrompt` entry type | Warning | Non-string entries in array skipped | +| Asset paths same as component paths | Warning | Same `./` and traversal rules | +| `capabilities` array required | Info | Defaults to empty array | +| `screenshots` array required | Info | Defaults to empty array | + +### Marketplace Rules + +Note: severity differs from Path Rules above. Plugin component paths (skills, mcpServers, apps) silently ignore invalid entries (Warning - plugin still installs). Marketplace source paths must be valid for install to proceed (Error - installation blocked). + +| Rule | Severity | Description | +|------|----------|-------------| +| Plugin source path `./` prefix | Error | Required for local sources | +| No `..` in source paths | Error | Traversal prevention | +| `policy.installation` valid enum | Warning | Must be known value | +| `policy.authentication` valid enum | Warning | Must be known value | +| Empty `products: []` blocks all | Warning | Likely unintentional | +| Missing `products` allows all | Info | Permissive default | +| Duplicate plugin names | Warning | First entry wins | +| Legacy policy fields ignored | Info | `installPolicy`/`authPolicy` top-level | +| Manifest name matches plugin name | Error | Must match during installation | + +### PluginId Rules + +| Rule | Severity | Description | +|------|----------|-------------| +| Format `name@marketplace` | Error | Required separator | +| Both segments non-empty | Error | No empty plugin or marketplace name | +| Segment characters | Error | ASCII alphanumeric, hyphens, underscores | + +## Common Pitfalls + +| Pitfall | Why It Happens | How to Avoid | +|---------|---------------|--------------| +| Paths without `./` prefix silently ignored | Developers use bare relative paths like `skills/` | Always prefix with `./` | +| `..` in paths silently rejected | Trying to reference files outside plugin root | Keep all files within plugin directory | +| Hooks in plugin.json not executed | Documentation suggests it works, runtime ignores it | Use global `~/.codex/hooks.json` instead | +| Empty `products: []` blocks everything | Confusion between missing and empty arrays | Omit `products` field to allow all products | +| Default prompts over 128 chars silently dropped | No clear error at authoring time | Keep prompts concise, check char count | +| Plugin name mismatch blocks installation | Manifest name must match marketplace plugin name | Keep names synchronized | +| Project config plugin settings ignored | Only user-level config is processed | Configure plugins in `~/.codex/config.toml` | +| Legacy policy fields have no effect | Old `installPolicy`/`authPolicy` at top level | Use nested `policy` object | +| Interface with all nulls becomes None | All-empty interface is normalized away | Ensure at least one interface field is set | + +## Best Practices + +1. **Use kebab-case names** - Convention followed by all official plugins (Source: developers.openai.com/codex/plugins/build) +2. **Always prefix paths with `./`** - The only accepted relative path format (Source: manifest.rs `resolve_manifest_path`) +3. **Keep assets in `./assets/`** - Organized convention for logos, icons, screenshots (Source: official build docs) +4. **Include `displayName` in interface** - Prioritized for rendering plugin labels (Source: PR #15606) +5. **Keep descriptions under 1024 chars** - Capability summary truncates at this limit (Source: manager_tests.rs) +6. **Keep default prompts under 128 chars** - Hard limit enforced during parsing (Source: manifest.rs constants) +7. **Limit default prompts to 3** - Hard maximum, excess silently dropped (Source: manifest.rs constants) +8. **Set explicit `policy.products`** - Avoid accidentally blocking or allowing unwanted products (Source: marketplace_tests.rs) +9. **Test with `codex /plugins`** - Verify plugin appears in the directory after installation (Source: official plugin docs) +10. **Restart Codex after changes** - Plugin cache requires restart to pick up modifications (Source: official build docs) + +## Further Reading + +| Resource | Type | Why Recommended | +|----------|------|-----------------| +| [manifest.rs](https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/manifest.rs) | Source | Definitive validation rules | +| [Build Plugins](https://developers.openai.com/codex/plugins/build) | Docs | Official development guide | +| [Plugin Overview](https://developers.openai.com/codex/plugins) | Docs | User-facing plugin documentation | +| [marketplace.rs](https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/marketplace.rs) | Source | Marketplace discovery and validation | +| [manager.rs](https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/manager.rs) | Source | Plugin lifecycle management | +| [store.rs](https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/store.rs) | Source | Cache storage and installation | +| [plugin_id.rs](https://github.com/openai/codex/blob/main/codex-rs/plugin/src/plugin_id.rs) | Source | PluginId parsing and validation | +| [v0.117.0 Release](https://github.com/openai/codex/releases) | Release | Plugin system introduction | +| [PR #14993](https://github.com/openai/codex/pull/14993) | PR | Product-aware policies design | +| [Issue #16430](https://github.com/openai/codex/issues/16430) | Issue | Plugin hooks limitation | + +--- + +*This guide was synthesized from 42 sources including Rust source code, TypeScript protocol definitions, official OpenAI documentation, GitHub PRs, and issues. See `resources/codex-plugin-manifest-sources.json` for full source list with quality scores.* diff --git a/git-map-spec.md b/git-map-spec.md deleted file mode 100644 index da24eb5..0000000 --- a/git-map-spec.md +++ /dev/null @@ -1,750 +0,0 @@ -# git-map: Static Git History Analysis Artifact - -> Specification for a cached, incrementally-updatable JSON artifact that extracts temporal, social, and behavioral signals from git history. Companion to repo-map (AST-based static analysis). - -**Status**: Design spec -**Date**: 2026-03-14 -**Research**: Synthesized from 91 sources across 3 learning guides - -## 1. Concept - -git-map creates a static JSON artifact from git history analysis. It sits alongside repo-map in agent-core: - -``` -repo-map -> WHAT exists (symbols, exports, imports) [static snapshot] -git-map -> WHAT HAPPENED (activity, ownership, trends) [temporal layer] -``` - -Combined, they give consumers a complete picture without re-running expensive analysis. - -### Design principles - -1. **Static, not live** - Pre-computed aggregates stored as JSON, not on-demand queries -2. **Incremental** - Git history is append-only; only process new commits since last scan -3. **No LLM** - Pure deterministic extraction and aggregation -4. **Query layer** - Consumers call functions over the cached map, not git commands -5. **AI-aware** - Detect and track AI-generated commits as a first-class concern - -### Validation model - -``` -HEAD === map.git.analyzedUpTo -> valid, no work needed -HEAD !== map.git.analyzedUpTo -> run delta extraction, merge into map -analyzedUpTo not in history -> history rewritten (force push), full rebuild -``` - -## 2. Architecture - -### Where it lives - -``` -agent-core/lib/ - git-map/ - index.js # init, update, status, load (public API) - extractor.js # runs git commands, returns raw delta - aggregator.js # merges delta into existing map - queries.js # pure functions over the cached map - cache.js # load/save git-map.json (reuses repo-map cache pattern) - ai-detect.js # AI commit detection logic - ai-signatures.json # updateable AI tool signature registry - collectors/ - index.js # add 'git' option to collector registry - git.js # thin wrapper: load map + run queries -> structured output -``` - -### Lifecycle - -``` -git-map init full scan - processes all history - | - v - git-map.json stored in .claude/ (platform-aware state dir) - | - v -git-map status HEAD === analyzedUpTo? -> valid / stale - | - v -git-map update process analyzedUpTo..HEAD only, merge into map -``` - -### Relationship to existing infrastructure - -| Component | Role | -|-----------|------| -| `agent-core/lib/git-map/` | Core library (extractor, aggregator, queries) | -| `agent-core/lib/collectors/git.js` | Thin consumer - loads map, runs queries, returns structured output | -| `agent-core/lib/collectors/index.js` | Registry - adds `'git'` as a collector option | -| `agent-core/lib/repo-map/` | Sibling - provides AST/symbol data that git-map complements | -| `agent-core/lib/collectors/github.js` | Sibling - provides GitHub API data (issues, PRs, milestones) | - -Consumers use it via the collector pattern: - -```js -const { collectors } = require('@agentsys/lib'); -const data = collectors.collect({ - collectors: ['github', 'docs', 'code', 'git'] // just add 'git' -}); -``` - -## 3. Data Model (Static Map Schema) - -```json -{ - "version": "1.0", - "generated": "2026-03-14T10:00:00Z", - "updated": "2026-03-14T12:00:00Z", - "partial": false, - - "git": { - "analyzedUpTo": "abc123def456", - "totalCommitsAnalyzed": 1847, - "firstCommitDate": "2024-01-15", - "lastCommitDate": "2026-03-13", - "scope": null, - "shallow": false - }, - - "contributors": { - "humans": { - "alice": { - "commits": 1180, - "firstSeen": "2024-01-15", - "lastSeen": "2026-03-13", - "aiAssistedCommits": 180 - }, - "bob": { - "commits": 390, - "firstSeen": "2024-03-20", - "lastSeen": "2026-03-10", - "aiAssistedCommits": 45 - } - }, - "bots": { - "dependabot[bot]": { "commits": 230, "firstSeen": "2024-02-01", "lastSeen": "2026-03-13" } - } - }, - - "fileActivity": { - "src/core/engine.rs": { - "changes": 47, - "authors": ["alice", "bob"], - "created": "2024-01-15", - "lastChanged": "2026-03-12", - "additions": 1200, - "deletions": 800, - "aiChanges": 12, - "aiAdditions": 480, - "aiDeletions": 320, - "bugFixChanges": 8 - } - }, - - "dirActivity": { - "src/core/": { - "changes": 120, - "authors": ["alice", "bob", "carol"], - "aiChanges": 28 - }, - "src/tools/": { - "changes": 85, - "authors": ["bob"], - "aiChanges": 40 - } - }, - - "coupling": { - "src/core/engine.rs": { - "src/core/engine_test.rs": { "cochanges": 38, "humanCochanges": 30, "aiCochanges": 8 }, - "src/config.rs": { "cochanges": 12, "humanCochanges": 10, "aiCochanges": 2 } - } - }, - - "commitShape": { - "sizeDistribution": { "tiny": 340, "small": 280, "medium": 150, "large": 60, "huge": 17 }, - "filesPerCommit": { "median": 3, "p90": 8 }, - "mergeCommits": 120 - }, - - "conventions": { - "prefixes": { "feat": 340, "fix": 280, "chore": 150, "docs": 77 }, - "style": "conventional", - "usesScopes": true, - "samples": ["feat(tools): add file search", "fix: handle empty input"] - }, - - "aiAttribution": { - "attributed": 180, - "heuristic": 45, - "none": 775, - "tools": { - "claude": 120, - "aider": 35, - "cursor": 25, - "unknown": 45 - } - }, - - "releases": { - "tags": [ - { "tag": "v0.5.0", "commit": "abc123", "date": "2026-03-01", "commitsSince": 12 } - ], - "cadence": "monthly" - }, - - "renames": [ - { "from": "src/util.rs", "to": "src/helpers.rs", "commit": "def456", "date": "2026-01-20" } - ], - - "deletions": [ - { "path": "src/old_parser.rs", "commit": "789abc", "date": "2026-02-15" } - ] -} -``` - -### What's stored vs derived - -**Stored (incrementally updatable with simple arithmetic):** - -| Field | Update operation | -|-------|-----------------| -| contributors | `count += delta`, update lastSeen | -| fileActivity per file | `changes += delta`, update lastChanged, union authors | -| dirActivity per dir | `changes += delta`, union authors | -| coupling pairs | `cochanges += delta` (prune pairs below threshold of 3) | -| commitShape.sizeDistribution | `bucket[size] += 1` per new commit | -| conventions.prefixes | `prefix[type] += delta` | -| aiAttribution | `attributed += delta`, update tool counts | -| releases.tags | append new tags | -| renames, deletions | append new entries | -| bugFixChanges per file | `count += delta` (commit message starts with `fix`) | - -**Derived at query time (pure JS over cached data, zero git commands):** - -| Query | Computation | -|-------|-------------| -| busFactor | Sort humans by commits, find N covering 80% | -| hotspots | Filter fileActivity by lastChanged > cutoff, sort by changes | -| coldspots | Filter fileActivity by lastChanged < cutoff | -| ownership per dir | Max contributor from dirActivity.authors + fileActivity | -| coupling strength | cochanges / totalChanges for file pair | -| recentContributors | Filter contributors by lastSeen > cutoff | -| isActive | lastCommitDate > 30 days ago | -| unreleasedWork | Commits since last tag | -| aiRatio per file | aiChanges / changes | -| aiRatio repo-wide | sum(aiChanges) / sum(changes) | -| commitFrequency | totalCommits / time span | -| adjustedBusFactor | Bus factor weighted by aiAssistedCommits ratio | - -## 4. Extractor - -The extractor runs git commands and returns a raw delta. On init, it processes full history. On update, only `analyzedUpTo..HEAD`. - -### Git commands (the only 4 needed) - -```js -function extractDelta(basePath, range) { - return { - // Commits with authors, dates, messages, and trailers - commits: execGit([ - 'log', range, - '--format=%H|%aN|%aE|%aI|%s%n%b%n---TRAILER---%n%(trailers:key=Co-authored-by,valueonly)%n---END---', - '--no-merges' - ]), - - // Files changed per commit with additions/deletions - files: execGit([ - 'log', range, - '--format=%H', - '--numstat', - '--no-merges' - ]), - - // Renames - renames: execGit([ - 'log', range, - '--diff-filter=R', '--format=%H|%aI', '-M', - '--name-status' - ]), - - // Deletions - deletions: execGit([ - 'log', range, - '--diff-filter=D', '--format=%H|%aI', - '--name-only' - ]) - }; -} -``` - -### AI detection (runs per-commit during extraction) - -Detection is string matching on data already being parsed. No extra git commands. - -```js -function detectAI(commit) { - const signals = { detected: false, tool: null, method: null }; - - // 1. Trailer check (highest confidence) - for (const trailer of commit.trailers) { - if (AI_SIGNATURES.trailerEmails.some(e => trailer.includes(e))) { - signals.detected = true; - signals.tool = identifyTool(trailer); - signals.method = 'trailer'; - return signals; - } - } - - // 2. Author email check - if (AI_SIGNATURES.authorEmails.some(e => commit.authorEmail.includes(e))) { - signals.detected = true; - signals.tool = identifyToolFromEmail(commit.authorEmail); - signals.method = 'author'; - return signals; - } - - // 3. Author name pattern check - if (AI_SIGNATURES.authorPatterns.some(p => new RegExp(p).test(commit.authorName))) { - signals.detected = true; - signals.tool = identifyToolFromAuthor(commit.authorName); - signals.method = 'author-pattern'; - return signals; - } - - // 4. Branch prefix check - if (commit.branch && AI_SIGNATURES.branchPrefixes.some(p => commit.branch.startsWith(p))) { - signals.detected = true; - signals.tool = identifyToolFromBranch(commit.branch); - signals.method = 'branch'; - return signals; - } - - // 5. Message body check - if (AI_SIGNATURES.messagePatterns.some(p => new RegExp(p).test(commit.body))) { - signals.detected = true; - signals.tool = identifyToolFromMessage(commit.body); - signals.method = 'message'; - return signals; - } - - return signals; -} -``` - -### AI Signature Registry (ai-signatures.json) - -Updateable data file - no code changes needed when tools change attribution: - -```json -{ - "version": "1.0", - "updated": "2026-03-14", - "trailerEmails": { - "noreply@anthropic.com": "claude", - "noreply@aider.chat": "aider", - "cursoragent@cursor.com": "cursor", - "agent@cursor.com": "cursor", - "bot@devin.ai": "devin" - }, - "authorEmails": { - "no-reply@replit.com": "replit" - }, - "authorPatterns": { - "\\(aider\\)$": "aider", - "\\[bot\\]$": "bot" - }, - "branchPrefixes": { - "copilot/": "copilot", - "cursor/": "cursor" - }, - "messagePatterns": { - "Generated with Claude Code": "claude", - "^aider: ": "aider" - }, - "trailerNames": { - "Claude": "claude", - "Cursor": "cursor", - "Copilot": "copilot", - "GitHub Copilot": "copilot", - "Codex": "codex", - "Jules": "jules", - "Devin": "devin", - "aider": "aider" - }, - "botAuthors": { - "dependabot[bot]": "dependabot", - "renovate[bot]": "renovate", - "github-actions[bot]": "github-actions", - "devin-ai-integration[bot]": "devin" - } -} -``` - -### Detection coverage (honest assessment) - -| Tool | Detectable | Method | Default state | -|------|-----------|--------|---------------| -| Claude Code | Yes | Trailer + body text | On by default, can be disabled | -| Aider | Yes | Author suffix + trailer | On by default, can be disabled | -| Cursor Agent | Yes | Trailer + branch prefix | On, hard to disable | -| Copilot Coding Agent | Yes | Author + branch prefix | On, hardcoded | -| Devin | Yes | Bot author | Always present | -| Codex CLI | Yes | Trailer | Recently added, configurable | -| Replit Agent | Yes | Author email | Always present | -| Copilot inline | No | No trace | Fundamentally undetectable | -| Windsurf/Codeium | No | No trace | No attribution | -| JetBrains AI | No | No trace | No attribution | -| Tabnine | No | No trace | No attribution | -| Cline/Continue | No | No trace | No attribution | -| Amazon Q | No | No trace | No attribution | - -`aiRatio` in the map is a **lower bound**. Industry data shows 41-46% of code is AI-generated, but only autonomous agents and some pair tools leave traces. Inline completions are invisible. - -### Noise filtering - -Excluded from coupling and hotspot analysis: - -```js -const NOISE_FILES = [ - /package-lock\.json$/, - /yarn\.lock$/, - /Cargo\.lock$/, - /go\.sum$/, - /pnpm-lock\.yaml$/, - /\.min\.(js|css)$/, - /dist\//, - /build\//, - /vendor\// -]; -``` - -Merge commits excluded via `--no-merges` flag to prevent double-counting. - -## 5. Aggregator - -Merges a delta into the existing map. Pure arithmetic - no git commands. - -```js -function mergeInto(existingMap, delta) { - const map = structuredClone(existingMap); - - for (const commit of delta.commits) { - // Update contributor counts - const isBot = AI_SIGNATURES.botAuthors[commit.authorName]; - const aiSignals = detectAI(commit); - const bucket = isBot ? 'bots' : 'humans'; - - if (!map.contributors[bucket][commit.authorName]) { - map.contributors[bucket][commit.authorName] = { - commits: 0, firstSeen: commit.date, lastSeen: commit.date - }; - if (bucket === 'humans') { - map.contributors.humans[commit.authorName].aiAssistedCommits = 0; - } - } - map.contributors[bucket][commit.authorName].commits++; - map.contributors[bucket][commit.authorName].lastSeen = commit.date; - - if (bucket === 'humans' && aiSignals.detected) { - map.contributors.humans[commit.authorName].aiAssistedCommits++; - } - - // Update AI attribution counts - if (aiSignals.detected) { - map.aiAttribution.attributed++; - const tool = aiSignals.tool || 'unknown'; - map.aiAttribution.tools[tool] = (map.aiAttribution.tools[tool] || 0) + 1; - } else { - map.aiAttribution.none++; - } - - // Update commit shape - const size = classifyCommitSize(commit.additions + commit.deletions); - map.commitShape.sizeDistribution[size]++; - - // Update conventions - const prefix = extractConventionalPrefix(commit.subject); - if (prefix) map.conventions.prefixes[prefix] = (map.conventions.prefixes[prefix] || 0) + 1; - - // Update per-file activity - for (const file of commit.files) { - if (isNoise(file.path)) continue; - - if (!map.fileActivity[file.path]) { - map.fileActivity[file.path] = { - changes: 0, authors: [], created: commit.date, - lastChanged: commit.date, additions: 0, deletions: 0, - aiChanges: 0, aiAdditions: 0, aiDeletions: 0, bugFixChanges: 0 - }; - } - const f = map.fileActivity[file.path]; - f.changes++; - f.lastChanged = commit.date; - f.additions += file.additions; - f.deletions += file.deletions; - if (!f.authors.includes(commit.authorName)) f.authors.push(commit.authorName); - - if (aiSignals.detected) { - f.aiChanges++; - f.aiAdditions += file.additions; - f.aiDeletions += file.deletions; - } - if (prefix === 'fix') f.bugFixChanges++; - } - - // Update coupling (co-occurrence within same commit) - const filePaths = commit.files.map(f => f.path).filter(p => !isNoise(p)); - for (let i = 0; i < filePaths.length; i++) { - for (let j = i + 1; j < filePaths.length; j++) { - const a = filePaths[i] < filePaths[j] ? filePaths[i] : filePaths[j]; - const b = filePaths[i] < filePaths[j] ? filePaths[j] : filePaths[i]; - if (!map.coupling[a]) map.coupling[a] = {}; - if (!map.coupling[a][b]) map.coupling[a][b] = { cochanges: 0, humanCochanges: 0, aiCochanges: 0 }; - map.coupling[a][b].cochanges++; - if (aiSignals.detected) { - map.coupling[a][b].aiCochanges++; - } else { - map.coupling[a][b].humanCochanges++; - } - } - } - - // Aggregate dir activity - for (const file of commit.files) { - const dir = dirname(file.path) + '/'; - if (!map.dirActivity[dir]) { - map.dirActivity[dir] = { changes: 0, authors: [], aiChanges: 0 }; - } - map.dirActivity[dir].changes++; - if (!map.dirActivity[dir].authors.includes(commit.authorName)) { - map.dirActivity[dir].authors.push(commit.authorName); - } - if (aiSignals.detected) map.dirActivity[dir].aiChanges++; - } - } - - // Merge renames and deletions - map.renames.push(...delta.renames); - map.deletions.push(...delta.deletions); - - // Prune low-signal coupling (below threshold of 3 cochanges) - for (const [fileA, pairs] of Object.entries(map.coupling)) { - for (const [fileB, counts] of Object.entries(pairs)) { - if (counts.cochanges < 3) delete pairs[fileB]; - } - if (Object.keys(pairs).length === 0) delete map.coupling[fileA]; - } - - // Update git metadata - map.git.analyzedUpTo = delta.head; - map.git.totalCommitsAnalyzed += delta.commits.length; - map.git.lastCommitDate = delta.commits[delta.commits.length - 1]?.date || map.git.lastCommitDate; - - map.updated = new Date().toISOString(); - return map; -} -``` - -## 6. Query API - -Pure functions over the cached map. No git commands. No LLM. - -```js -const gitMap = require('@agentsys/lib').gitMap; -const map = gitMap.load(basePath); - -// --- Health --- -gitMap.isActive(map) -// -> bool (commits in last 30 days) - -gitMap.getHealth(map) -// -> { active, busFactor, commitFrequency, age, aiRatio } - -gitMap.getBusFactor(map, { adjustForAI: false }) -// -> number (humans covering 80% of commits) -// adjustForAI: true weights by aiAssistedCommits ratio - -// --- Contributors --- -gitMap.getContributors(map, { months: 6 }) -// -> [{ name, commits, pct, aiAssistedPct }] filtered by lastSeen - -gitMap.getOwnership(map, 'src/core/') -// -> { primary: 'alice', pct: 78, contributors: [...], aiRatio: 0.23 } - -// --- Files --- -gitMap.getHotspots(map, { months: 3, limit: 10 }) -// -> [{ path, changes, authors, aiRatio, bugFixes }] - -gitMap.getColdspots(map, { months: 6 }) -// -> [{ path, lastChanged, daysSince }] - -gitMap.getFileHistory(map, 'src/engine.rs') -// -> { changes, authors, created, lastChanged, aiRatio, bugFixes, ... } - -gitMap.getCoupling(map, 'src/engine.rs', { humanOnly: false }) -// -> [{ path, strength, cochanges, humanCochanges }] -// humanOnly: true excludes AI session correlation - -// --- AI --- -gitMap.getAIRatio(map) -// -> { ratio: 0.18, attributed: 180, total: 1000, tools: {...} } -// NOTE: this is a lower bound - -gitMap.getAIRatio(map, 'src/tools/') -// -> per-directory AI ratio - -gitMap.getAIHotspots(map, { limit: 10 }) -// -> files with highest AI ratio AND high churn (risk signal) - -// --- Conventions --- -gitMap.getConventions(map) -// -> { style, prefixes, samples, aiAttribution } - -gitMap.getCommitShape(map) -// -> { typicalSize, filesPerCommit, mergeCommitRatio } - -// --- Releases --- -gitMap.getReleaseInfo(map) -// -> { cadence, lastRelease, unreleased, tags: [...] } - -// --- Changes --- -gitMap.getRecentDeletions(map, { months: 3 }) -// -> [{ path, date, commit }] - -gitMap.getRecentRenames(map, { months: 3 }) -// -> [{ from, to, date, commit }] -``` - -## 7. Consumers - -### Who uses what - -| Consumer | Fields consumed | Benefit | -|----------|----------------|---------| -| `/onboard` | Everything | Build mental model of unfamiliar codebase | -| `/can-i-help` | contributors, conventions, ownership, hotspots, releases | Guide OSS contributions | -| drift-detect | hotspots, churn, deletions, renames, ownership | Detect plan vs reality gaps | -| sync-docs | hotspots, deletions, renames, coupling (docs <-> code) | Find stale documentation | -| enhance | conventions, aiAttribution | Validate project patterns | -| next-task | hotspots, ownership, aiRatio | Assign work to right areas | -| perf | hotspots, churn, coupling | Identify optimization targets | -| code review | coupling, ownership, aiRatio, bugFixes | PR context and risk signals | - -### Example: drift-detect integration - -```js -const { collectors } = require('@agentsys/lib'); -const data = collectors.collect({ - collectors: ['github', 'docs', 'code', 'git'] -}); - -// drift-detect now gets git temporal data alongside GitHub API data -// "Module X has lost its primary author and churn increased 40%" -// "src/legacy/ hasn't been touched in 275 days but docs still reference it" -``` - -### Example: /onboard output - -``` -Project: opencode (Go TUI) -Active: Yes (daily commits) -Bus factor: 2 (alice: 65%, bob: 22%) -AI ratio: 18% detected (lower bound) -Convention: conventional commits (87% conformance) -Release cadence: monthly (last: v0.5.0, 12 unreleased commits) - -Hot areas (last 3 months): - src/core/ 120 changes, 3 authors, 23% AI - src/tools/ 85 changes, 1 author, 47% AI [WARN: single owner + high AI] - -Cold areas (untouched 6+ months): - src/legacy/ last modified 2025-06-12 - -Key couplings: - engine.rs <-> engine_test.rs (38 co-changes, 79% human) - engine.rs <-> config.rs (12 co-changes, 83% human) -``` - -## 8. Edge Cases - -| Edge case | Detection | Handling | -|-----------|-----------|----------| -| Shallow clone | `git rev-parse --is-shallow-repository` | Mark `partial: true`, analyze available history | -| Force push | `git cat-file -t analyzedUpTo` fails | Full rebuild | -| Monorepo | User passes `scope: 'packages/lib/'` | Filter commits to scope path | -| Merge commits | N/A | Excluded via `--no-merges` | -| Lockfiles/generated | Regex patterns | Excluded from coupling and hotspot analysis | -| Bot commits | Author name in botAuthors registry | Tracked separately, excluded from human metrics | -| AI attribution disabled | No trailer/metadata | Falls into `none` category; aiRatio is lower bound | - -## 9. Performance - -### Init (full scan) - -For a repo with 5000 commits: 4 git commands, each producing text output. Parsing and aggregation is O(commits * files_per_commit). Expected: < 10 seconds for most repos. - -### Update (incremental) - -For 50 new commits since last scan: same 4 git commands scoped to `analyzedUpTo..HEAD`. Expected: < 1 second. - -### Map size - -Estimated JSON size for a 5000-commit repo with 500 tracked files: -- fileActivity: ~50KB (100 bytes per file) -- coupling: ~20KB (sparse, threshold at 3) -- contributors: ~5KB -- Other: ~5KB -- Total: ~80KB compressed - -### Caching - -Cached at `{stateDir}/git-map.json` using the same platform-aware state directory as repo-map: -- Claude Code: `.claude/git-map.json` -- OpenCode: `config/.opencode/git-map.json` -- Codex: `.codex/git-map.json` - -## 10. Implementation Plan - -### Phase 1: Core (MVP) - -- [ ] `extractor.js` - 4 git commands, parse output -- [ ] `ai-detect.js` + `ai-signatures.json` - explicit signal detection -- [ ] `aggregator.js` - merge delta into map -- [ ] `cache.js` - load/save/status (reuse repo-map pattern) -- [ ] `index.js` - init/update/status/load API -- [ ] `collectors/git.js` - thin wrapper for collector registry -- [ ] Tests for all modules - -### Phase 2: Query Layer - -- [ ] `queries.js` - all query functions -- [ ] Integration with collector registry in `collectors/index.js` -- [ ] Tests for query edge cases (empty map, single contributor, no tags) - -### Phase 3: Consumers - -- [ ] `/onboard` command using git-map + repo-map -- [ ] `/can-i-help` command using git-map + GitHub collector -- [ ] drift-detect integration -- [ ] sync-docs integration - -### Phase 4: Enhancement - -- [ ] Heuristic AI detection (commit message patterns vs repo baseline) -- [ ] Agent Trace integration (if adopted by ecosystem) -- [ ] Deeper coupling analysis (cross-directory coupling highlighting) - -## 11. Research Foundation - -This spec is informed by 91 sources across 3 research guides: - -| Guide | Sources | Key contribution | -|-------|---------|-----------------| -| `git-history-analysis-developer-tools.md` | 24 | Core analysis techniques: hotspots, coupling, ownership, churn, CodeScene/GitClear/Hercules patterns | -| `ai-agent-commits-git-analysis-impact.md` | 25 | AI impact on metrics: 41-46% AI code, churn inflation, ownership breakdown, metric adjustments | -| `ai-commit-detection-forensics.md` | 42 | Detection methods: per-tool signatures, composite scoring, 97.2% F1-score fingerprinting study | - -Key academic references: -- "Fingerprinting AI Coding Agents on GitHub" (arXiv:2601.17406) - 97.2% F1, 41 behavioral features -- "Your Code as a Crime Scene" (Adam Tornhill) - behavioral code analysis paradigm -- GitClear 2025 research - 211M lines, AI churn impact data -- Agent Trace specification (agent-trace.dev) - emerging attribution standard diff --git a/rag-skill-indexing.md b/rag-skill-indexing.md new file mode 100644 index 0000000..094ce4f --- /dev/null +++ b/rag-skill-indexing.md @@ -0,0 +1,539 @@ +# Learning Guide: RAG Indexes and Knowledge Retrieval for AI Agent Skill Files + +**Generated**: 2026-03-31 +**Sources**: 18 resources analyzed +**Depth**: deep + +## Prerequisites + +- Familiarity with SKILL.md format and Agent Skills open standard +- Basic understanding of how AI coding agents (Claude Code, Cursor, Copilot) load context +- Experience writing CLAUDE.md or similar project instruction files + +## TL;DR + +- Agent skill files are **not traditional RAG** - they use **progressive disclosure** (metadata -> instructions -> resources) rather than embedding-based retrieval +- The **description field** is the single most important element - it is the gatekeeper that determines whether a skill activates +- **XML structured references outperform plain markdown tables** for LLM parsing when documents are injected into context, but **markdown tables are better as human-readable routing indexes** that the model reads from disk +- **Semantic routing triggers in descriptions** beat direct file links for document selection - the model decides based on natural language matching, not path-based lookup +- For the actual skill files that agents read: keep SKILL.md **under 500 lines / 5,000 tokens**, use **one-level-deep file references**, and put a **table of contents** at the top of any reference file over 100 lines +- The state of the art for knowledge retrieval in AI coding tools is **not embedding-based RAG** - it is **filesystem-based progressive disclosure** with **description-driven activation** + +## Core Insight: Skills Are Not Traditional RAG + +Traditional RAG systems chunk documents, embed them into vectors, retrieve relevant chunks via similarity search, and inject them into the LLM context. AI agent skill files work differently. + +The Agent Skills architecture uses a **three-tier progressive disclosure** model: + +| Tier | What Loads | When | Token Cost | +|------|-----------|------|------------| +| Metadata | name + description only | Session startup | ~50-100 tokens per skill | +| Instructions | Full SKILL.md body | When skill is activated | <5,000 tokens recommended | +| Resources | Referenced files, scripts, data | When instructions reference them | Varies (effectively unlimited) | + +This means the retrieval problem is really a **routing problem**: given a user's request, which skill should activate? The "index" is the collection of descriptions, not an embedding database. + +**Key implication**: Optimizing skill-based knowledge retrieval means optimizing descriptions, file structure, and reference patterns - not building vector indexes. + +## Question 1: XML Structured References vs Plain Markdown Tables + +### For Context Injection (API / System Prompt) + +When documents are programmatically injected into the LLM context (system prompt, tool results, API messages), **XML tags outperform plain markdown**. + +Anthropic's official guidance recommends wrapping documents in structured XML: + +```xml + + + api-reference.md + + {{CONTENT}} + + + +``` + +Why XML wins here: +- Claude parses XML tags **unambiguously** - no edge cases with pipe characters or alignment +- Nested tags create **natural hierarchy** (documents > document > content) +- Tags enable **ground-then-answer** patterns: "Find quotes from the documents first, then answer" +- The model can distinguish injected documents from instructions and conversation reliably + +### For Skill Routing Files (SKILL.md Body) + +When the model reads a SKILL.md file from the filesystem and needs to decide which reference documents to load next, **markdown tables and structured lists work well**. The model reads these as structured text and follows links. + +Example routing pattern that works: + +```markdown +## Available References + +**API patterns**: See [reference/api-patterns.md](reference/api-patterns.md) +**Data modeling**: See [reference/data-modeling.md](reference/data-modeling.md) +**Performance**: See [reference/performance.md](reference/performance.md) + +## Quick Search + +Find specific topics using grep: +grep -i "caching" reference/api-patterns.md +grep -i "indexing" reference/data-modeling.md +``` + +This works because the model is **reading a file and making a decision**, not parsing injected context. The links serve as a table of contents, not as a structured data format. + +### Recommendation + +Use **XML wrapping when injecting documents programmatically** into context. Use **markdown with clear section headers and file references** in SKILL.md routing files that the model reads from disk. Do not use XML inside SKILL.md files - they are markdown files read by both humans and models. + +## Question 2: Direct File Links vs Semantic Routing Triggers + +### How Skill Activation Actually Works + +Agent tools use **description-based matching**, not file path matching. The activation flow: + +1. At startup, the agent loads all skill **descriptions** into context (~50-100 tokens each) +2. When a user message arrives, the model reads all descriptions and decides which skill is relevant +3. The model loads the full SKILL.md of the matched skill +4. SKILL.md references point to deeper resources that the model loads on demand + +The description is the **only thing** that determines whether a skill activates. Direct file links play no role in the initial routing decision. + +### Semantic Routing Triggers Win + +A description like this activates reliably: + +```yaml +description: Build applications with Valkey, choose commands and data types, + implement caching, sessions, queuing, locking, rate-limiting, leaderboard, + counter, and search patterns. Use for streams, pub-sub, scripting, + transactions, performance optimization, and migration from Redis. +``` + +It works because it contains **domain-specific vocabulary** that matches what developers actually say when they need this knowledge. The model performs semantic matching against these terms. + +A file-path-based approach would not work: + +```yaml +# This does NOT work for activation +description: Reference files in reference/valkey-patterns/ +``` + +### Within SKILL.md: File References as Routing + +Once a skill is activated and the model reads SKILL.md, file references serve as **second-level routing**. The model reads the SKILL.md body, understands the user's specific need, and selectively loads only the relevant reference file. + +The pattern is: + +```markdown +## Caching Patterns +For TTL strategies, eviction policies, and cache-aside patterns, +see [reference/caching.md](reference/caching.md) + +## Data Structures +For choosing between Strings, Hashes, Sorted Sets, and Streams, +see [reference/data-structures.md](reference/data-structures.md) +``` + +The model reads this as a router: "The user asked about caching, so I'll load reference/caching.md." This is **semantic routing with file links as destinations** - the link itself is not the trigger, the surrounding description is. + +### Recommendation + +Use **semantic keywords in descriptions** for skill activation (tier 1 routing). Use **descriptive text + file links** in SKILL.md for second-level routing (tier 2). Never rely on file paths alone for routing - always pair with natural language context describing when to load each reference. + +## Question 3: Embedding-Based vs Keyword vs Structured Metadata Retrieval + +### State of the Art for General RAG + +For general-purpose RAG (searching over large document collections), the state of the art is **hybrid retrieval**: combining dense embeddings with sparse keyword matching (BM25), followed by a reranking step. + +Anthropic's Contextual Retrieval research shows: + +| Approach | Retrieval Failure Reduction | +|----------|---------------------------| +| Embeddings alone | Baseline | +| Contextual Embeddings | 35% fewer failures | +| Contextual Embeddings + BM25 | 49% fewer failures | +| + Reranking | 67% fewer failures | + +The key insight: **embedding-based and keyword-based retrieval are complementary**. Embeddings capture semantic similarity ("caching" matches "temporary storage"). BM25 captures exact terminology ("XADD" matches "XADD"). Combining both with a reranker gives the best results. + +### State of the Art for AI Agent Skills + +AI agent skill files do **not use embedding-based retrieval**. The retrieval mechanism is: + +1. **Description matching** (tier 1): The LLM reads all descriptions and decides based on natural language understanding. This is more powerful than embedding similarity because the LLM understands intent, not just semantic proximity. + +2. **Filesystem navigation** (tier 2): The model reads SKILL.md, sees references, and makes file-read tool calls. This is deterministic navigation, not retrieval. + +3. **Grep/search** (tier 3): For large reference files, the model can use grep to find specific content. This is keyword matching executed by the model. + +Some tools add embedding-based features on top: +- **Cursor** indexes the codebase with embeddings for `@codebase` semantic search +- **VS Code Copilot** has workspace indexing for `@workspace` queries +- **Claude Code** relies primarily on filesystem tools (Glob, Grep, Read) rather than embeddings + +### Structured Metadata + +Structured metadata in skill files improves routing precision: + +```yaml +--- +name: valkey-caching +description: Caching patterns with Valkey... +paths: + - "src/**/*.ts" + - "lib/cache/**" +metadata: + domain: database + technology: valkey +--- +``` + +The `paths` field enables **conditional activation** - the skill only loads when the model works with matching files. This is a form of metadata-based filtering that reduces false positives. + +### Recommendation + +For skill files, invest in **description quality** and **structured SKILL.md routing** over embedding-based retrieval. The LLM's own comprehension of descriptions is the primary retrieval mechanism. For very large knowledge bases (100+ reference documents), consider adding grep-based search hints in SKILL.md. Embedding-based retrieval is most useful at the tool level (Cursor codebase indexing, Copilot workspace search) rather than at the skill routing level. + +## Question 4: How AI Coding Tools Discover and Load Skill/Knowledge Files + +### Claude Code + +**Discovery**: Scans directories for SKILL.md files at startup. + +| Scope | Path | +|-------|------| +| Personal | `~/.claude/skills//SKILL.md` | +| Project | `.claude/skills//SKILL.md` | +| Plugin | `/skills//SKILL.md` | +| Enterprise | Managed settings | +| Nested | Subdirectory `.claude/skills/` auto-discovered | + +**Context loading**: +- Metadata (name + description) injected into system prompt at startup +- Full SKILL.md read via bash tool when triggered +- Referenced files read on demand via bash +- CLAUDE.md files loaded hierarchically from directory tree +- `.claude/rules/*.md` files loaded unconditionally or conditionally (with `paths` frontmatter) + +**Description budget**: 1% of context window (default). Descriptions truncated at 250 chars in listings. Configurable via `SLASH_COMMAND_TOOL_CHAR_BUDGET`. + +### Cursor + +**Discovery**: Scans `.cursor/rules/` for MDC (Markdown Configuration) files and `.agents/skills/` for Agent Skills. + +**Context loading**: +- `alwaysApply: true` rules loaded unconditionally +- `globs: "pattern"` rules loaded when working with matching files +- Codebase indexed with embeddings for `@codebase` semantic queries +- AGENTS.md and legacy `.cursorrules` supported for backward compatibility +- Agent Skills loaded via standard progressive disclosure + +### GitHub Copilot / VS Code + +**Discovery**: Scans `.github/copilot-instructions.md`, `.agents/skills/`, and `.github/skills/`. + +**Context loading**: +- Custom instructions always included in context +- Agent Skills follow the standard progressive disclosure pattern +- `@workspace` triggers embedding-based codebase search +- VS Code extensions can provide additional context via language server protocol + +### OpenAI Codex + +**Discovery**: Scans `.agents/skills/` and `.codex/skills/` directories. + +**Context loading**: Standard progressive disclosure. Full SKILL.md loaded on activation. + +### Cross-Tool Standard + +The `.agents/skills/` directory is the emerging cross-client convention. Skills placed there are discoverable by all compliant tools. Over 30 tools now support the Agent Skills open standard, including Gemini CLI, JetBrains Junie, OpenHands, Roo Code, Goose, Amp, Databricks, and Snowflake. + +### Recommendation + +For maximum cross-tool compatibility: +1. Place skills in `.agents/skills//SKILL.md` for cross-client discovery +2. Also place in `.claude/skills//SKILL.md` for Claude Code native support +3. Keep descriptions under 250 characters (front-load the key use case) +4. Use `paths` frontmatter for conditional activation when appropriate +5. Test with multiple tools to verify description matching works across different activation engines + +## Question 5: Optimal File Size and Structure for RAG Chunks + +### For Embedding-Based RAG Systems + +When building traditional RAG indexes over knowledge documents: + +| Parameter | Recommendation | Rationale | +|-----------|---------------|-----------| +| Chunk size | 256-512 tokens | Sweet spot for most embedding models | +| Overlap | 10-20% of chunk size | Preserves boundary context | +| Chunking strategy | Markdown-aware / semantic | Respects document structure | +| Context enrichment | Prepend parent section headers | Addresses the "lost context" problem | + +Anthropic's Contextual Retrieval adds chunk-level context by having a model prepend a brief explanation of where each chunk fits in the document. This reduces retrieval failures by 35% over plain chunking. + +For hierarchical knowledge (like the RAPTOR approach), multi-level summaries allow retrieval at different granularities - specific chunks for detailed questions, summary nodes for broad questions. + +### For Agent Skill Files (Not Embedding-Based) + +Skill files are read directly by the model, not chunked and embedded. The optimization targets are different: + +| File Type | Size Target | Rationale | +|-----------|------------|-----------| +| SKILL.md body | Under 500 lines / ~5,000 tokens | Fits in context alongside conversation without crowding | +| Reference files | Under 300 lines each | Model can read entire file in one tool call | +| Large references | Include table of contents at top | Model may use `head -100` to preview, TOC ensures it sees full scope | +| Total skill directory | No practical limit | Files only load on demand | + +### Structure for Optimal Model Consumption + +Models parse structured documents better when they follow these patterns: + +**1. Table of contents for files over 100 lines** + +```markdown +# API Reference + +## Contents +- Authentication (line 15) +- Core Methods: GET, SET, DEL (line 45) +- Advanced: Transactions, Scripting (line 120) +- Error Handling (line 180) +``` + +**2. Section headers as natural chunk boundaries** + +```markdown +## Caching Patterns + +### Cache-Aside (Lazy Loading) +[focused content for this pattern] + +### Write-Through +[focused content for this pattern] +``` + +**3. One concept per reference file** + +Instead of a 2000-line monolith reference, split into focused files: + +``` +reference/ + caching-patterns.md (150 lines) + data-structure-choice.md (200 lines) + performance-tuning.md (180 lines) + migration-from-redis.md (120 lines) +``` + +The model loads only the file relevant to the user's question. Four 150-line files are better than one 600-line file because the model only pays the context cost for the portion it needs. + +### Recommendation + +For SKILL.md routing files: stay under 500 lines, use progressive disclosure to deeper references. For reference documents: keep each under 300 lines with clear section headers. For very large knowledge bases: split by domain/topic into focused files, with SKILL.md serving as the router. Always include a table of contents for files over 100 lines. + +## Question 6: Designing Trigger Phrases for Maximum Recall Without False Positives + +### How Trigger Matching Works + +In AI agent skill systems, the model reads all skill descriptions at startup and uses its own language understanding to match user requests to skills. This is not keyword matching - it is semantic comprehension by the LLM. + +The description is the **single most important field** for correct activation. Writing it well is the highest-leverage optimization for skill retrieval. + +### Principles for High-Recall Descriptions + +**1. Include both "what" and "when"** + +```yaml +# Good - states both capability and trigger context +description: Build applications with Valkey, choose data types and commands. + Use when implementing caching, sessions, queuing, locking, rate-limiting, + leaderboards, or counters with Valkey or Redis. + +# Bad - only states capability +description: Valkey application patterns. +``` + +**2. Front-load the primary use case** + +Descriptions may be truncated at 250 characters. Put the most distinctive trigger phrase first: + +```yaml +# Good - primary trigger is first +description: Debug and profile Valkey performance bottlenecks. Use for + slow queries, memory analysis, latency spikes, or connection pool tuning. + +# Bad - generic preamble wastes the first 250 chars +description: A comprehensive skill that provides detailed guidance for + various aspects of working with Valkey performance issues... +``` + +**3. Use domain-specific vocabulary, not generic terms** + +The model matches on semantic meaning, but domain-specific terms reduce ambiguity: + +```yaml +# Good - specific vocabulary +description: Implement pub-sub messaging, Valkey Streams consumers, + XADD/XREAD patterns, consumer groups, and event-driven architectures. + +# Bad - too generic +description: Handle messaging and events with a database. +``` + +**4. Include common synonyms and alternative phrasings** + +Users express the same intent in different ways: + +```yaml +description: Deploy Valkey clusters, configure replication and failover, + set up Sentinel for high availability. Use when scaling, deploying, + or operating Valkey in production. Covers Docker, Kubernetes, and + bare-metal setups. +``` + +This catches "deploy valkey", "scale valkey", "valkey kubernetes", "valkey sentinel", "valkey HA", and "valkey production" as triggers. + +**5. Exclude explicitly to prevent false positives** + +If a skill is frequently confused with another, use the description or SKILL.md instructions to clarify boundaries: + +```yaml +description: Valkey client library usage with GLIDE SDK. Use when writing + application code that connects to Valkey. Does NOT cover server + administration, deployment, or internals. +``` + +### Trigger Phrase Design for CLAUDE.md Knowledge Indexes + +For CLAUDE.md files that act as routing indexes (like the `agent-knowledge/CLAUDE.md` pattern), trigger phrases serve a different purpose. They are loaded into context and help the model match user questions to knowledge files. + +The pattern: + +```markdown +## Trigger Phrases + +Use this knowledge when user asks about: +- "RAG chunking strategy" -> rag-skill-indexing.md +- "skill description quality" -> rag-skill-indexing.md +- "progressive disclosure" -> rag-skill-indexing.md +- "context window optimization" -> rag-skill-indexing.md +``` + +For these in-context triggers: +- Use **exact phrases users would type**, not academic descriptions +- Include **both short queries and longer questions** +- Add **tool-specific terms** ("CLAUDE.md", "SKILL.md", "cursor rules") +- Keep the total trigger section **under 30 entries per guide** to avoid context bloat + +### Recommendation + +Invest heavily in description quality - it has the highest impact on retrieval precision. Front-load the primary use case within 250 characters. Include domain-specific vocabulary and common synonyms. Use "Use when..." phrasing to explicitly state activation triggers. For CLAUDE.md routing indexes, map exact user phrases to knowledge files. + +## Synthesis: The Optimal SKILL.md Knowledge Architecture + +Combining all findings, here is the recommended architecture for AI agent skill files that route to reference knowledge: + +### Directory Structure + +``` +skill-name/ + SKILL.md # Router: <500 lines, progressive disclosure + reference/ + topic-a.md # Focused reference: <300 lines each + topic-b.md # One concept per file + topic-c.md # Include TOC if >100 lines + scripts/ + helper.py # Executable code (runs, not loaded into context) + examples/ + pattern-1.md # Example files for specific patterns +``` + +### SKILL.md Template + +```yaml +--- +name: domain-skill +description: , + , . Covers .> +--- + +# Domain Skill + +## Quick Start + +[Most common operation in <10 lines - gets the user productive immediately] + +## Reference Topics + +**Topic A**: [2-3 sentence description of what's covered] +See [reference/topic-a.md](reference/topic-a.md) + +**Topic B**: [2-3 sentence description of what's covered] +See [reference/topic-b.md](reference/topic-b.md) + +**Topic C**: [2-3 sentence description of what's covered] +See [reference/topic-c.md](reference/topic-c.md) + +## Quick Search + +Find specific items: +grep -i "keyword" reference/topic-a.md +grep -i "keyword" reference/topic-b.md +``` + +### Reference File Template + +```markdown +# Topic Title + +## Contents +- Section 1: [brief description] +- Section 2: [brief description] +- Section 3: [brief description] + +## Section 1 + +[Focused content, practical examples, code samples] + +## Section 2 + +[Focused content, practical examples, code samples] +``` + +### Key Principles + +1. **Description is everything** - Invest 80% of optimization effort in the description field +2. **Progressive disclosure** - Only load what's needed, when it's needed +3. **One concept per file** - Reference files should be independently useful +4. **Table of contents** - Every file over 100 lines gets a TOC at the top +5. **Grep-friendly** - Include search hints for large reference sections +6. **One-level deep** - SKILL.md references files; files do not reference other files +7. **Under 500 lines** - SKILL.md body stays concise; deep content goes in reference files +8. **Front-load triggers** - Primary use case in the first 250 characters of description +9. **Test across models** - Haiku needs more guidance, Opus needs less explanation +10. **Iterate from observation** - Watch how the model navigates your skill, adjust based on actual behavior + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Fix | +|-------------|---------|-----| +| Giant monolith SKILL.md (1000+ lines) | Crowds context window, reduces adherence | Split into SKILL.md router + reference files | +| Deeply nested references (3+ levels) | Model may use `head -100` and miss content | Keep references one level deep from SKILL.md | +| Vague descriptions ("helps with data") | Model cannot match to user intent | Include specific vocabulary and trigger phrases | +| Embedding-based retrieval for <50 docs | Over-engineering; filesystem navigation is faster | Use description-based routing with file references | +| XML tags inside SKILL.md body | SKILL.md is markdown read by humans and models | Use XML only for programmatic context injection | +| No table of contents in long files | Model may partially read and miss important sections | Add TOC at top of any file over 100 lines | +| Duplicate content across reference files | Wastes context when multiple files loaded | Single source of truth per concept | +| Description longer than 250 chars without front-loading | Key triggers truncated in skill listings | Put the most distinctive trigger phrase first | + +## Further Reading + +- Anthropic Skill Authoring Best Practices: https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices +- Agent Skills Open Standard: https://agentskills.io/specification +- Claude Code Skills Documentation: https://code.claude.com/docs/en/skills +- Claude Code Memory System: https://code.claude.com/docs/en/memory +- Anthropic Context Engineering: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents +- Anthropic Contextual Retrieval: https://www.anthropic.com/research/contextual-retrieval +- Anthropic Prompting Best Practices: https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices diff --git a/resources/codex-plugin-manifest-sources.json b/resources/codex-plugin-manifest-sources.json new file mode 100644 index 0000000..3979f5b --- /dev/null +++ b/resources/codex-plugin-manifest-sources.json @@ -0,0 +1,688 @@ +{ + "topic": "Codex CLI Plugin Manifest System", + "slug": "codex-plugin-manifest", + "generated": "2026-04-01T00:00:00Z", + "depth": "deep", + "totalSources": 42, + "sources": [ + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/manifest.rs", + "title": "manifest.rs - Plugin manifest parsing and validation (full source)", + "qualityScore": 100, + "authority": 10, + "recency": 10, + "depth": 10, + "examples": 10, + "uniqueness": 10, + "keyInsights": [ + "RawPluginManifest uses serde rename_all camelCase", + "Path fields are top-level strings, not nested under paths object", + "resolve_manifest_path requires ./ prefix, blocks .. traversal", + "Empty name falls back to directory name", + "MAX_DEFAULT_PROMPT_COUNT=3, MAX_DEFAULT_PROMPT_LEN=128", + "Interface omitted if all fields are empty/default", + "Asset paths use same validation as component paths", + "Whitespace normalization via split_whitespace().join(\" \")" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/manager.rs", + "title": "manager.rs - Plugin lifecycle management", + "qualityScore": 95, + "authority": 10, + "recency": 10, + "depth": 10, + "examples": 8, + "uniqueness": 9, + "keyInsights": [ + "PluginsManager orchestrates plugin lifecycle with caching", + "TTL-based caching for featured plugin IDs", + "Product-based access restrictions at admission time", + "MCP server deduplication across plugins", + "Skill path resolution with config-based disabling" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/marketplace.rs", + "title": "marketplace.rs - Marketplace discovery and plugin resolution", + "qualityScore": 95, + "authority": 10, + "recency": 10, + "depth": 10, + "examples": 8, + "uniqueness": 9, + "keyInsights": [ + "MarketplacePluginPolicy with installation and auth policies", + "Products array: missing=allow all, empty []=deny all", + "Local source paths require ./ prefix", + "Legacy installPolicy/authPolicy fields silently ignored", + "Marketplace discovery from home dir and git repo roots", + "First duplicate plugin entry wins" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/store.rs", + "title": "store.rs - Plugin storage and installation", + "qualityScore": 90, + "authority": 10, + "recency": 10, + "depth": 9, + "examples": 7, + "uniqueness": 8, + "keyInsights": [ + "Cache structure: plugins/cache/{marketplace}/{plugin}/{version}/", + "Atomic installation with staging directory and rollback", + "local version preferred over SHA versions", + "Manifest name must match marketplace plugin name", + "Path separator rejection in plugin/marketplace names" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/plugin/src/plugin_id.rs", + "title": "plugin_id.rs - PluginId parsing and segment validation", + "qualityScore": 92, + "authority": 10, + "recency": 10, + "depth": 9, + "examples": 7, + "uniqueness": 9, + "keyInsights": [ + "PluginId format: plugin_name@marketplace_name", + "Split on rightmost @ separator", + "validate_plugin_segment: non-empty, ASCII alphanumeric + hyphens + underscores", + "Both segments must pass validation" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/plugin/src/lib.rs", + "title": "codex_plugin crate - Plugin types and capability summary", + "qualityScore": 88, + "authority": 10, + "recency": 10, + "depth": 8, + "examples": 6, + "uniqueness": 8, + "keyInsights": [ + "PluginCapabilitySummary with config_name, display_name, has_skills, mcp_server_names, app_connector_ids", + "AppConnectorId is a newtype wrapper over String", + "PluginTelemetryMetadata for analytics" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/plugin/src/load_outcome.rs", + "title": "load_outcome.rs - PluginLoadOutcome and LoadedPlugin structs", + "qualityScore": 90, + "authority": 10, + "recency": 10, + "depth": 9, + "examples": 7, + "uniqueness": 8, + "keyInsights": [ + "LoadedPlugin tracks config_name, manifest_name, enabled, error, skill_roots, disabled_skill_paths", + "Plugin is_active requires enabled=true AND error=None", + "effective_skill_roots, effective_mcp_servers, effective_apps filter by active state", + "EffectiveSkillRoots trait for plugin outcome" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/plugin/src/plugin_namespace.rs", + "title": "plugin_namespace.rs - Skill namespace resolution", + "qualityScore": 85, + "authority": 10, + "recency": 10, + "depth": 8, + "examples": 7, + "uniqueness": 7, + "keyInsights": [ + "Walks up directory ancestors to find .codex-plugin/plugin.json", + "Extracts name field or falls back to directory name", + "Skills are namespaced as plugin_name:skill_name" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/discoverable.rs", + "title": "discoverable.rs - Plugin discovery for tool suggestions", + "qualityScore": 82, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 6, + "uniqueness": 7, + "keyInsights": [ + "Allowlist of 8 curated plugins: GitHub, Notion, Slack, Gmail, Google Calendar, Google Drive, Linear, Figma", + "Excludes already installed plugins from discovery", + "Sorted alphabetically by name then by ID" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/injection.rs", + "title": "injection.rs - Plugin injection into agent context", + "qualityScore": 80, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 5, + "uniqueness": 7, + "keyInsights": [ + "Builds DeveloperInstructions from mentioned plugins", + "Filters MCP servers by plugin display name", + "Locates enabled connectors matching plugin display name", + "Uses BTreeSet for sorted, deduplicated results" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/mentions.rs", + "title": "mentions.rs - Plugin mention collection from user input", + "qualityScore": 78, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 5, + "uniqueness": 6, + "keyInsights": [ + "Plugin mentions use @ sigil, tool mentions use $ sigil", + "Extracts from both structured Mention objects and linked text", + "App IDs extracted from app:// URI scheme", + "Plugin IDs extracted from plugin:// URI scheme" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/remote.rs", + "title": "remote.rs - Remote plugin status fetching", + "qualityScore": 75, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 5, + "uniqueness": 6, + "keyInsights": [ + "Requires ChatGPT authentication (not API key)", + "30-second timeout for requests", + "Featured plugins endpoint uses platform parameter", + "Enable/disable via POST to /plugins/{id}/{action}" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/render.rs", + "title": "render.rs - Plugin instruction rendering", + "qualityScore": 78, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 5, + "uniqueness": 6, + "keyInsights": [ + "Renders ## Plugins section with available plugin list", + "Per-plugin instructions list MCP servers and apps", + "Skills use plugin_name: prefix convention", + "Wrapped in XML tags for structured parsing" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/startup_sync.rs", + "title": "startup_sync.rs - Startup plugin synchronization", + "qualityScore": 80, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 5, + "uniqueness": 7, + "keyInsights": [ + "Git-first sync with HTTP fallback", + "Shallow clone depth=1 for efficiency", + "SHA-based caching in .tmp/plugins.sha", + "5-second prerequisite wait, 30-second operation timeout", + "Atomic rename with rollback capability", + "Marker file prevents duplicate syncs" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/toggles.rs", + "title": "toggles.rs - Plugin enabled/disabled state tracking", + "qualityScore": 72, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 4, + "uniqueness": 6, + "keyInsights": [ + "Three config write patterns: direct property, object, bulk", + "Later writes override earlier ones for same plugin", + "BTreeMap for deterministic ordering" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/mod.rs", + "title": "mod.rs - Plugin module structure and public API", + "qualityScore": 75, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 3, + "uniqueness": 7, + "keyInsights": [ + "11 internal modules: discoverable, injection, manager, manifest, marketplace, mentions, remote, render, startup_sync, store, toggles", + "Re-exports from codex_plugin: AppConnectorId, PluginId, PluginCapabilitySummary", + "OPENAI_CURATED_MARKETPLACE_NAME constant" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/test_support.rs", + "title": "test_support.rs - Plugin test fixtures and helpers", + "qualityScore": 82, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 8, + "uniqueness": 6, + "keyInsights": [ + "Standard test plugin has .codex-plugin/plugin.json, skills/SKILL.md, .mcp.json, .app.json", + "Test SHA constant for version validation", + "Marketplace fixture at .agents/plugins/marketplace.json", + "ConfigBuilder for loading test configurations" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/marketplace_tests.rs", + "title": "marketplace_tests.rs - 16 marketplace validation tests", + "qualityScore": 92, + "authority": 10, + "recency": 10, + "depth": 9, + "examples": 10, + "uniqueness": 8, + "keyInsights": [ + "Path traversal rejection for source paths", + "Display name parsing from interface", + "Error resilience for invalid marketplaces", + "Asset paths without ./ prefix silently excluded", + "Explicit empty products blocks all products", + "First duplicate entry wins", + "Product-based access control enforcement" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/store_tests.rs", + "title": "store_tests.rs - 11 store validation tests", + "qualityScore": 88, + "authority": 10, + "recency": 10, + "depth": 8, + "examples": 9, + "uniqueness": 7, + "keyInsights": [ + "Manifest name used for destination, not source dir name", + "local version preferred over hash versions", + "Path separator rejection in key segments", + "Manifest name must match marketplace plugin name" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/manager_tests.rs", + "title": "manager_tests.rs - 36 comprehensive plugin management tests", + "qualityScore": 95, + "authority": 10, + "recency": 10, + "depth": 10, + "examples": 10, + "uniqueness": 9, + "keyInsights": [ + "Custom component paths require ./ prefix", + "Description sanitization to single line", + "Description truncation at 1024 chars", + "Disabled plugins preserve metadata but contribute nothing", + "Plugin key format: sample@marketplace", + "Project config plugin settings ignored", + "Feature flag gates all plugin operations", + "Additive-only sync preserves existing plugins", + "App connector deduplication across plugins" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/render_tests.rs", + "title": "render_tests.rs - Plugin rendering tests", + "qualityScore": 72, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 6, + "uniqueness": 5, + "keyInsights": [ + "Empty plugins returns None", + "Wrapped in plugins_instructions XML tags", + "Skills prefixed with plugin_name:" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/mentions_tests.rs", + "title": "mentions_tests.rs - Plugin mention handling tests", + "qualityScore": 75, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 7, + "uniqueness": 6, + "keyInsights": [ + "Plugin mentions via plugin:// URI scheme", + "App mentions via app:// URI scheme", + "Deduplication of structured and linked mentions", + "Dollar-prefixed mentions excluded from plugin matches" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/discoverable_tests.rs", + "title": "discoverable_tests.rs - Plugin discovery tests", + "qualityScore": 72, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 6, + "uniqueness": 5, + "keyInsights": [ + "Description normalization in discovery results", + "Installed plugins omitted from discovery", + "Feature disabled returns empty" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/plugins/startup_sync_tests.rs", + "title": "startup_sync_tests.rs - Startup sync tests", + "qualityScore": 78, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 6, + "uniqueness": 6, + "keyInsights": [ + "SHA validation and comparison", + "Stale directory cleanup after 10 minutes", + "Git-first with HTTP fallback", + "Atomic activation with rollback" + ] + }, + { + "url": "https://developers.openai.com/codex/plugins", + "title": "OpenAI - Codex Plugins Overview (Official Docs)", + "qualityScore": 90, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 6, + "uniqueness": 9, + "keyInsights": [ + "Three component types: skills, apps, MCP servers", + "Install via /plugins command or Codex App directory", + "@prefix for explicit plugin mentions", + "Existing approval settings still apply after installation", + "Disable via enabled = false in config.toml", + "Apps may prompt for auth on setup or first use" + ] + }, + { + "url": "https://developers.openai.com/codex/plugins/build", + "title": "OpenAI - Build Codex Plugins (Official Docs)", + "qualityScore": 95, + "authority": 10, + "recency": 10, + "depth": 9, + "examples": 9, + "uniqueness": 10, + "keyInsights": [ + "$plugin-creator skill for scaffolding", + "Only plugin.json belongs in .codex-plugin/", + "Name must be stable kebab-case", + "Minimal manifest: name, version, description, skills", + "Complete manifest includes interface section for marketplace", + "Marketplace at .agents/plugins/marketplace.json", + "Source path resolves relative to marketplace location", + "Cache at ~/.codex/plugins/cache/$MARKETPLACE/$PLUGIN/local/", + "Restart required after changes" + ] + }, + { + "url": "https://github.com/openai/codex/releases", + "title": "Codex CLI Releases - v0.117.0 Plugin Introduction", + "qualityScore": 85, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 5, + "uniqueness": 8, + "keyInsights": [ + "v0.117.0 (March 26, 2026) introduced plugins as first-class workflow", + "Startup syncing of product-scoped plugins", + "/plugins command for discovery", + "MCP servers auto-install with plugin/install", + "Skill disable by name", + "Product-specific plugin filtering hardened" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginInterface.ts", + "title": "PluginInterface TypeScript type definition", + "qualityScore": 82, + "authority": 10, + "recency": 10, + "depth": 7, + "examples": 5, + "uniqueness": 7, + "keyInsights": [ + "All string fields nullable except capabilities and screenshots arrays", + "defaultPrompt capped at 3 entries, 128 chars each", + "AbsolutePathBuf for resolved asset paths" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginSummary.ts", + "title": "PluginSummary TypeScript type definition", + "qualityScore": 78, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 5, + "uniqueness": 6, + "keyInsights": [ + "Plugin summary includes id, name, source, installed, enabled, policies, interface" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginDetail.ts", + "title": "PluginDetail TypeScript type definition", + "qualityScore": 78, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 5, + "uniqueness": 6, + "keyInsights": [ + "Full plugin detail includes skills array, apps array, mcpServers array" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginMarketplaceEntry.ts", + "title": "PluginMarketplaceEntry TypeScript type definition", + "qualityScore": 72, + "authority": 10, + "recency": 10, + "depth": 5, + "examples": 4, + "uniqueness": 5, + "keyInsights": [ + "Marketplace entry has name, path, interface, plugins array" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginSource.ts", + "title": "PluginSource TypeScript type definition", + "qualityScore": 68, + "authority": 10, + "recency": 10, + "depth": 5, + "examples": 3, + "uniqueness": 5, + "keyInsights": [ + "Only local source type currently: { type: 'local', path: AbsolutePathBuf }" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginAuthPolicy.ts", + "title": "PluginAuthPolicy TypeScript type definition", + "qualityScore": 65, + "authority": 10, + "recency": 10, + "depth": 4, + "examples": 3, + "uniqueness": 5, + "keyInsights": [ + "Two values: ON_INSTALL or ON_USE" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginInstallPolicy.ts", + "title": "PluginInstallPolicy TypeScript type definition", + "qualityScore": 65, + "authority": 10, + "recency": 10, + "depth": 4, + "examples": 3, + "uniqueness": 5, + "keyInsights": [ + "Three values: NOT_AVAILABLE, AVAILABLE, INSTALLED_BY_DEFAULT" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginInstallParams.ts", + "title": "PluginInstallParams TypeScript type definition", + "qualityScore": 62, + "authority": 10, + "recency": 10, + "depth": 4, + "examples": 3, + "uniqueness": 4, + "keyInsights": [ + "Install requires marketplacePath and pluginName, optional forceRemoteSync" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginListParams.ts", + "title": "PluginListParams TypeScript type definition", + "qualityScore": 65, + "authority": 10, + "recency": 10, + "depth": 4, + "examples": 3, + "uniqueness": 5, + "keyInsights": [ + "cwds parameter discovers repo marketplaces, optional forceRemoteSync" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/schema/typescript/v2/PluginListResponse.ts", + "title": "PluginListResponse TypeScript type definition", + "qualityScore": 68, + "authority": 10, + "recency": 10, + "depth": 5, + "examples": 3, + "uniqueness": 5, + "keyInsights": [ + "Response includes marketplaces, load errors, remote sync error, featured plugin IDs" + ] + }, + { + "url": "https://github.com/openai/codex/pull/14993", + "title": "PR #14993 - Product-aware plugin policies and manifest naming cleanup", + "qualityScore": 82, + "authority": 9, + "recency": 10, + "depth": 8, + "examples": 5, + "uniqueness": 8, + "keyInsights": [ + "Raw serde shapes vs resolved in-memory models separation", + "Legacy policy fields silently ignored", + "Default to AVAILABLE/ON_INSTALL for missing policies", + "Gradual migration path for existing marketplace entries" + ] + }, + { + "url": "https://github.com/openai/codex/pull/13712", + "title": "PR #13712 - Curated plugin marketplace and metadata cleanup", + "qualityScore": 78, + "authority": 9, + "recency": 10, + "depth": 7, + "examples": 5, + "uniqueness": 7, + "keyInsights": [ + "Optional skills and hooks fields in manifest (hooks not consumed)", + "Validated absolute paths for improved security", + "Plugin config restricted to user config layer only", + "Curated marketplace sync as background operation concern" + ] + }, + { + "url": "https://github.com/openai/codex/pull/15606", + "title": "PR #15606 - Plugin display name priority and MCP tool refresh", + "qualityScore": 72, + "authority": 9, + "recency": 10, + "depth": 6, + "examples": 4, + "uniqueness": 6, + "keyInsights": [ + "interface.displayName prioritized for plugin labels", + "Plugin provenance preserved through MCP tool refresh", + "App mention deduplication in TUI" + ] + }, + { + "url": "https://github.com/openai/codex/issues/16430", + "title": "Issue #16430 - Plugin hooks not executed by runtime", + "qualityScore": 85, + "authority": 8, + "recency": 10, + "depth": 8, + "examples": 5, + "uniqueness": 9, + "keyInsights": [ + "Plugin-local hooks.json not executed - only global hooks.json", + "manifest.rs parses skills, mcpServers, apps but NOT hooks", + "hooks/discovery.rs scans only config-layer directories", + "Documentation/examples misleadingly suggest plugin hooks work" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/core/src/connectors.rs", + "title": "connectors.rs - App connector management module", + "qualityScore": 70, + "authority": 10, + "recency": 10, + "depth": 6, + "examples": 4, + "uniqueness": 5, + "keyInsights": [ + "Apps loaded from MCP tools with caching", + "App tool policy controls enable/disable state", + "30-second timeout for connectors on startup", + "Blocked connector IDs include connector_openai_ prefix" + ] + }, + { + "url": "https://github.com/openai/codex/blob/main/codex-rs/config/src/skills_config.rs", + "title": "skills_config.rs - Skills configuration types", + "qualityScore": 68, + "authority": 10, + "recency": 10, + "depth": 5, + "examples": 4, + "uniqueness": 5, + "keyInsights": [ + "SkillsConfig with bundled toggle and individual SkillConfig entries", + "Path-based and name-based skill identification", + "deny_unknown_fields for strict validation", + "TOML-based configuration" + ] + } + ] +} diff --git a/resources/rag-skill-indexing-sources.json b/resources/rag-skill-indexing-sources.json new file mode 100644 index 0000000..c96c1ae --- /dev/null +++ b/resources/rag-skill-indexing-sources.json @@ -0,0 +1,249 @@ +{ + "topic": "RAG skill indexing for AI agent knowledge files", + "slug": "rag-skill-indexing", + "generated": "2026-03-31T00:00:00Z", + "depth": "deep", + "totalSources": 18, + "sources": [ + { + "url": "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices", + "title": "Skill Authoring Best Practices - Anthropic", + "qualityScore": 99, + "scores": { "authority": 10, "recency": 10, "depth": 10, "examples": 10, "uniqueness": 10 }, + "keyInsights": [ + "Context window is a public good - only add what the model doesn't already know", + "Description field is the gatekeeper for skill discovery - must include both what and when", + "Keep SKILL.md under 500 lines, use progressive disclosure for deeper content", + "Match degrees of freedom to task fragility (high/medium/low)", + "Avoid deeply nested references - keep one level deep from SKILL.md", + "Structure longer reference files with table of contents for partial reads", + "Build evaluations before writing documentation", + "Test with all models (Haiku/Sonnet/Opus) - effectiveness depends on underlying model" + ] + }, + { + "url": "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview", + "title": "Agent Skills Overview - Anthropic", + "qualityScore": 98, + "scores": { "authority": 10, "recency": 10, "depth": 10, "examples": 9, "uniqueness": 10 }, + "keyInsights": [ + "Three-tier progressive disclosure: metadata (~100 tokens) -> instructions (<5k tokens) -> resources (as needed)", + "Skills leverage filesystem-based architecture for on-demand loading", + "Metadata loaded at startup, instructions when triggered, resources when referenced", + "Scripts executed via bash without loading code into context - only output consumes tokens", + "No practical limit on bundled content - context penalty only when accessed" + ] + }, + { + "url": "https://agentskills.io/specification", + "title": "Agent Skills Specification - Open Standard", + "qualityScore": 96, + "scores": { "authority": 10, "recency": 10, "depth": 9, "examples": 8, "uniqueness": 10 }, + "keyInsights": [ + "name: max 64 chars, lowercase + hyphens only, must match parent directory", + "description: max 1024 chars, must include what and when", + "Body content has no format restrictions - write whatever helps agents", + "Progressive disclosure: metadata ~100 tokens, instructions <5000 tokens recommended", + "File references should be one level deep from SKILL.md", + "Keep individual reference files focused - smaller files mean less context use" + ] + }, + { + "url": "https://agentskills.io/client-implementation/adding-skills-support", + "title": "How to Add Skills Support to Your Agent - Agent Skills", + "qualityScore": 95, + "scores": { "authority": 10, "recency": 10, "depth": 10, "examples": 9, "uniqueness": 9 }, + "keyInsights": [ + "Catalog approach: name + description disclosed at startup (~50-100 tokens per skill)", + "Model-driven activation: model reads catalog, decides relevance, loads full content", + "Structured wrapping with XML tags helps distinguish skill content from conversation", + "Protect skill content from context compaction - losing instructions silently degrades performance", + "Deduplicate activations to avoid same instructions appearing multiple times", + "File-read activation vs dedicated tool activation - both work" + ] + }, + { + "url": "https://code.claude.com/docs/en/skills", + "title": "Extend Claude with Skills - Claude Code Documentation", + "qualityScore": 94, + "scores": { "authority": 10, "recency": 10, "depth": 9, "examples": 9, "uniqueness": 8 }, + "keyInsights": [ + "Skills discovered from personal (~/.claude/skills/), project (.claude/skills/), and plugin directories", + "Automatic discovery from nested directories supports monorepo setups", + "Description front-loaded and truncated at 250 chars in skill listing", + "SLASH_COMMAND_TOOL_CHAR_BUDGET env var controls total description budget (default 1% of context)", + "disable-model-invocation: true removes skill from Claude context entirely", + "user-invocable: false hides from / menu but keeps in Claude context" + ] + }, + { + "url": "https://code.claude.com/docs/en/memory", + "title": "How Claude Remembers Your Project - Claude Code Memory", + "qualityScore": 93, + "scores": { "authority": 10, "recency": 10, "depth": 9, "examples": 8, "uniqueness": 8 }, + "keyInsights": [ + "CLAUDE.md files loaded hierarchically: managed policy > project > user", + "Target under 200 lines per CLAUDE.md file for best adherence", + "Auto memory: first 200 lines or 25KB of MEMORY.md loaded at session start", + "@path/to/import syntax for referencing external files (up to 5 hops deep)", + ".claude/rules/ directory for path-specific conditional rules with glob patterns", + "HTML comments stripped from CLAUDE.md before injection - use for human-only notes" + ] + }, + { + "url": "https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices", + "title": "Prompting Best Practices - Anthropic", + "qualityScore": 97, + "scores": { "authority": 10, "recency": 10, "depth": 10, "examples": 10, "uniqueness": 9 }, + "keyInsights": [ + "XML tags help Claude parse complex prompts unambiguously - use for documents, context, instructions", + "Structure with ......", + "Put longform data at the top of prompts, queries at the end (up to 30% quality improvement)", + "Ground responses in quotes from source documents to cut through noise", + "3-5 examples reliably steer output format, tone, and structure", + "Consistent, descriptive tag names reduce misinterpretation" + ] + }, + { + "url": "https://platform.claude.com/docs/en/build-with-claude/context-windows", + "title": "Context Windows - Anthropic Documentation", + "qualityScore": 92, + "scores": { "authority": 10, "recency": 10, "depth": 9, "examples": 7, "uniqueness": 8 }, + "keyInsights": [ + "Claude Opus 4.6 and Sonnet 4.6: 1M-token context window", + "Other Claude models (Sonnet 4.5, Sonnet 4): 200k-token context window", + "Context rot: accuracy and recall degrade as token count grows", + "Curating what's in context is as important as how much space is available", + "Context awareness: model tracks remaining budget with tags", + "Compaction provides server-side summarization for long conversations" + ] + }, + { + "url": "https://agentskills.io/what-are-skills", + "title": "What Are Skills? - Agent Skills", + "qualityScore": 90, + "scores": { "authority": 10, "recency": 10, "depth": 7, "examples": 8, "uniqueness": 7 }, + "keyInsights": [ + "Skills use progressive disclosure: discovery -> activation -> execution", + "Self-documenting format makes skills easy to audit and improve", + "Portable: just files, easy to edit, version, and share", + "Extensible from pure text instructions to executable code with assets" + ] + }, + { + "url": "https://agentskills.io", + "title": "Agent Skills Open Standard - Overview", + "qualityScore": 89, + "scores": { "authority": 10, "recency": 10, "depth": 6, "examples": 6, "uniqueness": 8 }, + "keyInsights": [ + "30+ AI tools support Agent Skills: Claude Code, Cursor, Copilot, Codex, Gemini CLI, JetBrains, VS Code, OpenHands, etc.", + "Open standard originally by Anthropic, now community-governed", + "Skills capture organizational knowledge in portable, version-controlled packages", + "Interoperability: same skill works across multiple agent products" + ] + }, + { + "url": "https://www.anthropic.com/research/contextual-retrieval", + "title": "Contextual Retrieval - Anthropic Research", + "qualityScore": 94, + "scores": { "authority": 10, "recency": 8, "depth": 10, "examples": 8, "uniqueness": 10 }, + "keyInsights": [ + "Traditional RAG loses context when chunking - individual chunks lack surrounding information", + "Contextual Retrieval prepends chunk-specific context using a language model before embedding", + "Contextual Embeddings + Contextual BM25 reduces retrieval failures by 67%", + "Adding reranking on top achieves 49% fewer failures overall", + "BM25 (keyword matching) and embeddings (semantic) are complementary - hybrid is best", + "Optimal chunk sizes: 100-300 tokens for embedding, with 10-20% overlap" + ] + }, + { + "url": "https://docs.cursor.com/context/rules", + "title": "Cursor Rules - Context Configuration", + "qualityScore": 85, + "scores": { "authority": 8, "recency": 9, "depth": 7, "examples": 7, "uniqueness": 7 }, + "keyInsights": [ + "Cursor supports .cursor/rules/ with MDC frontmatter including globs for path-specific activation", + "alwaysApply: true loads unconditionally, globs: pattern loads conditionally", + "Cursor also reads AGENTS.md and .cursorrules for backward compatibility", + "Cursor codebase indexing uses embeddings for semantic search over project files" + ] + }, + { + "url": "https://docs.github.com/en/copilot/customizing-copilot/adding-custom-instructions-for-github-copilot", + "title": "Custom Instructions for GitHub Copilot", + "qualityScore": 84, + "scores": { "authority": 9, "recency": 9, "depth": 6, "examples": 7, "uniqueness": 6 }, + "keyInsights": [ + "Copilot reads .github/copilot-instructions.md for project-level instructions", + "Copilot also supports Agent Skills in .agents/skills/ and .github/skills/", + "Copilot uses semantic matching against skill descriptions for activation", + "VS Code Copilot has built-in codebase indexing for @workspace queries" + ] + }, + { + "url": "https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents", + "title": "Effective Context Engineering for AI Agents - Anthropic Engineering", + "qualityScore": 96, + "scores": { "authority": 10, "recency": 10, "depth": 10, "examples": 9, "uniqueness": 10 }, + "keyInsights": [ + "Context engineering is the discipline of curating what goes into the context window", + "Quality of context matters more than quantity - context rot degrades with length", + "Progressive disclosure is the fundamental pattern: load what's needed when needed", + "Structured formats (XML, JSON) help models parse and locate information reliably", + "Compaction strategies should preserve high-signal content and drop low-signal content" + ] + }, + { + "url": "https://claude.com/blog/equipping-agents-for-the-real-world-with-agent-skills", + "title": "Equipping Agents for the Real World with Agent Skills - Anthropic Engineering", + "qualityScore": 93, + "scores": { "authority": 10, "recency": 10, "depth": 9, "examples": 8, "uniqueness": 9 }, + "keyInsights": [ + "Skills solve the gap between general AI capability and domain-specific expertise", + "Progressive disclosure enables large skill libraries without upfront context cost", + "Description quality is the primary factor in correct skill activation", + "Filesystem-based architecture enables on-demand loading without token waste" + ] + }, + { + "url": "https://arxiv.org/abs/2312.10997", + "title": "Retrieval-Augmented Generation for Large Language Models: A Survey", + "qualityScore": 88, + "scores": { "authority": 9, "recency": 7, "depth": 10, "examples": 7, "uniqueness": 8 }, + "keyInsights": [ + "Three RAG paradigms: Naive RAG, Advanced RAG, and Modular RAG", + "Chunk size profoundly affects retrieval quality - too small loses context, too large adds noise", + "Hybrid retrieval (dense + sparse) consistently outperforms either alone", + "Reranking stage critical for filtering noisy retrievals before generation", + "Metadata enrichment improves retrieval precision significantly" + ] + }, + { + "url": "https://www.pinecone.io/learn/chunking-strategies/", + "title": "Chunking Strategies for RAG - Pinecone", + "qualityScore": 86, + "scores": { "authority": 8, "recency": 8, "depth": 9, "examples": 8, "uniqueness": 7 }, + "keyInsights": [ + "Fixed-size chunking: simple but can break mid-sentence/concept", + "Semantic chunking: groups by topic coherence, better retrieval quality", + "Recursive character splitting: tries larger breaks first then smaller", + "Overlap of 10-20% between chunks preserves boundary context", + "Markdown/HTML-aware chunking respects document structure", + "256-512 tokens is the sweet spot for most embedding models" + ] + }, + { + "url": "https://research.google/blog/raptor-recursive-abstractive-processing-for-tree-organized-retrieval/", + "title": "RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval - Google Research", + "qualityScore": 87, + "scores": { "authority": 9, "recency": 7, "depth": 9, "examples": 7, "uniqueness": 9 }, + "keyInsights": [ + "Hierarchical summarization creates multi-level retrieval trees", + "Leaf nodes are original chunks, higher nodes are summaries of clusters", + "Retrieval at different tree levels answers both specific and broad queries", + "Outperforms flat chunking approaches on complex, multi-hop questions", + "Particularly effective for knowledge bases with mixed granularity needs" + ] + } + ] +} diff --git a/resources/resp-mq-server-rust-sources.json b/resources/resp-mq-server-rust-sources.json new file mode 100644 index 0000000..bc76268 --- /dev/null +++ b/resources/resp-mq-server-rust-sources.json @@ -0,0 +1,543 @@ +{ + "topic": "Building a RESP-Compatible Message Queue Server in Rust", + "slug": "resp-mq-server-rust", + "generated": "2026-04-03T00:00:00Z", + "depth": "deep", + "totalSources": 42, + "sources": [ + { + "url": "https://redis.io/docs/latest/develop/reference/protocol-spec/", + "title": "Redis RESP Protocol Specification", + "qualityScore": 98, + "scores": { "authority": 10, "recency": 10, "depth": 10, "examples": 8, "uniqueness": 10 }, + "keyInsights": [ + "RESP2 types: simple strings (+), errors (-), integers (:), bulk strings ($), arrays (*)", + "RESP3 adds: nulls (_), booleans (#), doubles (,), maps (%), sets (~), pushes (>)", + "All messages terminated with CRLF, binary-safe via length prefixes", + "HELLO command negotiates RESP2 vs RESP3", + "Pipelining sends multiple commands without waiting for responses" + ] + }, + { + "url": "https://github.com/tokio-rs/mini-redis", + "title": "mini-redis - Tokio educational Redis server", + "qualityScore": 95, + "scores": { "authority": 10, "recency": 8, "depth": 9, "examples": 10, "uniqueness": 8 }, + "keyInsights": [ + "Task-per-connection model with tokio::spawn", + "std::sync::Mutex preferred over tokio::sync::Mutex for short critical sections", + "Semaphore-based connection limiting (250 max)", + "Frame enum: Simple, Error, Integer, Bulk, Null, Array", + "Cursor-based RESP parsing with check-then-parse pattern" + ] + }, + { + "url": "https://docs.rs/redis-protocol/latest/redis_protocol/", + "title": "redis-protocol Rust crate", + "qualityScore": 88, + "scores": { "authority": 8, "recency": 8, "depth": 8, "examples": 9, "uniqueness": 8 }, + "keyInsights": [ + "Three frame interfaces: OwnedFrame, BytesFrame, RangeFrame/BorrowedFrame", + "Zero-copy parsing via BytesFrame with bytes crate", + "Supports both RESP2 and RESP3", + "int_as_bulkstring parameter for allocation-free integer encoding" + ] + }, + { + "url": "https://github.com/taskforcesh/bullmq/tree/master/src/commands", + "title": "BullMQ Lua Scripts Directory (51 scripts)", + "qualityScore": 96, + "scores": { "authority": 10, "recency": 9, "depth": 10, "examples": 10, "uniqueness": 10 }, + "keyInsights": [ + "51 Lua scripts for all queue operations", + "Key scripts: addStandardJob, moveToActive, moveToFinished, retryJob", + "Uses RPOPLPUSH, XADD, ZADD, HSET, INCR, EXISTS, LREM, SREM", + "9-14 keys per script, complex multi-key atomic operations", + "Scripts handle deduplication, rate limiting, priorities, parent-child deps" + ] + }, + { + "url": "https://github.com/taskforcesh/bullmq/blob/master/src/commands/moveToActive-11.lua", + "title": "BullMQ moveToActive Lua script", + "qualityScore": 92, + "scores": { "authority": 10, "recency": 9, "depth": 9, "examples": 10, "uniqueness": 8 }, + "keyInsights": [ + "RPOPLPUSH from wait to active list", + "Rate limit check via expireTime TTL", + "Promotes delayed jobs before fetching", + "Falls back to prioritized queue if standard empty", + "11 Redis keys accessed per invocation" + ] + }, + { + "url": "https://github.com/taskforcesh/bullmq/blob/master/src/commands/moveToFinished-14.lua", + "title": "BullMQ moveToFinished Lua script", + "qualityScore": 92, + "scores": { "authority": 10, "recency": 9, "depth": 9, "examples": 10, "uniqueness": 8 }, + "keyInsights": [ + "14 keys accessed - most complex BullMQ script", + "Uses EXISTS, SCARD, ZCARD, LREM, HMGET, HSET, ZADD, XADD", + "Handles completed/failed sets with retention policies", + "Manages parent-child dependency resolution", + "Optionally fetches next job for processing" + ] + }, + { + "url": "https://github.com/taskforcesh/bullmq/blob/master/src/commands/addStandardJob-9.lua", + "title": "BullMQ addStandardJob Lua script", + "qualityScore": 90, + "scores": { "authority": 10, "recency": 9, "depth": 8, "examples": 10, "uniqueness": 7 }, + "keyInsights": [ + "INCR for ID generation, EXISTS for parent validation", + "XADD for event emission, SADD for parent dependencies", + "RPUSH/LPUSH for FIFO/LIFO placement", + "Deduplication via deduplicateJob helper", + "9 Redis keys: wait, paused, meta, id, completed, delayed, active, events, marker" + ] + }, + { + "url": "https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/client.rb", + "title": "Sidekiq Client - Job Pushing", + "qualityScore": 85, + "scores": { "authority": 9, "recency": 8, "depth": 7, "examples": 8, "uniqueness": 8 }, + "keyInsights": [ + "LPUSH for immediate jobs to queue:{name}", + "SADD to queues set for queue discovery", + "ZADD to schedule sorted set for delayed jobs", + "Jobs are JSON: class, args, jid, queue, enqueued_at", + "Pipelined Redis operations, no Lua scripts" + ] + }, + { + "url": "https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/fetch.rb", + "title": "Sidekiq Fetch - Job Consumption", + "qualityScore": 84, + "scores": { "authority": 9, "recency": 8, "depth": 7, "examples": 8, "uniqueness": 7 }, + "keyInsights": [ + "BRPOP with 2-second timeout across multiple queues", + "Strict mode vs shuffled queue order for fairness", + "bulk_requeue for graceful shutdown job preservation", + "No Lua scripts - pure Redis commands" + ] + }, + { + "url": "https://github.com/celery/celery/blob/main/celery/backends/redis.py", + "title": "Celery Redis Backend", + "qualityScore": 82, + "scores": { "authority": 8, "recency": 7, "depth": 7, "examples": 8, "uniqueness": 8 }, + "keyInsights": [ + "GET, MGET, DEL, INCRBY, EXPIRE, SET, SETEX for results", + "Pub/Sub for result notification (task ID as channel)", + "ZADD/ZRANGE for chord result ordering", + "No Lua scripts - pipelined commands instead" + ] + }, + { + "url": "https://github.com/celery/kombu/blob/main/kombu/transport/redis.py", + "title": "Celery/Kombu Redis Transport", + "qualityScore": 88, + "scores": { "authority": 8, "recency": 7, "depth": 9, "examples": 8, "uniqueness": 9 }, + "keyInsights": [ + "LPUSH to insert, BRPOP to retrieve messages", + "Priority queues via separate keys per priority level", + "ZADD sorted sets for unacked message tracking", + "Visibility timeout: 3600s default with mutex-protected restoration", + "PSUBSCRIBE for fanout exchange patterns", + "No Lua scripts - Redis transactions for atomicity" + ] + }, + { + "url": "https://docs.rs/mlua/latest/mlua/", + "title": "mlua - Lua bindings for Rust", + "qualityScore": 93, + "scores": { "authority": 9, "recency": 9, "depth": 9, "examples": 9, "uniqueness": 9 }, + "keyInsights": [ + "Supports Lua 5.1-5.5, LuaJIT, and Luau", + "create_function() registers Rust callbacks callable from Lua", + "Async support via create_async_function", + "Sandboxing via StdLib flags and set_memory_limit", + "IntoLua/FromLua traits for safe type conversion", + "Send + Sync when feature-gated" + ] + }, + { + "url": "https://docs.rs/mlua/latest/mlua/struct.Lua.html", + "title": "mlua Lua struct API", + "qualityScore": 90, + "scores": { "authority": 9, "recency": 9, "depth": 9, "examples": 8, "uniqueness": 7 }, + "keyInsights": [ + "Lua::new_with(libs, options) for selective library loading", + "load().exec()/eval()/call() for script execution", + "create_table_from() for creating Lua tables from Rust iterators", + "UserData trait for exposing custom Rust types", + "scope() permits temporary non-static values" + ] + }, + { + "url": "https://docs.rs/piccolo/latest/piccolo/", + "title": "piccolo - Pure Rust Lua VM", + "qualityScore": 55, + "scores": { "authority": 5, "recency": 5, "depth": 4, "examples": 3, "uniqueness": 8 }, + "keyInsights": [ + "Stackless Lua VM in pure Rust - no C dependency", + "v0.3.3 with only 10% documentation coverage", + "Single developer, experimental stage", + "Not production-ready for server embedding" + ] + }, + { + "url": "https://redis.io/docs/latest/develop/interact/programmability/eval-intro/", + "title": "Redis Lua Scripting (EVAL/EVALSHA)", + "qualityScore": 95, + "scores": { "authority": 10, "recency": 9, "depth": 9, "examples": 9, "uniqueness": 9 }, + "keyInsights": [ + "EVAL executes script, EVALSHA executes by SHA1 hash", + "Scripts cached via SHA1 - 40 hex char identifier", + "redis.call() throws on error, redis.pcall() returns error as table", + "KEYS[] and ARGV[] arrays, numkeys boundary", + "Atomic execution - no other commands during script", + "SCRIPT LOAD, SCRIPT EXISTS, SCRIPT FLUSH for management" + ] + }, + { + "url": "https://redis.io/docs/latest/develop/interact/programmability/lua-api/", + "title": "Redis Lua API Reference", + "qualityScore": 93, + "scores": { "authority": 10, "recency": 9, "depth": 9, "examples": 8, "uniqueness": 8 }, + "keyInsights": [ + "Type conversion: integer to number, bulk to string, nil to false", + "cjson and cmsgpack libraries available", + "redis.log() with LOG_DEBUG/NOTICE/WARNING", + "redis.sha1hex() for hashing", + "redis.error_reply()/status_reply() for return types" + ] + }, + { + "url": "https://www.dragonflydb.io/blog/scaling-performance-redis-vs-dragonfly", + "title": "DragonflyDB Architecture - 25x Redis Throughput", + "qualityScore": 88, + "scores": { "authority": 8, "recency": 8, "depth": 8, "examples": 7, "uniqueness": 9 }, + "keyInsights": [ + "25x throughput over Redis with multi-threading", + "Drop-in Redis replacement, no code changes", + "Memory does not grow during snapshotting", + "proactor_threads configurable per CPU core", + "Inherently smaller memory usage for same dataset" + ] + }, + { + "url": "https://github.com/dragonflydb/dragonfly/blob/main/src/server/main_service.cc", + "title": "DragonflyDB Server Architecture (source)", + "qualityScore": 90, + "scores": { "authority": 9, "recency": 9, "depth": 9, "examples": 8, "uniqueness": 10 }, + "keyInsights": [ + "Shared-nothing with ProactorPool and EngineShardSet", + "Fiber-based concurrency within each proactor thread", + "MultiCommandSquasher batches pipeline commands", + "LOCK_AHEAD for single-shard, GLOBAL for multi-shard ops", + "Deterministic key hashing for shard distribution", + "Thread-local ServerState prevents lock contention" + ] + }, + { + "url": "https://microsoft.github.io/garnet/docs/welcome/features", + "title": "Microsoft Garnet Features", + "qualityScore": 85, + "scores": { "authority": 9, "recency": 9, "depth": 7, "examples": 5, "uniqueness": 8 }, + "keyInsights": [ + "RESP wire protocol compatible, works with unmodified Redis clients", + "Tsavorite storage engine with hybrid log-structured design", + "Lua scripting support plus C# extensibility", + "Sub-300us p99.9 latency with Accelerated Networking", + "Cluster mode with sharding, replication, failover" + ] + }, + { + "url": "https://github.com/apache/kvrocks", + "title": "Apache Kvrocks - Redis on RocksDB", + "qualityScore": 82, + "scores": { "authority": 8, "recency": 8, "depth": 7, "examples": 5, "uniqueness": 8 }, + "keyInsights": [ + "Redis-compatible protocol on RocksDB storage", + "Supports Lua via LuaJIT or standard Lua", + "Centralized cluster management (not gossip-based)", + "Namespace isolation with per-namespace auth", + "MySQL-style binlog replication" + ] + }, + { + "url": "https://github.com/tidwall/redcon.rs", + "title": "redcon.rs - Rust Redis server framework", + "qualityScore": 78, + "scores": { "authority": 6, "recency": 6, "depth": 7, "examples": 9, "uniqueness": 8 }, + "keyInsights": [ + "Multithreaded RESP server framework", + "Callback-based command handling", + "write_string/write_bulk/write_null/write_error API", + "Support for pipelining and telnet commands" + ] + }, + { + "url": "https://github.com/tidwall/redcon", + "title": "redcon (Go) - Redis server framework", + "qualityScore": 80, + "scores": { "authority": 7, "recency": 7, "depth": 7, "examples": 9, "uniqueness": 7 }, + "keyInsights": [ + "2x+ throughput over Redis (2M+ SET/sec)", + "Three callbacks: command handler, accept/reject, disconnect", + "Pre-parsed Command objects with byte slice args", + "TLS support via ListenAndServeTLS" + ] + }, + { + "url": "https://github.com/aembke/fred.rs", + "title": "fred.rs - Redis client for Rust", + "qualityScore": 78, + "scores": { "authority": 7, "recency": 8, "depth": 7, "examples": 7, "uniqueness": 6 }, + "keyInsights": [ + "Zero-copy frame parsing via custom protocol library", + "Round-robin and dynamic connection pooling", + "Automatic pipelining for throughput", + "SHA-1 feature for Lua script hashing" + ] + }, + { + "url": "https://tokio.rs/tokio/tutorial/shared-state", + "title": "Tokio Shared State Patterns", + "qualityScore": 92, + "scores": { "authority": 10, "recency": 9, "depth": 9, "examples": 9, "uniqueness": 7 }, + "keyInsights": [ + "std::sync::Mutex fine for short critical sections without .await", + "DashMap for sophisticated sharded hash maps", + "Sharding: db[hash(key) % db.len()] reduces contention", + "Message passing alternative: dedicated state manager task", + "Never hold MutexGuard across .await points" + ] + }, + { + "url": "https://tokio.rs/tokio/tutorial/channels", + "title": "Tokio Channel Patterns", + "qualityScore": 90, + "scores": { "authority": 10, "recency": 9, "depth": 8, "examples": 9, "uniqueness": 7 }, + "keyInsights": [ + "mpsc for commands, oneshot for responses", + "broadcast for pub/sub, watch for latest-value", + "Actor pattern: dedicated manager task with channel inbox", + "Enables command pipelining and better throughput" + ] + }, + { + "url": "https://docs.rs/dashmap/latest/dashmap/", + "title": "DashMap - Concurrent HashMap", + "qualityScore": 82, + "scores": { "authority": 7, "recency": 8, "depth": 7, "examples": 7, "uniqueness": 7 }, + "keyInsights": [ + "Sharded concurrent HashMap, thread-safe", + "DashSet wrapper using () values", + "Entry API for efficient conditional inserts", + "Better than RwLock for frequent concurrent access" + ] + }, + { + "url": "https://docs.rs/crossbeam-skiplist/latest/crossbeam_skiplist/", + "title": "crossbeam-skiplist - Concurrent Sorted Map", + "qualityScore": 85, + "scores": { "authority": 8, "recency": 8, "depth": 8, "examples": 7, "uniqueness": 9 }, + "keyInsights": [ + "Lock-free SkipMap and SkipSet (concurrent BTreeMap equivalent)", + "Mutating methods take &self (not &mut self)", + "Epoch-based memory reclamation prevents use-after-free", + "Excels with high write frequency; slower than BTree for reads" + ] + }, + { + "url": "https://docs.rs/crossbeam-deque/latest/crossbeam_deque/", + "title": "crossbeam-deque - Work-Stealing Queues", + "qualityScore": 83, + "scores": { "authority": 8, "recency": 8, "depth": 7, "examples": 7, "uniqueness": 8 }, + "keyInsights": [ + "Injector (shared FIFO), Worker (thread-local), Stealer pattern", + "FIFO and LIFO worker modes", + "steal_batch_and_pop for efficient batch stealing", + "Ideal for work-stealing schedulers in multi-threaded servers" + ] + }, + { + "url": "https://docs.rs/priority-queue/latest/priority_queue/", + "title": "priority-queue crate", + "qualityScore": 78, + "scores": { "authority": 6, "recency": 7, "depth": 7, "examples": 7, "uniqueness": 8 }, + "keyInsights": [ + "Heap-backed with HashMap index for O(log N) priority changes", + "DoublePriorityQueue for both min and max operations", + "O(1) peek, O(log N) insert/delete/change", + "Ideal for delayed job scheduling with rescheduling" + ] + }, + { + "url": "https://docs.rs/bumpalo/latest/bumpalo/", + "title": "bumpalo - Arena Allocator", + "qualityScore": 80, + "scores": { "authority": 7, "recency": 8, "depth": 8, "examples": 8, "uniqueness": 7 }, + "keyInsights": [ + "Bump allocation: pointer arithmetic for O(1) alloc", + "Mass deallocation by resetting arena pointer", + "Ideal for phase-oriented allocations (request batches)", + "Bump is !Sync; bumpalo-herd for thread-safe pools" + ] + }, + { + "url": "https://docs.rs/bytes/latest/bytes/", + "title": "bytes crate - Zero-Copy Buffers", + "qualityScore": 88, + "scores": { "authority": 9, "recency": 9, "depth": 7, "examples": 7, "uniqueness": 6 }, + "keyInsights": [ + "Bytes: immutable, reference-counted, zero-copy slicing", + "BytesMut: mutable buffer for construction", + "Buf/BufMut traits for structured read/write", + "Designed for networking - used throughout Tokio ecosystem" + ] + }, + { + "url": "https://docs.rs/slab/latest/slab/", + "title": "slab - Pre-allocated Storage", + "qualityScore": 75, + "scores": { "authority": 7, "recency": 7, "depth": 7, "examples": 7, "uniqueness": 6 }, + "keyInsights": [ + "Pre-allocated uniform storage with key-based access", + "O(1) insert via vacant slot reuse", + "Keys reused after removal - minimal fragmentation", + "Ideal for connection management in servers" + ] + }, + { + "url": "https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/", + "title": "Redis Persistence (RDB + AOF)", + "qualityScore": 95, + "scores": { "authority": 10, "recency": 9, "depth": 10, "examples": 8, "uniqueness": 8 }, + "keyInsights": [ + "RDB: fork-based copy-on-write snapshots", + "AOF: append-only write-ahead log with fsync policies", + "fsync policies: always (max durability), everysec (default), no", + "AOF rewrite compacts log via fork", + "Redis 7.0+ multi-part AOF with manifest" + ] + }, + { + "url": "https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/benchmarks/", + "title": "Redis Benchmarking Methodology", + "qualityScore": 90, + "scores": { "authority": 10, "recency": 8, "depth": 9, "examples": 9, "uniqueness": 7 }, + "keyInsights": [ + "redis-benchmark: -c clients, -n requests, -P pipeline, -d payload size", + "8-10x throughput improvement with 16-command pipelining", + "Unix domain sockets 50% better than TCP loopback", + "memtier_benchmark as alternative tool" + ] + }, + { + "url": "https://redis.io/docs/latest/develop/interact/transactions/", + "title": "Redis MULTI/EXEC Transactions", + "qualityScore": 90, + "scores": { "authority": 10, "recency": 9, "depth": 9, "examples": 8, "uniqueness": 7 }, + "keyInsights": [ + "MULTI queues commands, EXEC executes atomically", + "No rollback - failed commands do not abort transaction", + "WATCH for optimistic locking (CAS pattern)", + "Lua scripts preferred over MULTI/EXEC for complex logic" + ] + }, + { + "url": "https://github.com/valkey-io/valkey/blob/unstable/src/server.h", + "title": "Valkey/Redis server.h - Core Data Structures", + "qualityScore": 88, + "scores": { "authority": 9, "recency": 9, "depth": 9, "examples": 6, "uniqueness": 10 }, + "keyInsights": [ + "7 object types: STRING, LIST, SET, ZSET, HASH, MODULE, STREAM", + "Database has keys, expires, blocking_keys, ready_keys, watched_keys", + "Client flags: blocked, authenticated, executing_command, repl_state", + "blockingState tracks btype, timeout, keys for BRPOP etc." + ] + }, + { + "url": "https://github.com/valkey-io/valkey/blob/unstable/src/t_stream.c", + "title": "Valkey Stream Implementation", + "qualityScore": 88, + "scores": { "authority": 9, "recency": 9, "depth": 9, "examples": 7, "uniqueness": 9 }, + "keyInsights": [ + "Radix tree indexed by stream IDs containing listpacks", + "Consumer groups with group-level and consumer-level PELs", + "Delta-encoded IDs for space efficiency", + "streamNACK objects track delivery count and timestamp" + ] + }, + { + "url": "https://docs.rs/tokio-uring/latest/tokio_uring/", + "title": "tokio-uring - io_uring for Tokio", + "qualityScore": 75, + "scores": { "authority": 7, "recency": 7, "depth": 6, "examples": 6, "uniqueness": 8 }, + "keyInsights": [ + "Requires Linux kernel 5.10+", + "Ownership-based buffer management", + "BufResult type for buffer-based operations", + "v0.5.0 - maturing but not stable" + ] + }, + { + "url": "https://docs.rs/glommio/latest/glommio/", + "title": "Glommio - Thread-per-core Runtime", + "qualityScore": 82, + "scores": { "authority": 7, "recency": 7, "depth": 8, "examples": 7, "uniqueness": 9 }, + "keyInsights": [ + "Thread-per-core with dedicated io_uring per executor", + "Three ring types: main, latency, poll", + "No atomics needed - cooperative single-thread scheduling", + "Requires Linux 5.8+, 512+ KiB locked memory" + ] + }, + { + "url": "https://without.boats/blog/io-uring/", + "title": "Notes on io-uring (withoutboats)", + "qualityScore": 80, + "scores": { "authority": 8, "recency": 6, "depth": 8, "examples": 5, "uniqueness": 9 }, + "keyInsights": [ + "Completion-based API vs readiness-based (epoll)", + "Kernel writes to buffer even if future cancelled", + "Kernel buffer ownership model preferred over zero-copy borrowed", + "Future of all IO on Linux" + ] + }, + { + "url": "https://docs.rs/tokio-rustls/latest/tokio_rustls/", + "title": "tokio-rustls - TLS for Tokio", + "qualityScore": 78, + "scores": { "authority": 8, "recency": 8, "depth": 6, "examples": 7, "uniqueness": 6 }, + "keyInsights": [ + "TlsAcceptor wraps incoming TcpStream connections", + "LazyConfigAcceptor for per-connection certificate selection", + "TlsStream unified type for post-handshake I/O" + ] + }, + { + "url": "https://docs.rs/sha1/latest/sha1/", + "title": "sha1 crate for EVALSHA hashing", + "qualityScore": 72, + "scores": { "authority": 7, "recency": 7, "depth": 5, "examples": 8, "uniqueness": 5 }, + "keyInsights": [ + "Sha1::digest(script_bytes) for one-shot hashing", + "40 hex char output matches Redis EVALSHA format" + ] + }, + { + "url": "https://docs.rs/parking_lot/latest/parking_lot/", + "title": "parking_lot - Fast Synchronization Primitives", + "qualityScore": 80, + "scores": { "authority": 8, "recency": 8, "depth": 6, "examples": 6, "uniqueness": 6 }, + "keyInsights": [ + "Smaller, faster Mutex/RwLock than std::sync", + "FairMutex prevents thread starvation in high-concurrency", + "Const constructors for static initialization" + ] + } + ] +} diff --git a/resp-mq-server-rust.md b/resp-mq-server-rust.md new file mode 100644 index 0000000..ad2c72b --- /dev/null +++ b/resp-mq-server-rust.md @@ -0,0 +1,1075 @@ +# Learning Guide: Building a RESP-Compatible Message Queue Server in Rust + +**Generated**: 2026-04-03 +**Sources**: 42 resources analyzed +**Depth**: deep + +## Prerequisites + +- Proficiency in Rust (ownership, traits, async/await, tokio) +- Understanding of TCP networking and binary protocols +- Familiarity with Redis concepts (commands, data types, pipelining) +- Familiarity with at least one MQ library (BullMQ, Sidekiq, Celery, or glide-mq) +- Basic understanding of Lua scripting + +## TL;DR + +- RESP is a simple, CRLF-terminated, prefix-based wire protocol. Implementing a server requires parsing 5 RESP2 types (simple strings, errors, integers, bulk strings, arrays) plus optionally 8 RESP3 types. +- BullMQ is the hardest client to support - it ships 51 Lua scripts using EVAL/EVALSHA with up to 14 keys per script. Sidekiq and Celery are simpler (no Lua, pure commands). +- Use `mlua` with Lua 5.1/LuaJIT to embed Lua. Implement `redis.call()`/`redis.pcall()` as Rust functions registered into the Lua VM. Cache scripts by SHA1 hash. +- For queue-optimized storage, use skip lists for sorted sets (delayed/priority jobs), arena allocators for job payloads, and a sharded DashMap for the keyspace. +- A shared-nothing, thread-per-core architecture (like DragonflyDB) outperforms single-threaded Redis by 25x. Start simpler with Tokio multi-threaded runtime and shard data by key hash. + +## 1. RESP Protocol Implementation + +### Wire Format + +RESP (Redis Serialization Protocol) is a request-response protocol over TCP (default port 6379). Every element is terminated with `\r\n` (CRLF). The first byte identifies the type. + +**RESP2 Types (must implement)**: + +| First Byte | Type | Example | +|-----------|------|---------| +| `+` | Simple String | `+OK\r\n` | +| `-` | Error | `-ERR unknown command\r\n` | +| `:` | Integer (signed 64-bit) | `:1000\r\n` | +| `$` | Bulk String (length-prefixed) | `$5\r\nhello\r\n` | +| `*` | Array (count-prefixed) | `*2\r\n$3\r\nGET\r\n$3\r\nfoo\r\n` | + +Null values in RESP2: `$-1\r\n` (null bulk string) or `*-1\r\n` (null array). + +**RESP3 Types (implement for modern clients)**: + +| First Byte | Type | Example | +|-----------|------|---------| +| `_` | Null | `_\r\n` | +| `#` | Boolean | `#t\r\n` | +| `,` | Double | `,1.23\r\n` | +| `%` | Map | `%2\r\n+key\r\n:1\r\n+key2\r\n:2\r\n` | +| `~` | Set | `~2\r\n+a\r\n+b\r\n` | +| `>` | Push (out-of-band) | Server-initiated messages | +| `(` | Big Number | `(3492890328409238509324850943850943825024385\r\n` | +| `=` | Verbatim String | `=15\r\ntxt:Some string\r\n` | + +**Client requests** are always arrays of bulk strings: +``` +*3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\nbar\r\n +``` + +**Protocol negotiation**: Clients send `HELLO 3` to upgrade to RESP3. Respond with a map containing `server`, `version`, `proto`, `id`, `mode`, `role` fields. If you only support RESP2, respond with `-NOPROTO` error. + +**Inline commands**: For telnet debugging, support space-separated commands without RESP framing (detect by absence of `*` as first byte). + +**Pipelining**: Clients can send multiple commands without waiting for responses. Responses must be in the same order as requests - this is purely a buffering concern, not a special protocol mode. + +### Rust Implementation Strategy + +**Option A: Use `redis-protocol` crate** - provides zero-copy RESP2/RESP3 parsing with three frame interfaces: +- `OwnedFrame` - simplest, allocates per frame +- `BytesFrame` - zero-copy with `bytes::Bytes` +- `RangeFrame` - lowest overhead, borrowed references + +**Option B: Hand-roll (recommended for MQ server)** - mini-redis demonstrates the pattern: + +```rust +pub enum Frame { + Simple(String), + Error(String), + Integer(i64), + Bulk(Bytes), + Null, + Array(Vec), +} +``` + +**Parse-or-read-more loop** (from mini-redis): +1. Attempt to parse a complete frame from the buffer +2. If `Incomplete`, read more bytes from TCP into `BytesMut` +3. If complete, advance the buffer cursor and return the frame +4. If TCP returns 0 bytes, the connection is closed + +```rust +// Pseudocode for the core read loop +pub async fn read_frame(&mut self) -> Result> { + loop { + if let Some(frame) = self.parse_frame()? { + return Ok(Some(frame)); + } + if 0 == self.stream.read_buf(&mut self.buffer).await? { + return Ok(None); // Connection closed + } + } +} +``` + +**Key optimization**: Use `Cursor<&[u8]>` for parsing. Check completeness first (`Frame::check`) without allocating, then parse. This avoids partial allocations on incomplete reads. + +### TLS Support + +Use `tokio-rustls` with `TlsAcceptor`: +1. Load certificates via `rustls::ServerConfig` +2. Wrap each accepted `TcpStream` with `acceptor.accept(stream).await` +3. The resulting `TlsStream` implements `AsyncRead + AsyncWrite` - the rest of the code is unchanged + +For per-connection certificate selection (e.g., SNI-based), use `LazyConfigAcceptor`. + +### AUTH and Connection Commands + +Implement these connection-management commands: + +| Command | Behavior | +|---------|----------| +| `PING` | Return `+PONG\r\n` | +| `AUTH password` | Validate password, set connection authenticated flag | +| `HELLO [proto] [AUTH user pass]` | Protocol negotiation, optional inline auth | +| `CLIENT SETNAME name` | Store connection name for debugging | +| `CLIENT GETNAME` | Return stored name | +| `SELECT db` | For MQ, accept but ignore (or support 0-15 databases) | +| `QUIT` | Close connection gracefully | +| `INFO [section]` | Return server info (clients need this for health checks) | +| `CONFIG GET param` | Return dummy values for expected params | + +## 2. Redis Command Subset for MQ Workloads + +### BullMQ Command Surface (Most Complex) + +BullMQ ships 51 Lua scripts. The Redis commands called across all scripts: + +**Core Data Operations**: +- `RPOPLPUSH` / `LMOVE` - atomically move jobs between lists (wait -> active) +- `RPUSH` / `LPUSH` - add jobs to lists (FIFO/LIFO) +- `LREM` - remove specific items from lists +- `LLEN` - list length +- `ZADD` - add to sorted sets (delayed jobs, completed/failed with timestamps) +- `ZRANGEBYSCORE` / `ZRANGE` - query delayed jobs ready for execution +- `ZREM` - remove from sorted sets +- `ZCARD` / `SCARD` - count members +- `HSET` / `HMSET` / `HGET` / `HMGET` / `HGETALL` / `HDEL` - job data storage +- `INCR` - job ID generation +- `EXISTS` - key existence checks +- `SADD` / `SREM` / `SMEMBERS` / `SISMEMBER` - set operations for dependencies +- `XADD` - event stream emission +- `XREADGROUP` / `XACK` - consumer group reads (for QueueEvents) +- `XGROUP CREATE` - create consumer groups +- `EVAL` / `EVALSHA` - Lua script execution +- `SCRIPT LOAD` / `SCRIPT EXISTS` - script caching + +**Key Patterns** (prefix is configurable, default `bull`): +- `bull:{queueName}:wait` - waiting jobs list +- `bull:{queueName}:active` - processing jobs list +- `bull:{queueName}:delayed` - delayed jobs sorted set +- `bull:{queueName}:completed` - completed jobs sorted set +- `bull:{queueName}:failed` - failed jobs sorted set +- `bull:{queueName}:prioritized` - priority queue sorted set +- `bull:{queueName}:events` - event stream +- `bull:{queueName}:id` - job ID counter +- `bull:{queueName}:meta` - queue metadata hash +- `bull:{queueName}:{jobId}` - individual job hash + +### Sidekiq Command Surface (Simpler) + +Sidekiq uses no Lua scripts - pure Redis commands with pipelining: + +**Enqueue**: `LPUSH queue:{name} {json}` + `SADD queues {name}` +**Schedule**: `ZADD schedule {timestamp} {json}` +**Consume**: `BRPOP queue:{name1} queue:{name2} ... {timeout}` +**Retry**: `ZADD retry {timestamp} {json}` +**Track**: `SADD queues {name}` (queue discovery) + +**Key Patterns**: +- `queue:{name}` - job lists +- `schedule` - sorted set of scheduled jobs +- `retry` - sorted set of jobs to retry +- `queues` - set of all queue names + +### Celery Command Surface + +Celery (via Kombu transport) also uses no Lua scripts: + +**Enqueue**: `LPUSH {queue_name} {message}` +**Consume**: `BRPOP {queue1} {queue2} ... {timeout}` +**Results**: `SET celery-task-meta-{task_id} {result}` + `PUBLISH {task_id} {result}` +**Priority**: Separate keys per priority level: `{queue}\x06\x16{priority}` +**Unacked**: `ZADD unacked {timestamp} {message}` (visibility timeout tracking) +**Fanout**: `PUBLISH` / `PSUBSCRIBE` for fanout exchanges + +### Minimum Command Set + +Commands required to support all four client libraries: + +**Must Implement (Full)**: +- String: `SET`, `GET`, `DEL`, `EXISTS`, `INCR`, `INCRBY`, `EXPIRE`, `PEXPIRE`, `TTL`, `PTTL`, `SETEX`, `MGET`, `PERSIST` +- List: `LPUSH`, `RPUSH`, `LPOP`, `RPOP`, `BRPOP`, `BLPOP`, `LREM`, `LLEN`, `LRANGE`, `LINDEX`, `RPOPLPUSH`/`LMOVE` +- Hash: `HSET`, `HGET`, `HMSET`, `HMGET`, `HGETALL`, `HDEL`, `HINCRBY`, `HSETNX`, `HEXISTS` +- Set: `SADD`, `SREM`, `SMEMBERS`, `SISMEMBER`, `SCARD` +- Sorted Set: `ZADD`, `ZREM`, `ZRANGEBYSCORE`, `ZRANGE`, `ZREVRANGEBYSCORE`, `ZCARD`, `ZSCORE`, `ZREVRANGE` +- Stream: `XADD`, `XREADGROUP`, `XACK`, `XGROUP CREATE`, `XLEN`, `XRANGE`, `XDEL`, `XTRIM`, `XINFO` +- Scripting: `EVAL`, `EVALSHA`, `SCRIPT LOAD`, `SCRIPT EXISTS`, `SCRIPT FLUSH` +- Transaction: `MULTI`, `EXEC`, `DISCARD`, `WATCH`, `UNWATCH` +- Pub/Sub: `SUBSCRIBE`, `UNSUBSCRIBE`, `PSUBSCRIBE`, `PUNSUBSCRIBE`, `PUBLISH` +- Key: `DEL`, `EXISTS`, `EXPIRE`, `PEXPIRE`, `TTL`, `PTTL`, `TYPE`, `KEYS`, `SCAN`, `RENAME` + +**Stub/No-Op (return sensible defaults)**: +- `PING` - return PONG +- `SELECT` - accept, track per-connection DB index +- `INFO` - return minimal server info +- `CONFIG GET` - return expected defaults (e.g., `maxmemory-policy noeviction`) +- `CLIENT SETNAME` / `CLIENT GETNAME` / `CLIENT ID` - connection metadata +- `DBSIZE` - return key count +- `FLUSHDB` / `FLUSHALL` - clear data (implement fully or no-op with warning) +- `COMMAND` / `COMMAND DOCS` / `COMMAND COUNT` - command metadata +- `TIME` - return server time +- `WAIT` - return immediately (no replication) + +## 3. Lua Script Embedding + +### Why Lua Is Required + +BullMQ is the primary driver for Lua support. It ships 51 Lua scripts that must execute atomically. These scripts are the core of BullMQ's reliability guarantees - they ensure that multi-step queue operations (like moving a job from waiting to active while checking rate limits and priorities) happen without race conditions. + +Sidekiq and Celery do not use Lua scripts and can be supported without Lua. However, supporting Lua is essential for BullMQ (and many other Redis-based libraries that rely on atomic scripting). + +### Choosing a Lua Crate + +| Crate | Lua Version | Safety | Performance | Maturity | Recommendation | +|-------|-------------|--------|-------------|----------|----------------| +| **mlua** | 5.1-5.5, LuaJIT, Luau | Safe (IntoLua/FromLua) | Excellent with LuaJIT | Production | **Use this** | +| rlua | 5.3 | Safe | Good | Maintenance mode | Avoid | +| piccolo | ~5.3 | Pure Rust | Unknown | Experimental (v0.3) | Not ready | + +**Use mlua with Lua 5.1 feature** - Redis uses Lua 5.1, and BullMQ scripts are written against Lua 5.1 semantics. Enable LuaJIT for performance when available. + +### Implementation Architecture + +```rust +use mlua::prelude::*; + +struct LuaEngine { + /// Script cache: SHA1 hex -> compiled Lua chunk + scripts: DashMap>, + /// Pool of Lua VMs (one per worker thread) + vms: Vec, +} + +impl LuaEngine { + fn new(num_threads: usize) -> Self { + let vms: Vec = (0..num_threads) + .map(|_| { + let lua = Lua::new_with( + StdLib::TABLE | StdLib::STRING | StdLib::MATH | StdLib::BIT, + LuaOptions::default(), + ).unwrap(); + // Register redis.call() and redis.pcall() + Self::register_redis_api(&lua); + lua + }) + .collect(); + Self { scripts: DashMap::new(), vms } + } +} +``` + +### Implementing redis.call() and redis.pcall() + +The key challenge: Lua scripts call `redis.call("SET", key, value)` which must execute against your server's data structures, not a real Redis. + +```rust +fn register_redis_api(lua: &Lua) { + let redis_table = lua.create_table().unwrap(); + + // redis.call() - raises Lua error on Redis error + let call_fn = lua.create_function(|lua, args: LuaMultiValue| { + let cmd = extract_command(args)?; + match execute_command(&cmd) { + Ok(result) => redis_to_lua(lua, result), + Err(e) => Err(LuaError::RuntimeError(e.to_string())), + } + }).unwrap(); + + // redis.pcall() - returns error as table + let pcall_fn = lua.create_function(|lua, args: LuaMultiValue| { + let cmd = extract_command(args)?; + match execute_command(&cmd) { + Ok(result) => redis_to_lua(lua, result), + Err(e) => { + let err_table = lua.create_table()?; + err_table.set("err", e.to_string())?; + Ok(LuaValue::Table(err_table)) + } + } + }).unwrap(); + + redis_table.set("call", call_fn).unwrap(); + redis_table.set("pcall", pcall_fn).unwrap(); + lua.globals().set("redis", redis_table).unwrap(); +} +``` + +### Type Conversion (Redis <-> Lua) + +Follow the official Redis-Lua type mapping: + +| Redis Type | Lua Type | Notes | +|-----------|----------|-------| +| Integer | number | | +| Bulk String | string | | +| Array | table (1-indexed) | | +| Status Reply | table with `ok` field | `{ok = "OK"}` | +| Error Reply | table with `err` field | `{err = "ERR msg"}` | +| Nil | `false` | Critical - nil becomes false, not nil | + +The reverse (Lua -> Redis): + +| Lua Type | Redis Type | Notes | +|----------|-----------|-------| +| number | Integer | Truncated to i64 | +| string | Bulk String | | +| table (array) | Array | | +| table with `ok` | Status Reply | | +| table with `err` | Error Reply | | +| `false` / `nil` | Null | | +| boolean `true` | Integer 1 | | + +### EVAL/EVALSHA Flow + +``` +EVAL