diff --git a/docs/dir/.index b/docs/dir/.index index d2d376b3..f310fed5 100644 --- a/docs/dir/.index +++ b/docs/dir/.index @@ -3,18 +3,21 @@ nav: - Getting Started: getting-started.md - Features and Usage Scenarios: scenarios.md - Production Deployment: prod-deployment.md + - CLI Guide: directory-cli.md + - MCP Server: directory-mcp.md - Federation: - Running a Federated Directory Instance: partner-prod-federation.md - Profiles: federation-profiles.md - Best Practices and Troubleshooting: federation-troubleshooting.md + - Reference: + - CLI Reference: directory-cli-reference.md + - API Reference: dir-api-reference.md + - SDK Reference: directory-sdk.md + - GUI Application: directory-gui.md + - Records: ads-records.md + - Validation: validation.md - Architecture: architecture.md - Components: - Runtime Discovery: runtime.md - Event Streaming: events.md - - Records: ads-records.md - - Validation: validation.md - Trust Model: trust-model.md - - CLI Reference: directory-cli.md - - GUI Application: directory-gui.md - - SDK Reference: directory-sdk.md - - API Reference: dir-api-reference.md diff --git a/docs/dir/directory-cli-reference.md b/docs/dir/directory-cli-reference.md new file mode 100644 index 00000000..788f85bc --- /dev/null +++ b/docs/dir/directory-cli-reference.md @@ -0,0 +1,649 @@ +# Directory CLI Command Reference + +## Storage Operations + +### `dirctl push ` + +Stores records in the content-addressable store. Has the following features: + +- Supports OASF v1, v2, v3 record formats +- Content-addressable storage with CID generation +- Optional cryptographic signing +- Data integrity validation + +??? example + + ```bash + # Push from file + dirctl push agent-model.json + + # Push from stdin + cat agent-model.json | dirctl push --stdin + + # Push with signature + dirctl push agent-model.json --sign --key private.key + ``` + +### `dirctl pull ` + +Retrieves records by their Content Identifier (CID) or name reference. + +**Supported Reference Formats:** + +| Format | Description | +|--------|-------------| +| `` | Direct lookup by CID | +| `` | Retrieves the latest version | +| `:` | Retrieves the specified version | +| `@` | Hash-verified lookup (fails if resolved CID doesn't match) | +| `:@` | Hash-verified lookup for a specific version | + +??? example + + ```bash + # Pull by CID + dirctl pull baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi + + # Pull by name (latest version) + dirctl pull cisco.com/agent + + # Pull by name with specific version + dirctl pull cisco.com/agent:v1.0.0 + + # Pull with hash verification + dirctl pull cisco.com/agent@bafyreib... + dirctl pull cisco.com/agent:v1.0.0@bafyreib... + + # Pull with signature verification + dirctl pull --signature --public-key public.key + ``` + +**Hash Verification:** + +The `@` suffix enables hash verification. This command fails if the resolved CID doesn't match the expected digest: + +```bash +# Succeeds if cisco.com/agent:v1.0.0 resolves to bafyreib... +dirctl pull cisco.com/agent:v1.0.0@bafyreib... + +# Fails with error if CIDs don't match +dirctl pull cisco.com/agent@wrong-cid +# Error: hash verification failed: resolved CID "bafyreib..." does not match expected digest "wrong-cid" +``` + +**Version Resolution:** + +When no version is specified, commands return the most recently created record (by record's `created_at` field). This allows non-semver tags like `latest`, `dev`, or `stable`. + +### `dirctl delete ` + +Removes records from storage. + +??? example + + ```bash + # Delete a record + dirctl delete baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi + ``` + +### `dirctl info ` + +Displays metadata about stored records using CID or name reference. + +**Supported Reference Formats:** + +| Format | Description | +|--------|-------------| +| `` | Direct lookup by content address | +| `` | Displays the most recently created version | +| `:` | Displays the specified version | +| `@` | Hash-verified lookup | +| `:@` | Hash-verified lookup for a specific version | + +??? example + + ```bash + # Info by CID (existing) + dirctl info baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi + + # Info by name (latest version) + dirctl info cisco.com/agent --output json + + # Info by name with specific version + dirctl info cisco.com/agent:v1.0.0 --output json + ``` + +## Import Operations + +Import records from external registries into DIR. Supports automated batch imports from various registry types. + +### `dirctl import [flags]` + +Fetch and import records from external registries. + +**Supported Registries:** + +- `mcp` - Model Context Protocol registry v0.1 + +**Configuration Options:** + +| Flag | Environment Variable | Description | Required | Default | +|------|---------------------|-------------|----------|---------| +| `--type` | - | Registry type (mcp, a2a) | Yes | - | +| `--url` | - | Registry base URL | Yes | - | +| `--filter` | - | Registry-specific filters (key=value, repeatable) | No | - | +| `--limit` | - | Maximum records to import (0 = no limit) | No | 0 | +| `--dry-run` | - | Preview without importing | No | false | +| `--debug` | - | Enable debug output (shows MCP source and OASF record for failures) | No | false | +| `--force` | - | Force reimport of existing records (skip deduplication) | No | false | +| `--enrich-config` | - | Path to MCPHost configuration file (mcphost.json) | No | importer/enricher/mcphost.json | +| `--enrich-skills-prompt` | - | Optional: path to custom skills prompt template or inline prompt | No | "" (uses default) | +| `--enrich-domains-prompt` | - | Optional: path to custom domains prompt template or inline prompt | No | "" (uses default) | +| `--enrich-rate-limit` | - | Maximum LLM API requests per minute (to avoid rate limit errors) | No | 10 | +| `--sign` | - | Sign records after pushing (uses OIDC by default) | No | false | +| `--key` | - | Path to private key file for signing (requires `--sign`) | No | - | +| `--oidc-token` | - | OIDC token for non-interactive signing (requires `--sign`) | No | - | +| `--fulcio-url` | - | Sigstore Fulcio URL (requires `--sign`) | No | https://fulcio.sigstore.dev | +| `--rekor-url` | - | Sigstore Rekor URL (requires `--sign`) | No | https://rekor.sigstore.dev | +| `--server-addr` | DIRECTORY_CLIENT_SERVER_ADDRESS | DIR server address | No | localhost:8888 | + +!!! note + + By default, the importer performs deduplication: it builds a cache of existing records (by name and version) and skips importing records that already exist. This prevents duplicate imports when running the import command multiple times. Use `--force` to bypass deduplication and reimport existing records. Use `--debug` to see detailed output including which records were skipped and why imports failed. + +??? example + + ```bash + # Import from MCP registry + dirctl import --type=mcp --url=https://registry.modelcontextprotocol.io/v0.1 + + # Import with debug output (shows detailed diagnostics for failures) + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --debug + + # Force reimport of existing records (skips deduplication) + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --force + + # Import with time-based filter + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --filter=updated_since=2025-08-07T13:15:04.280Z + + # Combine multiple filters + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --filter=search=github \ + --filter=version=latest \ + --filter=updated_since=2025-08-07T13:15:04.280Z + + # Limit number of records + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --limit=50 + + # Preview without importing (dry run) + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --dry-run + + # Import and sign records with OIDC (opens browser for authentication) + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --sign + + # Import and sign records with a private key + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --sign \ + --key=/path/to/cosign.key + + # Import with rate limiting for LLM API calls + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --enrich-rate-limit=5 + ``` + +**MCP Registry Filters:** + +For the Model Context Protocol registry, available filters include: + +- `search` - Filter by server name (substring match) +- `version` - Filter by version ('latest' for latest version, or an exact version like '1.2.3') +- `updated_since` - Filter by updated time (RFC3339 datetime format, e.g., '2025-08-07T13:15:04.280Z') + +See the [MCP Registry API docs](https://registry.modelcontextprotocol.io/docs#/operations/list-servers#Query-Parameters) for the complete list of supported filters. + +### LLM-based Enrichment (Mandatory) + +**Enrichment is mandatory** — the import command automatically enriches MCP server records using LLM models to map them to appropriate OASF skills and domains. Records from the oasf-sdk translator are incomplete and require enrichment to be valid. This is powered by [mcphost](https://github.com/mark3labs/mcphost), which provides a Model Context Protocol (MCP) host that can run AI models with tool-calling capabilities. + +**Requirements:** + +- `dirctl` binary (includes the built-in MCP server with `agntcy_oasf_get_schema_skills` and `agntcy_oasf_get_schema_domains` tools) +- An LLM model with tool-calling support (GPT-4o, Claude, or compatible Ollama models) +- The `mcphost.json` configuration file must include the `dir-mcp-server` entry that runs `dirctl mcp serve`. This MCP server provides the schema tools needed for enrichment. + +**How it works:** + +1. The enricher starts an MCP server using `dirctl mcp serve` +2. The LLM uses the `agntcy_oasf_get_schema_skills` tool to browse available OASF skills +3. The LLM uses the `agntcy_oasf_get_schema_domains` tool to browse available OASF domains +4. Based on the MCP server description and capabilities, the LLM selects appropriate skills and domains +5. Selected skills and domains replace the defaults in the imported records + +**Setting up mcphost:** + +Edit a configuration file (default: `importer/enricher/mcphost.json`): + +```json +{ + "mcpServers": { + "dir-mcp-server": { + "command": "dirctl", + "args": ["mcp", "serve"], + "env": { + "OASF_API_VALIDATION_SCHEMA_URL": "https://schema.oasf.outshift.com" + } + } + }, + "model": "azure:gpt-4o", + "max-tokens": 4096, + "max-steps": 20 +} +``` + +**Recommended LLM providers:** + +- `azure:gpt-4o` - Azure OpenAI GPT-4o (recommended for speed and accuracy) +- `ollama:qwen3:8b` - Local Qwen3 via Ollama + +**Environment variables for LLM providers:** + +- Azure OpenAI: `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT` + +**Customizing Enrichment Prompts:** + +The enricher uses separate default prompt templates for skills and domains. You can customize these prompts for specific use cases: + +- **Skills**: Use default (omit `--enrich-skills-prompt`), or `--enrich-skills-prompt=/path/to/custom-skills-prompt.md`, or `--enrich-skills-prompt="Your custom prompt text..."` +- **Domains**: Use default (omit `--enrich-domains-prompt`), or `--enrich-domains-prompt=/path/to/custom-domains-prompt.md`, or `--enrich-domains-prompt="Your custom prompt text..."` + +The default prompt templates are available at `importer/enricher/enricher.skills.prompt.md` and `importer/enricher/enricher.domains.prompt.md`. + +??? example "Import with custom enrichment" + + ```bash + # Import with custom mcphost configuration + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --enrich-config=/path/to/custom-mcphost.json + + # Import with custom prompt templates + dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --enrich-skills-prompt=/path/to/custom-skills-prompt.md \ + --enrich-domains-prompt=/path/to/custom-domains-prompt.md \ + --debug + ``` + +### Signing Records During Import + +Records can be signed during import using the `--sign` flag. Signing options work the same as the standalone `dirctl sign` command (see [Security & Verification](#security-verification)). + +```bash +# Sign with OIDC (opens browser) +dirctl import --type=mcp --url=https://registry.modelcontextprotocol.io/v0.1 --sign + +# Sign with a private key +dirctl import --type=mcp --url=https://registry.modelcontextprotocol.io/v0.1 --sign --key=/path/to/cosign.key +``` + +### Rate Limiting for LLM API Calls + +When importing large batches of records, the enrichment process makes LLM API calls for each record. To avoid hitting rate limits from LLM providers, use the `--enrich-rate-limit` flag: + +```bash +# Import with reduced rate limit (5 requests per minute) +dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --enrich-rate-limit=5 + +# Import with higher rate limit for providers with generous limits +dirctl import --type=mcp \ + --url=https://registry.modelcontextprotocol.io/v0.1 \ + --enrich-rate-limit=30 +``` + +## Routing Operations + +The routing commands manage record announcement and discovery across the peer-to-peer network. + +### `dirctl routing publish ` + +Announces records to the network for discovery by other peers. The command does the following: + +- Announces record to DHT network. +- Makes record discoverable by other peers. +- Stores routing metadata locally. +- Enables network-wide discovery. + +??? example + + ```bash + # Publish a record to the network + dirctl routing publish baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi + ``` + +### `dirctl routing unpublish ` + +Removes records from network discovery while keeping them in local storage. The command does the following: + +- Removes DHT announcements. +- Stops network discovery. +- Keeps record in local storage. +- Cleans up routing metadata. + +??? example + + ```bash + # Remove from network discovery + dirctl routing unpublish baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi + ``` + +### `dirctl routing list [flags]` + +Queries local published records with optional filtering. + +The following flags are available: + +- `--skill ` - Filter by skill (repeatable) +- `--locator ` - Filter by locator type (repeatable) +- `--domain ` - Filter by domain (repeatable) +- `--module ` - Filter by module name (repeatable) +- `--cid ` - List specific record by CID +- `--limit ` - Limit number of results + +??? example + + ```bash + # List all local published records + dirctl routing list + + # List by skill + dirctl routing list --skill "AI" + dirctl routing list --skill "Natural Language Processing" + + # List by locator type + dirctl routing list --locator "docker-image" + + # List by module + dirctl routing list --module "runtime/framework" + + # Multiple criteria (AND logic) + dirctl routing list --skill "AI" --locator "docker-image" + dirctl routing list --domain "healthcare" --module "runtime/language" + + # Specific record by CID + dirctl routing list --cid baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi + + # Limit results + dirctl routing list --skill "AI" --limit 5 + ``` + +### `dirctl routing search [flags]` + +Discovers records from other peers across the network. + +The following flags are available: + +- `--skill ` - Search by skill (repeatable) +- `--locator ` - Search by locator type (repeatable) +- `--domain ` - Search by domain (repeatable) +- `--module ` - Search by module name (repeatable) +- `--limit ` - Maximum results to return +- `--min-score ` - Minimum match score threshold + +The output includes the following: + +- Record CID and provider peer information +- Match score showing query relevance +- Specific queries that matched +- Peer connection details + +??? example + + ```bash + # Search for AI records across the network + dirctl routing search --skill "AI" + + # Search with multiple criteria + dirctl routing search --skill "AI" --skill "ML" --min-score 2 + + # Search by locator type + dirctl routing search --locator "docker-image" + + # Search by module + dirctl routing search --module "runtime/framework" + + # Advanced search with scoring + dirctl routing search --skill "web-development" --limit 10 --min-score 1 + dirctl routing search --domain "finance" --module "validation" --min-score 2 + ``` + +**Output includes:** + +### `dirctl routing info` + +Shows routing statistics and summary information. + +The output includes the following: + +- Total published records count +- Skills distribution with counts +- Locators distribution with counts +- Helpful usage tips + +??? example + + ```bash + # Show local routing statistics + dirctl routing info + ``` + +## Search & Discovery + +### `dirctl search [flags]` + +General content search across all records using the search service. + +The following flags are available: + +- `--query ` - Search criteria (repeatable) +- `--limit ` - Maximum results +- `--offset ` - Result offset for pagination + +??? example + + ```bash + # Search by record name + dirctl search --query "name=my-agent" + + # Search by version + dirctl search --query "version=v1.0.0" + + # Search by skill ID + dirctl search --query "skill-id=10201" + + # Complex search with multiple criteria + dirctl search --limit 10 --offset 0 \ + --query "name=my-agent" \ + --query "skill-name=Text Completion" \ + --query "locator=docker-image:https://example.com/image" + ``` + +## Security & Verification + +### Name Verification + +Record name verification proves that the signing key is authorized by the domain claimed in the record's name field. + +**Requirements:** + +- Record name must include a protocol prefix: `https://domain/path` or `http://domain/path` +- A JWKS file must be hosted at `:///.well-known/jwks.json` +- The record must be signed with the private key corresponding to a public key present in that JWKS file + +**Workflow:** + +1. Push a record with a verifiable name. + + ```bash + dirctl push record.json --output raw + # Returns: bafyreib... + ``` + +2. Sign the record (triggers automatic verification). + + ```bash + dirctl sign --key private.key + ``` + +3. Check verification status using [`dirctl naming verify`](./directory-cli.md#dirctl-naming-verify-reference). + +### `dirctl sign [flags]` + +Signs records for integrity and authenticity. When signing a record with a verifiable name (e.g., `https://domain/path`), the system automatically attempts to verify domain authorization via JWKS. See [Name Verification](#name-verification) for details. + +??? example + + ```bash + # Sign with private key + dirctl sign --key private.key + + # Sign with OIDC (keyless signing) + dirctl sign --oidc --fulcio-url https://fulcio.example.com + ``` + +### `dirctl naming verify ` + +Verifies that a record's signing key is authorized by the domain claimed in its name field. Checks if the signing key matches a public key in the domain's JWKS file hosted at `/.well-known/jwks.json`. + +**Supported Reference Formats:** + +| Format | Description | +|--------|-------------| +| `` | Verify by content address | +| `` | Verify the most recently created version | +| `:` | Verify a specific version | + +??? example + + ```bash + # Verify by CID + dirctl naming verify bafyreib... --output json + + # Verify by name (latest version) + dirctl naming verify cisco.com/agent --output json + + # Verify by name with specific version + dirctl naming verify cisco.com/agent:v1.0.0 --output json + ``` + + Example verification response: + + ```json + { + "cid": "bafyreib...", + "verified": true, + "domain": "cisco.com", + "method": "jwks", + "key_id": "key-1", + "verified_at": "2026-01-21T10:30:00Z" + } + ``` + +### `dirctl verify [flags]` + +Verifies record signatures. + +??? example + + ```bash + # Verify with public key + dirctl verify record.json signature.sig --key public.key + ``` + +### `dirctl validate [] [flags]` + +Validates OASF record JSON from a file or stdin against the OASF schema. The JSON can be provided as a file path or piped from stdin (e.g., from `dirctl pull`). A schema URL must be provided via `--url` for API-based validation. + +| Flag | Description | +|------|-------------| +| `--url ` | OASF schema URL for API-based validation (required) | + +??? example + + ```bash + # Validate a file with API-based validation + dirctl validate record.json --url https://schema.oasf.outshift.com + + # Validate JSON piped from stdin + cat record.json | dirctl validate --url https://schema.oasf.outshift.com + + # Validate a record pulled from directory + dirctl pull --output json | dirctl validate --url https://schema.oasf.outshift.com + + # Validate all records (using shell scripting) + for cid in $(dirctl search --output jsonl | jq -r '.record_cid'); do + dirctl pull "$cid" | dirctl validate --url https://schema.oasf.outshift.com + done + ``` + +## Synchronization + +### `dirctl sync create ` + +Creates peer-to-peer synchronization. + +??? example + + ```bash + # Create sync with remote peer + dirctl sync create https://peer.example.com + ``` + +### `dirctl sync list` + +Lists active synchronizations. + +??? example + + ```bash + # Show all active syncs + dirctl sync list + ``` + +### `dirctl sync status ` + +Checks synchronization status. + +??? example + + ```bash + # Check specific sync status + dirctl sync status abc123-def456-ghi789 + ``` + +### `dirctl sync delete ` + +Removes synchronization. + +??? example + + ```bash + # Delete a sync + dirctl sync delete abc123-def456-ghi789 + ``` diff --git a/docs/dir/directory-cli.md b/docs/dir/directory-cli.md index 82af9e15..5dc6cfd9 100644 --- a/docs/dir/directory-cli.md +++ b/docs/dir/directory-cli.md @@ -1,1020 +1,87 @@ -# Directory CLI +# Directory CLI Guide -The Directory CLI (dirctl) provides comprehensive command-line tools for interacting with the Directory system, including storage, routing, search, and security operations. +The Directory CLI (`dirctl`) provides comprehensive command-line tools for interacting with the Directory system, including storage, routing, search, and security operations. -## Installation - -The Directory CLI can be installed in the following ways: - -### Using Homebrew - -```bash -brew tap agntcy/dir https://github.com/agntcy/dir/ -brew install dirctl -``` - -### Using Release Binaries - -```bash -# Download from GitHub Releases -curl -L https://github.com/agntcy/dir/releases/latest/download/dirctl-linux-amd64 -o dirctl -chmod +x dirctl -sudo mv dirctl /usr/local/bin/ -``` - -### Using Source - -```bash -git clone https://github.com/agntcy/dir -cd dir -task build-dirctl -``` - -### Using Container Image - -```bash -docker pull ghcr.io/agntcy/dir-ctl:latest -docker run --rm ghcr.io/agntcy/dir-ctl:latest --help -``` - -## Quick Start - -The following example demonstrates how to store, publish, search, and retrieve a record using the Directory CLI: - -1. Store a record - - ```bash - dirctl push my-agent.json - ``` - - Returns: `baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi` - -1. Publish for network discovery - - ```bash - dirctl routing publish baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi - ``` - -1. Search for records - - ```bash - dirctl routing search --skill "AI" --limit 10 - ``` - -1. Retrieve a record - - ```bash - # Pull by CID - dirctl pull baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi - - # Or pull by name (if the record has a verifiable name) - dirctl pull cisco.com/agent:v1.0.0 - ``` - -!!! note "Name-based References" - - The CLI supports Docker-style name references in addition to CIDs. Records can be pulled using formats like `name`, `name:version`, or `name:version@cid` for hash-verified lookups. See [Name Verification](#name-verification) for details. - -!!! note "Authentication for federation" - - When accessing Directory federation nodes, authenticate first with `dirctl auth login`. See [Authentication](#authentication) for details. - -## Directory MCP Server - -The Directory MCP Server provides a standardized interface for AI assistants and tools to interact with the AGNTCY Agent Directory and work with OASF agent records. - -The Directory MCP Server exposes Directory functionality through MCP, allowing AI assistants to: - -- Work with OASF schemas and validate agent records. -- Search and discover agent records from the Directory. -- Push and pull records to/from Directory servers. -- Navigate OASF skill and domain taxonomies. -- Generate agent records automatically from codebases. - -The MCP server runs via the `dirctl` CLI tool and acts as a bridge between AI development environments and the Directory infrastructure, making it easier to work with agent metadata in your development workflow. - -### Configuration - -**Binary Configuration:** - -Add the MCP server to your IDE's MCP configuration using the absolute path to the `dirctl` binary. - -**Example Cursor configuration (`~/.cursor/mcp.json`):** - -```json -{ - "mcpServers": { - "dir-mcp-server": { - "command": "/absolute/path/to/dirctl", - "args": ["mcp", "serve"] - } - } -} -``` - -**Docker Configuration:** - -Add the MCP server to your IDE's MCP configuration using Docker. - -??? example "Example Cursor configuration (`~/.cursor/mcp.json`)" - - ```json - { - "mcpServers": { - "dir-mcp-server": { - "command": "docker", - "args": [ - "run", - "--rm", - "-i", - "ghcr.io/agntcy/dir-ctl:latest", - "mcp", - "serve" - ] - } - } - } - ``` - -**Environment Variables:** - -Configure the MCP server behavior using environment variables: - -- `DIRECTORY_CLIENT_SERVER_ADDRESS` - Directory server address (default: `0.0.0.0:8888`) -- `DIRECTORY_CLIENT_AUTH_MODE` - Authentication mode: `none`, `x509`, `jwt`, `token` -- `DIRECTORY_CLIENT_SPIFFE_TOKEN` - Path to SPIFFE token file (for token authentication) -- `DIRECTORY_CLIENT_TLS_SKIP_VERIFY` - Skip TLS verification (set to `true` if needed) - -??? example "Example with environment variables" - - ```json - { - "mcpServers": { - "dir-mcp-server": { - "command": "/usr/local/bin/dirctl", - "args": ["mcp", "serve"], - "env": { - "DIRECTORY_CLIENT_SERVER_ADDRESS": "dir.example.com:8888", - "DIRECTORY_CLIENT_AUTH_MODE": "none", - "DIRECTORY_CLIENT_TLS_SKIP_VERIFY": "false" - } - } - } - } - ``` - -### Directory MCP Server Tools - -Using the Directory MCP Server, you can access the following tools: - -- `agntcy_oasf_list_versions` - Lists all available OASF schema versions supported by the server. -- `agntcy_oasf_get_schema` - Retrieves the complete OASF schema JSON content for the specified version. -- `agntcy_oasf_get_schema_skills` - Retrieves skills from the OASF schema with hierarchical navigation support. -- `agntcy_oasf_get_schema_domains` - Retrieves domains from the OASF schema with hierarchical navigation support. -- `agntcy_oasf_validate_record` - Validates an OASF agent record against the OASF schema. -- `agntcy_dir_push_record` - Pushes an OASF agent record to a Directory server. -- `agntcy_dir_pull_record` - Pulls an OASF agent record from the local Directory node by its CID (Content Identifier). -- `agntcy_dir_search_local` - Searches for agent records on the local directory node using structured query filters. -- `agntcy_dir_verify_name` - Verifies that a record's name is owned by the domain it claims (by CID or name; optional version). Checks that the record was signed with a key published in the domain's well-known JWKS file. - -For a full list of tools and usage examples, see the [Directory MCP Server documentation](https://github.com/agntcy/dir/blob/main/mcp/README.md). - -## Output Formats - -All `dirctl` commands support multiple output formats via the `--output` (or `-o`) flag, making it easy to switch between human-readable output and machine-processable formats. +This guide provides an overview of how to get started with the CLI, how to use its features, as well as common workflows and usage examples. -### Available Formats - -| Format | Description | Use Case | -|--------|-------------|----------| -| `human` | Human-readable, formatted output with colors and tables (default) | Interactive terminal use | -| `json` | Pretty-printed JSON with indentation | Debugging, single-record processing | -| `jsonl` | Newline-delimited JSON (compact, one object per line) | Streaming, batch processing, logging | -| `raw` | Raw values only (e.g., CIDs, IDs) | Shell scripting, piping to other commands | - -### Usage - -```bash -# Human-readable output (default) -dirctl routing list +For detailed reference information, see the [Directory CLI Reference](directory-cli-reference.md). -# JSON output (pretty-printed) -dirctl routing list --output json -dirctl routing list -o json - -# JSONL output (streaming-friendly) -dirctl events listen --output jsonl - -# Raw output (just values) -dirctl push my-agent.json --output raw -``` - -### Piping and Processing - -Structured formats (`json`, `jsonl`, `raw`) automatically route data to **stdout** and metadata messages to **stderr**, enabling clean piping to tools like `jq`: - -```bash -# Process JSON output with jq -dirctl routing search --skill "AI" -o json | jq '.[] | .cid' - -# Stream events and filter by type -dirctl events listen -o jsonl | jq -c 'select(.type == "EVENT_TYPE_RECORD_PUSHED")' - -# Capture CID for scripting -CID=$(dirctl push my-agent.json -o raw) -echo "Stored with CID: $CID" - -# Chain commands -dirctl routing list -o json | jq -r '.[].cid' | xargs -I {} dirctl pull {} -``` - -### Format Selection Guidelines - -- **`human`**: Default for terminal interaction, provides context and formatting -- **`json`**: Best for debugging or when you need readable structured data -- **`jsonl`**: Ideal for streaming events, logs, or processing large result sets line-by-line -- **`raw`**: Perfect for shell scripts and command chaining where you only need the value - -## Command Reference - -### Storage Operations - -#### `dirctl push ` - -Stores records in the content-addressable store. Has the following features: - -- Supports OASF v1, v2, v3 record formats -- Content-addressable storage with CID generation -- Optional cryptographic signing -- Data integrity validation - -??? example - - ```bash - # Push from file - dirctl push agent-model.json - - # Push from stdin - cat agent-model.json | dirctl push --stdin - - # Push with signature - dirctl push agent-model.json --sign --key private.key - ``` - -#### `dirctl pull ` - -Retrieves records by their Content Identifier (CID) or name reference. - -**Supported Reference Formats:** - -| Format | Description | -|--------|-------------| -| `` | Direct lookup by CID | -| `` | Retrieves the latest version | -| `:` | Retrieves the specified version | -| `@` | Hash-verified lookup (fails if resolved CID doesn't match) | -| `:@` | Hash-verified lookup for a specific version | - -??? example - - ```bash - # Pull by CID - dirctl pull baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi - - # Pull by name (latest version) - dirctl pull cisco.com/agent - - # Pull by name with specific version - dirctl pull cisco.com/agent:v1.0.0 - - # Pull with hash verification - dirctl pull cisco.com/agent@bafyreib... - dirctl pull cisco.com/agent:v1.0.0@bafyreib... - - # Pull with signature verification - dirctl pull --signature --public-key public.key - ``` - -**Hash Verification:** - -The `@` suffix enables hash verification. This command fails if the resolved CID doesn't match the expected digest: - -```bash -# Succeeds if cisco.com/agent:v1.0.0 resolves to bafyreib... -dirctl pull cisco.com/agent:v1.0.0@bafyreib... - -# Fails with error if CIDs don't match -dirctl pull cisco.com/agent@wrong-cid -# Error: hash verification failed: resolved CID "bafyreib..." does not match expected digest "wrong-cid" -``` - -**Version Resolution:** - -When no version is specified, commands return the most recently created record (by record's `created_at` field). This allows non-semver tags like `latest`, `dev`, or `stable`. - -#### `dirctl delete ` - -Removes records from storage. - -??? example - - ```bash - # Delete a record - dirctl delete baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi - ``` - -#### `dirctl info ` - -Displays metadata about stored records using CID or name reference. - -**Supported Reference Formats:** - -| Format | Description | -|--------|-------------| -| `` | Direct lookup by content address | -| `` | Displays the most recently created version | -| `:` | Displays the specified version | -| `@` | Hash-verified lookup | -| `:@` | Hash-verified lookup for a specific version | - -??? example - - ```bash - # Info by CID (existing) - dirctl info baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi - - # Info by name (latest version) - dirctl info cisco.com/agent --output json - - # Info by name with specific version - dirctl info cisco.com/agent:v1.0.0 --output json - ``` - -### Import Operations - -Import records from external registries into DIR. Supports automated batch imports from various registry types. - -#### `dirctl import [flags]` - -Fetch and import records from external registries. - -**Supported Registries:** - -- `mcp` - Model Context Protocol registry v0.1 - -**Configuration Options:** - -| Flag | Environment Variable | Description | Required | Default | -|------|---------------------|-------------|----------|---------| -| `--type` | - | Registry type (mcp, a2a) | Yes | - | -| `--url` | - | Registry base URL | Yes | - | -| `--filter` | - | Registry-specific filters (key=value, repeatable) | No | - | -| `--limit` | - | Maximum records to import (0 = no limit) | No | 0 | -| `--dry-run` | - | Preview without importing | No | false | -| `--debug` | - | Enable debug output (shows MCP source and OASF record for failures) | No | false | -| `--force` | - | Force reimport of existing records (skip deduplication) | No | false | -| `--enrich-config` | - | Path to MCPHost configuration file (mcphost.json) | No | importer/enricher/mcphost.json | -| `--enrich-skills-prompt` | - | Optional: path to custom skills prompt template or inline prompt | No | "" (uses default) | -| `--enrich-domains-prompt` | - | Optional: path to custom domains prompt template or inline prompt | No | "" (uses default) | -| `--enrich-rate-limit` | - | Maximum LLM API requests per minute (to avoid rate limit errors) | No | 10 | -| `--sign` | - | Sign records after pushing (uses OIDC by default) | No | false | -| `--key` | - | Path to private key file for signing (requires `--sign`) | No | - | -| `--oidc-token` | - | OIDC token for non-interactive signing (requires `--sign`) | No | - | -| `--fulcio-url` | - | Sigstore Fulcio URL (requires `--sign`) | No | https://fulcio.sigstore.dev | -| `--rekor-url` | - | Sigstore Rekor URL (requires `--sign`) | No | https://rekor.sigstore.dev | -| `--server-addr` | DIRECTORY_CLIENT_SERVER_ADDRESS | DIR server address | No | localhost:8888 | - -!!! note - - By default, the importer performs deduplication: it builds a cache of existing records (by name and version) and skips importing records that already exist. This prevents duplicate imports when running the import command multiple times. Use `--force` to bypass deduplication and reimport existing records. Use `--debug` to see detailed output including which records were skipped and why imports failed. - -??? example - - ```bash - # Import from MCP registry - dirctl import --type=mcp --url=https://registry.modelcontextprotocol.io/v0.1 - - # Import with debug output (shows detailed diagnostics for failures) - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --debug - - # Force reimport of existing records (skips deduplication) - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --force - - # Import with time-based filter - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --filter=updated_since=2025-08-07T13:15:04.280Z - - # Combine multiple filters - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --filter=search=github \ - --filter=version=latest \ - --filter=updated_since=2025-08-07T13:15:04.280Z - - # Limit number of records - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --limit=50 - - # Preview without importing (dry run) - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --dry-run - - # Import and sign records with OIDC (opens browser for authentication) - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --sign - - # Import and sign records with a private key - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --sign \ - --key=/path/to/cosign.key - - # Import with rate limiting for LLM API calls - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --enrich-rate-limit=5 - ``` - -**MCP Registry Filters:** - -For the Model Context Protocol registry, available filters include: - -- `search` - Filter by server name (substring match) -- `version` - Filter by version ('latest' for latest version, or an exact version like '1.2.3') -- `updated_since` - Filter by updated time (RFC3339 datetime format, e.g., '2025-08-07T13:15:04.280Z') - -See the [MCP Registry API docs](https://registry.modelcontextprotocol.io/docs#/operations/list-servers#Query-Parameters) for the complete list of supported filters. - -#### LLM-based Enrichment (Mandatory) - -**Enrichment is mandatory** — the import command automatically enriches MCP server records using LLM models to map them to appropriate OASF skills and domains. Records from the oasf-sdk translator are incomplete and require enrichment to be valid. This is powered by [mcphost](https://github.com/mark3labs/mcphost), which provides a Model Context Protocol (MCP) host that can run AI models with tool-calling capabilities. - -**Requirements:** - -- `dirctl` binary (includes the built-in MCP server with `agntcy_oasf_get_schema_skills` and `agntcy_oasf_get_schema_domains` tools) -- An LLM model with tool-calling support (GPT-4o, Claude, or compatible Ollama models) -- The `mcphost.json` configuration file must include the `dir-mcp-server` entry that runs `dirctl mcp serve`. This MCP server provides the schema tools needed for enrichment. - -**How it works:** - -1. The enricher starts an MCP server using `dirctl mcp serve` -2. The LLM uses the `agntcy_oasf_get_schema_skills` tool to browse available OASF skills -3. The LLM uses the `agntcy_oasf_get_schema_domains` tool to browse available OASF domains -4. Based on the MCP server description and capabilities, the LLM selects appropriate skills and domains -5. Selected skills and domains replace the defaults in the imported records - -**Setting up mcphost:** - -Edit a configuration file (default: `importer/enricher/mcphost.json`): - -```json -{ - "mcpServers": { - "dir-mcp-server": { - "command": "dirctl", - "args": ["mcp", "serve"], - "env": { - "OASF_API_VALIDATION_SCHEMA_URL": "https://schema.oasf.outshift.com" - } - } - }, - "model": "azure:gpt-4o", - "max-tokens": 4096, - "max-steps": 20 -} -``` - -**Recommended LLM providers:** - -- `azure:gpt-4o` - Azure OpenAI GPT-4o (recommended for speed and accuracy) -- `ollama:qwen3:8b` - Local Qwen3 via Ollama - -**Environment variables for LLM providers:** - -- Azure OpenAI: `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT` - -**Customizing Enrichment Prompts:** - -The enricher uses separate default prompt templates for skills and domains. You can customize these prompts for specific use cases: - -- **Skills**: Use default (omit `--enrich-skills-prompt`), or `--enrich-skills-prompt=/path/to/custom-skills-prompt.md`, or `--enrich-skills-prompt="Your custom prompt text..."` -- **Domains**: Use default (omit `--enrich-domains-prompt`), or `--enrich-domains-prompt=/path/to/custom-domains-prompt.md`, or `--enrich-domains-prompt="Your custom prompt text..."` - -The default prompt templates are available at `importer/enricher/enricher.skills.prompt.md` and `importer/enricher/enricher.domains.prompt.md`. - -??? example "Import with custom enrichment" - - ```bash - # Import with custom mcphost configuration - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --enrich-config=/path/to/custom-mcphost.json - - # Import with custom prompt templates - dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --enrich-skills-prompt=/path/to/custom-skills-prompt.md \ - --enrich-domains-prompt=/path/to/custom-domains-prompt.md \ - --debug - ``` - -#### Signing Records During Import - -Records can be signed during import using the `--sign` flag. Signing options work the same as the standalone `dirctl sign` command (see [Security & Verification](#security-verification)). - -```bash -# Sign with OIDC (opens browser) -dirctl import --type=mcp --url=https://registry.modelcontextprotocol.io/v0.1 --sign - -# Sign with a private key -dirctl import --type=mcp --url=https://registry.modelcontextprotocol.io/v0.1 --sign --key=/path/to/cosign.key -``` - -#### Rate Limiting for LLM API Calls - -When importing large batches of records, the enrichment process makes LLM API calls for each record. To avoid hitting rate limits from LLM providers, use the `--enrich-rate-limit` flag: - -```bash -# Import with reduced rate limit (5 requests per minute) -dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --enrich-rate-limit=5 - -# Import with higher rate limit for providers with generous limits -dirctl import --type=mcp \ - --url=https://registry.modelcontextprotocol.io/v0.1 \ - --enrich-rate-limit=30 -``` - -### Routing Operations - -The routing commands manage record announcement and discovery across the peer-to-peer network. - -#### `dirctl routing publish ` - -Announces records to the network for discovery by other peers. The command does the following: - -- Announces record to DHT network. -- Makes record discoverable by other peers. -- Stores routing metadata locally. -- Enables network-wide discovery. - -??? example - - ```bash - # Publish a record to the network - dirctl routing publish baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi - ``` - -#### `dirctl routing unpublish ` - -Removes records from network discovery while keeping them in local storage. The command does the following: - -- Removes DHT announcements. -- Stops network discovery. -- Keeps record in local storage. -- Cleans up routing metadata. - -??? example - - ```bash - # Remove from network discovery - dirctl routing unpublish baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi - ``` - -#### `dirctl routing list [flags]` - -Queries local published records with optional filtering. - -The following flags are available: - -- `--skill ` - Filter by skill (repeatable) -- `--locator ` - Filter by locator type (repeatable) -- `--domain ` - Filter by domain (repeatable) -- `--module ` - Filter by module name (repeatable) -- `--cid ` - List specific record by CID -- `--limit ` - Limit number of results - -??? example - - ```bash - # List all local published records - dirctl routing list - - # List by skill - dirctl routing list --skill "AI" - dirctl routing list --skill "Natural Language Processing" - - # List by locator type - dirctl routing list --locator "docker-image" - - # List by module - dirctl routing list --module "runtime/framework" - - # Multiple criteria (AND logic) - dirctl routing list --skill "AI" --locator "docker-image" - dirctl routing list --domain "healthcare" --module "runtime/language" - - # Specific record by CID - dirctl routing list --cid baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi - - # Limit results - dirctl routing list --skill "AI" --limit 5 - ``` - -#### `dirctl routing search [flags]` - -Discovers records from other peers across the network. - -The following flags are available: - -- `--skill ` - Search by skill (repeatable) -- `--locator ` - Search by locator type (repeatable) -- `--domain ` - Search by domain (repeatable) -- `--module ` - Search by module name (repeatable) -- `--limit ` - Maximum results to return -- `--min-score ` - Minimum match score threshold - -The output includes the following: - -- Record CID and provider peer information -- Match score showing query relevance -- Specific queries that matched -- Peer connection details - -??? example - - ```bash - # Search for AI records across the network - dirctl routing search --skill "AI" - - # Search with multiple criteria - dirctl routing search --skill "AI" --skill "ML" --min-score 2 - - # Search by locator type - dirctl routing search --locator "docker-image" - - # Search by module - dirctl routing search --module "runtime/framework" - - # Advanced search with scoring - dirctl routing search --skill "web-development" --limit 10 --min-score 1 - dirctl routing search --domain "finance" --module "validation" --min-score 2 - ``` - -**Output includes:** - -#### `dirctl routing info` - -Shows routing statistics and summary information. - -The output includes the following: - -- Total published records count -- Skills distribution with counts -- Locators distribution with counts -- Helpful usage tips - -??? example - - ```bash - # Show local routing statistics - dirctl routing info - ``` - -### Search & Discovery - -#### `dirctl search [flags]` - -General content search across all records using the search service. - -The following flags are available: - -- `--query ` - Search criteria (repeatable) -- `--limit ` - Maximum results -- `--offset ` - Result offset for pagination - -??? example - - ```bash - # Search by record name - dirctl search --query "name=my-agent" - - # Search by version - dirctl search --query "version=v1.0.0" - - # Search by skill ID - dirctl search --query "skill-id=10201" - - # Complex search with multiple criteria - dirctl search --limit 10 --offset 0 \ - --query "name=my-agent" \ - --query "skill-name=Text Completion" \ - --query "locator=docker-image:https://example.com/image" - ``` - -### Security & Verification - -#### Name Verification - -Record name verification proves that the signing key is authorized by the domain claimed in the record's name field. - -**Requirements:** - -- Record name must include a protocol prefix: `https://domain/path` or `http://domain/path` -- A JWKS file must be hosted at `:///.well-known/jwks.json` -- The record must be signed with the private key corresponding to a public key present in that JWKS file - -**Workflow:** - -1. Push a record with a verifiable name. - - ```bash - dirctl push record.json --output raw - # Returns: bafyreib... - ``` - -2. Sign the record (triggers automatic verification). - - ```bash - dirctl sign --key private.key - ``` - -3. Check verification status using [`dirctl naming verify`](./directory-cli.md#dirctl-naming-verify-reference). - -#### `dirctl sign [flags]` - -Signs records for integrity and authenticity. When signing a record with a verifiable name (e.g., `https://domain/path`), the system automatically attempts to verify domain authorization via JWKS. See [Name Verification](#name-verification) for details. - -??? example - - ```bash - # Sign with private key - dirctl sign --key private.key - - # Sign with OIDC (keyless signing) - dirctl sign --oidc --fulcio-url https://fulcio.example.com - ``` - -#### `dirctl naming verify ` - -Verifies that a record's signing key is authorized by the domain claimed in its name field. Checks if the signing key matches a public key in the domain's JWKS file hosted at `/.well-known/jwks.json`. - -**Supported Reference Formats:** - -| Format | Description | -|--------|-------------| -| `` | Verify by content address | -| `` | Verify the most recently created version | -| `:` | Verify a specific version | - -??? example - - ```bash - # Verify by CID - dirctl naming verify bafyreib... --output json - - # Verify by name (latest version) - dirctl naming verify cisco.com/agent --output json - - # Verify by name with specific version - dirctl naming verify cisco.com/agent:v1.0.0 --output json - ``` - - Example verification response: - - ```json - { - "cid": "bafyreib...", - "verified": true, - "domain": "cisco.com", - "method": "jwks", - "key_id": "key-1", - "verified_at": "2026-01-21T10:30:00Z" - } - ``` - -#### `dirctl verify [flags]` - -Verifies record signatures. - -??? example - - ```bash - # Verify with public key - dirctl verify record.json signature.sig --key public.key - ``` - -#### `dirctl validate [] [flags]` - -Validates OASF record JSON from a file or stdin against the OASF schema. The JSON can be provided as a file path or piped from stdin (e.g., from `dirctl pull`). A schema URL must be provided via `--url` for API-based validation. - -| Flag | Description | -|------|-------------| -| `--url ` | OASF schema URL for API-based validation (required) | - -??? example - - ```bash - # Validate a file with API-based validation - dirctl validate record.json --url https://schema.oasf.outshift.com - - # Validate JSON piped from stdin - cat record.json | dirctl validate --url https://schema.oasf.outshift.com - - # Validate a record pulled from directory - dirctl pull --output json | dirctl validate --url https://schema.oasf.outshift.com - - # Validate all records (using shell scripting) - for cid in $(dirctl search --output jsonl | jq -r '.record_cid'); do - dirctl pull "$cid" | dirctl validate --url https://schema.oasf.outshift.com - done - ``` - -### Synchronization - -#### `dirctl sync create ` - -Creates peer-to-peer synchronization. - -??? example - - ```bash - # Create sync with remote peer - dirctl sync create https://peer.example.com - ``` - -#### `dirctl sync list` +## Installation -Lists active synchronizations. +The Directory CLI can be installed in the following ways: -??? example +=== "Using Homebrew" ```bash - # Show all active syncs - dirctl sync list + brew tap agntcy/dir https://github.com/agntcy/dir/ + brew install dirctl ``` -#### `dirctl sync status ` - -Checks synchronization status. - -??? example +=== "Using Release Binaries" ```bash - # Check specific sync status - dirctl sync status abc123-def456-ghi789 + # Download from GitHub Releases + curl -L https://github.com/agntcy/dir/releases/latest/download/dirctl-linux-amd64 -o dirctl + chmod +x dirctl + sudo mv dirctl /usr/local/bin/ ``` -#### `dirctl sync delete ` - -Removes synchronization. - -??? example +=== "Using Source" ```bash - # Delete a sync - dirctl sync delete abc123-def456-ghi789 + git clone https://github.com/agntcy/dir + cd dir + task build-dirctl ``` -## Authentication - -Authentication is required when accessing Directory federation nodes. The CLI supports multiple authentication modes, with GitHub OAuth recommended for interactive use. - -| Command | Description | -|---------|-------------| -| `dirctl auth login` | Authenticate with GitHub | -| `dirctl auth logout` | Clear cached authentication credentials | -| `dirctl auth status` | Show current authentication status | - -### GitHub OAuth Authentication - -GitHub OAuth (Device Flow) enables secure, interactive authentication for accessing federation nodes. - -#### `dirctl auth login` - -Authenticate with GitHub using the OAuth 2.0 Device Flow. No prerequisites. - -```bash -# Start login (shows a code and link) -dirctl auth login -``` - -What happens: - -1. The CLI displays a short-lived **code** (e.g. `9190-173C`) and the URL **https://github.com/login/device** -2. You open that URL (on this machine or any device), enter the code, and authorize the application -3. After you authorize, the CLI receives a token and caches it at `~/.config/dirctl/auth-token.json` -4. Subsequent commands automatically use the cached token (no `--auth-mode` flag needed) - -```bash -# Force re-login even if already authenticated -dirctl auth login --force - -# Show code and URL only (do not open browser automatically) -dirctl auth login --no-browser -``` - -!!! note "Custom OAuth App (optional)" - To use your own GitHub OAuth App instead of the default, create an OAuth App in [GitHub Developer Settings](https://github.com/settings/developers) with Device Flow support and set `DIRECTORY_CLIENT_GITHUB_CLIENT_ID` (and optionally `DIRECTORY_CLIENT_GITHUB_CLIENT_SECRET`). For normal use, leave these unset. - -#### `dirctl auth status` - -Check your current authentication status. - -```bash -# Show authentication status -dirctl auth status - -# Validate token with GitHub API -dirctl auth status --validate -``` - -Example output: - -``` -Status: Authenticated - User: your-username - Organizations: agntcy, your-org - Cached at: 2025-12-22T10:30:00Z - Token: Valid ✓ - Estimated expiry: 2025-12-22T18:30:00Z - Cache file: /Users/you/.config/dirctl/auth-token.json -``` - -#### `dirctl auth logout` - -Clear cached authentication credentials. - -```bash -dirctl auth logout -``` - -#### Using Authenticated Commands +=== "Using Container Image" -Once authenticated via `dirctl auth login`, your cached credentials are automatically detected and used: + ```bash + docker pull ghcr.io/agntcy/dir-ctl:latest + docker run --rm ghcr.io/agntcy/dir-ctl:latest --help + ``` -```bash -# Push to federation (auto-detects and uses cached GitHub credentials) -dirctl push my-agent.json +## Quick Start -# Search federation nodes (auto-detects authentication) -dirctl --server-addr=federation.agntcy.org:443 search --skill "natural_language_processing" +The following example demonstrates how to store, publish, search, and retrieve a record using the Directory CLI: -# Pull from federation (auto-detects authentication) -dirctl pull baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi -``` +1. Store a record -**Authentication mode behavior:** + ```bash + dirctl push my-agent.json + ``` -- **No `--auth-mode` flag (default)**: Auto-detects authentication in this order: SPIFFE (if available in Kubernetes/SPIRE environment), cached GitHub credentials (if `dirctl auth login` was run), then insecure (for local development). -- **Explicit `--auth-mode=github`**: Forces GitHub authentication (e.g. to bypass SPIFFE in a SPIRE environment). -- **Other modes**: Use `--auth-mode=x509`, `--auth-mode=jwt`, or `--auth-mode=tls` for specific authentication methods. + Returns: `baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi` -```bash -# Force GitHub auth even if SPIFFE is available -dirctl --auth-mode=github push my-agent.json -``` +1. Publish for network discovery -### Other Authentication Modes + ```bash + dirctl routing publish baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi + ``` -| Mode | Description | Use Case | -|------|-------------|----------| -| `github` | GitHub OAuth (explicit) | Force GitHub auth, bypass SPIFFE auto-detect | -| `x509` | SPIFFE X.509 certificates | Kubernetes workloads with SPIRE | -| `jwt` | SPIFFE JWT tokens | Service-to-service authentication | -| `token` | SPIFFE token file | Pre-provisioned credentials | -| `tls` | mTLS with certificates | Custom PKI environments | -| `insecure` / `none` | No auth, skip auto-detect | Testing, local development | -| (empty) | Auto-detect: SPIFFE → cached GitHub → insecure | Default behavior (recommended) | +1. Search for records -## Configuration + ```bash + dirctl routing search --skill "AI" --limit 10 + ``` -### Server Connection +1. Retrieve a record -```bash -# Connect to specific server -dirctl --server-addr localhost:8888 routing list + ```bash + # Pull by CID + dirctl pull baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi -# Use environment variable -export DIRECTORY_CLIENT_SERVER_ADDRESS=localhost:8888 -dirctl routing list -``` + # Or pull by name (if the record has a verifiable name) + dirctl pull cisco.com/agent:v1.0.0 + ``` -### SPIFFE Authentication +!!! note "Name-based References" + + The CLI supports Docker-style name references in addition to CIDs. Records can be pulled using formats like `name`, `name:version`, or `name:version@cid` for hash-verified lookups. See [Name Verification](#name-verification) for details. -```bash -# Use SPIFFE Workload API -dirctl --spiffe-socket-path /run/spire/sockets/agent.sock routing list -``` +!!! note "Authentication for federation" + + When accessing Directory federation nodes, authenticate first with `dirctl auth login`. See [Authentication](#authentication) for details. ## Common Workflows @@ -1184,6 +251,193 @@ dirctl events listen --labels /skills/AI --output jsonl | \ dirctl events listen --output raw | tee event-cids.txt ``` +## Output Formats + +All `dirctl` commands support multiple output formats via the `--output` (or `-o`) flag, making it easy to switch between human-readable output and machine-processable formats. + +### Available Formats + +| Format | Description | Use Case | +|--------|-------------|----------| +| `human` | Human-readable, formatted output with colors and tables (default) | Interactive terminal use | +| `json` | Pretty-printed JSON with indentation | Debugging, single-record processing | +| `jsonl` | Newline-delimited JSON (compact, one object per line) | Streaming, batch processing, logging | +| `raw` | Raw values only (e.g., CIDs, IDs) | Shell scripting, piping to other commands | + +### Usage + +```bash +# Human-readable output (default) +dirctl routing list + +# JSON output (pretty-printed) +dirctl routing list --output json +dirctl routing list -o json + +# JSONL output (streaming-friendly) +dirctl events listen --output jsonl + +# Raw output (just values) +dirctl push my-agent.json --output raw +``` + +### Piping and Processing + +Structured formats (`json`, `jsonl`, `raw`) automatically route data to **stdout** and metadata messages to **stderr**, enabling clean piping to tools like `jq`: + +```bash +# Process JSON output with jq +dirctl routing search --skill "AI" -o json | jq '.[] | .cid' + +# Stream events and filter by type +dirctl events listen -o jsonl | jq -c 'select(.type == "EVENT_TYPE_RECORD_PUSHED")' + +# Capture CID for scripting +CID=$(dirctl push my-agent.json -o raw) +echo "Stored with CID: $CID" + +# Chain commands +dirctl routing list -o json | jq -r '.[].cid' | xargs -I {} dirctl pull {} +``` + +### Format Selection Guidelines + +- **`human`**: Default for terminal interaction, provides context and formatting +- **`json`**: Best for debugging or when you need readable structured data +- **`jsonl`**: Ideal for streaming events, logs, or processing large result sets line-by-line +- **`raw`**: Perfect for shell scripts and command chaining where you only need the value + +## Authentication + +Authentication is required when accessing Directory federation nodes. The CLI supports multiple authentication modes, with GitHub OAuth recommended for interactive use. + +| Command | Description | +|---------|-------------| +| `dirctl auth login` | Authenticate with GitHub | +| `dirctl auth logout` | Clear cached authentication credentials | +| `dirctl auth status` | Show current authentication status | + +### GitHub OAuth Authentication + +GitHub OAuth (Device Flow) enables secure, interactive authentication for accessing federation nodes. + +#### `dirctl auth login` + +Authenticate with GitHub using the OAuth 2.0 Device Flow. No prerequisites. + +```bash +# Start login (shows a code and link) +dirctl auth login +``` + +What happens: + +1. The CLI displays a short-lived **code** (e.g. `9190-173C`) and the URL **https://github.com/login/device** +2. You open that URL (on this machine or any device), enter the code, and authorize the application +3. After you authorize, the CLI receives a token and caches it at `~/.config/dirctl/auth-token.json` +4. Subsequent commands automatically use the cached token (no `--auth-mode` flag needed) + +```bash +# Force re-login even if already authenticated +dirctl auth login --force + +# Show code and URL only (do not open browser automatically) +dirctl auth login --no-browser +``` + +!!! note "Custom OAuth App (optional)" + To use your own GitHub OAuth App instead of the default, create an OAuth App in [GitHub Developer Settings](https://github.com/settings/developers) with Device Flow support and set `DIRECTORY_CLIENT_GITHUB_CLIENT_ID` (and optionally `DIRECTORY_CLIENT_GITHUB_CLIENT_SECRET`). For normal use, leave these unset. + +#### `dirctl auth status` + +Check your current authentication status. + +```bash +# Show authentication status +dirctl auth status + +# Validate token with GitHub API +dirctl auth status --validate +``` + +Example output: + +``` +Status: Authenticated + User: your-username + Organizations: agntcy, your-org + Cached at: 2025-12-22T10:30:00Z + Token: Valid ✓ + Estimated expiry: 2025-12-22T18:30:00Z + Cache file: /Users/you/.config/dirctl/auth-token.json +``` + +#### `dirctl auth logout` + +Clear cached authentication credentials. + +```bash +dirctl auth logout +``` + +#### Using Authenticated Commands + +Once authenticated via `dirctl auth login`, your cached credentials are automatically detected and used: + +```bash +# Push to federation (auto-detects and uses cached GitHub credentials) +dirctl push my-agent.json + +# Search federation nodes (auto-detects authentication) +dirctl --server-addr=federation.agntcy.org:443 search --skill "natural_language_processing" + +# Pull from federation (auto-detects authentication) +dirctl pull baeareihdr6t7s6sr2q4zo456sza66eewqc7huzatyfgvoupaqyjw23ilvi +``` + +**Authentication mode behavior:** + +- **No `--auth-mode` flag (default)**: Auto-detects authentication in this order: SPIFFE (if available in Kubernetes/SPIRE environment), cached GitHub credentials (if `dirctl auth login` was run), then insecure (for local development). +- **Explicit `--auth-mode=github`**: Forces GitHub authentication (e.g. to bypass SPIFFE in a SPIRE environment). +- **Other modes**: Use `--auth-mode=x509`, `--auth-mode=jwt`, or `--auth-mode=tls` for specific authentication methods. + +```bash +# Force GitHub auth even if SPIFFE is available +dirctl --auth-mode=github push my-agent.json +``` + +### Other Authentication Modes + +| Mode | Description | Use Case | +|------|-------------|----------| +| `github` | GitHub OAuth (explicit) | Force GitHub auth, bypass SPIFFE auto-detect | +| `x509` | SPIFFE X.509 certificates | Kubernetes workloads with SPIRE | +| `jwt` | SPIFFE JWT tokens | Service-to-service authentication | +| `token` | SPIFFE token file | Pre-provisioned credentials | +| `tls` | mTLS with certificates | Custom PKI environments | +| `insecure` / `none` | No auth, skip auto-detect | Testing, local development | +| (empty) | Auto-detect: SPIFFE → cached GitHub → insecure | Default behavior (recommended) | + +## Configuration + +### Server Connection + +```bash +# Connect to specific server +dirctl --server-addr localhost:8888 routing list + +# Use environment variable +export DIRECTORY_CLIENT_SERVER_ADDRESS=localhost:8888 +dirctl routing list +``` + +### Authentication + +```bash +# Use SPIFFE Workload API +dirctl --spiffe-socket-path /run/spire/sockets/agent.sock routing list +``` + ## Command Organization The CLI follows a clear service-based organization: diff --git a/docs/dir/directory-mcp.md b/docs/dir/directory-mcp.md new file mode 100644 index 00000000..542f917c --- /dev/null +++ b/docs/dir/directory-mcp.md @@ -0,0 +1,99 @@ +# Directory MCP Server + +The Directory MCP Server provides a standardized interface for AI assistants and tools to interact with the AGNTCY Agent Directory and work with OASF agent records. + +The Directory MCP Server exposes Directory functionality through MCP, allowing AI assistants to: + +- Work with OASF schemas and validate agent records. +- Search and discover agent records from the Directory. +- Push and pull records to/from Directory servers. +- Navigate OASF skill and domain taxonomies. +- Generate agent records automatically from codebases. + +The MCP server runs via the `dirctl` CLI tool and acts as a bridge between AI development environments and the Directory infrastructure, making it easier to work with agent metadata in your development workflow. + +## Configuration + +### Binary Configuration + +Add the MCP server to your IDE's MCP configuration using the absolute path to the `dirctl` binary. + +**Example Cursor configuration (`~/.cursor/mcp.json`):** + +```json +{ + "mcpServers": { + "dir-mcp-server": { + "command": "/absolute/path/to/dirctl", + "args": ["mcp", "serve"] + } + } +} +``` + +### Docker Configuration + +Add the MCP server to your IDE's MCP configuration using Docker. + +??? example "Example Cursor configuration (`~/.cursor/mcp.json`)" + + ```json + { + "mcpServers": { + "dir-mcp-server": { + "command": "docker", + "args": [ + "run", + "--rm", + "-i", + "ghcr.io/agntcy/dir-ctl:latest", + "mcp", + "serve" + ] + } + } + } + ``` + +### Environment Variables + +Configure the MCP server behavior using environment variables: + +- `DIRECTORY_CLIENT_SERVER_ADDRESS` - Directory server address (default: `0.0.0.0:8888`) +- `DIRECTORY_CLIENT_AUTH_MODE` - Authentication mode: `none`, `x509`, `jwt`, `token` +- `DIRECTORY_CLIENT_SPIFFE_TOKEN` - Path to SPIFFE token file (for token authentication) +- `DIRECTORY_CLIENT_TLS_SKIP_VERIFY` - Skip TLS verification (set to `true` if needed) + +??? example "Example with environment variables" + + ```json + { + "mcpServers": { + "dir-mcp-server": { + "command": "/usr/local/bin/dirctl", + "args": ["mcp", "serve"], + "env": { + "DIRECTORY_CLIENT_SERVER_ADDRESS": "dir.example.com:8888", + "DIRECTORY_CLIENT_AUTH_MODE": "none", + "DIRECTORY_CLIENT_TLS_SKIP_VERIFY": "false" + } + } + } + } + ``` + +## Directory MCP Server Tools + +Using the Directory MCP Server, you can access the following tools: + +- `agntcy_oasf_list_versions` - Lists all available OASF schema versions supported by the server. +- `agntcy_oasf_get_schema` - Retrieves the complete OASF schema JSON content for the specified version. +- `agntcy_oasf_get_schema_skills` - Retrieves skills from the OASF schema with hierarchical navigation support. +- `agntcy_oasf_get_schema_domains` - Retrieves domains from the OASF schema with hierarchical navigation support. +- `agntcy_oasf_validate_record` - Validates an OASF agent record against the OASF schema. +- `agntcy_dir_push_record` - Pushes an OASF agent record to a Directory server. +- `agntcy_dir_pull_record` - Pulls an OASF agent record from the local Directory node by its CID (Content Identifier). +- `agntcy_dir_search_local` - Searches for agent records on the local directory node using structured query filters. +- `agntcy_dir_verify_name` - Verifies that a record's name is owned by the domain it claims (by CID or name; optional version). Checks that the record was signed with a key published in the domain's well-known JWKS file. + +For a full list of tools and usage examples, see the [Directory MCP Server documentation](https://github.com/agntcy/dir/blob/main/mcp/README.md). diff --git a/docs/dir/getting-started.md b/docs/dir/getting-started.md index 4ceb7ae0..3109ecbc 100644 --- a/docs/dir/getting-started.md +++ b/docs/dir/getting-started.md @@ -334,7 +334,7 @@ For more configuration options, see [Validation](validation.md). ## Directory MCP Server -The Directory services are also accessible through the Directory MCP Server. It provides a standardized interface for AI assistants and tools to interact with the Directory system and work with OASF agent records. See the [Directory CLI Reference](directory-cli.md#directory-mcp-server) for more information. +The Directory services are also accessible through the Directory MCP Server. It provides a standardized interface for AI assistants and tools to interact with the Directory system and work with OASF agent records. See the [MCP Server documentation](directory-mcp.md) for more information. ## Next Steps diff --git a/docs/dir/scenarios.md b/docs/dir/scenarios.md index 7afa875f..dda1c595 100644 --- a/docs/dir/scenarios.md +++ b/docs/dir/scenarios.md @@ -7,7 +7,7 @@ scenarios. All code snippets below are tested against the Directory `v1.0.0` rel Although the following example is shown for a CLI-based usage scenario, the same functionality can be performed using language-specific SDKs. -The Agent Directory Service is also accessible through the Directory MCP Server. It provides a standardized interface for AI assistants and tools to interact with the Directory system and work with OASF agent records. See the [Directory CLI Reference](directory-cli.md#directory-mcp-server) for more information. +The Agent Directory Service is also accessible through the Directory MCP Server. It provides a standardized interface for AI assistants and tools to interact with the Directory system and work with OASF agent records. See the [MCP Server documentation](directory-mcp.md) for more information. ## Prerequisites diff --git a/docs/slim/.index b/docs/slim/.index index f3130beb..ed920b61 100644 --- a/docs/slim/.index +++ b/docs/slim/.index @@ -3,11 +3,12 @@ nav: - Getting Started with SLIM: - Getting Started: slim-howto.md - Configuration Reference: slim-data-plane-config.md + - SLIM Controller: slim-controller.md + - SLIM Controller Reference: slim-controller-reference.md - SLIM Messaging Layer: - Overview: slim-data-plane.md - SLIM Sessions: slim-session.md - SLIM Authentication: slim-authentication.md - - SLIM Controller: slim-controller.md - SLIM Groups: - Group Creation and Management: slim-group.md - Group Communication Tutorial: slim-group-tutorial.md diff --git a/docs/slim/slim-controller-reference.md b/docs/slim/slim-controller-reference.md new file mode 100644 index 00000000..e7667479 --- /dev/null +++ b/docs/slim/slim-controller-reference.md @@ -0,0 +1,186 @@ +# SLIM Controller Reference + +## `slimctl` Commands + +### `slim` - Local SLIM Instances + +Run standalone SLIM instances for development and testing using production configurations. + +**Start with a configuration file:** + +```bash +# Start with base configuration (insecure) +slimctl slim start --config data-plane/config/base/server-config.yaml + +# Start with TLS configuration +slimctl slim start --config data-plane/config/tls/server-config.yaml +``` + +**Quick start just specifying a custom listening endpoint:** + +```bash +slimctl slim start --endpoint 127.0.0.1:12345 +``` + +!!! note + The server will start without TLS when using this command. + +**Control log level:** + +```bash +RUST_LOG=debug slimctl slim start --config data-plane/config/base/server-config.yaml +``` + +The log level can also be set directly in the configuration file via the `tracing.log_level` field. +See [data-plane/config/base/server-config.yaml](https://github.com/agntcy/slim/blob/slim-v1.0.0/data-plane/config/base/server-config.yaml) +for an example. + +**Available flags:** + +- `--config` - Path to YAML configuration file (production SLIM format) +- `--endpoint` - Server endpoint (sets `SLIMCTL_SLIM_ENDPOINT` environment variable) + +**Configuration files:** See example configs from [data-plane/config/](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config): + +- [base](https://github.com/agntcy/slim/blob/slim-v1.0.0/data-plane/config/base) - Basic insecure configuration +- [tls](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/tls) - TLS-enabled server +- [mtls](https://github.com/agntcy/slim/blob/slim-v1.0.0/data-plane/config/mtls) - Mutual TLS authentication +- [basic-auth](https://github.com/agntcy/slim/blob/slim-v1.0.0/data-plane/config/basic-auth) - HTTP Basic authentication +- `jwt-auth-*` - JWT authentication ([RSA](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/jwt-auth-rsa), + [ECDSA](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/jwt-auth-ecdsa), + [HMAC](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/jwt-auth-hmac)) +- [spire](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/spire) - SPIFFE/SPIRE workload identity +- [proxy](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/proxy) - HTTP proxy configuration +- [telemetry](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/telemetry) - OpenTelemetry integration + +### `route` - Route Management + +Manage message routes on SLIM nodes via the Control Plane. + +**List routes:** + +```bash +slimctl controller route list --node-id=my-node +``` + +**Add a route:** + +```bash +# Create connection configuration +cat > connection_config.json < +``` + +**Add route directly to a node:** + +```bash +slimctl node route add org/namespace/service/0 via config.json --server= +``` + +### `version` - Version Information + +Display version and build information: + +```bash +slimctl version +``` + +### Getting Help + +Get detailed help for any command: + +```bash +slimctl --help +slimctl slim --help +slimctl slim start --help +slimctl route --help +``` + +## Usage Examples + +### Example 1: Create, Delete Route using node-id + +Add route for node `slim/a` to forward messages for `org/default/alice/0` to node `slim/b`. + +```bash +# List available nodes +slimctl controller node list +2 node(s) found +Node ID: slim/b status: CONNECTED + Connection details: + - Endpoint: 127.0.0.1:46457 + MtlsRequired: false + ExternalEndpoint: test-slim.default.svc.cluster.local:46457 +Node ID: slim/a status: CONNECTED + Connection details: + - Endpoint: 127.0.0.1:46357 + MtlsRequired: false + ExternalEndpoint: test-slim.default.svc.cluster.local:46357 + +# Add route to node slim/a +slimctl controller route add org/default/alice/0 via slim/b --node-id slim/a + +# Delete an existing route +slimctl controller route del org/default/alice/0 via slim/b --node-id slim/a +``` + +### Example 2: Create, Delete Route Using `connection_config.json` + +```bash +# Create connection configuration +cat > connection_config.json < + +Commands: + version Print version information + config Manage slimctl configuration + node Commands to interact with SLIM nodes directly + controller Commands to interact with the SLIM Control Plane + slim Commands for managing a local SLIM instance + help Print this message or the help of the given subcommand(s) +``` + +### Installing slimctl + +`slimctl` is available for multiple operating systems and architectures. + +=== "Pre-built Binaries" + + Download from the [GitHub releases page](https://github.com/agntcy/slim/releases): + + 1. Download the binary for your OS and architecture + 2. Extract the archive + 3. Move `slimctl` to a directory in your `PATH` + +=== "Homebrew (macOS)" + + If you are using macOS, you can install slimctl via Homebrew: + + ```bash + brew tap agntcy/slim https://github.com/agntcy/slim.git + brew install slimctl + ``` + +=== "Building from Source" + + **Prerequisites**: Go 1.20+, Task (taskfile.dev) + + ```bash + # From repository root + cd control-plane + task control-plane:slimctl:build + + # Binary location: .dist/bin/slimctl + ``` + +### Configuring slimctl + +`slimctl` supports configuration through a configuration file, environment variables, or command-line flags. It looks for the Control Plane configuration at the following locations: + +- `$HOME/.slimctl/config.yaml` +- `./config.yaml` (current directory) +- Via `--config` flag + +An example `config.yaml`: + +```yaml +server: "127.0.0.1:46358" +timeout: "10s" +tls: + insecure: false + ca_file: "/path/to/ca.pem" + cert_file: "/path/to/client.pem" + key_file: "/path/to/client.key" +``` + +The `server` endpoint should point to a [SLIM Control](https://github.com/agntcy/slim/tree/slim-v1.0.0/control-plane/control-plane) endpoint which is a central service managing SLIM node configurations. + +### Managing SLIM Nodes Directly + +SLIM nodes can be configured to expose a Controller endpoint. slimctl can connect to this endpoint to manage the SLIM instance directly using the `node` sub-command, bypassing the central Control Plane. + +To enable this, configure the node to host a server allowing the client to connect: + +```yaml + tracing: + log_level: info + display_thread_names: true + display_thread_ids: true + + runtime: + n_cores: 0 + thread_name: "slim-data-plane" + drain_timeout: 10s + + services: + slim/1: + dataplane: + servers: [] + clients: [] + controller: + servers: + - endpoint: "0.0.0.0:46358" + tls: + insecure: true # Or specify tls cert and key + clients: [] +``` + +**List connections on a SLIM instance:** + +```bash +slimctl node connection list --server= +``` + +**List routes on a SLIM instance:** + +```bash +slimctl node route list --server= +``` + +**Add a route to the SLIM instance:** + +```bash +slimctl node route add via --server= +``` + +**Delete a route from the SLIM instance:** + +```bash +slimctl node route del via --server= +``` + ## Key Features - **Centralized Node Management**: Register and manage multiple SLIM nodes from a single control point. @@ -146,66 +339,6 @@ sequenceDiagram Controller->>Controller: Update Node Status ``` -### Configuring the SLIM Controller - -The Controller can be configured through the `config.yaml` file. - -An example of a minimal configuration: - -```yaml -northbound: - httpHost: localhost - httpPort: 50051 - logging: - level: DEBUG - -southbound: - httpHost: localhost - httpPort: 50052 - # number of node reconciler threads - reconciler: - threads: 3 - -logging: - level: DEBUG -``` - -Example config to enable MTLS on Southbound endpoint using [Spire](https://spiffe.io/docs/latest/spire-about/spire-concepts/). - -```yaml - northbound: - httpHost: 0.0.0.0 - httpPort: 50051 - - southbound: - httpHost: 0.0.0.0 - httpPort: 50052 - tls: - useSpiffe: true - spire: - socketPath: "unix:///run/spire/agent-sockets/api.sock" - - logging: - level: DEBUG - - reconciler: - # Max number of times a failed reconcile will be retried - maxRequeues: 15 - # Max number of reconciles that can be run in parallel for different nodes - maxNumOfParallelReconciles: 1000 - - # Specifies the SQLite database file path for storing control plane data - database: - filePath: controlplane.db - - spire: - enabled: false - # Slim Controller SVIDs will be federated with these trust domains - trustedDomains: [] - # - cluster-a.example.org - # - cluster-b.example.org -``` - ## Usage ## Prerequisites @@ -282,321 +415,3 @@ Routes between SLIM nodes are automatically created by Controller upon receiving Nodes can be managed through slimctl. Although routes are automatically created for client subscription you can still add/remove routes manually. For more information, see the [slimctl](#slimctl). - -## slimctl - -`slimctl` is a unified command-line interface for managing SLIM instances and the SLIM control plane. It provides commands for: - -- **Local Development** - Run standalone SLIM instances for development and testing using production configurations -- **Route Management** - Configure message routing between services via the SLIM Control Plane -- **Connection Monitoring** - View and manage active connections on SLIM nodes -- **Direct Node Access** - Manage SLIM nodes directly without going through the Control Plane - -### Command Groups - -```bash -slimctl --help -Usage: slimctl [OPTIONS] - -Commands: - version Print version information - config Manage slimctl configuration - node Commands to interact with SLIM nodes directly - controller Commands to interact with the SLIM Control Plane - slim Commands for managing a local SLIM instance - help Print this message or the help of the given subcommand(s) -``` - -### Installing slimctl - -Slimctl is available for multiple operating systems and architectures. - -#### Pre-built Binaries - -Download from the [GitHub releases page](https://github.com/agntcy/slim/releases): - -1. Download the binary for your OS and architecture -2. Extract the archive -3. Move `slimctl` to a directory in your `PATH` - -#### Homebrew (macOS) - -If you are using macOS, you can install slimctl via Homebrew: - -```bash -brew tap agntcy/slim https://github.com/agntcy/slim.git -brew install slimctl -``` - -#### Building from Source - -**Prerequisites**: Go 1.20+, Task (taskfile.dev) - -```bash -# From repository root -cd control-plane -task control-plane:slimctl:build - -# Binary location: .dist/bin/slimctl -``` - -### Configuring slimctl - -`slimctl` supports configuration through a configuration file, environment variables, or command-line flags. - -slimctl looks for Control Plane configuration at: - -- `$HOME/.slimctl/config.yaml` -- `./config.yaml` (current directory) -- Via `--config` flag - -An example `config.yaml`: - -```yaml -server: "127.0.0.1:46358" -timeout: "10s" -tls: - insecure: false - ca_file: "/path/to/ca.pem" - cert_file: "/path/to/client.pem" - key_file: "/path/to/client.key" -``` - -The `server` endpoint should point to a [SLIM Control](https://github.com/agntcy/slim/tree/slim-v1.0.0/control-plane/control-plane) endpoint which is a central service managing SLIM node configurations. - -### Commands - -#### `slim` - Local SLIM Instances - -Run standalone SLIM instances for development and testing using production configurations. - -**Start with a configuration file:** - -```bash -# Start with base configuration (insecure) -slimctl slim start --config data-plane/config/base/server-config.yaml - -# Start with TLS configuration -slimctl slim start --config data-plane/config/tls/server-config.yaml -``` - -**Quick start just specifying a custom listening endpoint:** - -```bash -slimctl slim start --endpoint 127.0.0.1:12345 -``` - -!!! note - The server will start without TLS when using this command. - -**Control log level:** - -```bash -RUST_LOG=debug slimctl slim start --config data-plane/config/base/server-config.yaml -``` - -The log level can also be set directly in the configuration file via the `tracing.log_level` field. -See [data-plane/config/base/server-config.yaml](https://github.com/agntcy/slim/blob/slim-v1.0.0/data-plane/config/base/server-config.yaml) -for an example. - -**Available flags:** - -- `--config` - Path to YAML configuration file (production SLIM format) -- `--endpoint` - Server endpoint (sets `SLIMCTL_SLIM_ENDPOINT` environment variable) - -**Configuration files:** See example configs from [data-plane/config/](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config): - -- [base](https://github.com/agntcy/slim/blob/slim-v1.0.0/data-plane/config/base) - Basic insecure configuration -- [tls](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/tls) - TLS-enabled server -- [mtls](https://github.com/agntcy/slim/blob/slim-v1.0.0/data-plane/config/mtls) - Mutual TLS authentication -- [basic-auth](https://github.com/agntcy/slim/blob/slim-v1.0.0/data-plane/config/basic-auth) - HTTP Basic authentication -- `jwt-auth-*` - JWT authentication ([RSA](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/jwt-auth-rsa), - [ECDSA](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/jwt-auth-ecdsa), - [HMAC](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/jwt-auth-hmac)) -- [spire](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/spire) - SPIFFE/SPIRE workload identity -- [proxy](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/proxy) - HTTP proxy configuration -- [telemetry](https://github.com/agntcy/slim/tree/slim-v1.0.0/data-plane/config/telemetry) - OpenTelemetry integration - -#### `route` - Route Management - -Manage message routes on SLIM nodes via the Control Plane. - -**List routes:** - -```bash -slimctl controller route list --node-id=my-node -``` - -**Add a route:** - -```bash -# Create connection configuration -cat > connection_config.json < -``` - -**Add route directly to a node:** - -```bash -slimctl node route add org/namespace/service/0 via config.json --server= -``` - -#### `version` - Version Information - -Display version and build information: - -```bash -slimctl version -``` - -#### Getting Help - -Get detailed help for any command: - -```bash -slimctl --help -slimctl slim --help -slimctl slim start --help -slimctl route --help -``` - -### Example 1: Create, Delete Route using node-id - -Add route for node `slim/a` to forward messages for `org/default/alice/0` to node `slim/b`. - -```bash -# List available nodes -slimctl controller node list -2 node(s) found -Node ID: slim/b status: CONNECTED - Connection details: - - Endpoint: 127.0.0.1:46457 - MtlsRequired: false - ExternalEndpoint: test-slim.default.svc.cluster.local:46457 -Node ID: slim/a status: CONNECTED - Connection details: - - Endpoint: 127.0.0.1:46357 - MtlsRequired: false - ExternalEndpoint: test-slim.default.svc.cluster.local:46357 - -# Add route to node slim/a -slimctl controller route add org/default/alice/0 via slim/b --node-id slim/a - -# Delete an existing route -slimctl controller route del org/default/alice/0 via slim/b --node-id slim/a -``` - -### Example 2: Create, Delete Route Using `connection_config.json` - -```bash -# Create connection configuration -cat > connection_config.json < -``` - -**List routes on a SLIM instance:** - -```bash -slimctl node route list --server= -``` - -**Add a route to the SLIM instance:** - -```bash -slimctl node route add via --server= -``` - -**Delete a route from the SLIM instance:** - -```bash -slimctl node route del via --server= -``` diff --git a/docs/slim/slim-group-tutorial.md b/docs/slim/slim-group-tutorial.md index 18132540..8577ee12 100644 --- a/docs/slim/slim-group-tutorial.md +++ b/docs/slim/slim-group-tutorial.md @@ -535,7 +535,7 @@ application is still [group.py](https://github.com/agntcy/slim/tree/slim-v1.0.0/ #### Run the SLIM Controller -First, start the SLIM Controller. Full details are in the [Controller](./slim-controller.md) documentation; here we reproduce the minimal setup. Create a configuration file: +First, start the SLIM Controller. Full details are in the [SLIM Controller](./slim-controller.md) documentation; here we reproduce the minimal setup. Create a configuration file: ```bash cat << EOF > ./config-controller.yaml @@ -695,7 +695,7 @@ At this point all applications are waiting for a new session. #### Manage the Group with slimctl -Use `slimctl` (see [slim-controller](./slim-controller.md)) to send administrative commands to the controller. +Use `slimctl` (see [SLIM Controller](./slim-controller.md)) to send administrative commands to the controller. First, you need to run `slimctl`. To install it see the related [documentation](https://github.com/agntcy/slim/tree/slim-v1.0.0/control-plane/slimctl/README.md)