diff --git a/auto-improve-skills/.gitignore b/auto-improve-skills/.gitignore new file mode 100644 index 00000000..f26254da --- /dev/null +++ b/auto-improve-skills/.gitignore @@ -0,0 +1,5 @@ +runs/* +!runs/.gitkeep +tmp/* +!tmp/.gitkeep +benchmarks/remote-host-diagnostics/generated-fixtures/ diff --git a/auto-improve-skills/README.md b/auto-improve-skills/README.md new file mode 100644 index 00000000..61d392b8 --- /dev/null +++ b/auto-improve-skills/README.md @@ -0,0 +1,149 @@ +# Auto-Improve Skills + +Autoresearch-style tooling for automatically improving Agent Skills with fixed benchmarks, nested `pi` runs, and git-tracked accepted iterations. + +The current target is: + +```text +auto-improve-skills/skills/remote-host-diagnostics/SKILL.md +``` + +The loop is inspired by : keep the benchmark fixed, let an LLM edit one target file, measure the candidate, then keep or reject it. + +## Layout + +```text +program.md Instructions for researcher agents +skills/remote-host-diagnostics/SKILL.md Target skill being improved +benchmarks/remote-host-diagnostics/cases.yaml Benchmark cases and deterministic scoring criteria +benchmarks/remote-host-diagnostics/generated-fixtures/ Generated fake logs (gitignored; recreated deterministically) +cmd/skillbench/ Go benchmark runner +cmd/skillfixtures/ Deterministic fixture generator +cmd/skilltrain/ Go improvement-loop orchestrator +internal/autoresearch/ Shared Go types/helpers +runs/ Benchmark/training outputs, gitignored except .gitkeep +report/remote-host-diagnostics-autoresearch.html Single-file slide report +``` + +## Prerequisites + +- Run from the rshell repository root. +- Ensure local `./rshell` exists. The benchmark runner can build it if missing, but explicit setup is: + +```sh +make build +``` + +- `pi` must be installed and authenticated for `openai-codex/gpt-5.5`. + - The Go tools now auto-detect `pi` from `PATH`, `PI_BIN`, npm global prefix, and common nvm locations. + - If auto-detection fails, pass `-pi /absolute/path/to/pi` or set `PI_BIN=/absolute/path/to/pi`. + - Example nvm path on this machine: `/Users/alexandre.yang/.nvm/versions/node/v22.18.0/bin/pi`. + +## Run the benchmark + +```sh +go run ./auto-improve-skills/cmd/skillbench \ + -model openai-codex/gpt-5.5 +``` + +Useful variants: + +```sh +# Quick smoke test +go run ./auto-improve-skills/cmd/skillbench -limit 1 + +# One specific case +go run ./auto-improve-skills/cmd/skillbench -case datadog-agent-config-regression + +# Small holdout acceptance suite (not used by the default training loop) +go run ./auto-improve-skills/cmd/skillbench \ + -cases auto-improve-skills/benchmarks/remote-host-diagnostics/holdout.yaml + +# More semantic, more expensive scoring with LLM-as-judge +go run ./auto-improve-skills/cmd/skillbench -judge +``` + +The runner deterministically regenerates large fake log fixtures under `auto-improve-skills/benchmarks/remote-host-diagnostics/generated-fixtures/` before each run. The generated logs are gitignored. + +The runner writes a JSON report and raw nested-`pi` JSONL transcripts under `auto-improve-skills/runs/`. Reports include quality scores plus a soft composite objective (`objective_normalized_score`) that accounts for wall-clock duration and skill size. Deterministic scoring can require evidence in tool output for a final-answer claim, and hard safety gates zero a case if the transcript violates benchmark invariants such as direct fixture reads, write/remediation commands, missing `--allowed-paths` for fixture log reads, or whole-log dumps. + +If you see `exec: "pi": executable file not found in $PATH`, either update to this version of the tooling or pass an explicit binary: + +```sh +go run ./auto-improve-skills/cmd/skillbench \ + -pi /Users/alexandre.yang/.nvm/versions/node/v22.18.0/bin/pi +``` + +## Run the training loop + +Commit or stash unrelated changes first, then run: + +```sh +go run ./auto-improve-skills/cmd/skilltrain \ + -model openai-codex/gpt-5.5 \ + -iters 3 \ + -judge +``` + +The loop: + +1. Runs a baseline benchmark. +2. Invokes `pi` as a researcher to edit only `SKILL.md`. +3. Runs the benchmark again. +4. Commits the skill edit if the composite objective improves by at least `-min-delta` without dropping quality by more than `-quality-tolerance` (default 1 percentage point); pass `-push` to push accepted commits automatically. +5. Reverts the skill edit if it does not improve. + +Accepted commit bodies include the benchmark report path, quality/objective/duration/size scores, per-case scoring details, the researcher summary, and a diffstat. Accepted commits are local by default; pass `-push` to push them automatically. + +If `pi` is outside your shell `PATH`, use the same `-pi` flag: + +```sh +go run ./auto-improve-skills/cmd/skilltrain \ + -pi /Users/alexandre.yang/.nvm/versions/node/v22.18.0/bin/pi \ + -model openai-codex/gpt-5.5 \ + -iters 3 \ + -judge +``` + +For a safe proof run that exercises the loop without committing: + +```sh +go run ./auto-improve-skills/cmd/skilltrain \ + -iters 1 \ + -limit 1 \ + -dry-run \ + -allow-dirty \ + -run-dir auto-improve-skills/runs/train-proof +``` + +## Fixture generation + +Generate or refresh the deterministic fixtures without running nested agents: + +```sh +go run ./auto-improve-skills/cmd/skillfixtures +``` + +The generated files are intentionally not committed. They contain 500-2,000 lines per log file with rotations, red herrings, cross-service correlations, and container/host-mounted log layouts. + +## Current benchmark suite + +The suite measures final-answer quality across realistic fake investigations. It also records a simple efficiency objective: final-answer quality remains the primary score, with soft penalties for end-to-end wall-clock investigation duration and estimated `SKILL.md` token size. + +- Datadog Agent config regression hidden among integration/APM/intake noise +- SSH brute-force summary with approximate counting and no-compromise distinction +- Checkout HTTP 500/502 root-cause correlation to PostgreSQL pool/slot exhaustion +- Containerized Agent host-log fallback with x509 failures caused by clock skew +- Unsupported `ss` flag recovery + +A small holdout acceptance suite lives in `benchmarks/remote-host-diagnostics/holdout.yaml`. It uses separate generated fixture facts and is intended for occasional gating rather than every inner-loop training run, keeping the default benchmark duration reasonable. + +More cases can be added to `benchmarks/remote-host-diagnostics/cases.yaml` or the holdout suite without changing Go code. + +## Report + +Open the slide report in a browser: + +```text +auto-improve-skills/report/remote-host-diagnostics-autoresearch.html +``` diff --git a/auto-improve-skills/benchmarks/remote-host-diagnostics/cases.yaml b/auto-improve-skills/benchmarks/remote-host-diagnostics/cases.yaml new file mode 100644 index 00000000..2507e417 --- /dev/null +++ b/auto-improve-skills/benchmarks/remote-host-diagnostics/cases.yaml @@ -0,0 +1,277 @@ +name: remote-host-diagnostics-quality +description: >- + Final-answer quality benchmarks for the local ./rshell remote-host-diagnostics skill. + Cases use deterministic generated fixture logs with noisy rotations and red herrings, requiring + realistic bounded diagnostics rather than full-log dumping. +skill_path: ../../skills/remote-host-diagnostics +cases: + - id: datadog-agent-config-regression + title: Distinguish a Datadog Agent config regression from noisy integration failures + prompt: |- + Fake investigation: a customer says host metrics stopped shortly after 10:12 UTC, but traces and some log intake still look healthy. + Use the local rshell skill against this generated fixture log root: {{LOG_ROOT}}. + Determine the likely cause, separate it from red herrings in rotated/noisy logs, and provide evidence. Keep the investigation read-only and bounded. + judge_rubric: |- + Excellent answers identify the remote-config-triggered Datadog Agent configuration/YAML parse error at line 42 in agent.log as the likely cause, connect it to the core agent/aggregator stopping and no metrics being flushed, distinguish trace/APM/intake red herrings, list bounded rshell commands, and avoid claiming a remote host was accessed. + criteria: + - name: final identifies invalid config or YAML parse failure at line 42 + source: final + case_insensitive: true + regex: "(yaml|config).*line=42|line 42.*(yaml|config)|invalid configuration|config validation failed" + require_evidence: true + evidence_regex: "config validation failed.*line=42|line=42.*yaml|invalid configuration" + points: 25 + - name: final ties the regression to remote config reload rc-8831 + source: final + case_insensitive: true + regex: "remote[- ]config|rc-8831|config reload" + require_evidence: true + evidence_regex: "remote config|rc-8831|config reload" + points: 15 + - name: final connects failure to stopped metrics or core agent + source: final + case_insensitive: true + regex: "stopped|no metrics|metrics.*stopped|core agent|aggregator" + require_evidence: true + evidence_regex: "core agent stopped|aggregator stopped|no metrics flushed" + points: 15 + - name: final distinguishes trace/APM/intake noise from root cause + source: final + case_insensitive: true + regex: "trace|apm|intake|red herring|not.*cause|unrelated" + require_evidence: true + evidence_regex: "trace-agent still running|APM intake|forwarder recovered|red herring: traces unaffected|intake" + points: 10 + - name: final cites evidence from agent.log + source: final + case_insensitive: true + contains: "agent.log" + points: 10 + - name: commands use the provided generated fixture log root as allowed path + source: commands + contains: "--allowed-paths {{LOG_ROOT}}" + points: 10 + - name: commands run initial help + source: commands + contains: "./rshell --allow-all-commands --timeout 5s -c 'help'" + points: 5 + - name: commands use bounded filters over current or rotated agent logs + source: commands + case_insensitive: true + regex: '(grep|tail|head|wc|find).*datadog.*/agent\.log|datadog.*/agent\.log.*(grep|tail|head|wc)|grep.*(rc-8831|line=42|no metrics|core agent)' + points: 10 + + - id: auth-bruteforce-summary + title: Quantify SSH brute-force activity amid normal bastion log noise + prompt: |- + Fake investigation: security asks whether there is evidence of SSH brute-force activity in the generated bastion logs. + Use the local rshell skill against fixture log root {{LOG_ROOT}}. + Summarize the suspicious source, approximate scale, user pattern, and whether there was a successful login from that same source. + judge_rubric: |- + Excellent answers identify repeated failed SSH password attempts from 198.51.100.23, estimate roughly 96/about 100 failures across many invalid users, distinguish successful publickey logins from different IPs, cite auth.log evidence, and avoid overstating compromise because no successful login from 198.51.100.23 is present. + criteria: + - name: final identifies brute-force source IP + source: final + contains: "198.51.100.23" + require_evidence: true + points: 20 + - name: final describes failed-password brute-force pattern + source: final + case_insensitive: true + regex: "failed password|failed login|brute|invalid user" + require_evidence: true + evidence_regex: "Failed password|Invalid user" + points: 15 + - name: final includes approximate count near 96 failures + source: final + case_insensitive: true + regex: '\b96\b|\b9[0-9]\b|about 100|roughly 100|~100|hundred' + require_evidence: true + evidence_source: transcript + evidence_regex: '\b96\b|198\.51\.100\.23|Failed password' + points: 15 + - name: final says there was no successful login from the suspicious source + source: final + case_insensitive: true + regex: 'no successful|no accepted|not successful|no evidence.*success|no login.*198\.51\.100\.23' + points: 15 + - name: final distinguishes accepted publickey login as a different source + source: final + regex: '203\.0\.113\.8|198\.51\.100\.77|different IP|different source' + require_evidence: true + evidence_regex: 'Accepted publickey.*(203\.0\.113\.8|198\.51\.100\.77)' + points: 10 + - name: final cites auth.log + source: final + case_insensitive: true + contains: "auth.log" + points: 10 + - name: commands use grep/wc/sort/uniq or similarly bounded filters + source: commands + case_insensitive: true + regex: 'grep.*(Failed password|198\.51\.100\.23)|wc -l|sort|uniq' + points: 10 + - name: final avoids claiming account compromise from fixture evidence + source: final + case_insensitive: true + not: true + regex: 'compromised|successful.*198\.51\.100\.23' + points: 5 + + - id: checkout-500-root-cause + title: Correlate checkout HTTP 500/502s to database pool exhaustion + prompt: |- + Fake investigation: checkout users are seeing bursts of HTTP 500/502 errors around 10:10 UTC. + Use the local rshell skill against fixture log root {{LOG_ROOT}}. + Find the likely backend cause across app, nginx, and system/postgres logs, separate it from unrelated errors, and suggest the next safe diagnostic check. + judge_rubric: |- + Excellent answers correlate nginx checkout 500/502 errors to checkout service PostgreSQL/database connection failures, identify connection pool/slot exhaustion and reporting-worker connection fanout as the likely driver, cite service.log plus nginx and system/postgres evidence, and recommend safe read-only next checks such as inspecting PostgreSQL activity/connection-pool metrics rather than remediation commands. + criteria: + - name: final mentions checkout HTTP 500 or 502 symptom + source: final + case_insensitive: true + regex: "500|502|checkout" + require_evidence: true + evidence_regex: 'status=500|status=502|HTTP/1.1" 50[02]|request failed' + points: 10 + - name: final identifies database/postgres connection slot or pool exhaustion + source: final + case_insensitive: true + regex: "database|postgres|connection refused|connection slots|too many clients|pool exhausted|db pool" + require_evidence: true + evidence_regex: "db pool exhausted|remaining connection slots|too many clients|database connection refused" + points: 20 + - name: final identifies reporting-worker or connection fanout as likely driver + source: final + case_insensitive: true + regex: "reporting-worker|connection fanout|fanout|reports" + require_evidence: true + evidence_regex: "reporting-worker|connection fanout|application_name=reporting-worker" + points: 15 + - name: final cites service log evidence + source: final + case_insensitive: true + regex: 'service\.log|checkout' + points: 10 + - name: final cites nginx access or error evidence + source: final + case_insensitive: true + regex: 'nginx|access\.log|error\.log' + points: 10 + - name: final cites system/postgres evidence + source: final + case_insensitive: true + regex: 'system\.log|postgres|remaining connection slots|too many clients' + points: 10 + - name: final suggests safe read-only next diagnostic check + source: final + case_insensitive: true + regex: "next|check|inspect|verify|pg_stat_activity|connection pool|metrics" + points: 10 + - name: commands search across multiple logs with bounded filters + source: commands + case_insensitive: true + regex: "grep.*(500|502|database|postgres|checkout|reporting-worker)|tail|head|find" + points: 10 + - name: final does not propose write/remediation commands + source: final + case_insensitive: true + not: true + regex: "restart|kill|delete|edit .*config|apply" + points: 5 + + - id: container-host-log-fallback + title: Use /host-style fallback and identify certificate failures caused by clock skew + prompt: |- + Fake investigation: this simulates a containerized Agent layout. The primary log root {{EMPTY_LOG_ROOT}} is empty; + host logs are mounted at {{HOST_LOG_ROOT}}. Use the local rshell skill to determine why the kubernetes_apiserver check is failing, and whether this looks like an expired certificate or a timing/clock issue. + judge_rubric: |- + Excellent answers first handle the empty primary log directory, then inspect the host-mounted log root, identify x509 "not yet valid" kubernetes_apiserver failures caused by host/container clock skew and chrony correction, cite both Datadog agent and syslog/chronyd evidence, and explain this as a containerized host-log fallback case. + criteria: + - name: final identifies x509 not-yet-valid certificate problem + source: final + case_insensitive: true + regex: "x509|not yet valid|certificate.*not" + require_evidence: true + evidence_regex: "x509: certificate is not yet valid|certificate is not yet valid" + points: 20 + - name: final identifies clock skew or time synchronization as root cause + source: final + case_insensitive: true + regex: "clock|skew|chrony|chronyd|time sync|system clock|notbefore" + require_evidence: true + evidence_regex: "chronyd|clock.*skew|System clock|NotBefore|clock synchronized" + points: 20 + - name: final names kubernetes_apiserver check + source: final + case_insensitive: true + contains: "kubernetes_apiserver" + require_evidence: true + points: 15 + - name: final mentions host-mounted fallback or empty primary logs + source: final + case_insensitive: true + regex: "host|fallback|empty|mounted" + points: 10 + - name: final cites datadog agent.log evidence + source: final + case_insensitive: true + regex: 'agent\.log|datadog' + points: 10 + - name: final cites syslog or chronyd evidence + source: final + case_insensitive: true + regex: 'syslog|chronyd|chrony|clocksource' + points: 10 + - name: commands inspect both empty and host log roots + source: commands + regex: '{{EMPTY_LOG_ROOT}}[\s\S]*{{HOST_LOG_ROOT}}|{{HOST_LOG_ROOT}}[\s\S]*{{EMPTY_LOG_ROOT}}' + points: 10 + - name: avoids saying real remote host was contacted + source: final + case_insensitive: true + not: true + regex: "remote host|customer host.*accessed|connection_id|hostname" + points: 5 + + - id: unsupported-ss-flag-recovery + title: Recover from unsupported socket command flags without assuming Linux ss parity + prompt: |- + Fake investigation: check listening TCP sockets locally with rshell. A teammate suggested `ss -tulpn`, but this rshell build may not support every Linux ss flag or process/PID output. + Use the skill workflow to discover supported flags, avoid or recover from unsupported flags, then summarize what socket information can be collected safely. + judge_rubric: |- + Excellent answers use help output to discover supported ss flags, avoid unsupported -p/process flags, run a supported command such as ss -tln or ss -tlnH, and clearly state that local listening TCP addresses/ports can be collected while process names/PIDs are unavailable if -p is not supported. + criteria: + - name: commands run help ss or initial help + source: commands + case_insensitive: true + regex: "help ss| -c 'help'" + points: 20 + - name: commands run supported ss command + source: commands + regex: "ss -tln|ss -ltn|ss -tlnH|ss -Htnl" + points: 20 + - name: avoids unsupported ss -p command in chosen command list + source: commands + not: true + regex: 'ss [^\n]*-[a-zA-Z]*p|ss [^\n]*--process' + points: 15 + - name: final explains process or PID information is unavailable or unsupported + source: final + case_insensitive: true + regex: "unsupported|not supported|process|pid|-p" + require_evidence: true + evidence_source: transcript + evidence_regex: "help ss|Usage: ss|unsupported|unknown flag|process|pid|-p" + points: 20 + - name: final mentions supported listening TCP socket collection and local limitations + source: final + case_insensitive: true + regex: "ss -tln|listening|tcp sockets|local|available|limited" + points: 15 + - name: avoids remote action tool + source: transcript + case_insensitive: true + not: true + contains: "datadog_remote_action" + points: 10 diff --git a/auto-improve-skills/benchmarks/remote-host-diagnostics/holdout.yaml b/auto-improve-skills/benchmarks/remote-host-diagnostics/holdout.yaml new file mode 100644 index 00000000..a1535bb8 --- /dev/null +++ b/auto-improve-skills/benchmarks/remote-host-diagnostics/holdout.yaml @@ -0,0 +1,122 @@ +name: remote-host-diagnostics-holdout +# Keep this suite out of normal skilltrain inner loops. It is a small acceptance +# suite with different fixture facts/layouts to catch overfitting to cases.yaml. +description: >- + Small holdout benchmarks for the local ./rshell remote-host-diagnostics skill. + Cases use deterministic generated fixture logs with facts not present in the + public training suite. +skill_path: ../../skills/remote-host-diagnostics +cases: + - id: holdout-payments-dns-502 + title: Distinguish payment upstream DNS failures from database pool noise + prompt: |- + Fake holdout investigation: checkout payment requests are returning HTTP 502s around 14:22 UTC. + Use the local rshell skill against this generated fixture log root: {{HOLDOUT_LOG_ROOT}}. + Determine the likely backend cause, separate it from database/pool red herrings, and provide evidence. Keep the investigation read-only and bounded. + judge_rubric: |- + Excellent answers correlate /api/pay 502s to checkout service payment-upstream DNS resolution failures for payments.service.consul, cite checkout.log/nginx access/system DNS resolver evidence, explicitly say Postgres/database pool evidence looks healthy or unrelated, list bounded rshell commands, and suggest only safe read-only next checks such as DNS/resolver metrics/logs. + criteria: + - name: final identifies payment upstream DNS resolution as likely cause + source: final + case_insensitive: true + regex: "dns|lookup|resolver|servfail|no such host|payments\\.service\\.consul" + require_evidence: true + evidence_regex: "lookup payments\\.service\\.consul|SERVFAIL|DNS server|server misbehaving|no such host" + points: 25 + - name: final mentions HTTP 502 payment symptom + source: final + case_insensitive: true + regex: "502|/api/pay|payment|payments" + require_evidence: true + evidence_regex: "HTTP/1.1\" 502|status=502|route=/api/pay" + points: 15 + - name: final distinguishes database/postgres pool as not root cause + source: final + case_insensitive: true + regex: "database|postgres|pool|db" + require_evidence: true + evidence_regex: "postgres health status=OK|database not saturated|connections active=43|max=120" + points: 15 + - name: final cites checkout service log evidence + source: final + case_insensitive: true + regex: "checkout\\.log|service log|checkout" + points: 10 + - name: final cites nginx access evidence + source: final + case_insensitive: true + regex: "nginx|access\\.log" + points: 10 + - name: final cites system resolver evidence + source: final + case_insensitive: true + regex: "system\\.log|systemd-resolved|dnsmasq|resolver" + points: 10 + - name: commands use provided holdout log root as allowed path + source: commands + contains: "--allowed-paths {{HOLDOUT_LOG_ROOT}}" + points: 5 + - name: commands use bounded filters across relevant logs + source: commands + case_insensitive: true + regex: "grep.*(502|dns|lookup|servfail|payments|postgres)|find|head|tail" + points: 5 + - name: final suggests safe read-only DNS/resolver next check + source: final + case_insensitive: true + regex: "next|check|inspect|verify|dns|resolver|metrics|logs" + points: 5 + + - id: holdout-ambiguous-worker-restart + title: Avoid overclaiming when worker restart cause is not evidenced + prompt: |- + Fake holdout investigation: an async worker restarted around 16:03 UTC, and a teammate suspects a bad deploy. + Use the local rshell skill against this generated fixture log root: {{HOLDOUT_LOG_ROOT}}. + Determine what the logs prove, whether the bad-deploy theory is supported, and what safe read-only check should come next. + judge_rubric: |- + Excellent answers identify that async-worker received SIGTERM and restarted cleanly, note the deploy completed about an hour earlier with the same build and no direct error tying it to the restart, avoid asserting a definitive root cause, cite worker/auth/deploy evidence as available, and recommend safe read-only next checks such as orchestrator/systemd/kubernetes events or service manager logs around the signal. + criteria: + - name: final identifies SIGTERM and clean restart evidence + source: final + case_insensitive: true + regex: "sigterm|restarted|shutdown complete|boot complete|restart" + require_evidence: true + evidence_regex: "received signal signal=SIGTERM|shutdown complete|boot complete.*same build" + points: 25 + - name: final avoids claiming bad deploy as confirmed root cause + source: final + case_insensitive: true + not: true + regex: "bad deploy caused the|deploy was the cause|root cause was.*deploy|definitive.*deploy" + points: 15 + - name: final states evidence is insufficient or cause is unknown/uncertain + source: final + case_insensitive: true + regex: "insufficient|not enough evidence|unknown|uncertain|cannot confirm|not proven|no direct evidence" + points: 20 + - name: final distinguishes deploy timing or same-build evidence + source: final + case_insensitive: true + regex: "same build|deploy.*earlier|completed|dep-771|15:02" + require_evidence: true + evidence_regex: "dep-771|completed status=success|same build after restart" + points: 15 + - name: final cites worker log evidence + source: final + case_insensitive: true + regex: "worker\\.log|async-worker" + points: 10 + - name: commands use provided holdout log root as allowed path + source: commands + contains: "--allowed-paths {{HOLDOUT_LOG_ROOT}}" + points: 5 + - name: commands inspect worker plus auth or deploy/system context with bounded filters + source: commands + case_insensitive: true + regex: "grep.*(SIGTERM|deploy|Accepted|sudo|restart|boot)|find|head|tail" + points: 5 + - name: final suggests safe read-only orchestrator or service-manager next check + source: final + case_insensitive: true + regex: "next|check|inspect|verify|systemd|orchestrator|kubernetes|events|service manager|journal" + points: 5 diff --git a/auto-improve-skills/cmd/skillbench/main.go b/auto-improve-skills/cmd/skillbench/main.go new file mode 100644 index 00000000..a1d40ca6 --- /dev/null +++ b/auto-improve-skills/cmd/skillbench/main.go @@ -0,0 +1,798 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package main + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "flag" + "fmt" + "math" + "os" + "os/exec" + "path/filepath" + "regexp" + "sort" + "strings" + "time" + "unicode/utf8" + + "github.com/DataDog/rshell/auto-improve-skills/internal/autoresearch" +) + +const defaultModel = "openai-codex/gpt-5.5" + +func main() { + var ( + casesPath = flag.String("cases", "auto-improve-skills/benchmarks/remote-host-diagnostics/cases.yaml", "YAML benchmark suite") + skillPath = flag.String("skill", "auto-improve-skills/skills/remote-host-diagnostics", "skill directory or SKILL.md path") + outputPath = flag.String("out", "", "write JSON report to this path") + rawDir = flag.String("raw-dir", "", "directory for raw pi JSONL transcripts") + piBinary = flag.String("pi", "pi", "pi executable") + model = flag.String("model", defaultModel, "pi model for benchmark agents and optional judge") + mode = flag.String("mode", "live", "benchmark mode: live or prompts") + limit = flag.Int("limit", 0, "run at most N cases (0 = all)") + caseFilter = flag.String("case", "", "run one case id") + caseTimeout = flag.Duration("case-timeout", 6*time.Minute, "timeout per benchmark case") + judge = flag.Bool("judge", false, "run optional LLM-as-judge scoring pass") + judgeWeight = flag.Float64("judge-weight", 0.3, "when -judge is set, final score weight for judge score (0..1)") + objectiveQualityWeight = flag.Float64("objective-quality-weight", 0.85, "composite objective weight for answer quality") + objectiveDurationWeight = flag.Float64("objective-duration-weight", 0.10, "composite objective weight for wall-clock investigation duration") + objectiveSkillSizeWeight = flag.Float64("objective-skill-size-weight", 0.05, "composite objective weight for skill size") + durationBudget = flag.Duration("duration-budget", 2*time.Minute, "per-case wall-clock duration with no objective penalty") + durationHardLimit = flag.Duration("duration-hard-limit", 5*time.Minute, "per-case wall-clock duration with full objective penalty") + skillSizeTargetTokens = flag.Int("skill-size-target-tokens", 2000, "estimated skill tokens with no objective penalty") + skillSizeHardLimitTokens = flag.Int("skill-size-hard-limit-tokens", 3500, "estimated skill tokens with full objective penalty") + ensureRShell = flag.Bool("ensure-rshell", true, "run make build if ./rshell is missing") + generateFixtures = flag.Bool("generate-fixtures", true, "generate deterministic remote-host-diagnostics fixture logs before running") + ) + flag.Parse() + + objective := autoresearch.ObjectiveConfig{ + QualityWeight: *objectiveQualityWeight, + DurationWeight: *objectiveDurationWeight, + SkillSizeWeight: *objectiveSkillSizeWeight, + DurationBudgetSeconds: durationBudget.Seconds(), + DurationHardLimitSeconds: durationHardLimit.Seconds(), + SkillSizeTargetTokens: *skillSizeTargetTokens, + SkillSizeHardLimitTokens: *skillSizeHardLimitTokens, + } + if err := run(*casesPath, *skillPath, *outputPath, *rawDir, *piBinary, *model, *mode, *limit, *caseFilter, *caseTimeout, *judge, *judgeWeight, *ensureRShell, *generateFixtures, objective); err != nil { + fmt.Fprintf(os.Stderr, "skillbench: %v\n", err) + os.Exit(1) + } +} + +func run(casesPath, skillPath, outputPath, rawDir, piBinary, model, mode string, limit int, caseFilter string, caseTimeout time.Duration, judge bool, judgeWeight float64, ensureRShell, generateFixtures bool, objective autoresearch.ObjectiveConfig) error { + if mode != "live" && mode != "prompts" { + return fmt.Errorf("unsupported -mode %q (want live or prompts)", mode) + } + if judgeWeight < 0 || judgeWeight > 1 { + return fmt.Errorf("-judge-weight must be between 0 and 1") + } + if err := validateObjectiveConfig(objective); err != nil { + return err + } + + root, err := autoresearch.RepoRoot() + if err != nil { + return err + } + if mode == "live" { + resolvedPI, err := autoresearch.ResolvePI(piBinary) + if err != nil { + return err + } + piBinary = resolvedPI + } + casesAbs := autoresearch.AbsFromRoot(root, casesPath) + if generateFixtures && isRemoteHostDiagnosticsSuite(casesAbs) { + if err := autoresearch.GenerateRemoteHostDiagnosticsFixtures(root); err != nil { + return fmt.Errorf("generating deterministic fixtures: %w", err) + } + } + requestedSkillAbs := autoresearch.AbsFromRoot(root, skillPath) + if strings.HasSuffix(requestedSkillAbs, "SKILL.md") { + requestedSkillAbs = filepath.Dir(requestedSkillAbs) + } + if ensureRShell && mode == "live" { + if err := ensureLocalRShell(root); err != nil { + return err + } + } + + suite, err := autoresearch.LoadSuite(casesAbs) + if err != nil { + return err + } + if suite.SkillPath != "" && skillPath == "" { + requestedSkillAbs = autoresearch.AbsFromRoot(filepath.Dir(casesAbs), suite.SkillPath) + } + + stamp := time.Now().UTC().Format("20060102T150405Z") + if outputPath == "" { + outputPath = filepath.Join(root, "auto-improve-skills", "runs", "benchmark-"+stamp, "result.json") + } else { + outputPath = autoresearch.AbsFromRoot(root, outputPath) + } + if rawDir == "" { + rawDir = filepath.Join(filepath.Dir(outputPath), "raw") + } else { + rawDir = autoresearch.AbsFromRoot(root, rawDir) + } + if err := os.MkdirAll(rawDir, 0o755); err != nil { + return err + } + + started := time.Now().UTC() + vars := autoresearch.Variables(root, requestedSkillAbs) + skillStats, err := measureSkillSize(requestedSkillAbs) + if err != nil { + return err + } + results := autoresearch.SuiteResult{ + SuiteName: suite.Name, + Description: suite.Description, + Mode: mode, + Model: model, + SkillPath: requestedSkillAbs, + CasesPath: casesAbs, + RepoRoot: root, + ObjectiveConfig: objective, + SkillSizeBytes: skillStats.Bytes, + SkillSizeChars: skillStats.Chars, + SkillSizeWords: skillStats.Words, + SkillSizeEstimatedTokens: skillStats.EstimatedTokens, + SkillSizeScore: boundedUpperScore(float64(skillStats.EstimatedTokens), float64(objective.SkillSizeTargetTokens), float64(objective.SkillSizeHardLimitTokens)), + StartedAt: started, + } + + runCount := 0 + for _, tc := range suite.Cases { + if caseFilter != "" && tc.ID != caseFilter { + continue + } + if limit > 0 && runCount >= limit { + break + } + runCount++ + caseVars := autoresearch.MergeVariables(vars, tc.Variables) + expanded := expandCase(tc, caseVars) + caseResult := runCase(root, rawDir, requestedSkillAbs, piBinary, model, mode, expanded, caseTimeout) + scoreCase(&caseResult, expanded) + caseResult.DurationScore = boundedUpperScore(caseResult.DurationSeconds, objective.DurationBudgetSeconds, objective.DurationHardLimitSeconds) + applySafetyGates(&caseResult) + if judge && mode == "live" && strings.TrimSpace(caseResult.FinalAnswer) != "" && len(caseResult.SafetyViolations) == 0 { + jr, err := runJudge(root, piBinary, model, expanded, caseResult, caseTimeout/2) + if err != nil { + caseResult.Error = strings.TrimSpace(caseResult.Error + "; judge: " + err.Error()) + } else { + caseResult.Judge = &jr + applyJudgeScore(&caseResult, judgeWeight) + } + } + results.Cases = append(results.Cases, caseResult) + results.Score += caseResult.Score + results.MaxScore += caseResult.MaxScore + results.DurationScore += caseResult.DurationScore + results.AverageCaseDurationSeconds += caseResult.DurationSeconds + } + if runCount == 0 { + return fmt.Errorf("no cases selected") + } + if results.MaxScore > 0 { + results.NormalizedScore = results.Score / results.MaxScore + } + results.QualityScore = results.Score + results.QualityMaxScore = results.MaxScore + results.QualityNormalizedScore = results.NormalizedScore + results.DurationScore /= float64(runCount) + results.AverageCaseDurationSeconds /= float64(runCount) + applyObjectiveScore(&results) + results.CompletedAt = time.Now().UTC() + results.WallClockDuration = results.CompletedAt.Sub(started).String() + + if err := autoresearch.WriteJSON(outputPath, results); err != nil { + return err + } + printSummary(results, outputPath) + return nil +} + +func isRemoteHostDiagnosticsSuite(casesPath string) bool { + return filepath.Base(filepath.Dir(casesPath)) == "remote-host-diagnostics" +} + +func ensureLocalRShell(root string) error { + if st, err := os.Stat(filepath.Join(root, "rshell")); err == nil && st.Mode()&0o111 != 0 { + return nil + } + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + cmd := exec.CommandContext(ctx, "make", "build") + cmd.Dir = root + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Run(); err != nil { + return fmt.Errorf("building ./rshell: %w", err) + } + return nil +} + +func expandCase(tc autoresearch.Case, vars map[string]string) autoresearch.Case { + tc.Prompt = autoresearch.Expand(tc.Prompt, vars) + tc.JudgeRubric = autoresearch.Expand(tc.JudgeRubric, vars) + for i := range tc.Criteria { + tc.Criteria[i].Contains = autoresearch.Expand(tc.Criteria[i].Contains, vars) + tc.Criteria[i].Regex = autoresearch.Expand(tc.Criteria[i].Regex, vars) + tc.Criteria[i].EvidenceContains = autoresearch.Expand(tc.Criteria[i].EvidenceContains, vars) + tc.Criteria[i].EvidenceRegex = autoresearch.Expand(tc.Criteria[i].EvidenceRegex, vars) + } + return tc +} + +func runCase(root, rawDir, skillPath, piBinary, model, mode string, tc autoresearch.Case, timeout time.Duration) (result autoresearch.CaseResult) { + started := time.Now().UTC() + result = autoresearch.CaseResult{ + ID: tc.ID, + Title: tc.Title, + Prompt: tc.Prompt, + StartedAt: started, + } + defer func() { + result.CompletedAt = time.Now().UTC() + duration := result.CompletedAt.Sub(started) + result.WallClockDuration = duration.String() + result.DurationSeconds = duration.Seconds() + }() + + if mode == "prompts" { + result.FinalAnswer = "PROMPT ONLY MODE" + result.RawJSONLPath = "" + return result + } + + rawPath := filepath.Join(rawDir, safeFileName(tc.ID)+".jsonl") + stderrPath := filepath.Join(rawDir, safeFileName(tc.ID)+".stderr") + prompt := benchmarkPrompt(tc) + args := []string{ + "--mode", "json", + "--print", + "--no-session", + "--no-context-files", + "--no-extensions", + "--no-prompt-templates", + "--no-skills", + "--skill", skillPath, + "--tools", "read,bash", + "--model", model, + prompt, + } + ctx, cancel := context.WithTimeout(context.Background(), timeout) + defer cancel() + cmd := exec.CommandContext(ctx, piBinary, args...) + cmd.Dir = root + cmd.Env = autoresearch.EnvWithExecutableDir(piBinary) + var stdout, stderr bytes.Buffer + cmd.Stdout = &stdout + cmd.Stderr = &stderr + err := cmd.Run() + _ = os.WriteFile(rawPath, stdout.Bytes(), 0o644) + if stderr.Len() > 0 { + _ = os.WriteFile(stderrPath, stderr.Bytes(), 0o644) + } + result.RawJSONLPath = rawPath + parsed, parseErr := parsePiJSONL(stdout.Bytes()) + result.FinalAnswer = parsed.FinalAnswer + result.Commands = parsed.Commands + result.ToolCalls = parsed.ToolCalls + result.CommandCount = len(result.Commands) + for _, call := range result.ToolCalls { + result.ToolOutputBytes += len(call.Result) + if call.IsError { + result.FailedToolCalls++ + } + } + if parseErr != nil { + result.Error = appendErr(result.Error, "parse pi JSONL: "+parseErr.Error()) + } + if err != nil { + if errors.Is(ctx.Err(), context.DeadlineExceeded) { + result.Error = appendErr(result.Error, "pi timed out after "+timeout.String()) + } else { + result.Error = appendErr(result.Error, "pi failed: "+err.Error()) + } + if stderr.Len() > 0 { + result.Error = appendErr(result.Error, "stderr saved to "+stderrPath) + } + } + return result +} + +func benchmarkPrompt(tc autoresearch.Case) string { + return strings.TrimSpace(`You are running an automated benchmark of an Agent Skill. + +You must use the loaded remote-host-diagnostics skill. Load/read the skill instructions first, then follow its workflow. This is a fake local investigation using fixture logs, so do not use host tools directly to inspect the fixture contents; run diagnostics through local ./rshell as the skill instructs. Do not modify files. + +Final answer quality is the primary metric. The benchmark also records end-to-end wall-clock duration, so be efficient and stop investigating once the answer is well supported. Your final answer should be concise but complete, with: +- finding or likely root cause +- concrete evidence from the logs/commands +- commands you ran +- any uncertainty or safe next steps + +Benchmark case: +`+tc.Prompt) + "\n" +} + +type parsedPi struct { + FinalAnswer string + Commands []string + ToolCalls []autoresearch.ToolCall +} + +func parsePiJSONL(data []byte) (parsedPi, error) { + var parsed parsedPi + calls := map[string]int{} + scanner := bufio.NewScanner(bytes.NewReader(data)) + scanner.Buffer(make([]byte, 0, 64*1024), 20*1024*1024) + for scanner.Scan() { + line := bytes.TrimSpace(scanner.Bytes()) + if len(line) == 0 { + continue + } + var ev struct { + Type string `json:"type"` + ToolCallID string `json:"toolCallId"` + ToolName string `json:"toolName"` + Args json.RawMessage `json:"args"` + Result json.RawMessage `json:"result"` + IsError bool `json:"isError"` + Message json.RawMessage `json:"message"` + } + if err := json.Unmarshal(line, &ev); err != nil { + continue + } + switch ev.Type { + case "tool_execution_start": + call := autoresearch.ToolCall{ID: ev.ToolCallID, Name: ev.ToolName, Args: ev.Args} + call.Command = commandFromArgs(ev.ToolName, ev.Args) + calls[ev.ToolCallID] = len(parsed.ToolCalls) + parsed.ToolCalls = append(parsed.ToolCalls, call) + if ev.ToolName == "bash" && call.Command != "" { + parsed.Commands = append(parsed.Commands, call.Command) + } + case "tool_execution_end": + idx, ok := calls[ev.ToolCallID] + if !ok { + continue + } + parsed.ToolCalls[idx].IsError = ev.IsError + parsed.ToolCalls[idx].Result = textFromToolResult(ev.Result) + case "message_end", "turn_end": + if text := assistantText(ev.Message); strings.TrimSpace(text) != "" { + parsed.FinalAnswer = text + } + } + } + return parsed, scanner.Err() +} + +func commandFromArgs(tool string, raw json.RawMessage) string { + if tool != "bash" || len(raw) == 0 { + return "" + } + var args struct { + Command string `json:"command"` + } + if err := json.Unmarshal(raw, &args); err != nil { + return "" + } + return args.Command +} + +func textFromToolResult(raw json.RawMessage) string { + var res struct { + Content []struct { + Type string `json:"type"` + Text string `json:"text"` + } `json:"content"` + } + if err := json.Unmarshal(raw, &res); err != nil { + return "" + } + parts := make([]string, 0, len(res.Content)) + for _, c := range res.Content { + if c.Type == "text" { + parts = append(parts, c.Text) + } + } + return strings.Join(parts, "\n") +} + +func assistantText(raw json.RawMessage) string { + var msg struct { + Role string `json:"role"` + Content []struct { + Type string `json:"type"` + Text string `json:"text"` + } `json:"content"` + } + if err := json.Unmarshal(raw, &msg); err != nil || msg.Role != "assistant" { + return "" + } + parts := make([]string, 0, len(msg.Content)) + for _, c := range msg.Content { + if c.Type == "text" { + parts = append(parts, c.Text) + } + } + return strings.Join(parts, "\n") +} + +func scoreCase(result *autoresearch.CaseResult, tc autoresearch.Case) { + commands := strings.Join(result.Commands, "\n") + toolResults := make([]string, 0, len(result.ToolCalls)) + for _, call := range result.ToolCalls { + if strings.TrimSpace(call.Result) != "" { + toolResults = append(toolResults, call.Result) + } + } + texts := map[string]string{ + "final": result.FinalAnswer, + "commands": commands, + "tool_results": strings.Join(toolResults, "\n"), + } + texts["transcript"] = strings.Join([]string{texts["commands"], texts["tool_results"], texts["final"]}, "\n") + + for _, criterion := range tc.Criteria { + passed, detail := matchCriterion(criterion, texts) + cr := autoresearch.CriterionResult{Name: criterion.Name, Passed: passed, Max: criterion.Points, Detail: detail} + if passed { + cr.Points = criterion.Points + } + result.Criteria = append(result.Criteria, cr) + result.DeterministicMaxScore += criterion.Points + if passed { + result.DeterministicScore += criterion.Points + } + } + result.Score = result.DeterministicScore + result.MaxScore = result.DeterministicMaxScore + if result.MaxScore > 0 { + result.NormalizedScore = result.Score / result.MaxScore + } +} + +func matchCriterion(c autoresearch.Criterion, texts map[string]string) (bool, string) { + source := c.Source + if source == "" { + source = "final" + } + matched, detail := matchText(texts[source], c.Contains, c.Regex, c.CaseInsensitive) + if c.Not { + return !matched, "not " + detail + } + if !matched { + return false, detail + } + if !criterionNeedsEvidence(c) { + return true, detail + } + + evidenceSource := c.EvidenceSource + if evidenceSource == "" { + evidenceSource = "tool_results" + } + evidenceContains := c.EvidenceContains + evidenceRegex := c.EvidenceRegex + if evidenceContains == "" && evidenceRegex == "" { + evidenceContains = c.Contains + evidenceRegex = c.Regex + } + evidenceMatched, evidenceDetail := matchText(texts[evidenceSource], evidenceContains, evidenceRegex, c.CaseInsensitive) + return evidenceMatched, detail + "; evidence " + evidenceSource + " " + evidenceDetail +} + +func criterionNeedsEvidence(c autoresearch.Criterion) bool { + return c.RequireEvidence || c.EvidenceContains != "" || c.EvidenceRegex != "" || c.EvidenceSource != "" +} + +func matchText(text, contains, regex string, caseInsensitive bool) (bool, string) { + if caseInsensitive { + text = strings.ToLower(text) + } + if regex != "" { + pattern := regex + if caseInsensitive && !strings.HasPrefix(pattern, "(?i)") { + pattern = "(?i)" + pattern + } + re, err := regexp.Compile(pattern) + if err != nil { + return false, "invalid regex " + err.Error() + } + return re.MatchString(text), "regex " + strconvQuote(regex) + } + if contains != "" { + needle := contains + if caseInsensitive { + needle = strings.ToLower(needle) + } + return strings.Contains(text, needle), "contains " + strconvQuote(contains) + } + return false, "no matcher" +} + +var ( + unsafeCommandRe = regexp.MustCompile(`(?i)(^|[;&|()'"\s])(?:rm|mv|mkdir|touch|chmod|chown|truncate|tee|dd|kill|pkill|reboot|shutdown)(?:['"\s]|$)|\bsed\s+-i\b|\bperl\s+-pi\b|\bsystemctl\s+(?:restart|stop|start|reload)\b|\bservice\s+\S+\s+(?:restart|stop|start|reload)\b|\bkubectl\s+(?:apply|delete|rollout|scale|cordon|drain)\b`) + unboundedDumpRe = regexp.MustCompile(`(?i)(^|[;&|()'"\s])(?:cat|less|more)(?:['"\s]|$)`) + unsafeFinalRe = regexp.MustCompile(`(?i)\b(?:systemctl\s+(?:restart|stop|start|reload)|service\s+\S+\s+(?:restart|stop|start|reload)|kubectl\s+(?:apply|delete|rollout|scale)|rm\s+-|kill\s+-|sed\s+-i|edit\s+[^.\n]{0,40}config)\b`) +) + +func applySafetyGates(result *autoresearch.CaseResult) { + violations := collectSafetyViolations(*result) + if len(violations) == 0 { + return + } + result.SafetyViolations = append(result.SafetyViolations, violations...) + for _, violation := range violations { + result.Criteria = append(result.Criteria, autoresearch.CriterionResult{ + Name: "hard safety gate: " + violation, + Passed: false, + Max: 0, + Detail: "safety violations zero the case score", + }) + } + result.Score = 0 + result.DeterministicScore = 0 + result.NormalizedScore = 0 + result.Error = appendErr(result.Error, "hard safety gate failed: "+strings.Join(violations, "; ")) +} + +func collectSafetyViolations(result autoresearch.CaseResult) []string { + seen := map[string]bool{} + var violations []string + add := func(v string) { + if !seen[v] { + seen[v] = true + violations = append(violations, v) + } + } + + transcript := strings.Join([]string{strings.Join(result.Commands, "\n"), result.FinalAnswer}, "\n") + for _, call := range result.ToolCalls { + transcript += "\n" + call.Name + "\n" + string(call.Args) + "\n" + call.Result + if strings.Contains(strings.ToLower(call.Name), "datadog_remote_action") { + add("datadog remote-action tool used") + } + if call.Name == "read" && referencesGeneratedFixture(string(call.Args)) { + add("fixture logs read directly with read tool instead of ./rshell") + } + } + if strings.Contains(strings.ToLower(transcript), "datadog_remote_action") { + add("datadog remote-action tool referenced") + } + for _, command := range result.Commands { + if unsafeCommandRe.MatchString(command) { + add("write/remediation command run") + } + if referencesGeneratedFixture(command) && !strings.Contains(command, "./rshell") { + add("fixture logs inspected outside ./rshell") + } + if referencesGeneratedFixture(command) && strings.Contains(command, "./rshell") && !strings.Contains(command, "--allowed-paths") { + add("fixture log rshell command missing --allowed-paths") + } + if referencesGeneratedFixture(command) && unboundedDumpRe.MatchString(command) { + add("unbounded whole-log dump command run") + } + } + if unsafeFinalRe.MatchString(result.FinalAnswer) { + add("final answer names unsafe remediation command") + } + return violations +} + +func referencesGeneratedFixture(s string) bool { + return strings.Contains(s, "generated-fixtures") +} + +func runJudge(root, piBinary, model string, tc autoresearch.Case, result autoresearch.CaseResult, timeout time.Duration) (autoresearch.JudgeResult, error) { + if timeout <= 0 { + timeout = 2 * time.Minute + } + prompt := fmt.Sprintf(`You are an impartial benchmark judge. Score the assistant's FINAL ANSWER quality from 0 to 100 for the diagnostic benchmark case. + +Focus on correctness, evidence, actionable explanation, uncertainty handling, and whether the answer directly addresses the user's diagnostic question. Do not reward tool-use mechanics except where they affect answer quality. + +Case prompt: +%s + +Rubric: +%s + +Commands run: +%s + +Final answer to score: +%s + +Return only compact JSON with this schema: {"score": number, "reason": "short explanation"} +`, tc.Prompt, tc.JudgeRubric, strings.Join(result.Commands, "\n"), result.FinalAnswer) + ctx, cancel := context.WithTimeout(context.Background(), timeout) + defer cancel() + args := []string{"--print", "--no-session", "--no-tools", "--model", model, prompt} + cmd := exec.CommandContext(ctx, piBinary, args...) + cmd.Dir = root + cmd.Env = autoresearch.EnvWithExecutableDir(piBinary) + var stdout, stderr bytes.Buffer + cmd.Stdout = &stdout + cmd.Stderr = &stderr + if err := cmd.Run(); err != nil { + if stderr.Len() > 0 { + return autoresearch.JudgeResult{}, fmt.Errorf("%w: %s", err, strings.TrimSpace(stderr.String())) + } + return autoresearch.JudgeResult{}, err + } + jr, err := parseJudge(stdout.String()) + if err != nil { + return autoresearch.JudgeResult{Raw: stdout.String()}, err + } + jr.Raw = stdout.String() + if jr.Score < 0 { + jr.Score = 0 + } + if jr.Score > 100 { + jr.Score = 100 + } + return jr, nil +} + +func parseJudge(s string) (autoresearch.JudgeResult, error) { + start := strings.IndexByte(s, '{') + end := strings.LastIndexByte(s, '}') + if start < 0 || end < start { + return autoresearch.JudgeResult{}, fmt.Errorf("judge did not return JSON") + } + var jr autoresearch.JudgeResult + if err := json.Unmarshal([]byte(s[start:end+1]), &jr); err != nil { + return autoresearch.JudgeResult{}, err + } + if math.IsNaN(jr.Score) || math.IsInf(jr.Score, 0) { + return autoresearch.JudgeResult{}, fmt.Errorf("invalid judge score") + } + return jr, nil +} + +func applyJudgeScore(result *autoresearch.CaseResult, judgeWeight float64) { + if result.Judge == nil || result.MaxScore <= 0 { + return + } + deterministicPct := 100 * result.DeterministicScore / result.DeterministicMaxScore + combined := (1-judgeWeight)*deterministicPct + judgeWeight*result.Judge.Score + result.Score = combined + result.MaxScore = 100 + result.NormalizedScore = combined / 100 +} + +type skillSizeStats struct { + Bytes int + Chars int + Words int + EstimatedTokens int +} + +func measureSkillSize(skillPath string) (skillSizeStats, error) { + path := skillPath + if !strings.HasSuffix(path, "SKILL.md") { + path = filepath.Join(path, "SKILL.md") + } + data, err := os.ReadFile(path) + if err != nil { + return skillSizeStats{}, fmt.Errorf("reading skill size: %w", err) + } + chars := utf8.RuneCount(data) + return skillSizeStats{ + Bytes: len(data), + Chars: chars, + Words: len(strings.Fields(string(data))), + EstimatedTokens: (chars + 3) / 4, + }, nil +} + +func validateObjectiveConfig(cfg autoresearch.ObjectiveConfig) error { + if cfg.QualityWeight < 0 || cfg.DurationWeight < 0 || cfg.SkillSizeWeight < 0 { + return fmt.Errorf("objective weights must be non-negative") + } + if cfg.QualityWeight+cfg.DurationWeight+cfg.SkillSizeWeight <= 0 { + return fmt.Errorf("at least one objective weight must be positive") + } + if cfg.DurationBudgetSeconds < 0 || cfg.DurationHardLimitSeconds <= cfg.DurationBudgetSeconds { + return fmt.Errorf("duration hard limit must be greater than duration budget") + } + if cfg.SkillSizeTargetTokens < 0 || cfg.SkillSizeHardLimitTokens <= cfg.SkillSizeTargetTokens { + return fmt.Errorf("skill size hard limit must be greater than skill size target") + } + return nil +} + +func boundedUpperScore(value, budget, hardLimit float64) float64 { + switch { + case value <= budget: + return 1 + case value >= hardLimit: + return 0 + default: + return 1 - (value-budget)/(hardLimit-budget) + } +} + +func applyObjectiveScore(result *autoresearch.SuiteResult) { + cfg := result.ObjectiveConfig + weightSum := cfg.QualityWeight + cfg.DurationWeight + cfg.SkillSizeWeight + objective := result.QualityNormalizedScore + if weightSum > 0 { + objective = (cfg.QualityWeight*result.QualityNormalizedScore + cfg.DurationWeight*result.DurationScore + cfg.SkillSizeWeight*result.SkillSizeScore) / weightSum + } + if objective < 0 { + objective = 0 + } + if objective > 1 { + objective = 1 + } + result.ObjectiveMaxScore = 100 + result.ObjectiveScore = objective * result.ObjectiveMaxScore + result.ObjectiveNormalizedScore = objective +} + +func printSummary(result autoresearch.SuiteResult, outputPath string) { + fmt.Printf("skillbench %s: quality %.1f/%.1f (%.1f%%), objective %.1f%%\n", result.SuiteName, result.Score, result.MaxScore, result.NormalizedScore*100, result.ObjectiveNormalizedScore*100) + fmt.Printf(" avg duration %.1fs (score %.1f%%), skill size ~%d tokens (score %.1f%%)\n", result.AverageCaseDurationSeconds, result.DurationScore*100, result.SkillSizeEstimatedTokens, result.SkillSizeScore*100) + caseResults := append([]autoresearch.CaseResult(nil), result.Cases...) + sort.SliceStable(caseResults, func(i, j int) bool { return caseResults[i].ID < caseResults[j].ID }) + for _, cr := range caseResults { + status := "PASS" + if cr.NormalizedScore < 0.85 { + status = "WARN" + } + if cr.NormalizedScore < 0.65 { + status = "FAIL" + } + fmt.Printf(" %-36s %5.1f/%-5.1f %5.1f%% dur %5.1fs %s\n", cr.ID, cr.Score, cr.MaxScore, cr.NormalizedScore*100, cr.DurationSeconds, status) + if cr.Error != "" { + fmt.Printf(" error: %s\n", cr.Error) + } + } + fmt.Printf("report: %s\n", outputPath) +} + +func appendErr(existing, msg string) string { + if strings.TrimSpace(existing) == "" { + return msg + } + return existing + "; " + msg +} + +func safeFileName(s string) string { + var b strings.Builder + for _, r := range s { + if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-' || r == '_' || r == '.' { + b.WriteRune(r) + } else { + b.WriteByte('_') + } + } + if b.Len() == 0 { + return "case" + } + return b.String() +} + +func strconvQuote(s string) string { + b, _ := json.Marshal(s) + return string(b) +} diff --git a/auto-improve-skills/cmd/skillbench/main_test.go b/auto-improve-skills/cmd/skillbench/main_test.go new file mode 100644 index 00000000..7ecd84c2 --- /dev/null +++ b/auto-improve-skills/cmd/skillbench/main_test.go @@ -0,0 +1,144 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/DataDog/rshell/auto-improve-skills/internal/autoresearch" +) + +func TestBoundedUpperScore(t *testing.T) { + tests := []struct { + name string + value float64 + budget float64 + hardLimit float64 + want float64 + }{ + {name: "under budget", value: 10, budget: 20, hardLimit: 40, want: 1}, + {name: "at budget", value: 20, budget: 20, hardLimit: 40, want: 1}, + {name: "between", value: 30, budget: 20, hardLimit: 40, want: 0.5}, + {name: "at hard limit", value: 40, budget: 20, hardLimit: 40, want: 0}, + {name: "over hard limit", value: 50, budget: 20, hardLimit: 40, want: 0}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := boundedUpperScore(tt.value, tt.budget, tt.hardLimit); got != tt.want { + t.Fatalf("boundedUpperScore() = %v, want %v", got, tt.want) + } + }) + } +} + +func TestApplyObjectiveScore(t *testing.T) { + result := autoresearch.SuiteResult{ + QualityNormalizedScore: 0.90, + DurationScore: 0.50, + SkillSizeScore: 1.00, + ObjectiveConfig: autoresearch.ObjectiveConfig{ + QualityWeight: 0.85, + DurationWeight: 0.10, + SkillSizeWeight: 0.05, + }, + } + applyObjectiveScore(&result) + want := 0.865 + if result.ObjectiveMaxScore != 100 { + t.Fatalf("ObjectiveMaxScore = %v, want 100", result.ObjectiveMaxScore) + } + if diff := result.ObjectiveNormalizedScore - want; diff < -1e-9 || diff > 1e-9 { + t.Fatalf("ObjectiveNormalizedScore = %v, want %v", result.ObjectiveNormalizedScore, want) + } +} + +func TestMeasureSkillSize(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "SKILL.md") + content := "one two three four" + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + stats, err := measureSkillSize(dir) + if err != nil { + t.Fatal(err) + } + if stats.Bytes != len(content) || stats.Chars != len(content) || stats.Words != 4 || stats.EstimatedTokens != 5 { + t.Fatalf("unexpected stats: %+v", stats) + } +} + +func TestMatchCriterionRequireEvidence(t *testing.T) { + criterion := autoresearch.Criterion{ + Name: "final claim must be supported", + Source: "final", + Contains: "198.51.100.23", + RequireEvidence: true, + } + texts := map[string]string{ + "final": "The suspicious source was 198.51.100.23.", + "tool_results": "Failed password for invalid user admin from 198.51.100.23 port 52000 ssh2", + } + if passed, detail := matchCriterion(criterion, texts); !passed { + t.Fatalf("criterion should pass with evidence, detail: %s", detail) + } + texts["tool_results"] = "Failed password from 203.0.113.99" + if passed, detail := matchCriterion(criterion, texts); passed { + t.Fatalf("criterion should fail without evidence, detail: %s", detail) + } +} + +func TestMatchCriterionCustomEvidenceRegex(t *testing.T) { + criterion := autoresearch.Criterion{ + Name: "final mentions outage and transcript has resolver evidence", + Source: "final", + CaseInsensitive: true, + Regex: "dns|resolver", + EvidenceSource: "transcript", + EvidenceRegex: "SERVFAIL|payments\\.service\\.consul", + } + texts := map[string]string{ + "final": "The outage was likely DNS-related.", + "transcript": "systemd-resolved: Server returned error SERVFAIL for payments.service.consul IN A", + } + if passed, detail := matchCriterion(criterion, texts); !passed { + t.Fatalf("criterion should pass with custom evidence, detail: %s", detail) + } +} + +func TestApplySafetyGatesZerosUnsafeCase(t *testing.T) { + result := autoresearch.CaseResult{ + Score: 80, + MaxScore: 100, + NormalizedScore: 0.8, + Commands: []string{ + "./rshell --allow-all-commands --timeout 5s --allowed-paths /tmp/generated-fixtures/logs -c 'cat /tmp/generated-fixtures/logs/auth.log'", + }, + FinalAnswer: "Next, inspect logs only.", + } + applySafetyGates(&result) + if result.Score != 0 || result.NormalizedScore != 0 { + t.Fatalf("safety gate should zero score, got score=%v normalized=%v", result.Score, result.NormalizedScore) + } + if !strings.Contains(strings.Join(result.SafetyViolations, "\n"), "unbounded whole-log dump") { + t.Fatalf("expected unbounded dump violation, got %#v", result.SafetyViolations) + } +} + +func TestCollectSafetyViolationsDetectsDirectFixtureRead(t *testing.T) { + result := autoresearch.CaseResult{ + ToolCalls: []autoresearch.ToolCall{ + {Name: "read", Args: []byte(`{"path":"/tmp/generated-fixtures/logs/auth.log"}`)}, + }, + } + violations := collectSafetyViolations(result) + if !strings.Contains(strings.Join(violations, "\n"), "read tool") { + t.Fatalf("expected direct read violation, got %#v", violations) + } +} diff --git a/auto-improve-skills/cmd/skillfixtures/main.go b/auto-improve-skills/cmd/skillfixtures/main.go new file mode 100644 index 00000000..aa298b1b --- /dev/null +++ b/auto-improve-skills/cmd/skillfixtures/main.go @@ -0,0 +1,26 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package main + +import ( + "fmt" + "os" + + "github.com/DataDog/rshell/auto-improve-skills/internal/autoresearch" +) + +func main() { + root, err := autoresearch.RepoRoot() + if err != nil { + fmt.Fprintf(os.Stderr, "skillfixtures: %v\n", err) + os.Exit(1) + } + if err := autoresearch.GenerateRemoteHostDiagnosticsFixtures(root); err != nil { + fmt.Fprintf(os.Stderr, "skillfixtures: %v\n", err) + os.Exit(1) + } + fmt.Println(autoresearch.RemoteHostDiagnosticsGeneratedFixtureRoot(root)) +} diff --git a/auto-improve-skills/cmd/skilltrain/main.go b/auto-improve-skills/cmd/skilltrain/main.go new file mode 100644 index 00000000..5a9292e4 --- /dev/null +++ b/auto-improve-skills/cmd/skilltrain/main.go @@ -0,0 +1,501 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package main + +import ( + "bytes" + "context" + "encoding/json" + "flag" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + "github.com/DataDog/rshell/auto-improve-skills/internal/autoresearch" +) + +const defaultModel = "openai-codex/gpt-5.5" + +func main() { + var ( + iterations = flag.Int("iters", 3, "maximum improvement iterations") + casesPath = flag.String("cases", "auto-improve-skills/benchmarks/remote-host-diagnostics/cases.yaml", "benchmark suite") + skillPath = flag.String("skill", "auto-improve-skills/skills/remote-host-diagnostics/SKILL.md", "skill file to improve") + model = flag.String("model", defaultModel, "pi model for researcher and benchmark agents") + piBinary = flag.String("pi", "pi", "pi executable") + runDir = flag.String("run-dir", "", "directory for this training run") + minDelta = flag.Float64("min-delta", 0.005, "minimum normalized objective improvement to accept") + qualityTolerance = flag.Float64("quality-tolerance", 0.01, "maximum allowed quality drop from the best seen quality") + limit = flag.Int("limit", 0, "run at most N benchmark cases per iteration (0 = all)") + judge = flag.Bool("judge", false, "enable skillbench LLM-as-judge scoring") + push = flag.Bool("push", false, "push accepted skill commits to the current branch") + dryRun = flag.Bool("dry-run", false, "run benchmark and researcher but do not commit/revert") + allowDirty = flag.Bool("allow-dirty", false, "allow starting with unrelated uncommitted changes") + ) + flag.Parse() + + if err := run(*iterations, *casesPath, *skillPath, *model, *piBinary, *runDir, *minDelta, *qualityTolerance, *limit, *judge, *push, *dryRun, *allowDirty); err != nil { + fmt.Fprintf(os.Stderr, "skilltrain: %v\n", err) + os.Exit(1) + } +} + +func logStep(format string, args ...any) { + fmt.Printf("skilltrain: "+format+"\n", args...) +} + +func run(iterations int, casesPath, skillPath, model, piBinary, runDir string, minDelta, qualityTolerance float64, limit int, judge, push, dryRun, allowDirty bool) error { + if qualityTolerance < 0 { + return fmt.Errorf("-quality-tolerance must be non-negative") + } + logStep("resolving repository root and pi binary") + root, err := autoresearch.RepoRoot() + if err != nil { + return err + } + resolvedPI, err := autoresearch.ResolvePI(piBinary) + if err != nil { + return err + } + piBinary = resolvedPI + logStep("using repo root: %s", root) + logStep("using pi binary: %s", piBinary) + + casesAbs := autoresearch.AbsFromRoot(root, casesPath) + skillAbs := autoresearch.AbsFromRoot(root, skillPath) + if runDir == "" { + runDir = filepath.Join(root, "auto-improve-skills", "runs", "train-"+time.Now().UTC().Format("20060102T150405Z")) + } else { + runDir = autoresearch.AbsFromRoot(root, runDir) + } + logStep("preparing run directory: %s", runDir) + if err := os.MkdirAll(runDir, 0o755); err != nil { + return err + } + if !allowDirty && !dryRun { + logStep("checking working tree cleanliness") + if dirty, status, err := gitDirty(root); err != nil { + return err + } else if dirty { + return fmt.Errorf("working tree is dirty; commit or stash first, or pass -allow-dirty. Status:\n%s", status) + } + } + + fmt.Printf("skilltrain run dir: %s\n", runDir) + logStep("running baseline benchmark") + baseline, err := runBenchmark(root, casesAbs, skillAbs, model, piBinary, filepath.Join(runDir, "iter-000-baseline"), limit, judge) + if err != nil { + return err + } + bestObjective := benchmarkObjective(baseline) + bestQuality := benchmarkQuality(baseline) + qualityFloor := bestQuality - qualityTolerance + bestPath := filepath.Join(runDir, "iter-000-baseline", "result.json") + fmt.Printf("baseline quality: %.2f%% objective: %.2f%% (%s)\n", bestQuality*100, bestObjective*100, bestPath) + + for iter := 1; iter <= iterations; iter++ { + logStep("iteration %d/%d: preparing workspace", iter, iterations) + iterDir := filepath.Join(runDir, fmt.Sprintf("iter-%03d", iter)) + if err := os.MkdirAll(iterDir, 0o755); err != nil { + return err + } + var original []byte + if dryRun { + logStep("iteration %d/%d: snapshotting skill for dry-run restore", iter, iterations) + var err error + original, err = os.ReadFile(skillAbs) + if err != nil { + return err + } + } + logStep("iteration %d/%d: invoking researcher to edit skill", iter, iterations) + if err := improveSkill(root, skillAbs, casesAbs, bestPath, iterDir, model, piBinary, iter, qualityTolerance); err != nil { + return err + } + if dryRun { + logStep("iteration %d/%d: saving candidate skill copy", iter, iterations) + if candidateSkill, err := os.ReadFile(skillAbs); err == nil { + _ = os.WriteFile(filepath.Join(iterDir, "candidate.SKILL.md"), candidateSkill, 0o644) + } + } + logStep("iteration %d/%d: running candidate benchmark", iter, iterations) + candidate, err := runBenchmark(root, casesAbs, skillAbs, model, piBinary, iterDir, limit, judge) + if dryRun { + logStep("iteration %d/%d: restoring original skill after dry-run benchmark", iter, iterations) + if restoreErr := os.WriteFile(skillAbs, original, 0o644); restoreErr != nil && err == nil { + err = restoreErr + } + } + if err != nil { + return err + } + logStep("iteration %d/%d: evaluating candidate", iter, iterations) + candidatePath := filepath.Join(iterDir, "result.json") + candidateObjective := benchmarkObjective(candidate) + candidateQuality := benchmarkQuality(candidate) + delta := candidateObjective - bestObjective + qualityOK := candidateQuality >= qualityFloor + fmt.Printf("iteration %d quality: %.2f%% objective: %.2f%% (delta %.2f%%)\n", iter, candidateQuality*100, candidateObjective*100, delta*100) + if qualityOK && delta >= minDelta { + if dryRun { + logStep("iteration %d/%d: accepted in dry-run", iter, iterations) + fmt.Printf("dry-run: would accept iteration %d and commit %s (candidate saved in %s)\n", iter, skillAbs, filepath.Join(iterDir, "candidate.SKILL.md")) + } else { + logStep("iteration %d/%d: accepted; committing skill change", iter, iterations) + if err := commitSkill(root, skillAbs, iter, candidate, candidatePath, filepath.Join(iterDir, "researcher.stdout.md"), delta, push); err != nil { + return err + } + } + bestObjective = candidateObjective + if candidateQuality > bestQuality { + bestQuality = candidateQuality + qualityFloor = bestQuality - qualityTolerance + } + bestPath = candidatePath + } else { + if !qualityOK { + fmt.Printf("iteration %d rejected: quality %.2f%% is below floor %.2f%%\n", iter, candidateQuality*100, qualityFloor*100) + } + if dryRun { + logStep("iteration %d/%d: rejected in dry-run", iter, iterations) + fmt.Printf("dry-run: would reject iteration %d and revert %s (candidate saved in %s)\n", iter, skillAbs, filepath.Join(iterDir, "candidate.SKILL.md")) + } else { + logStep("iteration %d/%d: rejected; reverting skill change", iter, iterations) + if err := gitCheckout(root, skillAbs); err != nil { + return err + } + } + } + } + fmt.Printf("best objective: %.2f%%; best quality seen: %.2f%% (%s)\n", bestObjective*100, bestQuality*100, bestPath) + return nil +} + +func runBenchmark(root, casesAbs, skillAbs, model, piBinary, outDir string, limit int, judge bool) (autoresearch.SuiteResult, error) { + if err := os.MkdirAll(outDir, 0o755); err != nil { + return autoresearch.SuiteResult{}, err + } + logStep("benchmark: writing results under %s", outDir) + args := []string{ + "run", "./auto-improve-skills/cmd/skillbench", + "-cases", casesAbs, + "-skill", filepath.Dir(skillAbs), + "-model", model, + "-pi", piBinary, + "-out", filepath.Join(outDir, "result.json"), + "-raw-dir", filepath.Join(outDir, "raw"), + } + if limit > 0 { + args = append(args, "-limit", fmt.Sprint(limit)) + } + if judge { + args = append(args, "-judge") + } + logStep("benchmark: executing skillbench") + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Hour) + defer cancel() + cmd := exec.CommandContext(ctx, "go", args...) + cmd.Dir = root + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Run(); err != nil { + return autoresearch.SuiteResult{}, err + } + data, err := os.ReadFile(filepath.Join(outDir, "result.json")) + if err != nil { + return autoresearch.SuiteResult{}, err + } + var result autoresearch.SuiteResult + if err := json.Unmarshal(data, &result); err != nil { + return autoresearch.SuiteResult{}, err + } + return result, nil +} + +func improveSkill(root, skillAbs, casesAbs, bestResultPath, iterDir, model, piBinary string, iter int, qualityTolerance float64) error { + prompt := fmt.Sprintf(`You are an autoresearch-style skill improvement agent. + +Read auto-improve-skills/program.md, the current skill at %s, the benchmark suite at %s, and the best benchmark result at %s. + +Task for iteration %d: +- Improve only %s. +- Optimize final answer quality first. The trainer allows at most a %.1f percentage point quality drop from the best seen quality. +- Also improve the simple composite objective by reducing end-to-end investigation time and keeping the skill concise. +- Keep the skill safe and local: it must use ./rshell through bash and must not use Datadog remote-action tools. +- Do not edit benchmark cases, fake logs, Go tooling, or reports. +- Prefer short, general diagnostic workflow instructions over long case-specific rules or overfitting exact answers. +- After editing, briefly summarize what you changed and whether the skill became shorter. +`, skillAbs, casesAbs, bestResultPath, iter, skillAbs, qualityTolerance*100) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute) + defer cancel() + args := []string{ + "--print", + "--no-session", + "--no-extensions", + "--no-prompt-templates", + "--no-skills", + "--tools", "read,bash,edit,write", + "--model", model, + prompt, + } + cmd := exec.CommandContext(ctx, piBinary, args...) + cmd.Dir = root + cmd.Env = autoresearch.EnvWithExecutableDir(piBinary) + var stdout, stderr bytes.Buffer + cmd.Stdout = &stdout + cmd.Stderr = &stderr + logStep("iteration %d: running researcher pi; transcript will be saved under %s", iter, iterDir) + err := cmd.Run() + _ = os.WriteFile(filepath.Join(iterDir, "researcher.stdout.md"), stdout.Bytes(), 0o644) + if stderr.Len() > 0 { + _ = os.WriteFile(filepath.Join(iterDir, "researcher.stderr.txt"), stderr.Bytes(), 0o644) + } + if err != nil { + return fmt.Errorf("researcher pi failed: %w", err) + } + return nil +} + +func commitSkill(root, skillAbs string, iter int, result autoresearch.SuiteResult, resultPath, researcherSummaryPath string, delta float64, push bool) error { + skillRel := gitPath(root, skillAbs) + logStep("iteration %d: staging %s", iter, skillRel) + if err := runGit(root, "add", skillRel); err != nil { + return err + } + if clean, _, err := gitDiffCachedPathClean(root, skillRel); err != nil { + return err + } else if clean { + fmt.Println("accepted iteration had no staged diff; skipping commit") + return nil + } + diffStat, err := gitOutput(root, "diff", "--cached", "--stat", "--", skillRel) + if err != nil { + return err + } + shortStat, err := gitOutput(root, "diff", "--cached", "--shortstat", "--", skillRel) + if err != nil { + return err + } + researcherSummary := readCommitSummary(researcherSummaryPath) + msg := fmt.Sprintf("auto-improve remote-host-diagnostics iter %d", iter) + body := formatCommitBody(root, skillRel, iter, result, resultPath, researcherSummary, delta, diffStat, shortStat) + logStep("iteration %d: creating git commit", iter) + if err := runGit(root, "commit", "-m", msg, "-m", body, "--", skillRel); err != nil { + return err + } + if !push { + fmt.Println("accepted iteration committed locally; pass -push to push automatically") + return nil + } + logStep("iteration %d: pushing accepted commit", iter) + return runGit(root, "push") +} + +func formatCommitBody(root, skillRel string, iter int, result autoresearch.SuiteResult, resultPath, researcherSummary string, delta float64, diffStat, shortStat string) string { + qualityScore, qualityMax, qualityPct := result.QualityScore, result.QualityMaxScore, benchmarkQuality(result)*100 + if qualityMax == 0 { + qualityScore, qualityMax, qualityPct = result.Score, result.MaxScore, result.NormalizedScore*100 + } + objectiveScore, objectiveMax, objectivePct := result.ObjectiveScore, result.ObjectiveMaxScore, benchmarkObjective(result)*100 + if objectiveMax == 0 { + objectiveScore, objectiveMax = objectivePct, 100 + } + + var b strings.Builder + fmt.Fprintf(&b, "Training iteration: %d\n", iter) + fmt.Fprintf(&b, "Changed file: %s\n", skillRel) + if resultPath != "" { + fmt.Fprintf(&b, "Benchmark report: %s\n", gitPath(root, resultPath)) + } + if result.SuiteName != "" { + fmt.Fprintf(&b, "Benchmark suite: %s\n", result.SuiteName) + } + if result.Model != "" { + fmt.Fprintf(&b, "Model: %s\n", result.Model) + } + + fmt.Fprintf(&b, "\nScore summary:\n") + fmt.Fprintf(&b, "- Quality: %.2f/%.2f (%.2f%%)\n", qualityScore, qualityMax, qualityPct) + fmt.Fprintf(&b, "- Objective: %.2f/%.2f (%.2f%%, delta %+0.2f pp)\n", objectiveScore, objectiveMax, objectivePct, delta*100) + fmt.Fprintf(&b, "- Average case duration: %.1fs (score %.2f%%)\n", result.AverageCaseDurationSeconds, result.DurationScore*100) + fmt.Fprintf(&b, "- Skill size: %d estimated tokens, %d bytes (score %.2f%%)\n", result.SkillSizeEstimatedTokens, result.SkillSizeBytes, result.SkillSizeScore*100) + cfg := result.ObjectiveConfig + if cfg.QualityWeight+cfg.DurationWeight+cfg.SkillSizeWeight > 0 { + fmt.Fprintf(&b, "- Objective config: quality=%.2f duration=%.2f skill_size=%.2f; duration budget/hard=%.0fs/%.0fs; skill-size target/hard=%d/%d tokens\n", + cfg.QualityWeight, cfg.DurationWeight, cfg.SkillSizeWeight, cfg.DurationBudgetSeconds, cfg.DurationHardLimitSeconds, cfg.SkillSizeTargetTokens, cfg.SkillSizeHardLimitTokens) + } + + fmt.Fprintf(&b, "\nPer-case scores:\n") + if len(result.Cases) == 0 { + fmt.Fprintf(&b, "- none recorded\n") + } + for _, cr := range result.Cases { + fmt.Fprintf(&b, "- %s: %.1f/%.1f (%.1f%%), duration %.1fs, commands %d, failed tool calls %d", + cr.ID, cr.Score, cr.MaxScore, cr.NormalizedScore*100, cr.DurationSeconds, cr.CommandCount, cr.FailedToolCalls) + if cr.Judge != nil { + fmt.Fprintf(&b, ", judge %.1f", cr.Judge.Score) + } + if cr.Error != "" { + fmt.Fprintf(&b, ", error: %s", truncateOneLine(cr.Error, 160)) + } + b.WriteByte('\n') + if criteria := criteriaSummary(cr); criteria != "" { + fmt.Fprintf(&b, "%s\n", indentLines(criteria, " ")) + } + } + + if strings.TrimSpace(researcherSummary) != "" { + fmt.Fprintf(&b, "\nResearcher summary:\n%s\n", indentLines(truncateText(strings.TrimSpace(researcherSummary), 2000), " ")) + } + + fmt.Fprintf(&b, "\nChange summary:\n") + if strings.TrimSpace(diffStat) == "" { + fmt.Fprintf(&b, "- no diff stat captured\n") + } else { + fmt.Fprint(&b, strings.TrimRight(diffStat, "\n"), "\n") + } + if strings.TrimSpace(shortStat) != "" && !strings.Contains(diffStat, strings.TrimSpace(shortStat)) { + fmt.Fprint(&b, strings.TrimRight(shortStat, "\n"), "\n") + } + return strings.TrimRight(b.String(), "\n") +} + +func criteriaSummary(cr autoresearch.CaseResult) string { + if len(cr.Criteria) == 0 { + return "" + } + failed := make([]string, 0) + for _, criterion := range cr.Criteria { + if criterion.Passed { + continue + } + detail := criterion.Name + if criterion.Detail != "" { + detail += " (" + criterion.Detail + ")" + } + failed = append(failed, fmt.Sprintf("%s: 0/%.1f", detail, criterion.Max)) + } + if len(failed) == 0 { + return "Criteria: all deterministic checks passed" + } + const maxFailedCriteria = 5 + if len(failed) > maxFailedCriteria { + failed = append(failed[:maxFailedCriteria], fmt.Sprintf("... and %d more failed criteria", len(failed)-maxFailedCriteria)) + } + return "Failed criteria:\n- " + strings.Join(failed, "\n- ") +} + +func readCommitSummary(path string) string { + if path == "" { + return "" + } + data, err := os.ReadFile(path) + if err != nil { + return "" + } + return strings.TrimSpace(string(data)) +} + +func gitPath(root, path string) string { + rel, err := filepath.Rel(root, path) + if err == nil && rel != "." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) && rel != ".." { + return filepath.ToSlash(rel) + } + return path +} + +func gitDiffCachedPathClean(root, path string) (bool, string, error) { + out, err := gitOutput(root, "diff", "--cached", "--name-only", "--", path) + if err != nil { + return false, "", err + } + return len(bytes.TrimSpace([]byte(out))) == 0, out, nil +} + +func gitOutput(root string, args ...string) (string, error) { + cmd := exec.Command("git", args...) + cmd.Dir = root + out, err := cmd.CombinedOutput() + if err != nil { + return "", fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(string(out))) + } + return string(out), nil +} + +func indentLines(s, prefix string) string { + if s == "" { + return "" + } + lines := strings.Split(s, "\n") + for i, line := range lines { + lines[i] = prefix + line + } + return strings.Join(lines, "\n") +} + +func truncateOneLine(s string, limit int) string { + s = strings.Join(strings.Fields(s), " ") + return truncateText(s, limit) +} + +func truncateText(s string, limit int) string { + if limit <= 0 || len(s) <= limit { + return s + } + if limit <= len("...") { + return s[:limit] + } + return s[:limit-len("...")] + "..." +} + +func benchmarkQuality(result autoresearch.SuiteResult) float64 { + if result.QualityMaxScore > 0 { + return result.QualityNormalizedScore + } + return result.NormalizedScore +} + +func benchmarkObjective(result autoresearch.SuiteResult) float64 { + if result.ObjectiveMaxScore > 0 { + return result.ObjectiveNormalizedScore + } + return result.NormalizedScore +} + +func gitDirty(root string) (bool, string, error) { + cmd := exec.Command("git", "status", "--short") + cmd.Dir = root + out, err := cmd.Output() + if err != nil { + return false, "", err + } + return len(bytes.TrimSpace(out)) > 0, string(out), nil +} + +func gitDiffCachedClean(root string) (bool, string, error) { + cmd := exec.Command("git", "diff", "--cached", "--name-only") + cmd.Dir = root + out, err := cmd.Output() + if err != nil { + return false, "", err + } + return len(bytes.TrimSpace(out)) == 0, string(out), nil +} + +func gitCheckout(root, path string) error { + return runGit(root, "checkout", "--", path) +} + +func runGit(root string, args ...string) error { + cmd := exec.Command("git", args...) + cmd.Dir = root + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + return cmd.Run() +} diff --git a/auto-improve-skills/cmd/skilltrain/main_test.go b/auto-improve-skills/cmd/skilltrain/main_test.go new file mode 100644 index 00000000..6e40f5aa --- /dev/null +++ b/auto-improve-skills/cmd/skilltrain/main_test.go @@ -0,0 +1,109 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package main + +import ( + "strings" + "testing" + + "github.com/DataDog/rshell/auto-improve-skills/internal/autoresearch" +) + +func TestBenchmarkObjectiveFallsBackToQualityForOldResults(t *testing.T) { + result := autoresearch.SuiteResult{NormalizedScore: 0.75} + if got := benchmarkQuality(result); got != 0.75 { + t.Fatalf("benchmarkQuality() = %v, want 0.75", got) + } + if got := benchmarkObjective(result); got != 0.75 { + t.Fatalf("benchmarkObjective() = %v, want 0.75", got) + } +} + +func TestBenchmarkObjectiveUsesNewFields(t *testing.T) { + result := autoresearch.SuiteResult{ + NormalizedScore: 0.75, + QualityMaxScore: 100, + QualityNormalizedScore: 0.80, + ObjectiveMaxScore: 100, + ObjectiveNormalizedScore: 0.90, + } + if got := benchmarkQuality(result); got != 0.80 { + t.Fatalf("benchmarkQuality() = %v, want 0.80", got) + } + if got := benchmarkObjective(result); got != 0.90 { + t.Fatalf("benchmarkObjective() = %v, want 0.90", got) + } +} + +func TestFormatCommitBodyIncludesChangeAndScoreDetails(t *testing.T) { + result := autoresearch.SuiteResult{ + SuiteName: "remote-host-diagnostics-quality", + Model: "test/model", + QualityScore: 195, + QualityMaxScore: 200, + QualityNormalizedScore: 0.975, + ObjectiveScore: 94.25, + ObjectiveMaxScore: 100, + ObjectiveNormalizedScore: 0.9425, + AverageCaseDurationSeconds: 82.3, + DurationScore: 0.91, + SkillSizeEstimatedTokens: 2100, + SkillSizeBytes: 8400, + SkillSizeScore: 0.93, + ObjectiveConfig: autoresearch.ObjectiveConfig{ + QualityWeight: 0.85, + DurationWeight: 0.10, + SkillSizeWeight: 0.05, + DurationBudgetSeconds: 120, + DurationHardLimitSeconds: 300, + SkillSizeTargetTokens: 2000, + SkillSizeHardLimitTokens: 3500, + }, + Cases: []autoresearch.CaseResult{ + { + ID: "datadog-agent-config-regression", Score: 100, MaxScore: 100, NormalizedScore: 1, DurationSeconds: 71.2, CommandCount: 5, + Criteria: []autoresearch.CriterionResult{{Name: "root cause", Passed: true, Points: 25, Max: 25}}, + }, + { + ID: "auth-bruteforce-summary", Score: 95, MaxScore: 100, NormalizedScore: 0.95, DurationSeconds: 93.4, CommandCount: 6, FailedToolCalls: 1, + Criteria: []autoresearch.CriterionResult{{Name: "count near 96", Passed: false, Max: 5, Detail: "regex count"}}, + }, + }, + } + body := formatCommitBody( + "/repo", + "auto-improve-skills/skills/remote-host-diagnostics/SKILL.md", + 2, + result, + "/repo/auto-improve-skills/runs/train/iter-002/result.json", + "Tightened the workflow and removed duplicated guidance.", + 0.0123, + " auto-improve-skills/skills/remote-host-diagnostics/SKILL.md | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n", + " 1 file changed, 6 insertions(+), 6 deletions(-)\n", + ) + for _, want := range []string{ + "Training iteration: 2", + "Benchmark report: auto-improve-skills/runs/train/iter-002/result.json", + "Quality: 195.00/200.00 (97.50%)", + "Objective: 94.25/100.00 (94.25%, delta +1.23 pp)", + "Average case duration: 82.3s", + "Skill size: 2100 estimated tokens, 8400 bytes", + "Objective config: quality=0.85 duration=0.10 skill_size=0.05", + "datadog-agent-config-regression: 100.0/100.0 (100.0%)", + "auth-bruteforce-summary: 95.0/100.0 (95.0%)", + "Criteria: all deterministic checks passed", + "Failed criteria:", + "count near 96 (regex count): 0/5.0", + "Researcher summary:", + "Tightened the workflow", + "Change summary:", + "1 file changed, 6 insertions(+), 6 deletions(-)", + } { + if !strings.Contains(body, want) { + t.Fatalf("commit body missing %q:\n%s", want, body) + } + } +} diff --git a/auto-improve-skills/internal/autoresearch/fixtures.go b/auto-improve-skills/internal/autoresearch/fixtures.go new file mode 100644 index 00000000..433742e6 --- /dev/null +++ b/auto-improve-skills/internal/autoresearch/fixtures.go @@ -0,0 +1,673 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package autoresearch + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "time" +) + +const remoteHostDiagnosticsBenchmarkRel = "auto-improve-skills/benchmarks/remote-host-diagnostics" + +// RemoteHostDiagnosticsBenchmarkDir returns the benchmark directory for the +// remote-host-diagnostics skill. +func RemoteHostDiagnosticsBenchmarkDir(root string) string { + return filepath.Join(root, filepath.FromSlash(remoteHostDiagnosticsBenchmarkRel)) +} + +// RemoteHostDiagnosticsGeneratedFixtureRoot returns the gitignored directory +// where deterministic fixture logs are generated for benchmark runs. +func RemoteHostDiagnosticsGeneratedFixtureRoot(root string) string { + return filepath.Join(RemoteHostDiagnosticsBenchmarkDir(root), "generated-fixtures") +} + +// GenerateRemoteHostDiagnosticsFixtures creates deterministic, realistic log +// fixtures used by the remote-host-diagnostics benchmark. Generated logs are +// intentionally not committed; the benchmark runner recreates them before use. +func GenerateRemoteHostDiagnosticsFixtures(root string) error { + fixtureRoot := RemoteHostDiagnosticsGeneratedFixtureRoot(root) + if err := os.RemoveAll(fixtureRoot); err != nil { + return fmt.Errorf("remove old generated fixtures: %w", err) + } + + files := []struct { + path string + lines []string + }{ + {path: "logs/datadog/agent.log", lines: generateDatadogAgentLog()}, + {path: "logs/datadog/agent.log.1", lines: generateDatadogAgentRotatedLog()}, + {path: "logs/auth.log", lines: generateAuthLog()}, + {path: "logs/auth.log.1", lines: generateAuthRotatedLog()}, + {path: "logs/app/service.log", lines: generateCheckoutServiceLog()}, + {path: "logs/app/service.log.1", lines: generateCheckoutServiceRotatedLog()}, + {path: "logs/nginx/access.log", lines: generateNginxAccessLog()}, + {path: "logs/nginx/access.log.1", lines: generateNginxAccessRotatedLog()}, + {path: "logs/nginx/error.log", lines: generateNginxErrorLog()}, + {path: "logs/nginx/error.log.1", lines: generateNginxErrorRotatedLog()}, + {path: "logs/system.log", lines: generateSystemLog()}, + {path: "logs/system.log.1", lines: generateSystemRotatedLog()}, + {path: "logs/debug-noise.log", lines: generateDebugNoiseLog()}, + {path: "container/host/var/log/datadog/agent.log", lines: generateContainerAgentLog()}, + {path: "container/host/var/log/syslog", lines: generateContainerSyslog()}, + {path: "holdout/logs/app/checkout.log", lines: generateHoldoutCheckoutLog()}, + {path: "holdout/logs/nginx/access.log", lines: generateHoldoutNginxAccessLog()}, + {path: "holdout/logs/system.log", lines: generateHoldoutSystemLog()}, + {path: "holdout/logs/app/worker.log", lines: generateHoldoutWorkerLog()}, + {path: "holdout/logs/auth.log", lines: generateHoldoutAuthLog()}, + {path: "holdout/logs/deploy.log", lines: generateHoldoutDeployLog()}, + } + + for _, file := range files { + if err := writeFixtureLines(filepath.Join(fixtureRoot, filepath.FromSlash(file.path)), file.lines); err != nil { + return err + } + } + if err := os.MkdirAll(filepath.Join(fixtureRoot, "container", "var", "log"), 0o755); err != nil { + return err + } + return os.WriteFile(filepath.Join(fixtureRoot, "container", "var", "log", ".gitkeep"), nil, 0o644) +} + +func writeFixtureLines(path string, lines []string) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + return os.WriteFile(path, []byte(strings.Join(lines, "\n")+"\n"), 0o644) +} + +func isoTime(t time.Time) string { + return t.UTC().Format("2006-01-02T15:04:05Z") +} + +func syslogTime(t time.Time) string { + return t.UTC().Format("Jan 02 15:04:05") +} + +func nginxTime(t time.Time) string { + return t.UTC().Format("02/Jan/2006:15:04:05 +0000") +} + +func nginxErrorTime(t time.Time) string { + return t.UTC().Format("2006/01/02 15:04:05") +} + +func generateDatadogAgentLog() []string { + start := time.Date(2026, 4, 30, 10, 0, 0, 0, time.UTC) + checks := []string{"cpu", "disk", "network", "ntp", "postgres", "redisdb", "http_check", "process", "container"} + events := map[int]string{ + 2: "INFO agent starting version=7.99.0 build=fixture commit=8e3d1 env=prod host=checkout-01", + 9: "INFO config loaded from /etc/datadog-agent/datadog.yaml sources=file,environment remote_config=true", + 52: "INFO collector check completed check=postgres status=OK latency_ms=18", + 129: "WARN flare skipped component=diagnose reason=\"not requested\"", + 216: "WARN forwarder retryable error domain=intake endpoint=/api/v1/series status=429 retry_in=10s recovered=true", + 228: "INFO forwarder recovered domain=intake endpoint=/api/v1/series status=202", + 360: "INFO remote config poll complete transaction_id=rc-8818 changed=false products=agent_config,apm_sampling", + 454: "WARN collector check failed check=redisdb error=\"i/o timeout\" retrying=true", + 466: "INFO collector check recovered check=redisdb status=OK latency_ms=24", + 643: "INFO remote config applied transaction_id=rc-8830 product=apm_sampling version=314159 changed=true", + 650: "INFO trace-agent config reloaded transaction_id=rc-8830 status=OK", + 702: "INFO remote config applied transaction_id=rc-8831 product=agent_config version=271828 changed=true source=remote-config", + 714: "INFO config reload requested source=remote-config transaction_id=rc-8831 path=/etc/datadog-agent/datadog.yaml", + 722: "ERROR config validation failed file=/etc/datadog-agent/datadog.yaml line=42 column=17 key=logs_config error=\"yaml: mapping values are not allowed in this context\" transaction_id=rc-8831", + 723: "ERROR core agent stopped: invalid configuration after remote-config reload transaction_id=rc-8831", + 724: "WARN aggregator stopped; skipping metric flush last_success=2026-04-30T10:11:58Z", + 725: "WARN forwarder paused because aggregator is stopped pending_series=1842", + 731: "INFO trace-agent still running status=OK note=\"APM intake is healthy; core metrics agent is stopped\"", + 775: "INFO retrying config load attempt=1 source=remote-config transaction_id=rc-8831", + 776: "ERROR config validation failed file=/etc/datadog-agent/datadog.yaml line=42 column=17 key=logs_config error=\"yaml: mapping values are not allowed in this context\" transaction_id=rc-8831", + 846: "WARN no metrics flushed since 2026-04-30T10:12:03Z reason=\"core agent stopped\"", + 918: "INFO remote config poll complete transaction_id=rc-8832 changed=false products=agent_config,apm_sampling", + 969: "ERROR collector scheduler disabled because core agent is not running", + 1031: "WARN no metrics flushed since 2026-04-30T10:12:03Z reason=\"invalid configuration\"", + 1120: "INFO trace-agent heartbeat status=OK spans_sent=293 note=\"red herring: traces unaffected\"", + } + + lines := make([]string, 0, 1200) + for i := 0; i < 1200; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", isoTime(dt), event)) + continue + } + check := checks[(i*7)%len(checks)] + switch { + case i%137 == 0: + lines = append(lines, fmt.Sprintf("%s WARN collector slow check=%s duration_ms=%d sample_id=agent-noise-%04d", isoTime(dt), check, 180+i%90, i)) + case i%113 == 0: + lines = append(lines, fmt.Sprintf("%s ERROR log pipeline dropped message pipeline=app count=1 reason=\"invalid utf8\" sample_id=agent-noise-%04d", isoTime(dt), i)) + case i%29 == 0: + lines = append(lines, fmt.Sprintf("%s DEBUG remote config poll skipped jitter_ms=%d transaction_id=rc-noop-%04d", isoTime(dt), 50+i%400, i)) + default: + lines = append(lines, fmt.Sprintf("%s DEBUG collector check heartbeat check=%s status=OK sequence=%04d token=agent-noise", isoTime(dt), check, i)) + } + } + return lines +} + +func generateDatadogAgentRotatedLog() []string { + start := time.Date(2026, 4, 29, 23, 45, 0, 0, time.UTC) + checks := []string{"cpu", "disk", "network", "ntp", "postgres", "redisdb", "http_check", "process", "container"} + events := map[int]string{ + 14: "INFO agent starting version=7.98.1 build=fixture host=checkout-01", + 119: "ERROR config validation failed file=/etc/datadog-agent/conf.d/http_check.d/conf.yaml line=17 error=\"missing required field url\" check=http_check recovered=true", + 131: "INFO collector check recovered check=http_check status=OK after_fix=true", + 311: "WARN forwarder retryable error domain=logs endpoint=/api/v2/logs status=503 retry_in=15s recovered=true", + 325: "INFO forwarder recovered domain=logs endpoint=/api/v2/logs status=202", + 512: "INFO remote config poll complete transaction_id=rc-8799 changed=false", + } + + lines := make([]string, 0, 700) + for i := 0; i < 700; i++ { + dt := start.Add(time.Duration(i*2) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", isoTime(dt), event)) + } else if i%41 == 0 { + lines = append(lines, fmt.Sprintf("%s WARN collector transient check=%s error=\"temporary network timeout\" recovered=true token=old-noise-%04d", isoTime(dt), checks[i%len(checks)], i)) + } else { + lines = append(lines, fmt.Sprintf("%s DEBUG agent previous-rotation heartbeat sequence=%04d token=old-agent-noise", isoTime(dt), i)) + } + } + return lines +} + +func generateAuthLog() []string { + start := time.Date(2026, 4, 30, 9, 45, 0, 0, time.UTC) + users := []string{"admin", "root", "oracle", "postgres", "test", "ubuntu", "deploy", "backup", "guest", "ci", "jenkins", "support", "mysql", "elastic", "git", "prometheus"} + failures := map[int]int{} + for n := 0; n < 96; n++ { + failures[785+n*4] = n + } + events := map[int]string{ + 61: "bastion sshd[1410]: Accepted publickey for deploy from 203.0.113.8 port 61200 ssh2: RSA SHA256:fixture-deploy", + 130: "bastion sudo: deploy : TTY=pts/0 ; PWD=/srv/app ; USER=root ; COMMAND=/usr/bin/systemctl status checkout.service", + 405: "bastion sshd[1501]: Failed password for invalid user admin from 192.0.2.50 port 51220 ssh2", + 501: "bastion sshd[1502]: Failed password for invalid user root from 192.0.2.50 port 51221 ssh2", + 693: "bastion sshd[1510]: Accepted publickey for release from 198.51.100.77 port 49212 ssh2: ED25519 SHA256:fixture-release", + 754: "bastion sshd[1512]: Invalid user postgres from 198.51.100.23 port 52001", + 1172: "bastion sshd[1802]: maximum authentication attempts exceeded for invalid user support from 198.51.100.23 port 52320 ssh2 [preauth]", + 1244: "bastion sshd[1810]: Accepted publickey for deploy from 203.0.113.8 port 61244 ssh2: RSA SHA256:fixture-deploy", + 1328: "bastion sshd[1820]: Failed password for invalid user admin from 198.51.100.24 port 53220 ssh2", + 1398: "bastion sshd[1830]: Connection closed by authenticating user root 198.51.100.23 port 52444 [preauth]", + } + + lines := make([]string, 0, 1500) + for i := 0; i < 1500; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if n, ok := failures[i]; ok { + user := users[n%len(users)] + lines = append(lines, fmt.Sprintf("%s bastion sshd[%d]: Failed password for invalid user %s from 198.51.100.23 port %d ssh2", syslogTime(dt), 1600+n, user, 52000+n)) + } else if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", syslogTime(dt), event)) + } else if i%97 == 0 { + lines = append(lines, fmt.Sprintf("%s bastion sshd[%d]: Failed password for invalid user scanner from 203.0.113.44 port %d ssh2", syslogTime(dt), 2000+i, 40000+i)) + } else if i%83 == 0 { + lines = append(lines, fmt.Sprintf("%s bastion sshd[%d]: pam_unix(sshd:session): session opened for user deploy(uid=1001) by (uid=0)", syslogTime(dt), 2100+i)) + } else if i%67 == 0 { + lines = append(lines, fmt.Sprintf("%s bastion sudo: deploy : TTY=pts/0 ; PWD=/srv/app ; USER=root ; COMMAND=/usr/bin/journalctl -n 20", syslogTime(dt))) + } else if i%31 == 0 { + lines = append(lines, fmt.Sprintf("%s bastion sshd[%d]: Received disconnect from 203.0.113.%d port %d:11: disconnected by user", syslogTime(dt), 2200+i, 10+i%30, 41000+i)) + } else { + lines = append(lines, fmt.Sprintf("%s bastion CRON[%d]: pam_unix(cron:session): session closed for user root token=auth-noise-%04d", syslogTime(dt), 3000+i, i)) + } + } + return lines +} + +func generateAuthRotatedLog() []string { + start := time.Date(2026, 4, 29, 22, 0, 0, 0, time.UTC) + lines := make([]string, 0, 700) + for i := 0; i < 700; i++ { + dt := start.Add(time.Duration(i*3) * time.Second) + if i%89 == 0 { + lines = append(lines, fmt.Sprintf("%s bastion sshd[%d]: Failed password for invalid user temp from 203.0.113.%d port %d ssh2", syslogTime(dt), 4000+i, 60+i%20, 45000+i)) + } else if i == 321 { + lines = append(lines, fmt.Sprintf("%s bastion sshd[4455]: Accepted publickey for deploy from 203.0.113.8 port 61111 ssh2: RSA SHA256:fixture-deploy", syslogTime(dt))) + } else { + lines = append(lines, fmt.Sprintf("%s bastion CRON[%d]: pam_unix(cron:session): session closed for user root token=auth-rotated-noise-%04d", syslogTime(dt), 5000+i, i)) + } + } + return lines +} + +func generateCheckoutServiceLog() []string { + start := time.Date(2026, 4, 30, 10, 0, 0, 0, time.UTC) + routes := []string{"/api/cart", "/api/checkout", "/api/profile", "/api/promotions", "/health"} + events := map[int]string{ + 1: "INFO service=checkout boot complete version=2026.04.30 build=fc9e3b config_source=file", + 62: "INFO service=checkout handled request id=req-090062 route=/api/checkout status=200 latency_ms=44", + 183: "WARN service=checkout upstream retry id=req-090183 upstream=payments attempt=1 error=\"deadline exceeded\" recovered=true", + 197: "INFO service=checkout upstream recovered id=req-090197 upstream=payments status=OK", + 552: "WARN service=checkout db pool wait high pool=checkout_rw active=108 idle=0 max=120 wait_ms=450 db_host=db.internal db_port=5432", + 578: "WARN service=checkout dependency latency high dependency=postgres p95_ms=920 pool=checkout_rw active=116 max=120", + 594: "ERROR service=checkout db pool exhausted pool=checkout_rw active=120 max=120 wait_ms=3000 error=\"context deadline exceeded\" suspected_client=reporting-worker", + 595: "ERROR service=checkout request failed id=req-1015 route=/api/checkout status=500 error=\"database connection refused\" db_host=db.internal db_port=5432 pool=checkout_rw", + 601: "ERROR service=checkout request failed id=req-1016 route=/api/checkout status=500 error=\"pq: remaining connection slots are reserved for non-replication superuser connections\" db_host=db.internal db_port=5432 pool=checkout_rw", + 607: "ERROR service=checkout request failed id=req-1017 route=/api/checkout status=500 error=\"database connection refused\" db_host=db.internal db_port=5432 pool=checkout_rw", + 614: "WARN service=checkout circuit breaker opened dependency=postgres route=/api/checkout failure_rate=0.86 window=60s", + 639: "ERROR service=checkout request failed id=req-1021 route=/api/checkout status=502 error=\"upstream checkout worker unavailable after db timeout\"", + 683: "INFO service=checkout healthcheck status=degraded dependency=postgres pool=checkout_rw active=120 max=120", + 777: "WARN service=checkout cache miss spike cache=redis route=/api/cart note=\"not correlated with checkout 500s\"", + 910: "INFO service=checkout payment gateway status=OK note=\"red herring resolved before incident\"", + } + + lines := make([]string, 0, 1100) + for i := 0; i < 1100; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", isoTime(dt), event)) + continue + } + route := routes[(i*5+3)%len(routes)] + latency := 30 + (i*17)%180 + if i%149 == 0 { + lines = append(lines, fmt.Sprintf("%s WARN service=checkout slow request id=req-%06d route=%s status=200 latency_ms=%d token=svc-noise-%04d", isoTime(dt), 90000+i, route, latency+400, i)) + } else if i%211 == 0 { + lines = append(lines, fmt.Sprintf("%s ERROR service=checkout feature-flag refresh failed flag=promo_banner error=\"timeout\" recovered=true token=svc-noise-%04d", isoTime(dt), i)) + } else { + lines = append(lines, fmt.Sprintf("%s INFO service=checkout handled request id=req-%06d route=%s status=200 latency_ms=%d token=svc-noise", isoTime(dt), 90000+i, route, latency)) + } + } + return lines +} + +func generateCheckoutServiceRotatedLog() []string { + start := time.Date(2026, 4, 29, 23, 20, 0, 0, time.UTC) + lines := make([]string, 0, 650) + for i := 0; i < 650; i++ { + dt := start.Add(time.Duration(i*2) * time.Second) + switch { + case i == 188: + lines = append(lines, fmt.Sprintf("%s ERROR service=checkout request failed id=req-old-188 route=/api/checkout status=500 error=\"feature flag parse failed\" recovered=true", isoTime(dt))) + case i == 190: + lines = append(lines, fmt.Sprintf("%s INFO service=checkout recovered route=/api/checkout status=200 note=\"old rotation red herring\"", isoTime(dt))) + case i%73 == 0: + lines = append(lines, fmt.Sprintf("%s WARN service=checkout slow request id=req-old-%d route=/api/cart latency_ms=%d recovered=true", isoTime(dt), i, 500+i%50)) + default: + lines = append(lines, fmt.Sprintf("%s INFO service=checkout previous-rotation heartbeat sequence=%04d token=svc-rotated-noise", isoTime(dt), i)) + } + } + return lines +} + +func generateNginxAccessLog() []string { + start := time.Date(2026, 4, 30, 9, 50, 0, 0, time.UTC) + checkoutFailures := map[int]int{ + 1202: 500, 1205: 500, 1208: 500, 1211: 502, 1214: 500, 1217: 502, 1220: 500, 1224: 500, + 1230: 502, 1235: 500, 1240: 500, 1246: 502, 1252: 500, 1258: 500, 1264: 502, 1270: 500, + } + routes := []string{"/health", "/api/cart", "/api/checkout", "/api/profile", "/api/promotions"} + lines := make([]string, 0, 1800) + for i := 0; i < 1800; i++ { + dt := start.Add(time.Duration(i) * time.Second) + client := fmt.Sprintf("203.0.113.%d", 10+i%80) + if code, ok := checkoutFailures[i]; ok { + size := 148 + if code == 502 { + size = 167 + } + lines = append(lines, fmt.Sprintf("%s - - [%s] \"POST /api/checkout HTTP/1.1\" %d %d \"-\" \"fixture-client/%d\" request_id=req-%04d", client, nginxTime(dt), code, size, i%7, 1000+i)) + } else if i%227 == 0 { + lines = append(lines, fmt.Sprintf("%s - - [%s] \"GET /api/search?q=fixture HTTP/1.1\" 500 211 \"-\" \"fixture-client/%d\" request_id=search-red-herring-%04d", client, nginxTime(dt), i%7, i)) + } else if i%131 == 0 { + lines = append(lines, fmt.Sprintf("%s - - [%s] \"POST /api/login HTTP/1.1\" 429 98 \"-\" \"fixture-client/%d\" request_id=rate-noise-%04d", client, nginxTime(dt), i%7, i)) + } else { + route := routes[i%len(routes)] + method := "GET" + if route == "/api/checkout" { + method = "POST" + } + size := 400 + i%600 + userAgent := fmt.Sprintf("fixture-client/%d", i%7) + if route == "/health" { + size = 12 + userAgent = "kube-probe" + } + lines = append(lines, fmt.Sprintf("%s - - [%s] \"%s %s HTTP/1.1\" 200 %d \"-\" \"%s\" request_id=req-%04d", client, nginxTime(dt), method, route, size, userAgent, 1000+i)) + } + } + return lines +} + +func generateNginxAccessRotatedLog() []string { + start := time.Date(2026, 4, 29, 22, 30, 0, 0, time.UTC) + routes := []string{"/health", "/api/cart", "/api/checkout", "/static/app.js"} + lines := make([]string, 0, 900) + for i := 0; i < 900; i++ { + dt := start.Add(time.Duration(i*2) * time.Second) + client := fmt.Sprintf("198.51.100.%d", 30+i%30) + if i%173 == 0 { + lines = append(lines, fmt.Sprintf("%s - - [%s] \"GET /api/search?q=old HTTP/1.1\" 500 201 \"-\" \"fixture-client-old\" request_id=old-search-%04d", client, nginxTime(dt), i)) + } else { + route := routes[i%len(routes)] + method := "GET" + if route == "/api/checkout" { + method = "POST" + } + lines = append(lines, fmt.Sprintf("%s - - [%s] \"%s %s HTTP/1.1\" 200 %d \"-\" \"fixture-client-old\" request_id=old-%04d", client, nginxTime(dt), method, route, 100+i%500, i)) + } + } + return lines +} + +func generateNginxErrorLog() []string { + start := time.Date(2026, 4, 30, 10, 0, 0, 0, time.UTC) + events := map[int]string{ + 602: "[error] 100#100: *420 upstream prematurely closed connection while reading response header from upstream, client: 203.0.113.13, server: checkout.example, request: \"POST /api/checkout HTTP/1.1\", upstream: \"http://127.0.0.1:8080/api/checkout\", request_id=req-2202", + 611: "[error] 100#100: *421 connect() failed (111: Connection refused) while connecting to upstream, client: 203.0.113.16, server: checkout.example, request: \"POST /api/checkout HTTP/1.1\", upstream: \"http://127.0.0.1:8080/api/checkout\", request_id=req-2211", + 627: "[error] 100#100: *422 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 203.0.113.18, server: checkout.example, request: \"POST /api/checkout HTTP/1.1\", upstream: \"http://127.0.0.1:8080/api/checkout\", request_id=req-2227", + 660: "[warn] 100#100: *425 upstream server temporarily disabled while connecting to upstream, server: checkout.example, request: \"POST /api/checkout HTTP/1.1\", upstream: \"http://127.0.0.1:8080/api/checkout\"", + } + + lines := make([]string, 0, 800) + for i := 0; i < 800; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", nginxErrorTime(dt), event)) + } else if i%181 == 0 { + lines = append(lines, fmt.Sprintf("%s [error] 100#100: *%d open() \"/usr/share/nginx/html/favicon.ico\" failed (2: No such file or directory), client: 203.0.113.%d, server: checkout.example, request: \"GET /favicon.ico HTTP/1.1\"", nginxErrorTime(dt), 300+i, i%80)) + } else if i%97 == 0 { + lines = append(lines, fmt.Sprintf("%s [warn] 100#100: *%d an upstream response is buffered to a temporary file while reading upstream, client: 203.0.113.%d, request: \"GET /api/cart HTTP/1.1\"", nginxErrorTime(dt), 300+i, i%80)) + } else { + lines = append(lines, fmt.Sprintf("%s [info] 100#100: *%d client keepalive closed connection token=nginx-error-noise-%04d", nginxErrorTime(dt), 300+i, i)) + } + } + return lines +} + +func generateNginxErrorRotatedLog() []string { + start := time.Date(2026, 4, 29, 22, 30, 0, 0, time.UTC) + lines := make([]string, 0, 600) + for i := 0; i < 600; i++ { + dt := start.Add(time.Duration(i*2) * time.Second) + if i == 277 { + lines = append(lines, fmt.Sprintf("%s [error] 100#100: *88 upstream timed out while reading response header from upstream, request: \"GET /api/search HTTP/1.1\", recovered=true", nginxErrorTime(dt))) + } else { + lines = append(lines, fmt.Sprintf("%s [info] 100#100: *%d previous rotation keepalive closed token=nginx-rotated-noise-%04d", nginxErrorTime(dt), 80+i, i)) + } + } + return lines +} + +func generateSystemLog() []string { + start := time.Date(2026, 4, 30, 10, 0, 0, 0, time.UTC) + events := map[int]string{ + 0: "host kernel: boot fixture host kernel=6.8.0-fixture", + 192: "host systemd[1]: Started checkout.service.", + 510: "host postgres[2190]: LOG: checkpoint complete: wrote 142 buffers (0.9%); 0 WAL files added", + 574: "host postgres[2200]: LOG: connection received: host=10.0.44.19 port=45100 application_name=reporting-worker user=reports", + 575: "host postgres[2200]: LOG: connection received: host=10.0.44.19 port=45101 application_name=reporting-worker user=reports", + 576: "host postgres[2200]: LOG: connection received: host=10.0.44.19 port=45102 application_name=reporting-worker user=reports", + 594: "host kernel: TCP: request_sock_TCP: Possible SYN flooding on port 5432. Sending cookies. Check SNMP counters.", + 600: "host postgres[2201]: FATAL: remaining connection slots are reserved for non-replication superuser connections", + 601: "host postgres[2202]: FATAL: sorry, too many clients already application_name=checkout-service user=checkout_rw database=shop", + 603: "host postgres[2203]: LOG: could not accept SSL connection: Connection reset by peer", + 607: "host postgres[2204]: LOG: connection rejected application_name=checkout-service reason=\"remaining connection slots reserved\" active=120 max_connections=120", + 640: "host systemd[1]: checkout.service: Watchdog timeout ignored in fixture", + 690: "host postgres[2210]: LOG: connection received: host=10.0.44.19 port=45190 application_name=reporting-worker user=reports", + 810: "host cron[3333]: reporting-worker connection fanout job still running elapsed=15m db=db.internal", + } + + lines := make([]string, 0, 900) + for i := 0; i < 900; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", syslogTime(dt), event)) + } else if i%157 == 0 { + lines = append(lines, fmt.Sprintf("%s host kernel: audit: type=1400 apparmor=\"DENIED\" operation=\"open\" profile=\"fixture\" name=\"/tmp/noise-%d\" pid=%d comm=\"noise\"", syslogTime(dt), i, 6000+i)) + } else if i%103 == 0 { + lines = append(lines, fmt.Sprintf("%s host systemd[1]: logrotate.service: Deactivated successfully token=system-noise-%04d", syslogTime(dt), i)) + } else { + lines = append(lines, fmt.Sprintf("%s host systemd[1]: fixture heartbeat service=checkout.slice sequence=%04d token=system-noise", syslogTime(dt), i)) + } + } + return lines +} + +func generateSystemRotatedLog() []string { + start := time.Date(2026, 4, 29, 23, 0, 0, 0, time.UTC) + lines := make([]string, 0, 650) + for i := 0; i < 650; i++ { + dt := start.Add(time.Duration(i*2) * time.Second) + if i == 241 { + lines = append(lines, fmt.Sprintf("%s host postgres[1200]: FATAL: password authentication failed for user \"readonly\" recovered=true old_rotation=true", syslogTime(dt))) + } else { + lines = append(lines, fmt.Sprintf("%s host systemd[1]: previous rotation heartbeat sequence=%04d token=system-rotated-noise", syslogTime(dt), i)) + } + } + return lines +} + +func generateDebugNoiseLog() []string { + start := time.Date(2026, 4, 30, 8, 0, 0, 0, time.UTC) + lines := make([]string, 0, 1500) + for i := 0; i < 1500; i++ { + dt := start.Add(time.Duration(i*2) * time.Second) + level := "DEBUG" + message := "background sampler tick" + if i%211 == 0 { + level = "ERROR" + message = "synthetic canary failed but unrelated service=search" + } else if i%97 == 0 { + level = "WARN" + message = "slow DNS lookup for analytics endpoint recovered=true" + } + lines = append(lines, fmt.Sprintf("%s %s component=fixture-noise sequence=%04d message=\"%s\" token=not-relevant", isoTime(dt), level, i, message)) + } + return lines +} + +func generateContainerAgentLog() []string { + start := time.Date(2026, 4, 30, 3, 0, 0, 0, time.UTC) + checks := []string{"container", "docker", "kubelet", "process", "network", "kubernetes_state_core"} + events := map[int]string{ + 0: "INFO agent container boot version=7.99.0 container_id=fixture host_mount=/host/var/log", + 42: "INFO collector check completed check=kubelet status=OK latency_ms=31", + 127: "WARN collector check failed check=kubernetes_state_core error=\"context deadline exceeded\" recovered=true", + 134: "INFO collector check recovered check=kubernetes_state_core status=OK", + 314: "ERROR collector check failed check=kubernetes_apiserver error=\"x509: certificate is not yet valid: current time 2026-04-30T03:05:14Z is before 2026-04-30T10:58:00Z\" endpoint=https://10.96.0.1:443", + 315: "WARN collector skipped check=kubernetes_apiserver reason=\"tls handshake failure\" next_retry=15s", + 374: "ERROR collector check failed check=kubernetes_apiserver error=\"x509: certificate is not yet valid: current time 2026-04-30T03:06:14Z is before 2026-04-30T10:58:00Z\" endpoint=https://10.96.0.1:443", + 438: "ERROR collector check failed check=kubernetes_apiserver error=\"x509: certificate is not yet valid\" tls_server_name=kubernetes.default.svc", + 512: "INFO collector check completed check=container status=OK latency_ms=22 note=\"red herring: container check healthy\"", + 640: "WARN flare skipped reason=\"benchmark read-only fixture\"", + 714: "ERROR collector check failed check=kubernetes_apiserver error=\"x509: certificate is not yet valid\" endpoint=https://10.96.0.1:443", + } + + lines := make([]string, 0, 850) + for i := 0; i < 850; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", isoTime(dt), event)) + } else if i%109 == 0 { + lines = append(lines, fmt.Sprintf("%s WARN collector slow check=%s duration_ms=%d recovered=true token=container-agent-noise-%04d", isoTime(dt), checks[i%len(checks)], 250+i%80, i)) + } else if i%173 == 0 { + lines = append(lines, fmt.Sprintf("%s ERROR logs-agent tailer transient error file=/var/log/pods/noisy.log error=\"file rotated\" recovered=true token=container-agent-noise-%04d", isoTime(dt), i)) + } else { + lines = append(lines, fmt.Sprintf("%s DEBUG collector heartbeat check=%s status=OK sequence=%04d token=container-agent-noise", isoTime(dt), checks[i%len(checks)], i)) + } + } + return lines +} + +func generateContainerSyslog() []string { + start := time.Date(2026, 4, 30, 11, 0, 0, 0, time.UTC) + events := map[int]string{ + 4: "node systemd[1]: Started Datadog Agent container fixture.", + 116: "node chronyd[801]: Selected source 192.0.2.10 (time.example) but system clock is unsynchronised", + 128: "node kernel: clocksource: timekeeping watchdog on CPU0: Marking clocksource tsc as unstable because the skew is too large", + 132: "node chronyd[801]: System clock wrong by 07:53:46.217 seconds; waiting for makestep window", + 134: "node datadog-agent[17]: kubernetes_apiserver check failing: x509 certificate is not yet valid (agent clock before certificate NotBefore)", + 240: "node kubelet[22]: certificate rotation pending approval for client kubelet; unrelated to apiserver serving cert", + 256: "node chronyd[801]: System clock was stepped by +28426.217 seconds to correct skew", + 262: "node kubelet[22]: Node clock synchronized after chrony step", + 300: "node datadog-agent[17]: kubernetes_apiserver check retry still failing until next collector interval", + 420: "node datadog-agent[17]: kubernetes_apiserver check recovered after clock synchronization status=OK", + } + + lines := make([]string, 0, 750) + for i := 0; i < 750; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", syslogTime(dt), event)) + } else if i%127 == 0 { + lines = append(lines, fmt.Sprintf("%s node kubelet[22]: pod sandbox changed pod=fixture-noise-%d namespace=default", syslogTime(dt), i)) + } else if i%89 == 0 { + lines = append(lines, fmt.Sprintf("%s node containerd[33]: image garbage collection completed reclaimed=%dMB token=container-syslog-noise-%04d", syslogTime(dt), i%17, i)) + } else { + lines = append(lines, fmt.Sprintf("%s node systemd[1]: fixture heartbeat unit=container-runtime.service sequence=%04d token=container-syslog-noise", syslogTime(dt), i)) + } + } + return lines +} + +func generateHoldoutCheckoutLog() []string { + start := time.Date(2026, 5, 1, 14, 15, 0, 0, time.UTC) + events := map[int]string{ + 0: "INFO service=checkout boot complete version=2026.05.01 build=holdout-a config_source=file", + 312: "INFO service=checkout postgres health status=OK pool=checkout_rw active=42 idle=18 max=120 latency_ms=15", + 396: "WARN service=checkout dependency latency high dependency=payments route=/api/pay p95_ms=1800 request_id=pay-2198", + 414: "ERROR service=checkout request failed id=pay-2201 route=/api/pay status=502 upstream=payments error=\"lookup payments.service.consul: no such host\" resolver=10.0.0.53", + 421: "ERROR service=checkout request failed id=pay-2202 route=/api/pay status=502 upstream=payments error=\"dial tcp: lookup payments.service.consul: i/o timeout\" resolver=10.0.0.53", + 427: "WARN service=checkout circuit breaker opened dependency=payments reason=\"dns resolution failure\" window=60s", + 456: "INFO service=checkout postgres health status=OK pool=checkout_rw active=43 idle=17 max=120 latency_ms=17 note=\"database not saturated during payment errors\"", + 509: "ERROR service=checkout request failed id=pay-2211 route=/api/pay status=502 upstream=payments error=\"lookup payments.service.consul: server misbehaving\" resolver=10.0.0.53", + 690: "INFO service=checkout dependency=payments recovered status=OK dns_cache_refreshed=true", + } + routes := []string{"/api/cart", "/api/pay", "/api/profile", "/health"} + lines := make([]string, 0, 760) + for i := 0; i < 760; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", isoTime(dt), event)) + continue + } + route := routes[(i*3)%len(routes)] + if i%173 == 0 { + lines = append(lines, fmt.Sprintf("%s WARN service=checkout db pool wait elevated pool=analytics_ro active=%d max=20 recovered=true note=\"old analytics noise, not checkout_rw\"", isoTime(dt), 12+i%5)) + } else if i%137 == 0 { + lines = append(lines, fmt.Sprintf("%s ERROR service=checkout feature flag refresh failed flag=upsell recovered=true token=holdout-checkout-noise-%04d", isoTime(dt), i)) + } else { + lines = append(lines, fmt.Sprintf("%s INFO service=checkout handled request id=pay-noise-%04d route=%s status=200 latency_ms=%d token=holdout-checkout", isoTime(dt), i, route, 30+i%120)) + } + } + return lines +} + +func generateHoldoutNginxAccessLog() []string { + start := time.Date(2026, 5, 1, 14, 10, 0, 0, time.UTC) + failures := map[int]int{720: 502, 724: 502, 729: 502, 736: 502, 741: 502, 748: 502, 756: 502, 768: 502} + lines := make([]string, 0, 1050) + for i := 0; i < 1050; i++ { + dt := start.Add(time.Duration(i) * time.Second) + client := fmt.Sprintf("198.51.100.%d", 40+i%40) + if code, ok := failures[i]; ok { + lines = append(lines, fmt.Sprintf("%s - - [%s] \"POST /api/pay HTTP/1.1\" %d 173 \"-\" \"holdout-client/%d\" request_id=pay-%04d", client, nginxTime(dt), code, i%5, 2200+i-720)) + } else if i%211 == 0 { + lines = append(lines, fmt.Sprintf("%s - - [%s] \"GET /api/search?q=noise HTTP/1.1\" 500 211 \"-\" \"holdout-client/%d\" request_id=search-holdout-%04d", client, nginxTime(dt), i%5, i)) + } else { + route := "/api/cart" + method := "GET" + if i%4 == 0 { + route = "/api/pay" + method = "POST" + } + lines = append(lines, fmt.Sprintf("%s - - [%s] \"%s %s HTTP/1.1\" 200 %d \"-\" \"holdout-client/%d\" request_id=pay-noise-%04d", client, nginxTime(dt), method, route, 300+i%400, i%5, i)) + } + } + return lines +} + +func generateHoldoutSystemLog() []string { + start := time.Date(2026, 5, 1, 14, 15, 0, 0, time.UTC) + events := map[int]string{ + 378: "edge systemd-resolved[511]: DNS server 10.0.0.53 timed out, retrying transaction=payments.service.consul type=A", + 414: "edge systemd-resolved[511]: Server returned error SERVFAIL for payments.service.consul IN A", + 418: "edge dnsmasq[902]: query[A] payments.service.consul from 10.0.12.44", + 419: "edge dnsmasq[902]: forwarded payments.service.consul to 10.0.0.53", + 420: "edge dnsmasq[902]: reply payments.service.consul is SERVFAIL", + 456: "edge postgres[2300]: LOG: checkpoint complete: wrote 48 buffers; connections active=43 max=120", + 691: "edge systemd-resolved[511]: DNS lookup for payments.service.consul recovered status=NOERROR ttl=30", + } + lines := make([]string, 0, 760) + for i := 0; i < 760; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", syslogTime(dt), event)) + } else if i%149 == 0 { + lines = append(lines, fmt.Sprintf("%s edge kernel: audit: type=1400 apparmor=\"DENIED\" operation=\"open\" profile=\"fixture\" name=\"/tmp/holdout-noise-%d\"", syslogTime(dt), i)) + } else { + lines = append(lines, fmt.Sprintf("%s edge systemd[1]: fixture heartbeat unit=checkout.slice sequence=%04d token=holdout-system", syslogTime(dt), i)) + } + } + return lines +} + +func generateHoldoutWorkerLog() []string { + start := time.Date(2026, 5, 1, 15, 58, 0, 0, time.UTC) + events := map[int]string{ + 0: "INFO service=async-worker boot complete version=2026.05.01 build=77ac21 pid=4441", + 215: "WARN service=async-worker heartbeat delayed queue=emails lag_ms=2100 recovered=true", + 300: "INFO service=async-worker received signal signal=SIGTERM pid=4441 reason=unknown drain_started=true", + 303: "INFO service=async-worker shutdown complete pid=4441 jobs_inflight=0 exit_code=0", + 316: "INFO service=async-worker boot complete version=2026.05.01 build=77ac21 pid=4528 note=\"same build after restart\"", + 410: "INFO service=async-worker queue healthy queue=emails lag_ms=88", + } + lines := make([]string, 0, 620) + for i := 0; i < 620; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", isoTime(dt), event)) + } else if i%181 == 0 { + lines = append(lines, fmt.Sprintf("%s ERROR service=async-worker email provider transient timeout recovered=true token=worker-noise-%04d", isoTime(dt), i)) + } else { + lines = append(lines, fmt.Sprintf("%s DEBUG service=async-worker heartbeat pid=4441 sequence=%04d token=worker-holdout", isoTime(dt), i)) + } + } + return lines +} + +func generateHoldoutAuthLog() []string { + start := time.Date(2026, 5, 1, 15, 45, 0, 0, time.UTC) + events := map[int]string{ + 240: "ops sshd[3100]: Accepted publickey for deploy from 203.0.113.42 port 61022 ssh2: ED25519 SHA256:holdout-deploy", + 312: "ops sudo: deploy : TTY=pts/1 ; PWD=/srv/app ; USER=root ; COMMAND=/usr/bin/systemctl status async-worker.service", + 780: "ops sudo: deploy : TTY=pts/1 ; PWD=/srv/app ; USER=root ; COMMAND=/usr/bin/journalctl -u async-worker -n 50", + } + lines := make([]string, 0, 980) + for i := 0; i < 980; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", syslogTime(dt), event)) + } else if i%197 == 0 { + lines = append(lines, fmt.Sprintf("%s ops sshd[%d]: Failed password for invalid user temp from 198.51.100.%d port %d ssh2", syslogTime(dt), 4200+i, 80+i%10, 50000+i)) + } else { + lines = append(lines, fmt.Sprintf("%s ops CRON[%d]: pam_unix(cron:session): session closed for user root token=holdout-auth-%04d", syslogTime(dt), 5000+i, i)) + } + } + return lines +} + +func generateHoldoutDeployLog() []string { + start := time.Date(2026, 5, 1, 15, 0, 0, 0, time.UTC) + events := map[int]string{ + 10: "INFO deploy id=dep-771 service=async-worker version=2026.05.01 started_by=release-bot", + 132: "INFO deploy id=dep-771 service=async-worker version=2026.05.01 completed status=success finished_at=2026-05-01T15:02:12Z", + 540: "INFO deploy controller heartbeat service=checkout no_change=true", + } + lines := make([]string, 0, 620) + for i := 0; i < 620; i++ { + dt := start.Add(time.Duration(i) * time.Second) + if event, ok := events[i]; ok { + lines = append(lines, fmt.Sprintf("%s %s", isoTime(dt), event)) + } else { + lines = append(lines, fmt.Sprintf("%s DEBUG deploy controller idle sequence=%04d token=holdout-deploy", isoTime(dt), i)) + } + } + return lines +} diff --git a/auto-improve-skills/internal/autoresearch/fixtures_test.go b/auto-improve-skills/internal/autoresearch/fixtures_test.go new file mode 100644 index 00000000..91cb8296 --- /dev/null +++ b/auto-improve-skills/internal/autoresearch/fixtures_test.go @@ -0,0 +1,132 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package autoresearch + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestGenerateRemoteHostDiagnosticsFixtures(t *testing.T) { + root := t.TempDir() + if err := GenerateRemoteHostDiagnosticsFixtures(root); err != nil { + t.Fatalf("GenerateRemoteHostDiagnosticsFixtures() error = %v", err) + } + + fixtureRoot := RemoteHostDiagnosticsGeneratedFixtureRoot(root) + wantLineCounts := map[string]int{ + "logs/datadog/agent.log": 1200, + "logs/datadog/agent.log.1": 700, + "logs/auth.log": 1500, + "logs/auth.log.1": 700, + "logs/app/service.log": 1100, + "logs/app/service.log.1": 650, + "logs/nginx/access.log": 1800, + "logs/nginx/access.log.1": 900, + "logs/nginx/error.log": 800, + "logs/nginx/error.log.1": 600, + "logs/system.log": 900, + "logs/system.log.1": 650, + "logs/debug-noise.log": 1500, + "container/host/var/log/datadog/agent.log": 850, + "container/host/var/log/syslog": 750, + "holdout/logs/app/checkout.log": 760, + "holdout/logs/nginx/access.log": 1050, + "holdout/logs/system.log": 760, + "holdout/logs/app/worker.log": 620, + "holdout/logs/auth.log": 980, + "holdout/logs/deploy.log": 620, + "container/var/log/.gitkeep": 0, + } + for rel, want := range wantLineCounts { + data := readGeneratedFixture(t, fixtureRoot, rel) + if got := strings.Count(string(data), "\n"); got != want { + t.Fatalf("%s line count = %d, want %d", rel, got, want) + } + if want != 0 && (want < 500 || want > 2000) { + t.Fatalf("%s line count %d is outside expected benchmark fixture range", rel, want) + } + } + + agent := string(readGeneratedFixture(t, fixtureRoot, "logs/datadog/agent.log")) + assertContains(t, agent, "remote config applied transaction_id=rc-8831") + assertContains(t, agent, "line=42") + assertContains(t, agent, "no metrics flushed since 2026-04-30T10:12:03Z") + + auth := string(readGeneratedFixture(t, fixtureRoot, "logs/auth.log")) + if got := countLinesContaining(auth, "Failed password for invalid user", "from 198.51.100.23"); got != 96 { + t.Fatalf("suspicious brute-force failure count = %d, want 96", got) + } + assertContains(t, auth, "Accepted publickey for deploy from 203.0.113.8") + + service := string(readGeneratedFixture(t, fixtureRoot, "logs/app/service.log")) + assertContains(t, service, "db pool exhausted") + assertContains(t, service, "suspected_client=reporting-worker") + + system := string(readGeneratedFixture(t, fixtureRoot, "logs/system.log")) + assertContains(t, system, "remaining connection slots are reserved") + assertContains(t, system, "reporting-worker connection fanout") + + containerAgent := string(readGeneratedFixture(t, fixtureRoot, "container/host/var/log/datadog/agent.log")) + assertContains(t, containerAgent, "kubernetes_apiserver") + assertContains(t, containerAgent, "x509: certificate is not yet valid") + + containerSyslog := string(readGeneratedFixture(t, fixtureRoot, "container/host/var/log/syslog")) + assertContains(t, containerSyslog, "chronyd") + assertContains(t, containerSyslog, "clock") + assertContains(t, containerSyslog, "skew") + + holdoutCheckout := string(readGeneratedFixture(t, fixtureRoot, "holdout/logs/app/checkout.log")) + assertContains(t, holdoutCheckout, "lookup payments.service.consul") + assertContains(t, holdoutCheckout, "postgres health status=OK") + + holdoutSystem := string(readGeneratedFixture(t, fixtureRoot, "holdout/logs/system.log")) + assertContains(t, holdoutSystem, "SERVFAIL") + assertContains(t, holdoutSystem, "payments.service.consul") + + holdoutWorker := string(readGeneratedFixture(t, fixtureRoot, "holdout/logs/app/worker.log")) + assertContains(t, holdoutWorker, "received signal signal=SIGTERM") + assertContains(t, holdoutWorker, "same build after restart") + + holdoutDeploy := string(readGeneratedFixture(t, fixtureRoot, "holdout/logs/deploy.log")) + assertContains(t, holdoutDeploy, "dep-771") + assertContains(t, holdoutDeploy, "completed status=success") +} + +func readGeneratedFixture(t *testing.T, fixtureRoot, rel string) []byte { + t.Helper() + data, err := os.ReadFile(filepath.Join(fixtureRoot, filepath.FromSlash(rel))) + if err != nil { + t.Fatalf("read generated fixture %s: %v", rel, err) + } + return data +} + +func assertContains(t *testing.T, haystack, needle string) { + t.Helper() + if !strings.Contains(haystack, needle) { + t.Fatalf("generated fixture missing %q", needle) + } +} + +func countLinesContaining(s string, needles ...string) int { + count := 0 + for _, line := range strings.Split(s, "\n") { + matches := true + for _, needle := range needles { + if !strings.Contains(line, needle) { + matches = false + break + } + } + if matches { + count++ + } + } + return count +} diff --git a/auto-improve-skills/internal/autoresearch/pi.go b/auto-improve-skills/internal/autoresearch/pi.go new file mode 100644 index 00000000..82644272 --- /dev/null +++ b/auto-improve-skills/internal/autoresearch/pi.go @@ -0,0 +1,166 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package autoresearch + +import ( + "bytes" + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" +) + +// ResolvePI resolves the pi executable. It first respects an explicit -pi value, +// then PI_BIN, PATH, and common npm/nvm installation locations. The nvm fallback +// matters when Go is launched from a shell that did not source nvm, so "pi" is +// installed but not on PATH. +func ResolvePI(pi string) (string, error) { + pi = strings.TrimSpace(pi) + if pi == "" { + pi = "pi" + } + + if hasPathSeparator(pi) || filepath.IsAbs(pi) { + return resolveExecutablePath(pi) + } + + if pi != "pi" { + if path, err := exec.LookPath(pi); err == nil { + return path, nil + } + return "", fmt.Errorf("%q executable not found in PATH", pi) + } + + if env := strings.TrimSpace(os.Getenv("PI_BIN")); env != "" { + return resolveExecutablePath(env) + } + + if path, err := exec.LookPath("pi"); err == nil { + return path, nil + } + + for _, candidate := range piCandidates() { + if path, err := resolveExecutablePath(candidate); err == nil { + return path, nil + } + } + + return "", fmt.Errorf("pi executable not found. Install pi, pass -pi /path/to/pi, or set PI_BIN=/path/to/pi. Current PATH=%q", os.Getenv("PATH")) +} + +// EnvWithExecutableDir returns an environment that prepends the executable's +// directory to PATH. This is important for npm/nvm-installed pi scripts whose +// shebang uses /usr/bin/env node; node usually lives next to pi. +func EnvWithExecutableDir(executable string) []string { + env := os.Environ() + if executable == "" || !hasPathSeparator(executable) && !filepath.IsAbs(executable) { + return env + } + dir := filepath.Dir(executable) + if dir == "." || dir == string(filepath.Separator) { + return env + } + pathValue := os.Getenv("PATH") + newPath := dir + if pathValue != "" { + newPath += string(os.PathListSeparator) + pathValue + } + for i, kv := range env { + if strings.HasPrefix(kv, "PATH=") { + env[i] = "PATH=" + newPath + return env + } + } + return append(env, "PATH="+newPath) +} + +func resolveExecutablePath(path string) (string, error) { + path = strings.TrimSpace(path) + if path == "" { + return "", fmt.Errorf("empty executable path") + } + if !filepath.IsAbs(path) && hasPathSeparator(path) { + abs, err := filepath.Abs(path) + if err == nil { + path = abs + } + } + for _, candidate := range executableVariants(path) { + info, err := os.Stat(candidate) + if err != nil || info.IsDir() { + continue + } + if runtime.GOOS == "windows" || info.Mode()&0o111 != 0 { + return candidate, nil + } + } + return "", fmt.Errorf("executable not found or not executable: %s", path) +} + +func executableVariants(path string) []string { + if runtime.GOOS != "windows" || filepath.Ext(path) != "" { + return []string{path} + } + return []string{path, path + ".cmd", path + ".exe", path + ".bat"} +} + +func piCandidates() []string { + var candidates []string + if home := os.Getenv("HOME"); home != "" { + candidates = append(candidates, + filepath.Join(home, ".local", "bin", "pi"), + filepath.Join(home, ".npm-global", "bin", "pi"), + ) + if matches, err := filepath.Glob(filepath.Join(home, ".nvm", "versions", "node", "*", "bin", "pi")); err == nil { + // Prefer newest-looking versions by trying lexicographically later paths first. + for i := len(matches) - 1; i >= 0; i-- { + candidates = append(candidates, matches[i]) + } + } + } + candidates = append(candidates, + filepath.Join("/opt", "homebrew", "bin", "pi"), + filepath.Join("/usr", "local", "bin", "pi"), + ) + if npmPrefix := npmGlobalPrefix(); npmPrefix != "" { + candidates = append([]string{filepath.Join(npmPrefix, "bin", "pi")}, candidates...) + } + return dedupe(candidates) +} + +func npmGlobalPrefix() string { + npm, err := exec.LookPath("npm") + if err != nil { + return "" + } + cmd := exec.Command(npm, "prefix", "-g") + var out bytes.Buffer + cmd.Stdout = &out + cmd.Stderr = nil + if err := cmd.Run(); err != nil { + return "" + } + return strings.TrimSpace(out.String()) +} + +func dedupe(values []string) []string { + seen := make(map[string]bool, len(values)) + out := make([]string, 0, len(values)) + for _, v := range values { + if v == "" || seen[v] { + continue + } + seen[v] = true + out = append(out, v) + } + return out +} + +func hasPathSeparator(path string) bool { + return strings.ContainsRune(path, os.PathSeparator) || os.PathSeparator == '\\' && strings.ContainsRune(path, '/') +} diff --git a/auto-improve-skills/internal/autoresearch/types.go b/auto-improve-skills/internal/autoresearch/types.go new file mode 100644 index 00000000..74e4c99c --- /dev/null +++ b/auto-improve-skills/internal/autoresearch/types.go @@ -0,0 +1,257 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2026-present Datadog, Inc. + +package autoresearch + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + "gopkg.in/yaml.v3" +) + +// Suite describes a benchmark suite for one skill. +type Suite struct { + Name string `json:"name" yaml:"name"` + Description string `json:"description" yaml:"description"` + SkillPath string `json:"skill_path" yaml:"skill_path"` + Cases []Case `json:"cases" yaml:"cases"` +} + +// Case describes one benchmark prompt and its scoring rubric. +type Case struct { + ID string `json:"id" yaml:"id"` + Title string `json:"title" yaml:"title"` + Prompt string `json:"prompt" yaml:"prompt"` + JudgeRubric string `json:"judge_rubric,omitempty" yaml:"judge_rubric,omitempty"` + Variables map[string]string `json:"variables,omitempty" yaml:"variables,omitempty"` + Criteria []Criterion `json:"criteria" yaml:"criteria"` +} + +// Criterion is a deterministic check over the final answer, command list, tool +// results, or all transcript text. It is intentionally simple so new benchmark +// cases can be added without writing Go code. +type Criterion struct { + Name string `json:"name" yaml:"name"` + Source string `json:"source" yaml:"source"` // final, commands, tool_results, transcript + Contains string `json:"contains,omitempty" yaml:"contains,omitempty"` + Regex string `json:"regex,omitempty" yaml:"regex,omitempty"` + Not bool `json:"not,omitempty" yaml:"not,omitempty"` + CaseInsensitive bool `json:"case_insensitive,omitempty" yaml:"case_insensitive,omitempty"` + RequireEvidence bool `json:"require_evidence,omitempty" yaml:"require_evidence,omitempty"` + EvidenceSource string `json:"evidence_source,omitempty" yaml:"evidence_source,omitempty"` // defaults to tool_results + EvidenceContains string `json:"evidence_contains,omitempty" yaml:"evidence_contains,omitempty"` + EvidenceRegex string `json:"evidence_regex,omitempty" yaml:"evidence_regex,omitempty"` + Points float64 `json:"points" yaml:"points"` +} + +// ToolCall captures a tool invocation from pi's JSON event stream. +type ToolCall struct { + ID string `json:"id"` + Name string `json:"name"` + Args json.RawMessage `json:"args,omitempty"` + Command string `json:"command,omitempty"` + Result string `json:"result,omitempty"` + IsError bool `json:"is_error"` + Duration string `json:"duration,omitempty"` +} + +// CriterionResult records whether one rubric criterion passed. +type CriterionResult struct { + Name string `json:"name"` + Passed bool `json:"passed"` + Points float64 `json:"points"` + Max float64 `json:"max"` + Detail string `json:"detail,omitempty"` +} + +// JudgeResult is populated when skillbench runs an optional LLM-as-judge pass. +type JudgeResult struct { + Score float64 `json:"score"` + Reason string `json:"reason"` + Raw string `json:"raw,omitempty"` +} + +// ObjectiveConfig records the soft objective used to compare candidate +// skills. Quality remains the primary benchmark score; duration and skill size +// are soft penalties in the composite objective. +type ObjectiveConfig struct { + QualityWeight float64 `json:"quality_weight"` + DurationWeight float64 `json:"duration_weight"` + SkillSizeWeight float64 `json:"skill_size_weight"` + DurationBudgetSeconds float64 `json:"duration_budget_seconds"` + DurationHardLimitSeconds float64 `json:"duration_hard_limit_seconds"` + SkillSizeTargetTokens int `json:"skill_size_target_tokens"` + SkillSizeHardLimitTokens int `json:"skill_size_hard_limit_tokens"` +} + +// CaseResult contains all data needed to audit one case. +type CaseResult struct { + ID string `json:"id"` + Title string `json:"title"` + Prompt string `json:"prompt"` + Score float64 `json:"score"` + MaxScore float64 `json:"max_score"` + NormalizedScore float64 `json:"normalized_score"` + DeterministicScore float64 `json:"deterministic_score"` + DeterministicMaxScore float64 `json:"deterministic_max_score"` + FinalAnswer string `json:"final_answer"` + Commands []string `json:"commands"` + ToolCalls []ToolCall `json:"tool_calls"` + CommandCount int `json:"command_count"` + ToolOutputBytes int `json:"tool_output_bytes"` + FailedToolCalls int `json:"failed_tool_calls"` + DurationSeconds float64 `json:"duration_seconds"` + DurationScore float64 `json:"duration_score"` + Criteria []CriterionResult `json:"criteria"` + Judge *JudgeResult `json:"judge,omitempty"` + SafetyViolations []string `json:"safety_violations,omitempty"` + RawJSONLPath string `json:"raw_jsonl_path,omitempty"` + Error string `json:"error,omitempty"` + StartedAt time.Time `json:"started_at"` + CompletedAt time.Time `json:"completed_at"` + WallClockDuration string `json:"wall_clock_duration"` +} + +// SuiteResult is the machine-readable benchmark report. +type SuiteResult struct { + SuiteName string `json:"suite_name"` + Description string `json:"description"` + Mode string `json:"mode"` + Model string `json:"model"` + SkillPath string `json:"skill_path"` + CasesPath string `json:"cases_path"` + RepoRoot string `json:"repo_root"` + Score float64 `json:"score"` + MaxScore float64 `json:"max_score"` + NormalizedScore float64 `json:"normalized_score"` + QualityScore float64 `json:"quality_score"` + QualityMaxScore float64 `json:"quality_max_score"` + QualityNormalizedScore float64 `json:"quality_normalized_score"` + ObjectiveScore float64 `json:"objective_score"` + ObjectiveMaxScore float64 `json:"objective_max_score"` + ObjectiveNormalizedScore float64 `json:"objective_normalized_score"` + ObjectiveConfig ObjectiveConfig `json:"objective_config"` + AverageCaseDurationSeconds float64 `json:"average_case_duration_seconds"` + DurationScore float64 `json:"duration_score"` + SkillSizeBytes int `json:"skill_size_bytes"` + SkillSizeChars int `json:"skill_size_chars"` + SkillSizeWords int `json:"skill_size_words"` + SkillSizeEstimatedTokens int `json:"skill_size_estimated_tokens"` + SkillSizeScore float64 `json:"skill_size_score"` + Cases []CaseResult `json:"cases"` + StartedAt time.Time `json:"started_at"` + CompletedAt time.Time `json:"completed_at"` + WallClockDuration string `json:"wall_clock_duration"` +} + +// LoadSuite reads a YAML benchmark suite. +func LoadSuite(path string) (Suite, error) { + data, err := os.ReadFile(path) + if err != nil { + return Suite{}, err + } + var suite Suite + if err := yaml.Unmarshal(data, &suite); err != nil { + return Suite{}, err + } + if suite.Name == "" { + return Suite{}, fmt.Errorf("suite name is required") + } + if len(suite.Cases) == 0 { + return Suite{}, fmt.Errorf("suite %q has no cases", suite.Name) + } + for i, tc := range suite.Cases { + if tc.ID == "" { + return Suite{}, fmt.Errorf("case %d is missing id", i) + } + if tc.Prompt == "" { + return Suite{}, fmt.Errorf("case %q is missing prompt", tc.ID) + } + if len(tc.Criteria) == 0 { + return Suite{}, fmt.Errorf("case %q has no criteria", tc.ID) + } + } + return suite, nil +} + +// WriteJSON writes v as pretty JSON, creating parent directories. +func WriteJSON(path string, v any) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + data, err := json.MarshalIndent(v, "", " ") + if err != nil { + return err + } + data = append(data, '\n') + return os.WriteFile(path, data, 0o644) +} + +// RepoRoot returns the git repository root, falling back to cwd. +func RepoRoot() (string, error) { + cmd := exec.Command("git", "rev-parse", "--show-toplevel") + var out bytes.Buffer + cmd.Stdout = &out + cmd.Stderr = nil + if err := cmd.Run(); err == nil { + root := strings.TrimSpace(out.String()) + if root != "" { + return root, nil + } + } + return os.Getwd() +} + +// AbsFromRoot returns path if absolute, otherwise root/path. +func AbsFromRoot(root, path string) string { + if filepath.IsAbs(path) { + return filepath.Clean(path) + } + return filepath.Clean(filepath.Join(root, path)) +} + +// Variables returns the default benchmark template variables. +func Variables(root, skillPath string) map[string]string { + autoDir := filepath.Join(root, "auto-improve-skills") + benchDir := RemoteHostDiagnosticsBenchmarkDir(root) + fixtureRoot := RemoteHostDiagnosticsGeneratedFixtureRoot(root) + return map[string]string{ + "ROOT": root, + "AUTO_DIR": autoDir, + "BENCH_DIR": benchDir, + "SKILL_PATH": skillPath, + "LOG_ROOT": filepath.Join(fixtureRoot, "logs"), + "EMPTY_LOG_ROOT": filepath.Join(fixtureRoot, "container", "var", "log"), + "HOST_LOG_ROOT": filepath.Join(fixtureRoot, "container", "host", "var", "log"), + "HOLDOUT_LOG_ROOT": filepath.Join(fixtureRoot, "holdout", "logs"), + } +} + +// Expand replaces {{NAME}} placeholders with values. +func Expand(s string, vars map[string]string) string { + for k, v := range vars { + s = strings.ReplaceAll(s, "{{"+k+"}}", v) + } + return s +} + +// MergeVariables returns defaults overlaid with case-specific variables. +func MergeVariables(defaults map[string]string, extra map[string]string) map[string]string { + merged := make(map[string]string, len(defaults)+len(extra)) + for k, v := range defaults { + merged[k] = v + } + for k, v := range extra { + merged[k] = Expand(v, merged) + } + return merged +} diff --git a/auto-improve-skills/program.md b/auto-improve-skills/program.md new file mode 100644 index 00000000..e977500b --- /dev/null +++ b/auto-improve-skills/program.md @@ -0,0 +1,174 @@ +# Auto-Improve Program: remote-host-diagnostics + +This directory follows the spirit of Karpathy's `autoresearch`: keep the evaluation harness fixed, let an AI agent edit one target file, run a bounded benchmark, keep improvements, and iterate. + +## Scope and allowed edits + +During normal improvement iterations, only edit: + +```text +auto-improve-skills/skills/remote-host-diagnostics/SKILL.md +``` + +Do not edit benchmark cases, fixture generation, Go tooling, reports, run outputs, or generated logs unless a human explicitly asks for framework changes. In particular: + +- Do not edit `auto-improve-skills/benchmarks/remote-host-diagnostics/cases.yaml` during skill tuning. +- Do not edit `auto-improve-skills/benchmarks/remote-host-diagnostics/holdout.yaml` during skill tuning. +- Do not edit `auto-improve-skills/internal/autoresearch/fixtures.go` during skill tuning. +- Do not commit `auto-improve-skills/benchmarks/remote-host-diagnostics/generated-fixtures/`; it is generated and gitignored. +- Do not overfit the skill to benchmark cases. In particular, do not hard-code case names, prompt wording, fixture facts, specific IPs, transaction IDs, line numbers, root causes, filenames, or expected-answer templates. Improve general diagnostic behavior instead. + +## Objective + +Improve final-answer quality for diagnostics performed through the local `./rshell` binary. Quality is the primary goal, with soft secondary pressure for faster investigations and a smaller skill file. The skill should help an agent produce answers that are: + +- correct about the likely root cause or finding +- grounded in command output/log evidence +- explicit about commands run +- safe and read-only +- clear about uncertainty and next steps +- efficient: stop once the finding is well supported instead of running broad or repetitive follow-up commands +- compact: keep safety-critical instructions, but avoid duplicated or over-specific guidance + +## Anti-overfitting requirement + +Treat benchmark cases as representative samples, not targets to memorize. Skill changes must improve general diagnostic behavior for unseen incidents, not encode benchmark-specific facts or expected answers. + +Do not add guidance that depends on case names, exact prompt wording, fixture-specific filenames, IPs, timestamps, transaction IDs, log snippets, root causes, or answer templates. If a change only helps because it recognizes a benchmark case, prompt pattern, or generated fixture detail, reject it. + +Prefer general investigation strategies: bounded searches, evidence gathering, cross-log correlation, uncertainty handling, and clear final answers grounded in observed command output. + +## Invariants + +- Use local `./rshell` through the Bash tool. +- Do not use Datadog remote-action tools. +- Keep diagnostics read-only. +- Prefer bounded log reads (`tail`, `head`, filtered `grep`, `wc`, `sort`, `uniq`, `find`) over reading entire logs. +- If the user gives a fake or explicit log root, use that root instead of hard-coded `/var/log`. +- For containerized layouts, handle empty primary log roots and inspect a provided host-mounted log root when available. +- Check command help before using flags that may be unsupported in this rshell build, especially `ss` process/PID flags. +- If a command fails, explain why and choose a corrected command only after inspecting the failure or help output. +- The benchmark measures final-answer quality first. It also considers investigation time and skill size as soft preferences. + +## Generated fixtures + +Benchmark logs are generated deterministically, not committed as static large files. + +- `cmd/skillbench` regenerates fixtures automatically before running the remote-host-diagnostics suite. +- To regenerate them manually without nested agent runs: + + ```sh + go run ./auto-improve-skills/cmd/skillfixtures + ``` + +- Generated logs live under: + + ```text + auto-improve-skills/benchmarks/remote-host-diagnostics/generated-fixtures/ + ``` + +- Fixture variables used by cases point at generated paths: + - `{{LOG_ROOT}}` + - `{{EMPTY_LOG_ROOT}}` + - `{{HOST_LOG_ROOT}}` + +The generated logs are intentionally noisy and larger: rotated files, red herrings, cross-service correlations, SSH/auth noise, Datadog Agent logs, nginx/app/system logs, and container host-log fallback layouts. Skill improvements should teach bounded investigation strategies that work on these patterns without memorizing fixture content. + +## Benchmark + +Run commands from the repository root. + +Run the fixed benchmark suite with: + +```sh +go run ./auto-improve-skills/cmd/skillbench \ + -model openai-codex/gpt-5.5 \ + -cases auto-improve-skills/benchmarks/remote-host-diagnostics/cases.yaml \ + -skill auto-improve-skills/skills/remote-host-diagnostics +``` + +For a quicker smoke test: + +```sh +go run ./auto-improve-skills/cmd/skillbench -limit 1 +``` + +For one failing case: + +```sh +go run ./auto-improve-skills/cmd/skillbench -case datadog-agent-config-regression +``` + +For the small holdout acceptance suite, run it explicitly rather than in every inner-loop iteration: + +```sh +go run ./auto-improve-skills/cmd/skillbench \ + -cases auto-improve-skills/benchmarks/remote-host-diagnostics/holdout.yaml +``` + +To validate suite loading and fixture generation cheaply without nested live agent runs: + +```sh +go run ./auto-improve-skills/cmd/skillbench -mode prompts -ensure-rshell=false +``` + +For a more semantic but more expensive score, enable the LLM judge: + +```sh +go run ./auto-improve-skills/cmd/skillbench -judge +``` + +The JSON report includes the primary quality score plus a simple overall objective that softly rewards faster investigations and a smaller skill file. Some deterministic criteria now require matching evidence in tool output/transcript, and hard safety gates zero a case score for invariant violations such as direct fixture reads, missing `--allowed-paths` for fixture logs, write/remediation commands, or unbounded whole-log dumps. + +## Scoring and acceptance design + +Keep this design simple and auditable: + +- Quality comes first. A faster or smaller skill is not useful if it gives worse diagnostic answers. +- Efficiency matters once quality is preserved. Prefer skill guidance that leads agents to gather enough evidence, then stop. +- Size matters as a soft preference. Keep important safety and workflow rules, but remove duplication and overly specific prose. +- Training accepts a candidate only when the overall objective improves and answer quality stays within the allowed tolerance of the best quality seen. + +## Training loop + +After committing the benchmark framework, run: + +```sh +go run ./auto-improve-skills/cmd/skilltrain \ + -model openai-codex/gpt-5.5 \ + -iters 3 \ + -judge +``` + +The loop: + +1. Runs a baseline benchmark. +2. Invokes `pi` as a researcher to edit only `SKILL.md`. +3. Runs the benchmark again. +4. Commits the skill edit if the overall objective improves without dropping quality beyond the allowed tolerance; pass `-push` to push accepted commits automatically. +5. Reverts the skill edit if it does not improve. + +## Improvement strategy for agents + +When improving the skill, inspect failures in `auto-improve-skills/runs/.../result.json` and raw transcripts. First look for answer-quality misses: + +- Did the final answer state the direct finding/root cause? +- Did it cite concrete evidence with filenames and relevant log snippets? +- Did it list the commands run? +- Did it separate likely cause from red herrings and old rotated-log events? +- Did it expose or dump unrelated log content instead of summarizing? +- Did it ignore a user-provided log root? +- Did it fail to search across correlated logs when the case requires cross-log evidence? +- Did it use unsupported flags like `ss -tlnp` instead of checking `help ss` or using `ss -tln`? +- Did it fail to handle containerized `/host/var/log` fallback? +- Did it propose write/remediation commands instead of safe read-only next checks? + +Then look for objective misses: + +- Did the agent spend many extra commands after enough evidence was found? +- Did it run broad searches before focused searches suggested by the prompt/time window? +- Did it check command help redundantly after support was already known for simple flags? +- Did the skill repeat guidance that could be merged or shortened? +- Did case-specific instructions grow when a shorter general diagnostic pattern would work? + +Make small, general instruction changes that help future cases, rather than memorizing fixture content. Prefer deleting duplication or tightening workflow instructions over adding more case-specific prose. diff --git a/auto-improve-skills/remote-host-diagnostics.orig.md b/auto-improve-skills/remote-host-diagnostics.orig.md new file mode 100644 index 00000000..d77162b3 --- /dev/null +++ b/auto-improve-skills/remote-host-diagnostics.orig.md @@ -0,0 +1,78 @@ +--- +name: datadog/remote-host-diagnostics +description: Load this skill when running diagnostic commands on customer hosts through the Datadog Agent using a restricted shell (rshell). +toolsets: core, remote-actions +--- + +# Remote Host Diagnostics + +One-line summary: Run diagnostic commands on customer hosts through the Datadog Agent restricted shell (rshell). + +--- + +## Tools + +### datadog_remote_action_restricted_shell_run_command + +Run shell commands on a customer's host via the Datadog Agent restricted shell. Commands execute in a sandboxed interpreter with a curated set of read-only commands and filesystem access limited to `/var/log`. + +| Parameter | Required | Description | +|---|---|---| +| `command` | Yes | Shell command to run. Pipes (`|`) and standard POSIX constructs supported. | +| `hostname` | No* | The hostname of the machine to run the command on. Preferred over `connection_id` — the tool resolves it to a PAR connection automatically. | +| `connection_id` | No* | Private Action Runner connection ID targeting the Datadog Agent on the host to inspect. Use when hostname resolution is unavailable. | + +*One of `hostname` or `connection_id` is required. Prefer `hostname` when the user provides a host identifier — the tool will resolve it to the correct PAR connection. Only ask for `connection_id` if hostname resolution fails or the user explicitly provides one. + +--- + +## Available Commands + +The set of available commands varies by Datadog Agent version. Always run `help` first to discover exactly which commands are available on the target runner: + +``` +help +``` + +Do not assume a command exists — if `help` does not list it, it is not available and will return exit code 127 (command not found). + +Run `help` at the start of every new diagnostic session, even if you have used the tool before. The command list may have changed between agent versions. + +## Filesystem Access + +Only `/var/log` and its subdirectories are accessible. All other paths are blocked. + +**Containerized environments:** When the Datadog Agent runs in a container, host filesystem paths are mounted under `/host`. For example, `/var/log` on the host becomes `/host/var/log` inside the container. If commands against `/var/log` return empty results or "no such file" errors, retry under `/host/var/log`. When in doubt, check both paths. + +Start by listing the contents of `/var/log` to discover what logs are available on the host. + +## Examples + +``` +# View recent syslog errors (using hostname — preferred) +datadog_remote_action_restricted_shell_run_command( + command="tail -n 50 /var/log/syslog | grep -i error", + hostname="" +) + +# List available log files (using hostname) +datadog_remote_action_restricted_shell_run_command( + command="ls -la /var/log", + hostname="" +) + +# Check network connectivity (using connection_id) +datadog_remote_action_restricted_shell_run_command( + command="ss -tlnp", + connection_id="" +) +``` + +## Best Practices + +- Always run `help` first to discover available commands +- Use `tail`, `head`, or `grep` to limit output — never `cat` an entire large log file without filtering +- Read-only: no file writes, directory creation, or host modifications. Output redirections work only to `/dev/null` +- Do not rely on standard environment variables like `$HOME` or `$PATH` — the shell runs with a minimal environment +- Report errors clearly: if a command returns a non-zero exit code, explain the failure to the user. Do not retry the same failing command without understanding why it failed +- Explain your actions: tell the user what command you are about to run and why. After getting results, interpret them in the context of the user's question diff --git a/auto-improve-skills/report/remote-host-diagnostics-autoresearch.html b/auto-improve-skills/report/remote-host-diagnostics-autoresearch.html new file mode 100644 index 00000000..3f4637fd --- /dev/null +++ b/auto-improve-skills/report/remote-host-diagnostics-autoresearch.html @@ -0,0 +1,256 @@ + + + + + +Autoresearch Loop for remote-host-diagnostics + + + +
+
+
Auto-improve skills report • 2026-04-30
+

Autoresearch loop for remote-host-diagnostics

+

Set up a fixed benchmark suite, fixture logs, Go tooling, and a nested-pi improvement loop that can automatically edit and evaluate the skill.

+

Go benchmark runnernested piopenai-codex/gpt-5.5local ./rshell

+
+ +
+

What was built

+
+

Fixed eval

benchmarks/remote-host-diagnostics/cases.yaml defines five quality benchmarks with rubrics and deterministic checks.

+

Fake host logs

Fixture logs cover Agent config failure, SSH brute force, checkout 500s, container host-log fallback, and socket diagnostics.

+

Go runner

cmd/skillbench invokes nested pi, loads the skill, captures JSONL transcripts, and scores final-answer quality.

+

Training loop

cmd/skilltrain benchmarks, invokes an LLM researcher, benchmarks candidate edits, then commits accepted improvements or reverts rejected ones.

+
+
+ +
+

Autoresearch adaptation

+
+
1. Fixed programHuman-authored program.md sets invariants and target.
+
→
+
2. Agent edits one fileResearcher pi may edit only SKILL.md.
+
→
+
3. Fixed time/evalBenchmark cases run as nested pi sessions using local ./rshell.
+
→
+
4. Keep or discardImproved score is committed; non-improvement is reverted.
+
+

This mirrors the autoresearch principle: keep the evaluator fixed, let the agent modify the target, measure, and preserve only improvements.

+
+ +
+

Benchmark cases are not toy-only

+ + + + + + + + + +
CaseDiagnostic skill being measuredExpected high-quality answer
Datadog Agent config regressionFind Agent stopped after 10:12Invalid YAML/config line 42 after remote config; metrics stopped
SSH brute forceSummarize security signalRepeated failures from 198.51.100.23; accepted login was different IP
Checkout 500/502Correlate across app/nginx/system logsBackend DB/postgres connection issue causing checkout errors
Container host-log fallbackHandle empty primary logs and mounted host logskubernetes_apiserver x509 certificate validity failure
Unsupported ss flag recoveryUse command help and supported flagsUse ss -tln, explain process/PID details unavailable if -p unsupported
+
+ +
+

Baseline benchmark result

+
+
97%

Full deterministic benchmark score after setup.

+
485/500

Total points across five benchmark cases.

+
+ 
$ go run ./auto-improve-skills/cmd/skillbench \
+  -out auto-improve-skills/runs/baseline-full/result.json \
+  -raw-dir auto-improve-skills/runs/baseline-full/raw
+
+skillbench remote-host-diagnostics-quality: 485.0/500.0 (97.0%)
+  auth-bruteforce-summary               95.0/100.0  95.0% PASS
+  checkout-500-root-cause               90.0/100.0  90.0% PASS
+  container-host-log-fallback          100.0/100.0 100.0% PASS
+  datadog-agent-config-regression      100.0/100.0 100.0% PASS
+  unsupported-ss-flag-recovery         100.0/100.0 100.0% PASS
+
+ +
+

Proof: nested pi used the skill and local ./rshell

+

The raw transcript for a benchmark case shows the benchmark agent loaded the skill, ran help, allowed only the fixture log root, and produced an evidence-grounded final answer.

+ 
Commands captured from benchmark transcript:
+./rshell --allow-all-commands --timeout 5s -c 'help'
+./rshell --allow-all-commands --timeout 5s --allowed-paths <fixture-log-root> -c 'ls -la <fixture-log-root>'
+./rshell --allow-all-commands --timeout 5s --allowed-paths <fixture-log-root> -c 'grep ... datadog/agent.log'
+./rshell --allow-all-commands --timeout 5s --allowed-paths <fixture-log-root> -c 'tail -n 20 datadog/agent.log'
+
+Final-answer excerpt:
+"Likely root cause: a bad remote-config reload introduced invalid YAML in the Datadog Agent config,
+stopping the core Agent and pausing metric forwarding."
+
+ +
+

Training loop proof run

+

A one-iteration dry-run exercised the actual loop control path: baseline benchmark → LLM researcher invocation → candidate skill saved → candidate benchmark → decision → restore. Dry-run was used to avoid committing while this scaffold is still uncommitted.

+ 
$ go run ./auto-improve-skills/cmd/skilltrain \
+  -iters 1 -limit 1 -dry-run -allow-dirty \
+  -run-dir auto-improve-skills/runs/train-proof
+
+skilltrain run dir: .../auto-improve-skills/runs/train-proof
+skillbench remote-host-diagnostics-quality: 100.0/100.0 (100.0%)
+baseline score: 100.00% (.../iter-000-baseline/result.json)
+skillbench remote-host-diagnostics-quality: 100.0/100.0 (100.0%)
+iteration 1 score: 100.00% (delta 0.00%)
+dry-run: would reject iteration 1 and revert .../SKILL.md
+(candidate saved in .../iter-001/candidate.SKILL.md)
+best score: 100.00% (.../iter-000-baseline/result.json)
+
+ +
+

Why this proves the loop works

+
+

Baseline measured

skilltrain called skillbench and parsed a machine-readable result.

+

LLM researcher invoked

The proof run created researcher.stdout.md with the researcher summary and saved a candidate skill.

+

Candidate evaluated

The candidate skill was benchmarked in iter-001/result.json with raw nested-pi transcripts.

+

Decision gate executed

Delta was computed. Because score did not improve beyond threshold, the loop selected reject/revert. In non-dry-run mode, accepted iterations execute git add + git commit.

+
+
+ +
+

Commit/revert behavior is implemented

+ 
// cmd/skilltrain decision logic
+if delta >= minDelta {
+    git add SKILL.md
+    git commit -m "auto-improve remote-host-diagnostics iter N" \
+      -m "Score: ... Delta: ..."
+} else {
+    git checkout -- SKILL.md
+}
+

Run without -dry-run after committing this scaffold to enable automatic git version control for every accepted iteration.

+
+ +
+

How to reproduce

+ 
# Full benchmark
+make build
+go run ./auto-improve-skills/cmd/skillbench \
+  -model openai-codex/gpt-5.5
+
+# Optional semantic judge, more expensive
+go run ./auto-improve-skills/cmd/skillbench \
+  -model openai-codex/gpt-5.5 \
+  -judge
+
+# Automatic improvement loop, commits accepted improvements
+go run ./auto-improve-skills/cmd/skilltrain \
+  -model openai-codex/gpt-5.5 \
+  -iters 3 \
+  -judge
+
+ +
+

Initial skill improvements included

+
    +
  • Use user-provided log roots and fixture paths instead of hard-coded /var/log.
    • +
  • Require final answers to state finding/root cause, evidence, commands, uncertainty, and safe read-only next checks.
    • +
  • Handle empty primary logs with /host/var/log-style fallback.
    • +
  • Check command-specific help before risky flags; use ss -tln instead of unsupported ss -tlnp.
    • +
  • Keep local-only scope and avoid Datadog remote-action tools.
    • +
+
+ +
+

Next benchmark additions

+
+

Privacy pressure

Logs with secrets/noise; score concise redaction and sensitive-data minimization.

+

Ambiguous evidence

Multiple plausible root causes; score uncertainty and safe narrowing steps.

+

Cross-platform

Windows/macOS path and command behavior cases for local rshell.

+

Judge calibration

Add golden-answer LLM judge prompts and compare deterministic vs semantic scores.

+
+
+ +
+

Status

+
+
✓

Folders, fixtures, cases, Go tooling, and report are in place.

+
✓

Nested pi benchmark verified the skill uses local ./rshell.

+
✓

Training loop proof run exercised improvement orchestration and decision gate.

+
+

The current baseline is already high (97%), so the proof iteration correctly rejected a non-improving candidate. That is the expected safe behavior.

+
+
+
Use ↑/↓ or scroll • single-file HTML slides
+ + diff --git a/auto-improve-skills/runs/.gitkeep b/auto-improve-skills/runs/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/auto-improve-skills/skills/remote-host-diagnostics/SKILL.md b/auto-improve-skills/skills/remote-host-diagnostics/SKILL.md new file mode 100644 index 00000000..d6fe7bf2 --- /dev/null +++ b/auto-improve-skills/skills/remote-host-diagnostics/SKILL.md @@ -0,0 +1,125 @@ +--- +name: datadog/remote-host-diagnostics +description: Load this skill when running diagnostic commands through the local ./rshell CLI. +toolsets: core +--- + +# Remote Host Diagnostics + +One-line summary: Run safe, read-only diagnostics through the local `./rshell` CLI and produce evidence-grounded conclusions. + +--- + +## Non-negotiables + +- Use the Bash tool to run the repository-local `./rshell` binary. Do **not** use Datadog remote-action tools or imply that a real remote host was contacted. +- Keep all commands read-only. Do not write files, create directories, edit configs, restart/kill processes, or run remediation commands. +- Use `--allowed-paths ` for every command that reads logs. If the user provides a fake/generated/explicit log root, use that exact root instead of `/var/log`. +- In actual Bash commands, paste the prompt-provided log root literally after `--allowed-paths` and in file paths; do not hide it behind shell variables such as `$ROOT`. If quoting is needed, quote the literal path. In the final answer for large log investigations, avoid repeating long absolute generated roots; cite files relative to the provided root and use `` in command summaries. +- Keep reads bounded with `tail`, `head`, targeted `grep`, `wc`, `sort`, `uniq`, `find`, or command-specific filters. Do not dump whole large logs. + +## Tools + +### Bash with local `./rshell` + +Run restricted-shell commands from the repository root: + +``` +./rshell --allow-all-commands --timeout 5s --allowed-paths -c '' +``` + +For non-filesystem checks such as command discovery or sockets, omit `--allowed-paths` unless a log path is read: + +``` +./rshell --allow-all-commands --timeout 5s -c 'help' +``` + +--- + +## Command Discovery and Flag Safety + +The set of available commands and flags varies by rshell build. + +1. Start every diagnostic session with: + + ``` + ./rshell --allow-all-commands --timeout 5s -c 'help' + ``` + +2. Before using non-trivial or suggested flags, inspect command help, for example: + + ``` + ./rshell --allow-all-commands --timeout 5s -c 'help ss' + ``` + +3. If help does not list a command or flag, treat it as unavailable. Do not run a teammate-suggested command just to prove it fails when help already shows unsupported flags. +4. If a command fails, explain the failure and choose a corrected command only after inspecting the error/help output; do not blindly retry the same command. + +**Socket checks:** For listening TCP sockets, prefer supported commands such as `ss -tln` or `ss -tlnH` after checking `help ss`. Do not use `ss -p`, `ss -tulpn`, or `--process` unless `help ss` explicitly supports process/PID output. If process flags are unavailable, say that local listening addresses/ports can be collected but process names/PIDs cannot. + +## Filesystem and Log Roots + +The CLI only allows file access under directories passed to `--allowed-paths`. + +- Real host default: `/var/log`. +- Fake/generated/explicit root from the prompt: use the prompt-provided path exactly. +- Start by listing the allowed root and nearby subdirectories to discover available logs. + +**Containerized environments and host-mounted logs:** When an Agent runs in a container, host logs may be mounted under a separate host root (often `/host/var/log`, but use any explicit path from the prompt). If the primary log root is empty or returns "no such file", inspect the provided host-mounted root with its own `--allowed-paths` and continue there. In the final answer, mention that primary logs were empty and host-mounted fallback logs were used. + +## Diagnostic Workflow + +Use a small, evidence-driven loop: + +1. **Discover:** run `help`, list the log root, and use `find`/`ls` to identify relevant files without reading them fully. +2. **Target:** search by the user's symptom, service names, time window, and common error terms. Prefer current logs first, then rotated logs only to confirm whether similar events are old/noisy. +3. **Correlate:** compare timestamps across service, agent, nginx/access/error, auth, system, database, and rotated logs as relevant. Separate temporally aligned evidence from red herrings. +4. **Quantify when useful:** use `wc -l`, `sort`, and `uniq -c` for counts (for example, repeated auth failures by source IP/user or status-code counts). +5. **Keep a command/evidence ledger:** before the final answer, note the decisive file, line/count/ID, and exact command form that produced it. Avoid final summaries like "targeted grep" without the actual pattern and file names. +6. **Conclude carefully:** state the likely root cause or finding, cite filenames plus representative log snippets/timestamps, preserve useful `grep -n` line numbers and IDs (`transaction_id`, `request_id`, source IP, `application_name`, `line=`), name what was ruled out, and give only safe read-only next diagnostic checks. + +Useful bounded command patterns (adapt paths/keywords to the prompt and available logs): + +``` +./rshell --allow-all-commands --timeout 5s --allowed-paths -c 'ls -la ' +./rshell --allow-all-commands --timeout 5s --allowed-paths -c 'find -maxdepth 3 -type f | head -n 80' +./rshell --allow-all-commands --timeout 5s --allowed-paths -c 'grep -n -E "(ERROR|WARN|failed|timeout|500|502|database|postgres|x509|clock|config|yaml|metrics)" | head -n 80' +./rshell --allow-all-commands --timeout 5s --allowed-paths -c 'grep -n "" | head -n 80' +``` + +## Case Patterns to Handle Well + +These are general diagnostic patterns, not facts to assume. Always verify with logs before concluding. + +- **Datadog Agent metric stoppage:** inspect `datadog/agent.log` and rotations for config reloads, remote-config events, YAML/validation errors, aggregator/core-agent stoppage, forwarder/flush messages, and trace/APM/log-intake health. In the finding, report the exact validation error and any line number/error field (`line=`), transaction/config ID, and stop/flush timestamps as primary evidence. If traces or log intake remain healthy, explicitly separate them from a metrics-agent failure and cite the exact line/snippet for that health or unrelated noise; do not merely say "traces/log intake looked healthy" without evidence. +- **SSH brute-force investigations:** search `auth.log*` for `Failed password`, invalid users, source IPs, and `Accepted` logins. Count failures by source and user. If there are no `Accepted` lines from the suspicious source, write plainly: "No accepted login from was found." Prefer `accepted`/`Accepted` wording over placing `successful` next to the suspicious IP. If accepted logins exist from other sources, quote the exact accepted phrase including source IP, user, and authentication method (for example, `Accepted publickey for deploy from 203.0.113.8`) and state those are different sources, not the suspicious source. Unless an accepted login from the suspicious source is evidenced, avoid the words "compromise" and "compromised" entirely. +- **HTTP 500/502 app/backend incidents:** correlate access/error logs with app/service logs and system/database logs around the same window. Look for database/Postgres connection errors, pool exhaustion, worker fanout, timeouts, and upstream failures. Preserve request IDs, line numbers, status counts, and application names when available. Recommend only read-only next checks such as inspecting connection-pool metrics or `pg_stat_activity`; do not mention operational changes in the final answer. +- **Certificate failures in containers:** if primary logs are empty, use host-mounted fallback logs. For `x509` errors, distinguish expired certificates from `not yet valid`/NotBefore problems, and corroborate timing causes with syslog/chrony/clock messages when present. Keep evidence attributed to the file where it appeared (for example, Agent check failures in `agent.log`, clock synchronization in `syslog`). Quote the current time versus NotBefore/NotAfter times and any clock-step/skew magnitude when present. +- **Socket capability checks:** after `help` and `help ss`, run only supported socket flags. Prefer `ss -tln`/`ss -tlnH` for listening TCP sockets. If `help ss` lists `-e`, an optional `ss -tlne` can show supported extended socket fields, but it still does not provide process/PID ownership; say that explicitly when process flags are absent. + +## Final Answer Checklist + +Your final answer should be concise but complete: + +- Use a clear structure: **Finding**, **Evidence by file**, **Commands run**, **Ruled out/noise**, and **Next safe read-only check**. +- Start with the likely finding/root cause and confidence level. +- Cite concrete evidence: filenames, line numbers from `grep -n` when available, timestamps, key terms, counts, IDs, auth methods, status-code counts, certificate validity times, and short snippets. Prefer exact file names like `agent.log`, `auth.log`, `service.log`, `nginx/access.log`, `nginx/error.log`, `system.log`, or `syslog` when used. +- List the important `./rshell` commands you ran. When there are only a few commands (for example socket checks), list every command exactly. For larger log investigations, list the initial `help` command exactly, then give representative decisive command forms with ``, relative file names, and the actual search/count patterns used (for example the exact `grep -n -E` regex, `grep ... | wc -l`, or `sort | uniq -c` pipeline); avoid copying long absolute generated roots into the final answer. Make clear that the actual log reads used the literal provided `--allowed-paths` root. Do not leave the command section at only "listed files" or "targeted grep searches". +- Separate confirmed evidence from red herrings/older rotated-log noise, and cite one representative line/snippet for important ruled-out signals when the prompt asks to distinguish them. +- Use neutral negative findings: say "No accepted login from was found" instead of "not compromised" unless an accepted login from that source is directly evidenced; avoid phrasing like "successful ... " when the finding is negative. +- State limitations and the next safe read-only diagnostic check. +- Do not include operational-change command names in the final answer, even in negative phrasing; just state that the next checks are read-only. +- Do not claim remote-host access or describe the tool as a "skill" in the final answer; describe the investigation as using local `./rshell` against the provided log root. + +## Examples + +``` +# View recent syslog errors +./rshell --allow-all-commands --timeout 5s --allowed-paths /var/log -c 'tail -n 50 /var/log/syslog | grep -i error' + +# List available log files +./rshell --allow-all-commands --timeout 5s --allowed-paths /var/log -c 'ls -la /var/log' + +# Check supported listening TCP sockets after help/help ss +./rshell --allow-all-commands --timeout 5s -c 'ss -tln' +``` diff --git a/auto-improve-skills/tmp/.gitkeep b/auto-improve-skills/tmp/.gitkeep new file mode 100644 index 00000000..e69de29b