From a5e3a02d839b28e7398be841d32cb6f557105bca Mon Sep 17 00:00:00 2001 From: DJ Date: Tue, 7 Apr 2026 10:46:26 -0700 Subject: [PATCH 1/6] test(feature-ideation): extract bash to scripts, add schema + 92 bats tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refactors the reusable feature-ideation workflow's parsing surface from an inline 600-line YAML heredoc into testable scripts with deterministic contracts. Every defect that previously required post-merge review can now fail in CI before adopters notice. Why --- The prior reusable workflow used `2>/dev/null || echo '[]'` for every gh / GraphQL call, which silently downgraded auth failures, rate limits, network outages, and GraphQL schema drift to empty arrays. The pipeline would "succeed" while producing useless signals — and Mary's Discussion posts would silently degrade across every BMAD repo on the org. The prompt also instructed Mary to "use fuzzy matching" against existing Ideas Discussions in her head, which is non-deterministic and untestable. Risk register (probability × impact, scale 1–9): R1=9 swallow-all-errors gh wrapper R2=6 literal $() inside YAML direct prompt R3=6 no signals.json schema R4=6 jq --argjson crash on empty input R5=6 fuzzy match in Mary's prompt → duplicate Discussions R6=6 retry idempotency hole R7=6 GraphQL errors[]/null data not detected R8=4 GraphQL partial errors silently accepted R10=3 bot filter only catches dependabot/github-actions R11=4 pagination silently truncates What's new ---------- .github/scripts/feature-ideation/ collect-signals.sh Orchestrator (replaces inline heredoc) validate-signals.py JSON Schema 2020-12 validator match-discussions.sh Deterministic Jaccard matcher (kills R5/R6) discussion-mutations.sh create/comment/label wrappers + DRY_RUN mode lint-prompt.sh Catches unescaped $() / ${VAR} in prompt blocks lib/gh-safe.sh Defensive gh wrapper, fails loud on every documented failure mode (kills R1, R7, R8) lib/compose-signals.sh Validates JSON inputs before jq composition lib/filter-bots.sh Extensible bot author filter (kills R10) lib/date-utils.sh Cross-platform date helpers README.md Maintainer docs .github/schemas/signals.schema.json Pinned producer/consumer contract for signals.json (Draft 2020-12). CI rejects any drift; the runtime signals.json is also validated by the workflow before being handed to Mary. .github/workflows/feature-ideation-reusable.yml Rewritten. Adds a self-checkout of petry-projects/.github so the scripts above are available in the runner. Replaces inline bash with collect-signals.sh + validate-signals.py. Adds RUN_DATE / SIGNALS_PATH / PROPOSALS_PATH / MATCH_PLAN_PATH / TOOLING_DIR env vars passed to claude-code-action via env: instead of unescaped shell expansions in the prompt body. Adds dry_run input that flows through to discussion-mutations.sh, which logs every planned action to a JSONL audit log instead of executing — uploaded as the dry-run-log artifact. .github/workflows/feature-ideation-tests.yml New CI gate, path-filtered. Runs shellcheck, lint-prompt, schema fixture validation, and the full bats suite on every PR that touches the feature-ideation surface. standards/workflows/feature-ideation.yml Updated caller stub template. Adds dry_run workflow_dispatch input so adopters get safe smoke-testing for free. Existing TalkTerm caller stub continues to work unchanged (dry_run defaults to false). test/workflows/feature-ideation/ 92 bats tests across 9 suites. 14 GraphQL/REST response fixtures. 5 expected signals.json fixtures (3 valid + 2 INVALID for negative schema testing). Programmable gh PATH stub with single-call and multi-call modes for integration testing. | Suite | Tests | Risks closed | |-----------------------------|------:|--------------------| | gh-safe.bats | 19 | R1, R7, R8 | | compose-signals.bats | 8 | R3, R4 | | filter-bots.bats | 5 | R10 | | date-utils.bats | 7 | R9 | | collect-signals.bats | 14 | R1, R3, R4, R7, R11| | match-discussions.bats | 13 | R5, R6 | | discussion-mutations.bats | 10 | DRY_RUN contract | | lint-prompt.bats | 8 | R2 | | signals-schema.bats | 8 | R3 | | TOTAL | 92 | | Test results: 92 passing, 0 failing, 0 skipped. Run with: bats test/workflows/feature-ideation/ Backwards compatibility ----------------------- The reusable workflow's input surface is unchanged for existing callers (TalkTerm continues to work with no edits). The new dry_run input is optional and defaults to false. Adopters who copy the new standards caller stub get dry_run support automatically. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/schemas/signals.schema.json | 187 ++++++ .github/scripts/feature-ideation/README.md | 81 +++ .../feature-ideation/collect-signals.sh | 222 +++++++ .../feature-ideation/discussion-mutations.sh | 159 +++++ .../feature-ideation/lib/compose-signals.sh | 83 +++ .../feature-ideation/lib/date-utils.sh | 36 ++ .../feature-ideation/lib/filter-bots.sh | 50 ++ .../scripts/feature-ideation/lib/gh-safe.sh | 166 +++++ .../scripts/feature-ideation/lint-prompt.sh | 116 ++++ .../feature-ideation/match-discussions.sh | 178 ++++++ .../feature-ideation/validate-signals.py | 79 +++ .../workflows/feature-ideation-reusable.yml | 568 +++++++----------- .github/workflows/feature-ideation-tests.yml | 112 ++++ standards/workflows/feature-ideation.yml | 6 + .../feature-ideation/collect-signals.bats | 204 +++++++ .../feature-ideation/compose-signals.bats | 102 ++++ .../feature-ideation/date-utils.bats | 50 ++ .../discussion-mutations.bats | 102 ++++ .../feature-ideation/filter-bots.bats | 62 ++ .../expected/INVALID-bad-repo.signals.json | 13 + .../INVALID-missing-field.signals.json | 6 + .../fixtures/expected/empty-repo.signals.json | 13 + .../fixtures/expected/populated.signals.json | 93 +++ .../fixtures/expected/truncated.signals.json | 24 + .../gh-responses/graphql-categories.json | 13 + .../graphql-discussions-truncated.json | 20 + .../gh-responses/graphql-discussions.json | 20 + .../gh-responses/graphql-errors-envelope.json | 10 + .../graphql-no-ideas-category.json | 11 + .../gh-responses/issue-list-closed.json | 14 + .../gh-responses/issue-list-open.json | 23 + .../fixtures/gh-responses/pr-list-merged.json | 14 + .../fixtures/gh-responses/release-list.json | 8 + test/workflows/feature-ideation/gh-safe.bats | 163 +++++ .../feature-ideation/helpers/setup.bash | 47 ++ .../feature-ideation/lint-prompt.bats | 122 ++++ .../feature-ideation/match-discussions.bats | 181 ++++++ .../feature-ideation/signals-schema.bats | 52 ++ test/workflows/feature-ideation/stubs/gh | 71 +++ 39 files changed, 3141 insertions(+), 340 deletions(-) create mode 100644 .github/schemas/signals.schema.json create mode 100644 .github/scripts/feature-ideation/README.md create mode 100755 .github/scripts/feature-ideation/collect-signals.sh create mode 100755 .github/scripts/feature-ideation/discussion-mutations.sh create mode 100755 .github/scripts/feature-ideation/lib/compose-signals.sh create mode 100755 .github/scripts/feature-ideation/lib/date-utils.sh create mode 100755 .github/scripts/feature-ideation/lib/filter-bots.sh create mode 100755 .github/scripts/feature-ideation/lib/gh-safe.sh create mode 100755 .github/scripts/feature-ideation/lint-prompt.sh create mode 100755 .github/scripts/feature-ideation/match-discussions.sh create mode 100755 .github/scripts/feature-ideation/validate-signals.py create mode 100644 .github/workflows/feature-ideation-tests.yml create mode 100644 test/workflows/feature-ideation/collect-signals.bats create mode 100644 test/workflows/feature-ideation/compose-signals.bats create mode 100644 test/workflows/feature-ideation/date-utils.bats create mode 100644 test/workflows/feature-ideation/discussion-mutations.bats create mode 100644 test/workflows/feature-ideation/filter-bots.bats create mode 100644 test/workflows/feature-ideation/fixtures/expected/INVALID-bad-repo.signals.json create mode 100644 test/workflows/feature-ideation/fixtures/expected/INVALID-missing-field.signals.json create mode 100644 test/workflows/feature-ideation/fixtures/expected/empty-repo.signals.json create mode 100644 test/workflows/feature-ideation/fixtures/expected/populated.signals.json create mode 100644 test/workflows/feature-ideation/fixtures/expected/truncated.signals.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/graphql-categories.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/graphql-discussions-truncated.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/graphql-discussions.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/graphql-errors-envelope.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/graphql-no-ideas-category.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/issue-list-closed.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/issue-list-open.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/pr-list-merged.json create mode 100644 test/workflows/feature-ideation/fixtures/gh-responses/release-list.json create mode 100644 test/workflows/feature-ideation/gh-safe.bats create mode 100644 test/workflows/feature-ideation/helpers/setup.bash create mode 100644 test/workflows/feature-ideation/lint-prompt.bats create mode 100644 test/workflows/feature-ideation/match-discussions.bats create mode 100644 test/workflows/feature-ideation/signals-schema.bats create mode 100755 test/workflows/feature-ideation/stubs/gh diff --git a/.github/schemas/signals.schema.json b/.github/schemas/signals.schema.json new file mode 100644 index 0000000..51d5f65 --- /dev/null +++ b/.github/schemas/signals.schema.json @@ -0,0 +1,187 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/petry-projects/TalkTerm/.github/schemas/signals.schema.json", + "title": "Feature Ideation Signals", + "description": "Canonical contract between collect-signals.sh and the BMAD Analyst (Mary) prompt. Any change to this schema is a breaking change to the workflow.", + "type": "object", + "required": [ + "schema_version", + "scan_date", + "repo", + "open_issues", + "closed_issues_30d", + "ideas_discussions", + "releases", + "merged_prs_30d", + "feature_requests", + "bug_reports", + "truncation_warnings" + ], + "additionalProperties": false, + "properties": { + "schema_version": { + "type": "string", + "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$" + }, + "scan_date": { + "type": "string", + "format": "date-time" + }, + "repo": { + "type": "string", + "pattern": "^[^/]+/[^/]+$" + }, + "open_issues": { + "$ref": "#/$defs/issueBucket" + }, + "closed_issues_30d": { + "$ref": "#/$defs/issueBucket" + }, + "ideas_discussions": { + "type": "object", + "required": ["count", "items"], + "additionalProperties": false, + "properties": { + "count": { "type": "integer", "minimum": 0 }, + "items": { + "type": "array", + "items": { "$ref": "#/$defs/discussion" } + } + } + }, + "releases": { + "type": "array", + "items": { "$ref": "#/$defs/release" } + }, + "merged_prs_30d": { + "type": "object", + "required": ["count", "items"], + "additionalProperties": false, + "properties": { + "count": { "type": "integer", "minimum": 0 }, + "items": { + "type": "array", + "items": { "$ref": "#/$defs/pullRequest" } + } + } + }, + "feature_requests": { + "$ref": "#/$defs/issueBucket" + }, + "bug_reports": { + "$ref": "#/$defs/issueBucket" + }, + "truncation_warnings": { + "type": "array", + "items": { + "type": "object", + "required": ["source", "limit", "message"], + "additionalProperties": false, + "properties": { + "source": { "type": "string" }, + "limit": { "type": "integer", "minimum": 0 }, + "message": { "type": "string" } + } + } + } + }, + "$defs": { + "issueBucket": { + "type": "object", + "required": ["count", "items"], + "additionalProperties": false, + "properties": { + "count": { "type": "integer", "minimum": 0 }, + "items": { + "type": "array", + "items": { "$ref": "#/$defs/issue" } + } + } + }, + "issue": { + "type": "object", + "required": ["number", "title", "labels"], + "additionalProperties": true, + "properties": { + "number": { "type": "integer" }, + "title": { "type": "string" }, + "labels": { + "type": "array", + "items": { + "type": "object", + "required": ["name"], + "properties": { + "name": { "type": "string" } + } + } + }, + "createdAt": { "type": "string" }, + "closedAt": { "type": ["string", "null"] }, + "author": { + "type": ["object", "null"], + "properties": { + "login": { "type": "string" } + } + } + } + }, + "discussion": { + "type": "object", + "required": ["id", "number", "title"], + "additionalProperties": true, + "properties": { + "id": { "type": "string" }, + "number": { "type": "integer" }, + "title": { "type": "string" }, + "createdAt": { "type": "string" }, + "updatedAt": { "type": "string" }, + "labels": { + "type": "object", + "properties": { + "nodes": { + "type": "array", + "items": { + "type": "object", + "properties": { "name": { "type": "string" } } + } + } + } + }, + "comments": { + "type": "object", + "properties": { + "totalCount": { "type": "integer" } + } + } + } + }, + "release": { + "type": "object", + "required": ["tagName"], + "additionalProperties": true, + "properties": { + "tagName": { "type": "string" }, + "name": { "type": ["string", "null"] }, + "publishedAt": { "type": "string" }, + "isPrerelease": { "type": "boolean" } + } + }, + "pullRequest": { + "type": "object", + "required": ["number", "title"], + "additionalProperties": true, + "properties": { + "number": { "type": "integer" }, + "title": { "type": "string" }, + "labels": { + "type": "array", + "items": { + "type": "object", + "properties": { "name": { "type": "string" } } + } + }, + "mergedAt": { "type": "string" } + } + } + } +} diff --git a/.github/scripts/feature-ideation/README.md b/.github/scripts/feature-ideation/README.md new file mode 100644 index 0000000..9ef03b9 --- /dev/null +++ b/.github/scripts/feature-ideation/README.md @@ -0,0 +1,81 @@ +# Feature Ideation — Scripts & Test Strategy + +This directory contains the bash + Python helpers that back +`.github/workflows/feature-ideation.yml`. Every line of logic that used to +live inside the workflow's heredoc has been extracted here so it can be unit +tested with bats. + +## Why this exists + +The original workflow was a 500+ line YAML file with bash, jq, and GraphQL +queries inlined into a heredoc, plus a `direct_prompt:` block that asked +the BMAD Analyst (Mary) to call `gh api` and parse responses with no schema +or error handling. Every defect was discovered post-merge by reviewers. +The risks (R1–R11) are documented in the test architect's risk register — +search for "Murat" in the project history. + +This refactor moves the parsing surface into testable units so failures are +caught **before UAT** instead of after. + +## File map + +| File | Purpose | Killed risk | +|------|---------|------------| +| `lib/gh-safe.sh` | Wraps every `gh` and `gh api graphql` call. Fails loud on auth, rate-limit, network, GraphQL errors envelope, or `data: null`. Replaces the original `2>/dev/null \|\| echo '[]'` swallow. | R1, R7, R8 | +| `lib/compose-signals.sh` | Validates JSON inputs before `jq --argjson` and assembles the canonical signals.json document. | R3, R4 | +| `lib/filter-bots.sh` | Configurable bot allowlist via `FEATURE_IDEATION_BOT_AUTHORS`. | R10 | +| `lib/date-utils.sh` | Cross-platform date arithmetic helpers. | R9 | +| `collect-signals.sh` | Orchestrator: drives all `gh` calls, composes signals.json, emits truncation warnings. | R1, R3, R4, R11 | +| `validate-signals.py` | JSON Schema 2020-12 validator for signals.json against `../schemas/signals.schema.json`. | R3 | +| `match-discussions.sh` | Deterministic Jaccard-similarity matcher between Mary's proposals and existing Ideas Discussions. Replaces the prose "use fuzzy matching" instruction. | R5, R6 | +| `discussion-mutations.sh` | `create_discussion`, `comment_on_discussion`, `add_label_to_discussion` wrappers with `DRY_RUN=1` audit-log mode. | Smoke testing | +| `lint-prompt.sh` | Scans every workflow file for unescaped `$()` / `${VAR}` inside `direct_prompt:` blocks (which YAML and `claude-code-action` pass verbatim instead of expanding). | R2 | + +## Running the tests + +```bash +# Install once +brew install bats-core shellcheck jq # macOS +sudo apt-get install bats shellcheck jq # Ubuntu (CI) +pip install jsonschema # Python schema validator + +# Run everything +bats test/workflows/feature-ideation/ + +# Run a single suite +bats test/workflows/feature-ideation/gh-safe.bats + +# Lint shell scripts +(cd .github/scripts/feature-ideation && \ + shellcheck -x collect-signals.sh lint-prompt.sh match-discussions.sh \ + discussion-mutations.sh lib/*.sh) + +# Lint the workflow's direct_prompt block +bash .github/scripts/feature-ideation/lint-prompt.sh +``` + +CI runs all of the above on every PR that touches this directory or the +workflow file. See `.github/workflows/feature-ideation-tests.yml`. + +## DRY_RUN mode + +To smoke-test the workflow on a fork without writing to GitHub Discussions: + +1. Trigger via `workflow_dispatch` with `dry_run: true`. +2. The `analyze` job will source `discussion-mutations.sh` with `DRY_RUN=1`, + so every `create_discussion` / `comment_on_discussion` / `add_label_to_discussion` + call writes a JSONL entry to `$DRY_RUN_LOG` instead of executing. +3. The dry-run log is uploaded as the `dry-run-log` artifact for human review. + +## Schema as contract + +`/.github/schemas/signals.schema.json` is the **producer/consumer contract** +between `collect-signals.sh` and Mary's prompt. Any change to the shape of +signals.json is a breaking change and must: + +1. Bump `SCHEMA_VERSION` in `collect-signals.sh`. +2. Update fixtures under `test/workflows/feature-ideation/fixtures/expected/`. +3. Update Mary's prompt in `feature-ideation.yml` if any field references move. + +CI validates every fixture against the schema, and the workflow validates +the runtime output before handing it to Mary. diff --git a/.github/scripts/feature-ideation/collect-signals.sh b/.github/scripts/feature-ideation/collect-signals.sh new file mode 100755 index 0000000..cf51229 --- /dev/null +++ b/.github/scripts/feature-ideation/collect-signals.sh @@ -0,0 +1,222 @@ +#!/usr/bin/env bash +# collect-signals.sh — gather project signals into signals.json. +# +# Replaces the inline bash heredoc that previously lived in +# .github/workflows/feature-ideation.yml. Each step is delegated to a +# library function so it can be unit-tested in isolation. +# +# Required env: +# REPO "owner/name" +# GH_TOKEN GitHub token with read scopes +# +# Optional env: +# SIGNALS_OUTPUT Path to write signals.json (default: ./signals.json) +# ISSUE_LIMIT Max issues per query (default: 50) +# PR_LIMIT Max PRs per query (default: 30) +# DISCUSSION_LIMIT Max Ideas discussions to fetch (default: 100) +# FEATURE_IDEATION_BOT_AUTHORS Extra comma-separated bot logins to filter +# +# Exit codes: +# 0 signals.json written successfully +# 64 bad usage / missing env +# 65 data validation error +# 66 GraphQL refused our request (errors / null data) +# 1+ underlying gh failure (auth, rate limit, network) + +set -euo pipefail + +SCHEMA_VERSION="1.0.0" + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib/gh-safe.sh +. "${SCRIPT_DIR}/lib/gh-safe.sh" +# shellcheck source=lib/filter-bots.sh +. "${SCRIPT_DIR}/lib/filter-bots.sh" +# shellcheck source=lib/compose-signals.sh +. "${SCRIPT_DIR}/lib/compose-signals.sh" +# shellcheck source=lib/date-utils.sh +. "${SCRIPT_DIR}/lib/date-utils.sh" + +main() { + if [ -z "${REPO:-}" ]; then + printf '[collect-signals] REPO env var is required\n' >&2 + return 64 + fi + if [ -z "${GH_TOKEN:-}" ]; then + printf '[collect-signals] GH_TOKEN env var is required\n' >&2 + return 64 + fi + + local issue_limit="${ISSUE_LIMIT:-50}" + local pr_limit="${PR_LIMIT:-30}" + local discussion_limit="${DISCUSSION_LIMIT:-100}" + local output_path="${SIGNALS_OUTPUT:-./signals.json}" + + local owner repo_name + owner="${REPO%%/*}" + repo_name="${REPO##*/}" + if [ "$owner" = "$REPO" ] || [ -z "$repo_name" ]; then + printf '[collect-signals] REPO must be in owner/name format, got: %s\n' "$REPO" >&2 + return 64 + fi + + local thirty_days_ago + thirty_days_ago=$(date_days_ago 30) + local scan_date + scan_date=$(date_now_iso) + + local truncation_warnings='[]' + + # --- Open issues ----------------------------------------------------------- + printf '[collect-signals] fetching open issues (limit=%s)\n' "$issue_limit" >&2 + local open_issues_raw + open_issues_raw=$(gh_safe_rest issue list --repo "$REPO" --state open --limit "$issue_limit" \ + --json number,title,labels,createdAt,author) + local open_issues + open_issues=$(printf '%s' "$open_issues_raw" | filter_bots_apply) + + if [ "$(printf '%s' "$open_issues" | jq 'length')" -ge "$issue_limit" ]; then + truncation_warnings=$(printf '%s' "$truncation_warnings" \ + | jq --arg src "open_issues" --argjson lim "$issue_limit" \ + '. + [{source: $src, limit: $lim, message: "result count equals limit; possible truncation"}]') + fi + + # --- Recently closed issues ------------------------------------------------ + printf '[collect-signals] fetching closed issues (since %s)\n' "$thirty_days_ago" >&2 + local closed_issues_raw + closed_issues_raw=$(gh_safe_rest issue list --repo "$REPO" --state closed --limit "$issue_limit" \ + --json number,title,labels,closedAt) + local closed_issues + closed_issues=$(printf '%s' "$closed_issues_raw" \ + | jq --arg cutoff "$thirty_days_ago" '[.[] | select(.closedAt >= $cutoff)]') + + # --- Ideas discussions (paginated, category-filtered) ---------------------- + printf '[collect-signals] resolving Ideas discussion category\n' >&2 + local categories_query + read -r -d '' categories_query <<'GRAPHQL' || true +query($repo: String!, $owner: String!) { + repository(name: $repo, owner: $owner) { + discussionCategories(first: 25) { + nodes { id name } + } + } +} +GRAPHQL + + local categories + categories=$(gh_safe_graphql -f query="$categories_query" \ + -f owner="$owner" -f repo="$repo_name" \ + --jq '.data.repository.discussionCategories.nodes') + + local ideas_cat_id + ideas_cat_id=$(printf '%s' "$categories" \ + | jq -r '[.[] | select(.name == "Ideas")][0].id // empty') + + local ideas_discussions='[]' + if [ -n "$ideas_cat_id" ]; then + printf '[collect-signals] fetching Ideas discussions (limit=%s)\n' "$discussion_limit" >&2 + local discussions_query + read -r -d '' discussions_query <<'GRAPHQL' || true +query($repo: String!, $owner: String!, $categoryId: ID!, $limit: Int!) { + repository(name: $repo, owner: $owner) { + discussions(first: $limit, orderBy: {field: UPDATED_AT, direction: DESC}, categoryId: $categoryId) { + pageInfo { hasNextPage } + nodes { + id + number + title + createdAt + updatedAt + labels(first: 10) { nodes { name } } + comments(first: 1) { totalCount } + } + } + } +} +GRAPHQL + + local discussions_full + discussions_full=$(gh_safe_graphql -f query="$discussions_query" \ + -f owner="$owner" -f repo="$repo_name" -f categoryId="$ideas_cat_id" \ + -F limit="$discussion_limit") + + ideas_discussions=$(printf '%s' "$discussions_full" \ + | jq -c '.data.repository.discussions.nodes // []') + + local has_next_page + has_next_page=$(printf '%s' "$discussions_full" \ + | jq -r '.data.repository.discussions.pageInfo.hasNextPage // false') + if [ "$has_next_page" = "true" ]; then + truncation_warnings=$(printf '%s' "$truncation_warnings" \ + | jq --arg src "ideas_discussions" --argjson lim "$discussion_limit" \ + '. + [{source: $src, limit: $lim, message: "hasNextPage=true; results truncated"}]') + fi + else + printf '[collect-signals] no "Ideas" category found, skipping discussions\n' >&2 + fi + + # --- Releases -------------------------------------------------------------- + printf '[collect-signals] fetching recent releases\n' >&2 + local releases + releases=$(gh_safe_rest release list --repo "$REPO" --limit 5 \ + --json tagName,name,publishedAt,isPrerelease) + + # --- Merged PRs (last 30 days) --------------------------------------------- + printf '[collect-signals] fetching merged PRs (limit=%s)\n' "$pr_limit" >&2 + local merged_prs_raw + merged_prs_raw=$(gh_safe_rest pr list --repo "$REPO" --state merged --limit "$pr_limit" \ + --json number,title,labels,mergedAt) + local merged_prs + merged_prs=$(printf '%s' "$merged_prs_raw" \ + | jq --arg cutoff "$thirty_days_ago" '[.[] | select(.mergedAt >= $cutoff)]') + + # --- Derived: feature requests + bug reports ------------------------------- + local feature_requests + feature_requests=$(printf '%s' "$open_issues" \ + | jq -c '[.[] | select(.labels | map(.name) | any(test("enhancement|feature|idea"; "i")))]') + local bug_reports + bug_reports=$(printf '%s' "$open_issues" \ + | jq -c '[.[] | select(.labels | map(.name) | any(test("bug"; "i")))]') + + # --- Compose --------------------------------------------------------------- + local signals + signals=$(compose_signals \ + "$open_issues" \ + "$closed_issues" \ + "$ideas_discussions" \ + "$releases" \ + "$merged_prs" \ + "$feature_requests" \ + "$bug_reports" \ + "$REPO" \ + "$scan_date" \ + "$SCHEMA_VERSION" \ + "$truncation_warnings") + + printf '%s\n' "$signals" >"$output_path" + printf '[collect-signals] wrote %s\n' "$output_path" >&2 + + # --- Step summary (only when running inside GitHub Actions) ---------------- + if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then + { + printf '## Signals Collected\n\n' + printf -- '- **Schema version:** %s\n' "$SCHEMA_VERSION" + printf -- '- **Open issues:** %s\n' "$(jq '.open_issues.count' "$output_path")" + printf -- '- **Feature requests:** %s\n' "$(jq '.feature_requests.count' "$output_path")" + printf -- '- **Bug reports:** %s\n' "$(jq '.bug_reports.count' "$output_path")" + printf -- '- **Merged PRs (30d):** %s\n' "$(jq '.merged_prs_30d.count' "$output_path")" + printf -- '- **Existing Ideas discussions:** %s\n' "$(jq '.ideas_discussions.count' "$output_path")" + local warn_count + warn_count=$(jq '.truncation_warnings | length' "$output_path") + if [ "$warn_count" -gt 0 ]; then + printf -- '- **⚠️ Truncation warnings:** %s\n' "$warn_count" + jq -r '.truncation_warnings[] | " - " + .source + " (limit " + (.limit|tostring) + "): " + .message' "$output_path" + fi + } >>"$GITHUB_STEP_SUMMARY" + fi +} + +# Allow `source`-ing for tests; only run main when executed directly. +if [ "${BASH_SOURCE[0]}" = "${0}" ]; then + main "$@" +fi diff --git a/.github/scripts/feature-ideation/discussion-mutations.sh b/.github/scripts/feature-ideation/discussion-mutations.sh new file mode 100755 index 0000000..843d9d6 --- /dev/null +++ b/.github/scripts/feature-ideation/discussion-mutations.sh @@ -0,0 +1,159 @@ +#!/usr/bin/env bash +# discussion-mutations.sh — wrappers around the GraphQL mutations Mary uses +# to create / comment on Ideas Discussions, with a DRY_RUN switch. +# +# Why this exists: +# The original prompt instructed Mary to call `gh api graphql` directly for +# createDiscussion / addDiscussionComment. There was no way to: +# - safely smoke-test the workflow on a sandbox repo +# - replay a run without writing to GitHub +# - audit what Mary was about to do before she did it +# +# With DRY_RUN=1, every mutation logs a structured "planned action" entry +# instead of executing. The plan log is the contract Mary's prompt now +# references. +# +# Env: +# DRY_RUN "1" → log instead of execute +# DRY_RUN_LOG path to JSONL log file (default: ./dry-run.jsonl) +# GH_TOKEN required when not in DRY_RUN +# +# Functions: +# create_discussion <body> +# comment_on_discussion <discussion_id> <body> +# add_label_to_discussion <discussion_id> <label_id> + +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib/gh-safe.sh +. "${SCRIPT_DIR}/lib/gh-safe.sh" + +_dry_run_log() { + local action_json="$1" + local log_path="${DRY_RUN_LOG:-./dry-run.jsonl}" + printf '%s\n' "$action_json" >>"$log_path" +} + +_is_dry_run() { + [ "${DRY_RUN:-0}" = "1" ] +} + +create_discussion() { + if [ "$#" -ne 4 ]; then + printf '[create_discussion] expected 4 args (repo_id category_id title body), got %d\n' "$#" >&2 + return 64 + fi + local repo_id="$1" + local category_id="$2" + local title="$3" + local body="$4" + + if _is_dry_run; then + local entry + entry=$(jq -nc \ + --arg op "create_discussion" \ + --arg repo_id "$repo_id" \ + --arg category_id "$category_id" \ + --arg title "$title" \ + --arg body "$body" \ + '{op: $op, repo_id: $repo_id, category_id: $category_id, title: $title, body: $body}') + _dry_run_log "$entry" + printf '%s' "$entry" + return 0 + fi + + local query + read -r -d '' query <<'GRAPHQL' || true +mutation($repoId: ID!, $categoryId: ID!, $title: String!, $body: String!) { + createDiscussion(input: { + repositoryId: $repoId, + categoryId: $categoryId, + title: $title, + body: $body + }) { + discussion { id number url } + } +} +GRAPHQL + + gh_safe_graphql -f query="$query" \ + -f repoId="$repo_id" \ + -f categoryId="$category_id" \ + -f title="$title" \ + -f body="$body" +} + +comment_on_discussion() { + if [ "$#" -ne 2 ]; then + printf '[comment_on_discussion] expected 2 args (discussion_id body), got %d\n' "$#" >&2 + return 64 + fi + local discussion_id="$1" + local body="$2" + + if _is_dry_run; then + local entry + entry=$(jq -nc \ + --arg op "comment_on_discussion" \ + --arg discussion_id "$discussion_id" \ + --arg body "$body" \ + '{op: $op, discussion_id: $discussion_id, body: $body}') + _dry_run_log "$entry" + printf '%s' "$entry" + return 0 + fi + + local query + read -r -d '' query <<'GRAPHQL' || true +mutation($discussionId: ID!, $body: String!) { + addDiscussionComment(input: { + discussionId: $discussionId, + body: $body + }) { + comment { id url } + } +} +GRAPHQL + + gh_safe_graphql -f query="$query" \ + -f discussionId="$discussion_id" \ + -f body="$body" +} + +add_label_to_discussion() { + if [ "$#" -ne 2 ]; then + printf '[add_label_to_discussion] expected 2 args (discussion_id label_id), got %d\n' "$#" >&2 + return 64 + fi + local discussion_id="$1" + local label_id="$2" + + if _is_dry_run; then + local entry + entry=$(jq -nc \ + --arg op "add_label_to_discussion" \ + --arg discussion_id "$discussion_id" \ + --arg label_id "$label_id" \ + '{op: $op, discussion_id: $discussion_id, label_id: $label_id}') + _dry_run_log "$entry" + printf '%s' "$entry" + return 0 + fi + + local query + read -r -d '' query <<'GRAPHQL' || true +mutation($labelableId: ID!, $labelIds: [ID!]!) { + addLabelsToLabelable(input: { + labelableId: $labelableId, + labelIds: $labelIds + }) { + clientMutationId + } +} +GRAPHQL + + gh_safe_graphql -f query="$query" \ + -f labelableId="$discussion_id" \ + -f labelIds="[\"$label_id\"]" +} diff --git a/.github/scripts/feature-ideation/lib/compose-signals.sh b/.github/scripts/feature-ideation/lib/compose-signals.sh new file mode 100755 index 0000000..6a0f5bd --- /dev/null +++ b/.github/scripts/feature-ideation/lib/compose-signals.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# compose-signals.sh — assemble the canonical signals.json document. +# +# Why this exists: +# The original `jq -n --argjson` block crashed if any input variable was an +# empty string. By centralizing composition here, we (a) validate every input +# is JSON before passing to --argjson, (b) emit a stable shape regardless of +# which sub-queries returned empty, and (c) keep the schema source-of-truth +# in one place. +# +# Inputs (all JSON arrays, never empty strings): +# $1 open_issues +# $2 closed_issues +# $3 ideas_discussions +# $4 releases +# $5 merged_prs +# $6 feature_requests +# $7 bug_reports +# $8 repo (string, e.g. "petry-projects/talkterm") +# $9 scan_date (ISO-8601 string) +# $10 schema_version (string) +# $11 truncation_warnings (JSON array, may be []) +# +# Output: signals.json document on stdout. + +set -euo pipefail + +compose_signals() { + if [ "$#" -ne 11 ]; then + printf '[compose-signals] expected 11 args, got %d\n' "$#" >&2 + return 64 # EX_USAGE + fi + + local open_issues="$1" + local closed_issues="$2" + local ideas_discussions="$3" + local releases="$4" + local merged_prs="$5" + local feature_requests="$6" + local bug_reports="$7" + local repo="$8" + local scan_date="$9" + local schema_version="${10}" + local truncation_warnings="${11}" + + # Validate every JSON input before composition. Better to fail loudly here + # than to let `jq --argjson` produce a cryptic parse error. + local idx=0 + for input in "$open_issues" "$closed_issues" "$ideas_discussions" "$releases" \ + "$merged_prs" "$feature_requests" "$bug_reports" "$truncation_warnings"; do + idx=$((idx + 1)) + if ! printf '%s' "$input" | jq -e . >/dev/null 2>&1; then + printf '[compose-signals] arg #%d is not valid JSON: %s\n' "$idx" "${input:0:120}" >&2 + return 65 # EX_DATAERR + fi + done + + jq -n \ + --arg scan_date "$scan_date" \ + --arg repo "$repo" \ + --arg schema_version "$schema_version" \ + --argjson open_issues "$open_issues" \ + --argjson closed_issues "$closed_issues" \ + --argjson ideas_discussions "$ideas_discussions" \ + --argjson releases "$releases" \ + --argjson merged_prs "$merged_prs" \ + --argjson feature_requests "$feature_requests" \ + --argjson bug_reports "$bug_reports" \ + --argjson truncation_warnings "$truncation_warnings" \ + '{ + schema_version: $schema_version, + scan_date: $scan_date, + repo: $repo, + open_issues: { count: ($open_issues | length), items: $open_issues }, + closed_issues_30d: { count: ($closed_issues | length), items: $closed_issues }, + ideas_discussions: { count: ($ideas_discussions | length), items: $ideas_discussions }, + releases: $releases, + merged_prs_30d: { count: ($merged_prs | length), items: $merged_prs }, + feature_requests: { count: ($feature_requests | length), items: $feature_requests }, + bug_reports: { count: ($bug_reports | length), items: $bug_reports }, + truncation_warnings: $truncation_warnings + }' +} diff --git a/.github/scripts/feature-ideation/lib/date-utils.sh b/.github/scripts/feature-ideation/lib/date-utils.sh new file mode 100755 index 0000000..8c4b5a1 --- /dev/null +++ b/.github/scripts/feature-ideation/lib/date-utils.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# date-utils.sh — cross-platform date arithmetic helpers. +# +# Why this exists: +# `date -u -d '30 days ago'` is GNU; `date -u -v-30d` is BSD. The original +# workflow had both forms separated by `||`, but only one branch ever ran +# on `ubuntu-latest`. Centralizing here gives us one tested helper. + +set -euo pipefail + +# Print an ISO date (YYYY-MM-DD) for N days ago in UTC. +date_days_ago() { + local days="$1" + if [ -z "$days" ] || ! printf '%s' "$days" | grep -Eq '^[0-9]+$'; then + printf '[date-utils] days must be a non-negative integer, got: %s\n' "$days" >&2 + return 64 + fi + if date -u -d "${days} days ago" +%Y-%m-%d >/dev/null 2>&1; then + date -u -d "${days} days ago" +%Y-%m-%d + elif date -u -v-"${days}"d +%Y-%m-%d >/dev/null 2>&1; then + date -u -v-"${days}"d +%Y-%m-%d + else + printf '[date-utils] no supported date(1) variant available\n' >&2 + return 69 # EX_UNAVAILABLE + fi +} + +# Print the current UTC timestamp in ISO-8601 (Zulu). +date_now_iso() { + date -u +%Y-%m-%dT%H:%M:%SZ +} + +# Print today's date in UTC (YYYY-MM-DD). +date_today() { + date -u +%Y-%m-%d +} diff --git a/.github/scripts/feature-ideation/lib/filter-bots.sh b/.github/scripts/feature-ideation/lib/filter-bots.sh new file mode 100755 index 0000000..95aabfe --- /dev/null +++ b/.github/scripts/feature-ideation/lib/filter-bots.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# filter-bots.sh — strip bot-authored issues/PRs from a JSON array. +# +# Why this exists: +# The original gather-signals filter only excluded `dependabot[bot]` and +# `github-actions[bot]`. New bots (renovate, copilot, coderabbit, claude) +# pollute the signals payload and crowd out real user feedback. +# +# Configurable via FEATURE_IDEATION_BOT_AUTHORS (comma-separated, optional). +# Defaults to a sensible allowlist of known automation accounts. + +set -euo pipefail + +# Default bot author logins. Override via env to add project-specific bots. +DEFAULT_BOT_AUTHORS=( + "dependabot[bot]" + "github-actions[bot]" + "renovate[bot]" + "copilot[bot]" + "coderabbitai[bot]" + "coderabbit[bot]" + "claude[bot]" + "claude-bot[bot]" + "sonarcloud[bot]" + "sonarqubecloud[bot]" + "codeql[bot]" + "snyk-bot" + "imgbot[bot]" + "allcontributors[bot]" +) + +# Build the active bot list from defaults + env override. +filter_bots_build_list() { + local list=("${DEFAULT_BOT_AUTHORS[@]}") + if [ -n "${FEATURE_IDEATION_BOT_AUTHORS:-}" ]; then + local IFS=',' + # shellcheck disable=SC2206 + local extras=($FEATURE_IDEATION_BOT_AUTHORS) + list+=("${extras[@]}") + fi + printf '%s\n' "${list[@]}" | jq -R . | jq -sc . +} + +# Filter a JSON array of items, removing any whose .author.login is in the bot list. +# Reads JSON from stdin, writes filtered JSON to stdout. +filter_bots_apply() { + local bot_list + bot_list=$(filter_bots_build_list) + jq --argjson bots "$bot_list" '[.[] | select((.author.login // "") as $a | ($bots | index($a)) | not)]' +} diff --git a/.github/scripts/feature-ideation/lib/gh-safe.sh b/.github/scripts/feature-ideation/lib/gh-safe.sh new file mode 100755 index 0000000..7bbdf2c --- /dev/null +++ b/.github/scripts/feature-ideation/lib/gh-safe.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# gh-safe.sh — defensive wrapper around `gh` calls used by feature-ideation. +# +# Why this exists: +# The original workflow used the `2>/dev/null || echo '[]'` pattern, which +# silently swallows ALL errors — auth failures, rate limits, network outages, +# GraphQL schema drift — and continues with empty data. The pipeline would +# "succeed" while producing useless signals. +# +# Contract: +# - On a documented empty result (200 OK + []) → exit 0, stdout is the empty array +# - On any infrastructure error → exit non-zero, stderr explains why +# - On a GraphQL response containing an "errors" field → exit non-zero +# - On a GraphQL response with `data: null` → exit non-zero +# +# Usage: +# source .github/scripts/feature-ideation/lib/gh-safe.sh +# ISSUES=$(gh_safe_rest issue list --repo "$REPO" --state open --limit 50 \ +# --json number,title,labels) +# DISCUSSIONS=$(gh_safe_graphql "$query" -f owner=foo -f repo=bar \ +# --jq '.data.repository.discussions.nodes') + +set -euo pipefail + +# Marker emitted by callers when they want to default to "[]" — explicit, not silent. +GH_SAFE_EMPTY_ARRAY='[]' + +# Internal: emit a structured error to stderr. +_gh_safe_err() { + local code="$1" + shift + printf '[gh-safe][%s] %s\n' "$code" "$*" >&2 +} + +# Validate that a string is well-formed JSON. Returns 0 if valid. +gh_safe_is_json() { + local input="$1" + [ -n "$input" ] || return 1 + printf '%s' "$input" | jq -e . >/dev/null 2>&1 +} + +# Run a `gh` REST/CLI call (gh issue list, gh pr list, gh release list, etc.). +# Captures stdout, stderr, and exit code separately so we can distinguish: +# - empty result (success, output is "[]" or empty) +# - hard failure (non-zero exit) — never silently downgraded +# +# Arguments are passed verbatim to `gh`. +gh_safe_rest() { + local stdout stderr rc tmp_err + tmp_err=$(mktemp) + # shellcheck disable=SC2034 + set +e + stdout=$(gh "$@" 2>"$tmp_err") + rc=$? + set -e + stderr=$(cat "$tmp_err") + rm -f "$tmp_err" + + if [ "$rc" -ne 0 ]; then + _gh_safe_err "rest-failure" "exit=$rc args=$* stderr=$stderr" + return "$rc" + fi + + # Empty stdout from a successful gh call means the result set is empty. + # Normalize to an empty JSON array so downstream jq composition never sees "". + if [ -z "$stdout" ]; then + printf '%s' "$GH_SAFE_EMPTY_ARRAY" + return 0 + fi + + # Validate JSON shape — if gh ever returns non-JSON on success, fail loud. + if ! gh_safe_is_json "$stdout"; then + _gh_safe_err "rest-bad-json" "args=$* stdout (first 200 bytes)=${stdout:0:200}" + return 65 # EX_DATAERR + fi + + printf '%s' "$stdout" +} + +# Run a `gh api graphql` call. Same defensive contract as gh_safe_rest, plus: +# - Reject responses where the parsed result is the literal "null" +# - Reject responses with a top-level "errors" field (only meaningful when +# called WITHOUT --jq, i.e. when callers ask for the full response) +# +# When --jq is supplied, we cannot inspect the full response for an errors[] +# field, so we additionally call gh once WITHOUT --jq to validate the envelope. +# This costs an extra round-trip; for the feature-ideation workflow's volume +# (a handful of calls per run) this is the right trade-off. +gh_safe_graphql() { + local args=("$@") + local has_jq=0 + local jq_filter="" + local i=0 + while [ "$i" -lt "${#args[@]}" ]; do + if [ "${args[$i]}" = "--jq" ]; then + has_jq=1 + jq_filter="${args[$((i + 1))]}" + break + fi + i=$((i + 1)) + done + + # Build a no-jq variant for envelope validation. + local raw_args=() + i=0 + while [ "$i" -lt "${#args[@]}" ]; do + if [ "${args[$i]}" = "--jq" ]; then + i=$((i + 2)) + continue + fi + raw_args+=("${args[$i]}") + i=$((i + 1)) + done + + local raw stderr rc tmp_err + tmp_err=$(mktemp) + set +e + raw=$(gh api graphql "${raw_args[@]}" 2>"$tmp_err") + rc=$? + set -e + stderr=$(cat "$tmp_err") + rm -f "$tmp_err" + + if [ "$rc" -ne 0 ]; then + _gh_safe_err "graphql-failure" "exit=$rc stderr=$stderr" + return "$rc" + fi + + if ! gh_safe_is_json "$raw"; then + _gh_safe_err "graphql-bad-json" "first 200 bytes: ${raw:0:200}" + return 65 + fi + + # Reject error envelopes — GraphQL returns 200 OK even on partial errors. + if printf '%s' "$raw" | jq -e '.errors // empty | if . then true else false end' >/dev/null 2>&1; then + if printf '%s' "$raw" | jq -e '(.errors | type) == "array" and (.errors | length > 0)' >/dev/null 2>&1; then + local errs + errs=$(printf '%s' "$raw" | jq -c '.errors') + _gh_safe_err "graphql-errors" "$errs" + return 66 # EX_NOINPUT — repurposed: "GraphQL refused our request" + fi + fi + + # Reject `data: null` — usually means the path didn't resolve (permissions, + # missing field, renamed repo). + if printf '%s' "$raw" | jq -e '.data == null' >/dev/null 2>&1; then + _gh_safe_err "graphql-null-data" "data field is null" + return 66 + fi + + # If caller asked for a jq filter, apply it now and return that. + if [ "$has_jq" -eq 1 ]; then + local filtered + filtered=$(printf '%s' "$raw" | jq -c "$jq_filter" 2>/dev/null || true) + if [ -z "$filtered" ] || [ "$filtered" = "null" ]; then + # The filter resolved to null/empty — caller probably wants "[]" semantics + # for "no nodes found". Return the empty array sentinel. + printf '%s' "$GH_SAFE_EMPTY_ARRAY" + return 0 + fi + printf '%s' "$filtered" + return 0 + fi + + printf '%s' "$raw" +} diff --git a/.github/scripts/feature-ideation/lint-prompt.sh b/.github/scripts/feature-ideation/lint-prompt.sh new file mode 100755 index 0000000..8381b61 --- /dev/null +++ b/.github/scripts/feature-ideation/lint-prompt.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# lint-prompt.sh — guard against unescaped shell expansions in claude-code-action +# `direct_prompt:` blocks. +# +# Why this exists: +# The original feature-ideation.yml contained: +# Date: $(date -u +%Y-%m-%d) +# inside the `direct_prompt:` heredoc. YAML does NOT expand shell, and +# claude-code-action passes the prompt verbatim — Mary received the literal +# string `$(date -u +%Y-%m-%d)` instead of an actual date. This is R2. +# +# This linter scans every workflow file under .github/workflows/ for +# `direct_prompt:` blocks and flags any unescaped `$(...)` or `${VAR}` that +# YAML/the action will not interpolate. ${{ ... }} (GitHub expression syntax) +# is allowed because GitHub Actions evaluates it before the prompt is sent. +# +# Usage: +# lint-prompt.sh [<workflow.yml> ...] +# +# Exit codes: +# 0 no issues +# 1 one or more findings +# 2 bad usage / file error + +set -euo pipefail + +scan_file() { + local file="$1" + + python3 - "$file" <<'PY' +import re +import sys + +path = sys.argv[1] +try: + with open(path, "r", encoding="utf-8") as f: + lines = f.readlines() +except OSError as exc: + sys.stderr.write(f"[lint-prompt] cannot read {path}: {exc}\n") + sys.exit(2) + +# Find direct_prompt: blocks. We treat everything indented MORE than the +# `direct_prompt:` line as part of that block, until we hit a less-indented +# non-blank line. +in_block = False +block_indent = -1 +findings = [] + +# Pattern matches $(...) and ${VAR} but NOT GitHub Actions ${{ ... }} +# because that's evaluated before the prompt is rendered. +shell_expansion = re.compile(r'(?<!\\)\$\([^)]*\)|(?<!\$)\$\{[A-Za-z_][A-Za-z0-9_]*\}') + +for lineno, raw in enumerate(lines, start=1): + stripped = raw.lstrip(" ") + indent = len(raw) - len(stripped) + + if not in_block: + # Look for `direct_prompt:` or `direct_prompt: |` or `direct_prompt: >` + if re.match(r'direct_prompt:\s*[|>]?\s*$', stripped): + in_block = True + block_indent = indent + continue + else: + # Blank lines stay in the block. + if stripped.strip() == "": + continue + # If we drop back to or below the marker indent, the block ended. + if indent <= block_indent: + in_block = False + block_indent = -1 + continue + + # We're inside the prompt body. Scan for shell expansions. + # First, strip out any GitHub Actions expressions so they don't trip us. + no_gh = re.sub(r'\$\{\{[^}]*\}\}', '', raw) + for match in shell_expansion.finditer(no_gh): + findings.append((lineno, match.group(0), raw.rstrip())) + +if findings: + sys.stderr.write(f"[lint-prompt] {len(findings)} unescaped shell expansion(s) in {path}:\n") + for lineno, expr, line in findings: + sys.stderr.write(f" line {lineno}: {expr}\n") + sys.stderr.write(f" {line}\n") + sys.exit(1) +sys.exit(0) +PY + return $? +} + +main() { + if [ "$#" -eq 0 ]; then + # Default: scan every workflow file. + local repo_root + repo_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../../.." && pwd)" + local files=() + while IFS= read -r f; do + files+=("$f") + done < <(find "${repo_root}/.github/workflows" -type f \( -name '*.yml' -o -name '*.yaml' \)) + set -- "${files[@]}" + fi + + local exit=0 + for file in "$@"; do + if [ ! -f "$file" ]; then + printf '[lint-prompt] not found: %s\n' "$file" >&2 + exit=2 + continue + fi + if ! scan_file "$file"; then + exit=1 + fi + done + return "$exit" +} + +main "$@" diff --git a/.github/scripts/feature-ideation/match-discussions.sh b/.github/scripts/feature-ideation/match-discussions.sh new file mode 100755 index 0000000..826f29b --- /dev/null +++ b/.github/scripts/feature-ideation/match-discussions.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# match-discussions.sh — deterministic matching between Mary's proposed ideas +# and existing Ideas Discussions, replacing the prose "use fuzzy matching" +# instruction in the original prompt. +# +# Why this exists: +# The original Phase-5 instruction told Mary to "use fuzzy matching" against +# existing Discussion titles in her head. There is no way to test, replay, +# or audit that. Two runs could create duplicate Discussions for the same +# idea with slightly different titles. This is R5 + R6. +# +# Approach: +# 1. Normalize titles to a canonical form (strip emoji, punctuation, +# lowercase, collapse whitespace, drop common stopwords). +# 2. Tokenize and compute Jaccard similarity over token sets. +# 3. Match if similarity >= MATCH_THRESHOLD (default 0.6). +# +# Inputs: +# $1 Path to signals.json (must contain .ideas_discussions.items) +# $2 Path to proposals.json — array of { title, summary, ... } +# +# Output (stdout, JSON): +# { +# "matched": [ { "proposal": {...}, "discussion": {...}, "similarity": 0.83 } ], +# "new_candidates": [ { "proposal": {...} } ], +# "threshold": 0.6 +# } +# +# Env: +# MATCH_THRESHOLD Override default Jaccard similarity threshold. + +set -euo pipefail + +normalize_title() { + python3 -c ' +import re, sys, unicodedata +raw = sys.argv[1] +# Strip emoji and other symbol/other categories. +no_emoji = "".join(ch for ch in raw if unicodedata.category(ch)[0] not in ("S",)) +# Lowercase + ascii fold. +ascii_text = unicodedata.normalize("NFKD", no_emoji).encode("ascii", "ignore").decode() +ascii_text = ascii_text.lower() +# Replace non-alphanumeric with space. +collapsed = re.sub(r"[^a-z0-9]+", " ", ascii_text).strip() +# Drop stopwords. +stopwords = {"a","an","the","of","for","to","and","or","with","in","on","by","via","as","is","are","be","support","add","new","feature","idea"} +tokens = [t for t in collapsed.split() if t and t not in stopwords] +print(" ".join(tokens)) +' "$1" +} +export -f normalize_title + +# Compute Jaccard similarity between two normalized token strings. +jaccard_similarity() { + python3 -c ' +import sys +a = set(sys.argv[1].split()) +b = set(sys.argv[2].split()) +if not a and not b: + print("1.0") +elif not a or not b: + print("0.0") +else: + inter = len(a & b) + union = len(a | b) + print(f"{inter / union:.4f}") +' "$1" "$2" +} +export -f jaccard_similarity + +match_discussions_main() { + local signals_path="$1" + local proposals_path="$2" + local threshold="${MATCH_THRESHOLD:-0.6}" + + if [ ! -f "$signals_path" ]; then + printf '[match-discussions] signals not found: %s\n' "$signals_path" >&2 + return 64 + fi + if [ ! -f "$proposals_path" ]; then + printf '[match-discussions] proposals not found: %s\n' "$proposals_path" >&2 + return 64 + fi + + python3 - "$signals_path" "$proposals_path" "$threshold" <<'PY' +import json +import re +import sys +import unicodedata + +signals_path, proposals_path, threshold_str = sys.argv[1:4] +threshold = float(threshold_str) + +STOPWORDS = { + "a", "an", "the", "of", "for", "to", "and", "or", "with", "in", "on", + "by", "via", "as", "is", "are", "be", "support", "add", "new", + "feature", "idea", +} + + +def normalize(title: str) -> set[str]: + no_sym = "".join(ch for ch in title if unicodedata.category(ch)[0] != "S") + folded = ( + unicodedata.normalize("NFKD", no_sym).encode("ascii", "ignore").decode() + ) + folded = folded.lower() + cleaned = re.sub(r"[^a-z0-9]+", " ", folded).strip() + return {t for t in cleaned.split() if t and t not in STOPWORDS} + + +def jaccard(a: set[str], b: set[str]) -> float: + if not a and not b: + return 1.0 + if not a or not b: + return 0.0 + return len(a & b) / len(a | b) + + +with open(signals_path) as f: + signals = json.load(f) +with open(proposals_path) as f: + proposals = json.load(f) + +if not isinstance(proposals, list): + sys.stderr.write("[match-discussions] proposals must be a JSON array\n") + sys.exit(65) + +discussions = signals.get("ideas_discussions", {}).get("items", []) or [] +disc_norm = [(d, normalize(d.get("title", ""))) for d in discussions] + +matched = [] +new_candidates = [] +seen_disc_ids = set() + +for proposal in proposals: + if not isinstance(proposal, dict) or "title" not in proposal: + sys.stderr.write(f"[match-discussions] skipping malformed proposal: {proposal!r}\n") + continue + p_norm = normalize(proposal["title"]) + + best = None + best_sim = 0.0 + for disc, d_norm in disc_norm: + if disc.get("id") in seen_disc_ids: + continue + sim = jaccard(p_norm, d_norm) + if sim > best_sim: + best_sim = sim + best = disc + + if best is not None and best_sim >= threshold: + matched.append( + { + "proposal": proposal, + "discussion": best, + "similarity": round(best_sim, 4), + } + ) + seen_disc_ids.add(best.get("id")) + else: + new_candidates.append({"proposal": proposal, "best_similarity": round(best_sim, 4)}) + +result = { + "matched": matched, + "new_candidates": new_candidates, + "threshold": threshold, +} +print(json.dumps(result, indent=2)) +PY +} + +if [ "${BASH_SOURCE[0]}" = "${0}" ]; then + if [ "$#" -ne 2 ]; then + printf 'usage: %s <signals.json> <proposals.json>\n' "$0" >&2 + exit 64 + fi + match_discussions_main "$1" "$2" +fi diff --git a/.github/scripts/feature-ideation/validate-signals.py b/.github/scripts/feature-ideation/validate-signals.py new file mode 100755 index 0000000..b71a1bd --- /dev/null +++ b/.github/scripts/feature-ideation/validate-signals.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +"""Validate a signals.json file against signals.schema.json. + +Usage: + validate-signals.py <signals.json> [<schema.json>] + +Exit codes: + 0 valid + 1 invalid (validation error printed to stderr) + 2 usage / file error +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +try: + from jsonschema import Draft202012Validator +except ImportError: + sys.stderr.write( + "[validate-signals] python jsonschema not installed. " + "Install with: pip install jsonschema\n" + ) + sys.exit(2) + + +def main(argv: list[str]) -> int: + if len(argv) < 2 or len(argv) > 3: + sys.stderr.write( + "usage: validate-signals.py <signals.json> [<schema.json>]\n" + ) + return 2 + + signals_path = Path(argv[1]) + if len(argv) == 3: + schema_path = Path(argv[2]) + else: + schema_path = ( + Path(__file__).resolve().parent.parent.parent + / "schemas" + / "signals.schema.json" + ) + + if not signals_path.exists(): + sys.stderr.write(f"[validate-signals] not found: {signals_path}\n") + return 2 + if not schema_path.exists(): + sys.stderr.write(f"[validate-signals] schema not found: {schema_path}\n") + return 2 + + try: + signals = json.loads(signals_path.read_text()) + except json.JSONDecodeError as exc: + sys.stderr.write(f"[validate-signals] invalid JSON in {signals_path}: {exc}\n") + return 1 + + try: + schema = json.loads(schema_path.read_text()) + except json.JSONDecodeError as exc: + sys.stderr.write(f"[validate-signals] invalid schema JSON: {exc}\n") + return 2 + + validator = Draft202012Validator(schema) + errors = sorted(validator.iter_errors(signals), key=lambda e: list(e.absolute_path)) + if not errors: + print(f"[validate-signals] OK: {signals_path}") + return 0 + + sys.stderr.write(f"[validate-signals] {len(errors)} validation error(s) in {signals_path}:\n") + for err in errors: + path = "/".join(str(p) for p in err.absolute_path) or "<root>" + sys.stderr.write(f" - {path}: {err.message}\n") + return 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/.github/workflows/feature-ideation-reusable.yml b/.github/workflows/feature-ideation-reusable.yml index 9c70cbe..6c49a7a 100644 --- a/.github/workflows/feature-ideation-reusable.yml +++ b/.github/workflows/feature-ideation-reusable.yml @@ -5,6 +5,18 @@ # project_context — see standards/workflows/feature-ideation.yml for the # template stub and ci-standards.md §8 for the full standard. # +# Architecture: +# - All bash / jq / GraphQL parsing logic lives in +# .github/scripts/feature-ideation/ in THIS repo and is unit-tested +# via bats (see test/workflows/feature-ideation/). +# - The signals.json contract is pinned by .github/schemas/signals.schema.json +# and validated in CI before being handed to Mary. +# - The direct prompt block is linted by lint-prompt.sh to prevent the +# class of "$(date) inside YAML" bugs (R2). +# - DRY_RUN mode lets adopters smoke-test on a sandbox repo without +# writing real Discussions — every mutation is logged to a JSONL file +# and uploaded as an artifact. +# # Why a reusable workflow: # - The 5-phase ideation pipeline (Market Research → Brainstorming → Party # Mode → Adversarial → Publish) is universal. Tuning the pattern in one @@ -45,6 +57,16 @@ on: required: false default: 60 type: number + dry_run: + description: 'Skip Discussion mutations and log them to a JSONL artifact instead.' + required: false + default: false + type: boolean + tooling_ref: + description: 'Ref of petry-projects/.github to source the feature-ideation scripts from. Defaults to main; override only for testing forks.' + required: false + default: 'main' + type: string secrets: CLAUDE_CODE_OAUTH_TOKEN: description: 'Claude Code OAuth token (org-level secret)' @@ -66,129 +88,36 @@ jobs: env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - name: Checkout repository + - name: Checkout calling repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + fetch-depth: 1 + + - name: Checkout feature-ideation tooling uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: + repository: petry-projects/.github + ref: ${{ inputs.tooling_ref }} + path: .feature-ideation-tooling fetch-depth: 1 + - name: Install Python jsonschema + run: pip install --quiet 'jsonschema>=4' + - name: Collect project signals + env: + REPO: ${{ github.repository }} + SIGNALS_OUTPUT: ${{ runner.temp }}/signals.json run: | set -euo pipefail - REPO="${{ github.repository }}" - SIGNALS_FILE=$(mktemp) - - # --- Open issues (exclude bot-generated, omit body to stay within output limits) --- - OPEN_ISSUES=$(gh issue list --repo "$REPO" --state open --limit 50 \ - --json number,title,labels,createdAt,author \ - -q '[.[] | select(.author.login != "dependabot[bot]" and .author.login != "github-actions[bot]")]' \ - 2>/dev/null || echo '[]') - - # --- Recently closed issues (last 30 days) --- - THIRTY_DAYS_AGO=$(date -u -d '30 days ago' +%Y-%m-%d 2>/dev/null || date -u -v-30d +%Y-%m-%d) - CLOSED_ISSUES=$(gh issue list --repo "$REPO" --state closed --limit 30 \ - --json number,title,labels,closedAt \ - -q "[.[] | select(.closedAt >= \"${THIRTY_DAYS_AGO}\")]" \ - 2>/dev/null || echo '[]') - - # --- Existing Ideas discussions (to detect what's already tracked) --- - # NOTE: GraphQL queries use heredocs with quoted terminators so the - # `$repo` / `$owner` / `$categoryId` references stay literal (they - # are GraphQL variables, not shell expansions). - IDEAS_CAT_QUERY=$(cat <<'GRAPHQL' - query($repo: String!, $owner: String!) { - repository(name: $repo, owner: $owner) { - discussionCategories(first: 20) { - nodes { id name } - } - } - } - GRAPHQL - ) - IDEAS_CAT_ID=$(gh api graphql -f query="$IDEAS_CAT_QUERY" \ - -f owner="${{ github.repository_owner }}" \ - -f repo="${{ github.event.repository.name }}" \ - --jq '[.data.repository.discussionCategories.nodes[] | select(.name == "Ideas")][0].id' \ - 2>/dev/null || echo "") - - if [ -n "$IDEAS_CAT_ID" ] && [ "$IDEAS_CAT_ID" != "null" ]; then - IDEAS_DISC_QUERY=$(cat <<'GRAPHQL' - query($repo: String!, $owner: String!, $categoryId: ID!) { - repository(name: $repo, owner: $owner) { - discussions(first: 100, orderBy: {field: UPDATED_AT, direction: DESC}, - categoryId: $categoryId) { - nodes { - id number title - createdAt updatedAt - labels(first: 10) { nodes { name } } - comments(first: 1) { totalCount } - } - } - } - } - GRAPHQL - ) - IDEAS_DISCUSSIONS=$(gh api graphql -f query="$IDEAS_DISC_QUERY" \ - -f owner="${{ github.repository_owner }}" \ - -f repo="${{ github.event.repository.name }}" \ - -f categoryId="$IDEAS_CAT_ID" \ - --jq '.data.repository.discussions.nodes' \ - 2>/dev/null || echo '[]') - else - IDEAS_DISCUSSIONS='[]' - fi + bash .feature-ideation-tooling/.github/scripts/feature-ideation/collect-signals.sh - # --- Recent releases --- - RELEASES=$(gh release list --repo "$REPO" --limit 5 \ - --json tagName,name,publishedAt,isPrerelease \ - 2>/dev/null || echo '[]') - - # --- PR activity (merged last 30 days) --- - MERGED_PRS=$(gh pr list --repo "$REPO" --state merged --limit 30 \ - --json number,title,labels,mergedAt \ - -q "[.[] | select(.mergedAt >= \"${THIRTY_DAYS_AGO}\")]" \ - 2>/dev/null || echo '[]') - - # --- Feature requests / bug reports --- - FEATURE_REQUESTS=$(echo "$OPEN_ISSUES" | jq -c \ - '[.[] | select(.labels | map(.name) | any(test("enhancement|feature|idea"; "i")))]') - BUG_REPORTS=$(echo "$OPEN_ISSUES" | jq -c \ - '[.[] | select(.labels | map(.name) | any(test("bug"; "i")))]') - - jq -n \ - --arg scan_date "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg repo "$REPO" \ - --argjson open_issues "$OPEN_ISSUES" \ - --argjson closed_issues "$CLOSED_ISSUES" \ - --argjson ideas_discussions "$IDEAS_DISCUSSIONS" \ - --argjson releases "$RELEASES" \ - --argjson merged_prs "$MERGED_PRS" \ - --argjson feature_requests "$FEATURE_REQUESTS" \ - --argjson bug_reports "$BUG_REPORTS" \ - '{ - scan_date: $scan_date, - repo: $repo, - open_issues: { count: ($open_issues | length), items: $open_issues }, - closed_issues_30d: { count: ($closed_issues | length), items: $closed_issues }, - ideas_discussions: { count: ($ideas_discussions | length), items: $ideas_discussions }, - releases: $releases, - merged_prs_30d: { count: ($merged_prs | length), items: $merged_prs }, - feature_requests: { count: ($feature_requests | length), items: $feature_requests }, - bug_reports: { count: ($bug_reports | length), items: $bug_reports } - }' > "$SIGNALS_FILE" - - cp "$SIGNALS_FILE" "${{ runner.temp }}/signals.json" - - { - echo "## Signals Collected" - echo "" - echo "- **Open issues:** $(jq '.open_issues.count' "$SIGNALS_FILE")" - echo "- **Feature requests:** $(jq '.feature_requests.count' "$SIGNALS_FILE")" - echo "- **Bug reports:** $(jq '.bug_reports.count' "$SIGNALS_FILE")" - echo "- **Merged PRs (30d):** $(jq '.merged_prs_30d.count' "$SIGNALS_FILE")" - echo "- **Existing Ideas discussions:** $(jq '.ideas_discussions.count' "$SIGNALS_FILE")" - } >> "$GITHUB_STEP_SUMMARY" - - rm -f "$SIGNALS_FILE" + - name: Validate signals.json against schema + run: | + set -euo pipefail + python3 .feature-ideation-tooling/.github/scripts/feature-ideation/validate-signals.py \ + "${{ runner.temp }}/signals.json" \ + .feature-ideation-tooling/.github/schemas/signals.schema.json - name: Upload signals artifact uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 @@ -210,9 +139,17 @@ jobs: discussions: write id-token: write steps: - - name: Checkout repository + - name: Checkout calling repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + fetch-depth: 1 + + - name: Checkout feature-ideation tooling uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: + repository: petry-projects/.github + ref: ${{ inputs.tooling_ref }} + path: .feature-ideation-tooling fetch-depth: 1 - name: Download signals artifact @@ -246,6 +183,14 @@ jobs: - name: Run Claude Code — BMAD Analyst env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + DRY_RUN: ${{ inputs.dry_run && '1' || '0' }} + DRY_RUN_LOG: ${{ runner.temp }}/dry-run.jsonl + SIGNALS_PATH: ${{ runner.temp }}/signals.json + PROPOSALS_PATH: ${{ runner.temp }}/proposals.json + MATCH_PLAN_PATH: ${{ runner.temp }}/match-plan.json + TOOLING_DIR: ${{ github.workspace }}/.feature-ideation-tooling/.github/scripts/feature-ideation + FOCUS_AREA: ${{ inputs.focus_area || '' }} + RESEARCH_DEPTH: ${{ inputs.research_depth }} uses: anthropics/claude-code-action@6e2bd52842c65e914eba5c8badd17560bd26b5de # v1.0.89 with: claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} @@ -254,14 +199,12 @@ jobs: # this, the action uses its own GitHub App token (claude[bot]) which # lacks discussions:write and silently fails every mutation. github_token: ${{ secrets.GITHUB_TOKEN }} - # Model selection goes through claude_args per the v1 action's - # documented interface. Opus 4.6 is the default — see inputs.model. claude_args: | --model ${{ inputs.model }} --allowedTools Bash,Read,Glob,Grep,WebSearch,WebFetch prompt: | You are **Mary**, the BMAD Strategic Business Analyst, running as an automated - weekly feature research and ideation agent for the **${{ github.repository }}** project. + weekly feature research and ideation agent for the ${{ github.repository }} project. ## Thinking & Research Directive @@ -288,20 +231,29 @@ jobs: ## Environment - - Repository: ${{ github.repository }} - - Workflow run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - - Focus area: ${{ inputs.focus_area || 'Open exploration' }} - - Research depth: ${{ inputs.research_depth }} + All paths and toggles are passed via environment variables — read them at runtime + with `printenv` or `$VAR_NAME`. Do NOT try to interpolate shell expressions in + this prompt body; YAML does not expand them and you would receive literal text. + + - `$SIGNALS_PATH` — JSON file with this week's signals (schema-validated) + - `$PROPOSALS_PATH` — where to write your proposals.json (you create this) + - `$MATCH_PLAN_PATH` — where the matcher writes its decision plan + - `$TOOLING_DIR` — path to `.github/scripts/feature-ideation/` (canonical scripts) + - `$DRY_RUN` — "1" = mutations are logged to `$DRY_RUN_LOG`, not executed + - `$FOCUS_AREA` — optional focus area (empty = open exploration) + - `$RESEARCH_DEPTH` — quick | standard | deep + - `$GH_TOKEN` — workflow token with `discussions: write` - (The current date is in `signals.json` as `scan_date` — read it - from there when you need it for Discussion timestamps.) + The current date is in `$SIGNALS_PATH` as `.scan_date` — read it from there + when you need it for Discussion timestamps. ## Phase 1: Load Context 1. Read the project signals collected by the previous job: ```bash - cat ${{ runner.temp }}/signals.json + cat "$SIGNALS_PATH" ``` + The shape is pinned by `signals.schema.json` — every field is guaranteed present. 2. Discover and read planning artifacts. BMAD installations vary — use Glob to find them, then Read the ones that exist: @@ -324,8 +276,10 @@ jobs: - TODO / FIXME / HACK comments — known gaps the team has flagged. - Files named `*config*`, `*settings*` — surfaces user-facing knobs. - 5. **Read existing Ideas discussions from signals.** These are ideas already being - tracked. You will update them if you have new findings, or leave them alone. + 5. **Read existing Ideas discussions from `$SIGNALS_PATH`** (`.ideas_discussions.items`). + These are ideas already being tracked. The deterministic matcher in Phase 6 will + decide which proposals are new vs updates — you do NOT need to do fuzzy matching + in your head. ## Phase 2: Market Research Skill (Iterative Evidence Gathering) @@ -347,12 +301,12 @@ jobs: - Technology shifts that change what is feasible or affordable ### User Need Signals - - Patterns in open issues and feature requests - - Common pain points from bug reports that suggest UX gaps - - What shipped recently (merged PRs) and what gaps remain - - Community discussion themes + - Patterns in `.open_issues` and `.feature_requests` + - Common pain points from `.bug_reports` that suggest UX gaps + - What shipped recently (`.merged_prs_30d`) and what gaps remain + - Discussion themes from `.ideas_discussions` - Adjust depth based on RESEARCH_DEPTH: + Adjust depth based on `$RESEARCH_DEPTH`: - **quick**: Skim signals, 1-2 web searches, focus on low-hanging fruit - **standard**: Thorough signal analysis, 5-8 web searches, balanced exploration - **deep**: Exhaustive research, 10+ web searches, competitor deep dives, trend analysis @@ -363,9 +317,8 @@ jobs: ## Phase 3: Brainstorming Skill (Divergent Ideation) - **Switch to the "Brainstorming" mindset.** Now generate as many ideas as possible — - quantity over quality. No idea is too wild at this stage. Build on the evidence - from Phase 2 but let your imagination run. + **Switch to the "Brainstorming" mindset.** Generate as many ideas as possible — + quantity over quality. Build on the evidence from Phase 2 but let your imagination run. Generate **8-15 raw feature ideas** across these lenses: - What gaps do competitors leave that this project could fill? @@ -376,32 +329,21 @@ jobs: For each raw idea, write a one-line description and note which evidence supports it. - **Iterate:** Review your list. Can you combine two weaker ideas into one stronger - idea? Did generating ideas reveal a research gap? If so, go back to Phase 2 for - a targeted search, then return here. Repeat until the ideas feel grounded and - diverse. + **Iterate:** Review your list. Combine weaker ideas. Spot research gaps and go + back to Phase 2 for targeted searches if needed. Repeat until grounded and diverse. ## Phase 4: Party Mode Skill (Collaborative Refinement) - **Switch to the "Party Mode" mindset.** Imagine you are a room of enthusiastic - product people, engineers, and designers who are all excited about these ideas. - The energy is high. Each person builds on the previous person's suggestion. + **Switch to the "Party Mode" mindset.** Imagine a room of enthusiastic product + people, engineers, and designers. Each person builds on the previous suggestion. Take the top 5-8 ideas from Phase 3 and refine them collaboratively: - 1. **Amplify:** For each idea, ask "What if we went even further?" Push the idea - to its most ambitious form. - 2. **Connect:** Look for synergies — can two ideas combine into something greater - than the sum of their parts? - 3. **Ground:** For each amplified idea, ask "What is the minimum version that still - delivers the core value?" Find the sweet spot between ambition and feasibility. - 4. **Score:** For each refined idea, assess: - - | Dimension | Question | - |-----------|----------| - | **Feasibility** | Can we build this with the current stack? How much effort? | - | **Impact** | How many users does this serve? How much value does it add? | - | **Urgency** | Is the market moving here now? Is there competitive pressure? | + 1. **Amplify:** "What if we went even further?" + 2. **Connect:** Look for synergies — combine ideas into something greater. + 3. **Ground:** "What is the minimum version that still delivers core value?" + 4. **Score:** For each refined idea, assess Feasibility / Impact / Urgency on + a high/med/low scale. Select the **top 5 ideas** scoring high on at least 2 of 3 dimensions. @@ -409,119 +351,169 @@ jobs: **Switch to the "Adversarial" mindset.** You are now a skeptical VP of Product, a resource-constrained engineering lead, and a demanding end-user — all at once. - Your job is to tear these ideas apart. Only the ones that survive are worth proposing. - - For each of the top 5 ideas from Phase 4, apply these challenges: - - 1. **"So what?"** — Why would a user actually switch to or stay with this project - because of this? If you can't articulate a compelling user story, cut it. - 2. **"Who else?"** — Is someone else already doing this better, or about to? - If this project can't differentiate, cut it or find the unique angle. - 3. **"At what cost?"** — What is the true engineering cost (not the optimistic - estimate)? What would we have to stop doing to build this? If the opportunity - cost is too high, cut it. - 4. **"What breaks?"** — What existing functionality or architectural assumptions - does this violate? If it creates technical debt or UX inconsistency, can that - be mitigated? - 5. **"Prove it."** — What evidence from Phase 2 actually supports demand for this? - Gut feelings and "it would be cool" are not evidence. Cite specific signals. + Tear these ideas apart. Only the survivors are worth proposing. + + For each top-5 idea, apply: **So what? Who else? At what cost? What breaks? Prove it.** **After the adversarial pass, you should have 3-5 ROBUST and DEFENSIBLE proposals.** Each surviving idea must have: - - At least one concrete market signal (competitor move, user request, or trend) + - At least one concrete market signal - A clear user story explaining why someone would care - A feasible technical path grounded in the current architecture - A rebuttal to the strongest objection against it - If fewer than 3 ideas survive, that is fine — quality over quantity. - If a focus area was specified, weight proposals toward that area but don't - exclude other high-scoring discoveries. + Quality over quantity — fewer than 3 is fine if that is what survives. + + ## Phase 6: Write proposals.json + run the deterministic matcher + + Write your surviving proposals to `$PROPOSALS_PATH` as a JSON array. Each entry + must include at minimum a `title` field (use the `💡 ` prefix). Recommended shape: + + ```json + [ + { + "title": "💡 Concise Idea Title", + "summary": "2-3 sentence description", + "market_signal": "What competitive or market trend supports this", + "user_signal": "Related issues / feedback / themes", + "technical_opportunity": "How architecture enables this", + "adversarial_objection": "Strongest objection", + "rebuttal": "Why the idea survives", + "feasibility": "high|med|low", + "impact": "high|med|low", + "urgency": "high|med|low", + "next_step": "Concrete action" + } + ] + ``` + + Then run the matcher. It uses normalized Jaccard similarity, so you do NOT + need to do any fuzzy matching yourself: + + ```bash + bash "$TOOLING_DIR/match-discussions.sh" \ + "$SIGNALS_PATH" \ + "$PROPOSALS_PATH" \ + > "$MATCH_PLAN_PATH" + ``` + + The output has two arrays: + - `.matched` — proposals aligning with an existing Discussion (update path) + - `.new_candidates` — proposals with no existing match (create path) - ## Phase 6: Resolve Discussion Category & Repository IDs + Each `.matched` entry includes a `similarity` score and the matched discussion. - **Auth note:** All `gh` and `gh api` commands in the phases below use - the `GH_TOKEN` environment variable (already set to the workflow's - `GITHUB_TOKEN`), which has `discussions: write` permission via the - job-level grant. You do NOT need to authenticate manually. + ## Phase 7: Resolve repository + category IDs - Before creating or updating Discussions, you need the repository ID and the - "Ideas" discussion category ID. + **Auth note:** All `gh` and `gh api` commands use the `GH_TOKEN` environment + variable (already set to the workflow's `GITHUB_TOKEN`), which has + `discussions: write` permission via the job-level grant. You do NOT need to + authenticate manually. + + Fetch the repository ID and the "Ideas" category ID: ```bash gh api graphql -f query=' query($repo: String!, $owner: String!) { repository(name: $repo, owner: $owner) { id - discussionCategories(first: 20) { + discussionCategories(first: 25) { nodes { id name } } } - }' -f owner="${{ github.repository_owner }}" \ - -f repo="${{ github.event.repository.name }}" + }' -f owner="${GITHUB_REPOSITORY%%/*}" \ + -f repo="${GITHUB_REPOSITORY##*/}" ``` Save the repository ID and the "Ideas" category ID. If "Ideas" does not exist, - fall back to "General". If neither exists, skip Discussion creation and write - an error to the step summary. + fall back to "General". If neither exists, write an error to step summary and + skip Phase 8. - ## Phase 7: Create or Update Per-Idea Discussions + ## Phase 8: Execute the plan via discussion-mutations.sh - For **each** feature proposal surviving Phase 5, check whether a matching Discussion - already exists in the target category (Ideas, or General as fallback). + Source the mutation helpers and call them per the match plan. The helpers + automatically honor `$DRY_RUN` — when `$DRY_RUN=1` they log a JSONL entry to + `$DRY_RUN_LOG` instead of calling GitHub. - ### Matching Logic + ```bash + source "$TOOLING_DIR/discussion-mutations.sh" + ``` - **Re-query before each create.** The signals snapshot only contains the first - 100 Ideas discussions and may miss the General fallback category. Before - creating a new Discussion for a proposal, run a fresh GraphQL query for - existing discussions in the target category, filtered to titles starting with - `💡`. This avoids duplicates if the snapshot was incomplete or if a previous - run in this same execution already created a related Discussion. + ### For each entry in `.new_candidates` from `$MATCH_PLAN_PATH` - Search the fresh results (and the signals snapshot) for a title that - matches the proposal's core concept. Use fuzzy matching — the exact title - may differ but the idea should be recognizably the same. + Build the body using the template at the bottom of this prompt, then: - **Discussion title format:** `💡 <Concise Idea Title>` + ```bash + create_discussion "$REPO_ID" "$CAT_ID" "$TITLE" "$BODY" + ``` - ### If the Discussion DOES NOT exist → Create it + Then add the `enhancement` label if your repo has one (resolve the label ID + first; skip silently if it doesn't exist): ```bash - gh api graphql -f query=' - mutation($repoId: ID!, $categoryId: ID!, $title: String!, $body: String!) { - createDiscussion(input: { - repositoryId: $repoId, - categoryId: $categoryId, - title: $title, - body: $body - }) { - discussion { id number url } - } - }' -f repoId="<REPO_ID>" \ - -f categoryId="<CATEGORY_ID>" \ - -f title="💡 <Idea Title>" \ - -f body="<BODY>" + add_label_to_discussion "$DISCUSSION_ID" "$LABEL_ID" + ``` + + ### For each entry in `.matched` with genuinely new information + + ```bash + comment_on_discussion "$DISCUSSION_ID" "$UPDATE_BODY" + ``` + + **Only post updates if there is genuinely new information.** No empty updates. + + ### Retirement recommendations + + If research shows a previously proposed idea is no longer viable, post a final + comment recommending closure (do NOT close the Discussion — only recommend): + + ```bash + comment_on_discussion "$DISCUSSION_ID" "$RETIREMENT_BODY" ``` - **New Discussion body format:** + ## Phase 9: Step Summary + + Write to `$GITHUB_STEP_SUMMARY` a markdown table summarizing: + - New ideas proposed (with similarity scores from `$MATCH_PLAN_PATH`) + - Existing ideas updated (link to the comment) + - Ideas reviewed with no update needed + - Top 2-3 market trend bullets + - **If `$DRY_RUN=1`,** append the contents of `$DRY_RUN_LOG` (formatted) so + reviewers can audit what would have been written. + + ## Rules + + - **Do NOT create issues or PRs.** Output goes to Discussions only. + - **Do NOT implement code.** Research and ideation only. + - **Max 5 new Discussions per run.** Quality over quantity. + - **Always go through the deterministic matcher** in Phase 6 — never invent + your own fuzzy match logic. + - **Always go through `discussion-mutations.sh`** — never call createDiscussion + or addDiscussionComment GraphQL mutations directly. The helpers handle DRY_RUN. + - **Ground every proposal in evidence** from the signals file or your research. + - **Be specific about next steps.** + - **Respect the project's architecture** documented in AGENTS.md and planning + artifacts. + - **Use the 💡 prefix** in all idea Discussion titles. + + ## Discussion body template (new proposals) ```markdown ## Summary - <2-3 sentence description of the feature idea> + <2-3 sentence description> ## Market Signal - <What competitive or market trend supports this idea> + <Competitive or market trend> ## User Signal - <Related issues, feedback, discussion themes, or bug patterns> + <Related issues / feedback themes> ## Technical Opportunity - <How the current architecture enables or supports this — reference specific - interfaces, abstractions, or extension points from the codebase> + <How current architecture enables this — cite ports/interfaces> ## Assessment @@ -538,54 +530,17 @@ jobs: ## Suggested Next Step - <Concrete action: "Create a product brief", "Spike on X", "Discuss with team", etc.> - - --- - *Proposed by the [BMAD Analyst (Mary)](/${{ github.repository }}/actions/workflows/feature-ideation.yml) on <DATE>.* - *[Workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})* - ``` - - After creating, add the `enhancement` label to the discussion if your repo - has one. Resolve the label ID with: - ```bash - gh api graphql -f query=' - query($repo: String!, $owner: String!) { - repository(name: $repo, owner: $owner) { - label(name: "enhancement") { id } - } - }' -f owner="${{ github.repository_owner }}" \ - -f repo="${{ github.event.repository.name }}" - ``` - Then apply via `addLabelsToLabelable`. Skip silently if the label doesn't exist. - - ### If the Discussion ALREADY exists → Add an update comment - - When a prior run already created a Discussion for the same idea, add a comment - with new developments rather than creating a duplicate. - - ```bash - gh api graphql -f query=' - mutation($discussionId: ID!, $body: String!) { - addDiscussionComment(input: { - discussionId: $discussionId, - body: $body - }) { - comment { id url } - } - }' -f discussionId="<DISCUSSION_ID>" \ - -f body="<COMMENT>" + <Concrete action> ``` - **Update comment format:** + ## Discussion comment template (updates) ```markdown - ## Weekly Update — <DATE> + ## Weekly Update ### What Changed - <New market signals, competitive moves, technology developments, or - user feedback since the last update. Be specific about what's new — - don't repeat the original proposal.> + <New signals since last update — be specific> ### Updated Assessment @@ -597,81 +552,14 @@ jobs: ### Recommendation - <Should this idea be advanced, kept watching, or retired? Why?> - - --- - *Updated by the [BMAD Analyst (Mary)](/${{ github.repository }}/actions/workflows/feature-ideation.yml) on <DATE>.* - *[Workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})* + <Advance / keep watching / retire — and why> ``` - **Important:** Only add an update comment if there is genuinely new information. - If nothing has changed for an existing idea, skip it — don't post "no updates" comments. - - ### If an existing idea should be retired - - If research shows a previously proposed idea is no longer viable (market shifted, - already implemented, superseded by another approach), add a final comment - recommending closure: - - ```markdown - ## Retirement Recommendation — <DATE> - - **Recommendation:** Close this proposal. - - **Reason:** <Why the idea is no longer viable or relevant> - - <Details on what changed> - - --- - *Reviewed by the [BMAD Analyst (Mary)](/${{ github.repository }}/actions/workflows/feature-ideation.yml) on <DATE>.* - ``` - - Do NOT close the Discussion — only recommend closure for human decision. - - ## Phase 8: Step Summary - - Write to $GITHUB_STEP_SUMMARY: - - ```markdown - ## Feature Research & Ideation — <DATE> - - **Focus:** <area or "Open exploration"> - **Research depth:** <quick|standard|deep> - - ### New Ideas Proposed - | Discussion | Title | Feasibility | Impact | Urgency | - |------------|-------|-------------|--------|---------| - | #N | 💡 ... | ... | ... | ... | - - ### Existing Ideas Updated - | Discussion | Title | Key Change | - |------------|-------|------------| - | #N | 💡 ... | <what's new> | - - ### Ideas Reviewed — No Update Needed - | Discussion | Title | Reason | - |------------|-------|--------| - | #N | 💡 ... | <no new signal> | - - ### Market & Trend Summary - <2-3 bullet points on the most notable trends this week> - - --- - *[Workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})* - ``` - - ## Rules - - - **Do NOT create issues or PRs.** Output goes to Discussion threads only. - - **Do NOT implement code.** This is research and ideation only. - - **Max 5 new Discussions per run.** Fewer is fine — quality over quantity. - - **Only propose ideas scoring high on 2+ dimensions** (feasibility/impact/urgency). - - **One Discussion per idea.** Never create duplicates — always check existing first. - - **Only comment on existing Discussions when there is new information.** No empty updates. - - **Ground proposals in evidence.** Every proposal must cite at least one market signal, - user signal, or technical opportunity — not just abstract ideas. - - **Be specific about next steps.** Each proposal should suggest a concrete action - (product brief, spike, discussion, etc.) not just "we should consider this." - - **Respect the project's architecture.** Proposals should be compatible with the - patterns documented in AGENTS.md and the planning artifacts. - - **Use the 💡 prefix** in all idea Discussion titles for easy identification. + - name: Upload dry-run log (if present) + if: ${{ inputs.dry_run }} + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: dry-run-log + path: ${{ runner.temp }}/dry-run.jsonl + if-no-files-found: ignore + retention-days: 7 diff --git a/.github/workflows/feature-ideation-tests.yml b/.github/workflows/feature-ideation-tests.yml new file mode 100644 index 0000000..91c1a62 --- /dev/null +++ b/.github/workflows/feature-ideation-tests.yml @@ -0,0 +1,112 @@ +# Quality gates for the reusable feature-ideation workflow and its +# supporting scripts. +# +# Triggered on any PR that touches: +# - .github/workflows/feature-ideation*.yml +# - .github/scripts/feature-ideation/** +# - .github/schemas/signals.schema.json +# - test/workflows/feature-ideation/** +# - standards/workflows/feature-ideation.yml +# +# Gates (all must pass before merge): +# 1. shellcheck — static analysis for every script in the suite +# 2. lint-prompt — direct prompt blocks have no unescaped shell expansions +# 3. schema check — fixtures validate against signals.schema.json +# 4. bats — full unit + integration test suite (~92 tests) +# +# Standard: this workflow enforces the contract documented in +# .github/scripts/feature-ideation/README.md. + +name: Feature Ideation Tests + +on: + pull_request: + paths: + - '.github/workflows/feature-ideation*.yml' + - '.github/scripts/feature-ideation/**' + - '.github/schemas/signals.schema.json' + - 'test/workflows/feature-ideation/**' + - 'standards/workflows/feature-ideation.yml' + push: + branches: + - main + paths: + - '.github/workflows/feature-ideation*.yml' + - '.github/scripts/feature-ideation/**' + - '.github/schemas/signals.schema.json' + - 'test/workflows/feature-ideation/**' + - 'standards/workflows/feature-ideation.yml' + +permissions: + contents: read + +concurrency: + group: feature-ideation-tests-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + name: Lint, schema, and bats + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + fetch-depth: 1 + + - name: Install bats, shellcheck, and jq + run: | + set -euo pipefail + sudo apt-get update -qq + sudo apt-get install -y --no-install-recommends bats shellcheck jq + + - name: Install Python jsonschema + run: pip install --quiet 'jsonschema>=4' + + - name: shellcheck + run: | + set -euo pipefail + cd .github/scripts/feature-ideation + shellcheck -x \ + collect-signals.sh \ + lint-prompt.sh \ + match-discussions.sh \ + discussion-mutations.sh \ + lib/gh-safe.sh \ + lib/filter-bots.sh \ + lib/compose-signals.sh \ + lib/date-utils.sh + + - name: Lint direct prompt blocks + run: bash .github/scripts/feature-ideation/lint-prompt.sh + + - name: Validate schema fixtures + run: | + set -euo pipefail + for fixture in test/workflows/feature-ideation/fixtures/expected/*.signals.json; do + base=$(basename "$fixture") + if [[ "$base" == INVALID-* ]]; then + if python3 .github/scripts/feature-ideation/validate-signals.py \ + "$fixture" .github/schemas/signals.schema.json; then + echo "::error file=$fixture::expected validation failure but it passed" + exit 1 + fi + else + python3 .github/scripts/feature-ideation/validate-signals.py \ + "$fixture" .github/schemas/signals.schema.json + fi + done + + - name: Run bats suite + run: bats --print-output-on-failure test/workflows/feature-ideation/ + + - name: Upload bats output on failure + if: failure() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: bats-output + path: | + test/workflows/feature-ideation/**/*.log + if-no-files-found: ignore + retention-days: 7 diff --git a/standards/workflows/feature-ideation.yml b/standards/workflows/feature-ideation.yml index fa7b5ba..7d78a79 100644 --- a/standards/workflows/feature-ideation.yml +++ b/standards/workflows/feature-ideation.yml @@ -35,6 +35,11 @@ on: - quick - standard - deep + dry_run: + description: 'Skip Discussion mutations and log them to a JSONL artifact instead. Use this on a fork to smoke-test before going live.' + required: false + default: false + type: boolean permissions: {} @@ -69,5 +74,6 @@ jobs: include A, B, C. Key emerging trends in this space: X, Y, Z." focus_area: ${{ inputs.focus_area || '' }} research_depth: ${{ inputs.research_depth || 'standard' }} + dry_run: ${{ inputs.dry_run || false }} secrets: CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} diff --git a/test/workflows/feature-ideation/collect-signals.bats b/test/workflows/feature-ideation/collect-signals.bats new file mode 100644 index 0000000..9543b97 --- /dev/null +++ b/test/workflows/feature-ideation/collect-signals.bats @@ -0,0 +1,204 @@ +#!/usr/bin/env bats +# Integration test for collect-signals.sh end-to-end. +# +# Uses the multi-call gh stub to script the exact sequence of gh invocations +# the collector makes, then validates the resulting signals.json against the +# JSON schema. + +load 'helpers/setup' + +setup() { + tt_make_tmpdir + tt_install_gh_stub + export REPO="petry-projects/talkterm" + export GH_TOKEN="fake-token-for-tests" + export SIGNALS_OUTPUT="${TT_TMP}/signals.json" +} + +teardown() { + tt_cleanup_tmpdir +} + +# Build a multi-call gh script for the standard happy path. +# Order MUST match collect-signals.sh: +# 1. gh issue list --state open +# 2. gh issue list --state closed +# 3. gh api graphql (categories) +# 4. gh api graphql (discussions) +# 5. gh release list +# 6. gh pr list --state merged +build_happy_script() { + local script="${TT_TMP}/gh-script.tsv" + : >"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/issue-list-open.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/issue-list-closed.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/graphql-categories.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/graphql-discussions.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/release-list.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/pr-list-merged.json" >>"$script" + export GH_STUB_SCRIPT="$script" + rm -f "${TT_TMP}/.gh-stub-counter" +} + +# --------------------------------------------------------------------------- +# Happy path +# --------------------------------------------------------------------------- + +@test "collect-signals: happy path produces valid signals.json" { + build_happy_script + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -eq 0 ] + [ -f "$SIGNALS_OUTPUT" ] + run python3 "${TT_SCRIPTS_DIR}/validate-signals.py" "$SIGNALS_OUTPUT" + [ "$status" -eq 0 ] +} + +@test "collect-signals: bot author is filtered out of open_issues" { + build_happy_script + bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + count=$(jq '.open_issues.count' "$SIGNALS_OUTPUT") + [ "$count" = "2" ] + bot_present=$(jq '[.open_issues.items[] | select(.author.login == "dependabot[bot]")] | length' "$SIGNALS_OUTPUT") + [ "$bot_present" = "0" ] +} + +@test "collect-signals: closed issues are filtered by 30-day cutoff" { + build_happy_script + bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + # The fixture has one recent (2099) and one ancient (2020) closed issue. + # Only the future-dated one should survive the 30-day-ago cutoff. + count=$(jq '.closed_issues_30d.count' "$SIGNALS_OUTPUT") + [ "$count" = "1" ] + num=$(jq '.closed_issues_30d.items[0].number' "$SIGNALS_OUTPUT") + [ "$num" = "95" ] +} + +@test "collect-signals: feature_requests derives from labeled open issues" { + build_happy_script + bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + fr_count=$(jq '.feature_requests.count' "$SIGNALS_OUTPUT") + [ "$fr_count" = "1" ] + fr_num=$(jq '.feature_requests.items[0].number' "$SIGNALS_OUTPUT") + [ "$fr_num" = "101" ] +} + +@test "collect-signals: bug_reports derives from labeled open issues" { + build_happy_script + bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + br_count=$(jq '.bug_reports.count' "$SIGNALS_OUTPUT") + [ "$br_count" = "1" ] + br_num=$(jq '.bug_reports.items[0].number' "$SIGNALS_OUTPUT") + [ "$br_num" = "102" ] +} + +@test "collect-signals: ideas_discussions populated when category exists" { + build_happy_script + bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + idc=$(jq '.ideas_discussions.count' "$SIGNALS_OUTPUT") + [ "$idc" = "1" ] + title=$(jq -r '.ideas_discussions.items[0].title' "$SIGNALS_OUTPUT") + [[ "$title" == *"Streaming voice"* ]] +} + +@test "collect-signals: scan_date is ISO-8601 Zulu" { + build_happy_script + bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + d=$(jq -r '.scan_date' "$SIGNALS_OUTPUT") + [[ "$d" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ ]] +} + +# --------------------------------------------------------------------------- +# Failure modes — pipeline must fail loud, NOT silently produce empty data +# --------------------------------------------------------------------------- + +@test "collect-signals: FAILS LOUD on open-issues auth failure" { + script="${TT_TMP}/gh-script.tsv" + err_file="${TT_TMP}/auth-err.txt" + printf 'HTTP 401: Bad credentials\n' >"$err_file" + printf '4\t-\t%s\n' "$err_file" >"$script" + export GH_STUB_SCRIPT="$script" + rm -f "${TT_TMP}/.gh-stub-counter" + + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -ne 0 ] + [ ! -f "$SIGNALS_OUTPUT" ] +} + +@test "collect-signals: FAILS LOUD on GraphQL errors envelope (categories)" { + script="${TT_TMP}/gh-script.tsv" + : >"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/issue-list-open.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/issue-list-closed.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/graphql-errors-envelope.json" >>"$script" + export GH_STUB_SCRIPT="$script" + rm -f "${TT_TMP}/.gh-stub-counter" + + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -ne 0 ] + [ ! -f "$SIGNALS_OUTPUT" ] +} + +@test "collect-signals: missing REPO env causes usage error" { + unset REPO + build_happy_script + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -eq 64 ] +} + +@test "collect-signals: missing GH_TOKEN env causes usage error" { + unset GH_TOKEN + build_happy_script + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -eq 64 ] +} + +@test "collect-signals: malformed REPO causes usage error" { + export REPO="not-a-slug" + build_happy_script + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -eq 64 ] +} + +# --------------------------------------------------------------------------- +# Truncation warnings +# --------------------------------------------------------------------------- + +@test "collect-signals: emits truncation warning when discussions hasNextPage=true" { + script="${TT_TMP}/gh-script.tsv" + : >"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/issue-list-open.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/issue-list-closed.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/graphql-categories.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/graphql-discussions-truncated.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/release-list.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/pr-list-merged.json" >>"$script" + export GH_STUB_SCRIPT="$script" + rm -f "${TT_TMP}/.gh-stub-counter" + + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -eq 0 ] + warn_count=$(jq '.truncation_warnings | length' "$SIGNALS_OUTPUT") + [ "$warn_count" -ge 1 ] + jq -e '.truncation_warnings[] | select(.source == "ideas_discussions")' "$SIGNALS_OUTPUT" >/dev/null +} + +# --------------------------------------------------------------------------- +# No "Ideas" category — graceful skip, not a hard failure +# --------------------------------------------------------------------------- + +@test "collect-signals: skips discussions when Ideas category absent" { + script="${TT_TMP}/gh-script.tsv" + : >"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/issue-list-open.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/issue-list-closed.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/graphql-no-ideas-category.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/release-list.json" >>"$script" + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/pr-list-merged.json" >>"$script" + export GH_STUB_SCRIPT="$script" + rm -f "${TT_TMP}/.gh-stub-counter" + + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -eq 0 ] + idc=$(jq '.ideas_discussions.count' "$SIGNALS_OUTPUT") + [ "$idc" = "0" ] +} diff --git a/test/workflows/feature-ideation/compose-signals.bats b/test/workflows/feature-ideation/compose-signals.bats new file mode 100644 index 0000000..4dced2c --- /dev/null +++ b/test/workflows/feature-ideation/compose-signals.bats @@ -0,0 +1,102 @@ +#!/usr/bin/env bats +# Tests for .github/scripts/feature-ideation/lib/compose-signals.sh +# +# Kills R4 (jq --argjson crash on empty inputs) and R3 (no contract for +# the signals payload). Schema validation lives in signals-schema.bats; +# this file pins the composition logic itself. + +load 'helpers/setup' + +setup() { + # shellcheck source=/dev/null + . "${TT_SCRIPTS_DIR}/lib/compose-signals.sh" +} + +# Helper: invoke compose_signals with all-empty buckets and a fixed scan_date. +compose_empty() { + compose_signals \ + '[]' '[]' '[]' '[]' '[]' '[]' '[]' \ + 'foo/bar' \ + '2026-04-07T00:00:00Z' \ + '1.0.0' \ + '[]' +} + +# --------------------------------------------------------------------------- +# Argument validation (R4) +# --------------------------------------------------------------------------- + +@test "compose: rejects wrong arg count" { + run compose_signals '[]' '[]' + [ "$status" -ne 0 ] +} + +@test "compose: rejects empty string for any JSON arg" { + run compose_signals \ + '' '[]' '[]' '[]' '[]' '[]' '[]' \ + 'foo/bar' '2026-04-07T00:00:00Z' '1.0.0' '[]' + [ "$status" -ne 0 ] +} + +@test "compose: rejects non-JSON for any JSON arg" { + run compose_signals \ + 'not json' '[]' '[]' '[]' '[]' '[]' '[]' \ + 'foo/bar' '2026-04-07T00:00:00Z' '1.0.0' '[]' + [ "$status" -ne 0 ] +} + +# --------------------------------------------------------------------------- +# Output shape (R3) +# --------------------------------------------------------------------------- + +@test "compose: produces all required top-level fields with empty inputs" { + run compose_empty + [ "$status" -eq 0 ] + for field in schema_version scan_date repo open_issues closed_issues_30d \ + ideas_discussions releases merged_prs_30d feature_requests \ + bug_reports truncation_warnings; do + printf '%s' "$output" | jq -e "has(\"$field\")" >/dev/null + done +} + +@test "compose: count fields equal items length" { + open='[{"number":1,"title":"a","labels":[]},{"number":2,"title":"b","labels":[]}]' + run compose_signals \ + "$open" '[]' '[]' '[]' '[]' '[]' '[]' \ + 'foo/bar' '2026-04-07T00:00:00Z' '1.0.0' '[]' + [ "$status" -eq 0 ] + count=$(printf '%s' "$output" | jq '.open_issues.count') + items_len=$(printf '%s' "$output" | jq '.open_issues.items | length') + [ "$count" = "2" ] + [ "$items_len" = "2" ] +} + +@test "compose: schema_version is preserved verbatim" { + run compose_signals \ + '[]' '[]' '[]' '[]' '[]' '[]' '[]' \ + 'foo/bar' '2026-04-07T00:00:00Z' '2.5.1' '[]' + [ "$status" -eq 0 ] + v=$(printf '%s' "$output" | jq -r '.schema_version') + [ "$v" = "2.5.1" ] +} + +@test "compose: truncation_warnings round-trip" { + warnings='[{"source":"open_issues","limit":50,"message":"truncated"}]' + run compose_signals \ + '[]' '[]' '[]' '[]' '[]' '[]' '[]' \ + 'foo/bar' '2026-04-07T00:00:00Z' '1.0.0' "$warnings" + [ "$status" -eq 0 ] + src=$(printf '%s' "$output" | jq -r '.truncation_warnings[0].source') + [ "$src" = "open_issues" ] +} + +@test "compose: scan_date and repo round-trip exactly" { + run compose_signals \ + '[]' '[]' '[]' '[]' '[]' '[]' '[]' \ + 'octocat/hello-world' '2030-01-15T12:34:56Z' '1.0.0' '[]' + [ "$status" -eq 0 ] + d=$(printf '%s' "$output" | jq -r '.scan_date') + r=$(printf '%s' "$output" | jq -r '.repo') + [ "$d" = "2030-01-15T12:34:56Z" ] + [ "$r" = "octocat/hello-world" ] +} diff --git a/test/workflows/feature-ideation/date-utils.bats b/test/workflows/feature-ideation/date-utils.bats new file mode 100644 index 0000000..dfb132b --- /dev/null +++ b/test/workflows/feature-ideation/date-utils.bats @@ -0,0 +1,50 @@ +#!/usr/bin/env bats +# Tests for .github/scripts/feature-ideation/lib/date-utils.sh + +load 'helpers/setup' + +setup() { + # shellcheck source=/dev/null + . "${TT_SCRIPTS_DIR}/lib/date-utils.sh" +} + +@test "date_days_ago: returns ISO date format" { + run date_days_ago 30 + [ "$status" -eq 0 ] + [[ "$output" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] +} + +@test "date_days_ago: 0 returns today" { + today=$(date -u +%Y-%m-%d) + run date_days_ago 0 + [ "$status" -eq 0 ] + [ "$output" = "$today" ] +} + +@test "date_days_ago: rejects non-integer input" { + run date_days_ago "abc" + [ "$status" -ne 0 ] +} + +@test "date_days_ago: rejects empty input" { + run date_days_ago "" + [ "$status" -ne 0 ] +} + +@test "date_days_ago: 30 days ago is earlier than today" { + today=$(date_today) + past=$(date_days_ago 30) + [ "$past" \< "$today" ] +} + +@test "date_now_iso: returns ISO-8601 Zulu" { + run date_now_iso + [ "$status" -eq 0 ] + [[ "$output" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ ]] +} + +@test "date_today: returns YYYY-MM-DD" { + run date_today + [ "$status" -eq 0 ] + [[ "$output" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] +} diff --git a/test/workflows/feature-ideation/discussion-mutations.bats b/test/workflows/feature-ideation/discussion-mutations.bats new file mode 100644 index 0000000..b42dabe --- /dev/null +++ b/test/workflows/feature-ideation/discussion-mutations.bats @@ -0,0 +1,102 @@ +#!/usr/bin/env bats +# Tests for discussion-mutations.sh — verifies DRY_RUN logging contract +# and live-mode delegation to gh-safe. + +bats_require_minimum_version 1.5.0 + +load 'helpers/setup' + +setup() { + tt_make_tmpdir + tt_install_gh_stub + export DRY_RUN_LOG="${TT_TMP}/dry-run.jsonl" + # shellcheck source=/dev/null + . "${TT_SCRIPTS_DIR}/discussion-mutations.sh" +} + +teardown() { + tt_cleanup_tmpdir +} + +# --------------------------------------------------------------------------- +# DRY_RUN mode +# --------------------------------------------------------------------------- + +@test "dry-run: create_discussion logs entry and returns it on stdout" { + DRY_RUN=1 run create_discussion "R_1" "C_ideas" "💡 New idea" "Body text" + [ "$status" -eq 0 ] + [[ "$output" == *'"create_discussion"'* ]] + [[ "$output" == *'"R_1"'* ]] + [[ "$output" == *'"💡 New idea"'* ]] + [ -f "$DRY_RUN_LOG" ] + [ "$(wc -l <"$DRY_RUN_LOG")" -eq 1 ] +} + +@test "dry-run: comment_on_discussion logs entry" { + DRY_RUN=1 run comment_on_discussion "D_1" "Update text" + [ "$status" -eq 0 ] + [[ "$output" == *'"comment_on_discussion"'* ]] + [[ "$output" == *'"D_1"'* ]] + [[ "$output" == *'"Update text"'* ]] +} + +@test "dry-run: add_label_to_discussion logs entry" { + DRY_RUN=1 run add_label_to_discussion "D_1" "L_enhancement" + [ "$status" -eq 0 ] + [[ "$output" == *'"add_label_to_discussion"'* ]] + [[ "$output" == *'"L_enhancement"'* ]] +} + +@test "dry-run: log file is JSONL (one valid object per line)" { + DRY_RUN=1 create_discussion "R_1" "C_1" "title 1" "body 1" + DRY_RUN=1 comment_on_discussion "D_1" "comment 1" + DRY_RUN=1 add_label_to_discussion "D_1" "L_1" + [ "$(wc -l <"$DRY_RUN_LOG")" -eq 3 ] + while IFS= read -r line; do + printf '%s' "$line" | jq -e . >/dev/null + done <"$DRY_RUN_LOG" +} + +@test "dry-run: never invokes gh" { + log="${TT_TMP}/gh-invocations.log" + GH_STUB_LOG="$log" + DRY_RUN=1 create_discussion "R_1" "C_1" "t" "b" + [ ! -f "$log" ] +} + +# --------------------------------------------------------------------------- +# Argument validation +# --------------------------------------------------------------------------- + +@test "create_discussion: rejects wrong arg count" { + DRY_RUN=1 run create_discussion "R_1" "C_1" "title" + [ "$status" -eq 64 ] +} + +@test "comment_on_discussion: rejects wrong arg count" { + DRY_RUN=1 run comment_on_discussion "D_1" + [ "$status" -eq 64 ] +} + +@test "add_label_to_discussion: rejects wrong arg count" { + DRY_RUN=1 run add_label_to_discussion "D_1" + [ "$status" -eq 64 ] +} + +# --------------------------------------------------------------------------- +# Live mode (gh stub returns success) +# --------------------------------------------------------------------------- + +@test "live: create_discussion calls gh and returns its output" { + GH_STUB_STDOUT='{"data":{"createDiscussion":{"discussion":{"id":"D_new","number":42,"url":"https://x"}}}}' \ + run create_discussion "R_1" "C_ideas" "title" "body" + [ "$status" -eq 0 ] + [[ "$output" == *'"D_new"'* ]] + [ ! -f "$DRY_RUN_LOG" ] +} + +@test "live: comment_on_discussion fails loudly when gh returns errors envelope" { + GH_STUB_STDOUT='{"errors":[{"message":"forbidden"}],"data":null}' \ + run comment_on_discussion "D_1" "body" + [ "$status" -ne 0 ] +} diff --git a/test/workflows/feature-ideation/filter-bots.bats b/test/workflows/feature-ideation/filter-bots.bats new file mode 100644 index 0000000..6f4e3ff --- /dev/null +++ b/test/workflows/feature-ideation/filter-bots.bats @@ -0,0 +1,62 @@ +#!/usr/bin/env bats +# Tests for .github/scripts/feature-ideation/lib/filter-bots.sh +# +# Pins R10: bot filter must catch all known automation accounts and be +# extensible via env var. + +load 'helpers/setup' + +setup() { + # shellcheck source=/dev/null + . "${TT_SCRIPTS_DIR}/lib/filter-bots.sh" +} + +input_with_bots() { + cat <<'JSON' +[ + {"number":1,"title":"real issue","author":{"login":"alice"}}, + {"number":2,"title":"dependabot bump","author":{"login":"dependabot[bot]"}}, + {"number":3,"title":"renovate bump","author":{"login":"renovate[bot]"}}, + {"number":4,"title":"copilot suggestion","author":{"login":"copilot[bot]"}}, + {"number":5,"title":"coderabbit","author":{"login":"coderabbitai[bot]"}}, + {"number":6,"title":"another real","author":{"login":"bob"}}, + {"number":7,"title":"github actions","author":{"login":"github-actions[bot]"}} +] +JSON +} + +@test "filter-bots: removes default bot authors" { + result=$(input_with_bots | filter_bots_apply) + count=$(printf '%s' "$result" | jq 'length') + [ "$count" = "2" ] + printf '%s' "$result" | jq -e '.[] | select(.author.login == "alice")' >/dev/null + printf '%s' "$result" | jq -e '.[] | select(.author.login == "bob")' >/dev/null +} + +@test "filter-bots: leaves human authors untouched" { + result=$(input_with_bots | filter_bots_apply) + for user in alice bob; do + printf '%s' "$result" | jq -e --arg u "$user" '.[] | select(.author.login == $u)' >/dev/null + done +} + +@test "filter-bots: handles items without author field" { + echo '[{"number":1,"title":"orphan"}]' | filter_bots_apply | jq -e '.[0].number == 1' >/dev/null +} + +@test "filter-bots: env extension adds custom bot logins" { + input='[{"number":1,"title":"x","author":{"login":"my-custom-bot"}},{"number":2,"title":"y","author":{"login":"alice"}}]' + result=$(FEATURE_IDEATION_BOT_AUTHORS="my-custom-bot" sh -c " + . '${TT_SCRIPTS_DIR}/lib/filter-bots.sh' + printf '%s' '$input' | filter_bots_apply + ") + count=$(printf '%s' "$result" | jq 'length') + [ "$count" = "1" ] + login=$(printf '%s' "$result" | jq -r '.[0].author.login') + [ "$login" = "alice" ] +} + +@test "filter-bots: empty input array round-trips" { + result=$(echo '[]' | filter_bots_apply) + [ "$result" = "[]" ] +} diff --git a/test/workflows/feature-ideation/fixtures/expected/INVALID-bad-repo.signals.json b/test/workflows/feature-ideation/fixtures/expected/INVALID-bad-repo.signals.json new file mode 100644 index 0000000..697af83 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/expected/INVALID-bad-repo.signals.json @@ -0,0 +1,13 @@ +{ + "schema_version": "1.0.0", + "scan_date": "2026-04-07T07:00:00Z", + "repo": "not-a-valid-slug", + "open_issues": { "count": 0, "items": [] }, + "closed_issues_30d": { "count": 0, "items": [] }, + "ideas_discussions": { "count": 0, "items": [] }, + "releases": [], + "merged_prs_30d": { "count": 0, "items": [] }, + "feature_requests": { "count": 0, "items": [] }, + "bug_reports": { "count": 0, "items": [] }, + "truncation_warnings": [] +} diff --git a/test/workflows/feature-ideation/fixtures/expected/INVALID-missing-field.signals.json b/test/workflows/feature-ideation/fixtures/expected/INVALID-missing-field.signals.json new file mode 100644 index 0000000..897f5a2 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/expected/INVALID-missing-field.signals.json @@ -0,0 +1,6 @@ +{ + "schema_version": "1.0.0", + "scan_date": "2026-04-07T07:00:00Z", + "repo": "petry-projects/talkterm", + "open_issues": { "count": 0, "items": [] } +} diff --git a/test/workflows/feature-ideation/fixtures/expected/empty-repo.signals.json b/test/workflows/feature-ideation/fixtures/expected/empty-repo.signals.json new file mode 100644 index 0000000..4101afb --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/expected/empty-repo.signals.json @@ -0,0 +1,13 @@ +{ + "schema_version": "1.0.0", + "scan_date": "2026-04-07T07:00:00Z", + "repo": "petry-projects/talkterm", + "open_issues": { "count": 0, "items": [] }, + "closed_issues_30d": { "count": 0, "items": [] }, + "ideas_discussions": { "count": 0, "items": [] }, + "releases": [], + "merged_prs_30d": { "count": 0, "items": [] }, + "feature_requests": { "count": 0, "items": [] }, + "bug_reports": { "count": 0, "items": [] }, + "truncation_warnings": [] +} diff --git a/test/workflows/feature-ideation/fixtures/expected/populated.signals.json b/test/workflows/feature-ideation/fixtures/expected/populated.signals.json new file mode 100644 index 0000000..4c92921 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/expected/populated.signals.json @@ -0,0 +1,93 @@ +{ + "schema_version": "1.0.0", + "scan_date": "2026-04-07T07:00:00Z", + "repo": "petry-projects/talkterm", + "open_issues": { + "count": 2, + "items": [ + { + "number": 101, + "title": "Add dark mode", + "labels": [{ "name": "enhancement" }], + "createdAt": "2026-03-15T10:00:00Z", + "author": { "login": "alice" } + }, + { + "number": 102, + "title": "Crash on startup", + "labels": [{ "name": "bug" }], + "createdAt": "2026-03-20T11:00:00Z", + "author": { "login": "bob" } + } + ] + }, + "closed_issues_30d": { + "count": 1, + "items": [ + { + "number": 95, + "title": "Old issue", + "labels": [], + "closedAt": "2026-03-25T12:00:00Z" + } + ] + }, + "ideas_discussions": { + "count": 1, + "items": [ + { + "id": "D_1", + "number": 7, + "title": "💡 Streaming voice responses", + "createdAt": "2026-03-01T00:00:00Z", + "updatedAt": "2026-04-01T00:00:00Z", + "labels": { "nodes": [{ "name": "enhancement" }] }, + "comments": { "totalCount": 3 } + } + ] + }, + "releases": [ + { + "tagName": "v0.1.0", + "name": "Initial preview", + "publishedAt": "2026-03-30T00:00:00Z", + "isPrerelease": true + } + ], + "merged_prs_30d": { + "count": 1, + "items": [ + { + "number": 60, + "title": "feat: weekly ideation workflow", + "labels": [], + "mergedAt": "2026-04-01T00:00:00Z" + } + ] + }, + "feature_requests": { + "count": 1, + "items": [ + { + "number": 101, + "title": "Add dark mode", + "labels": [{ "name": "enhancement" }], + "createdAt": "2026-03-15T10:00:00Z", + "author": { "login": "alice" } + } + ] + }, + "bug_reports": { + "count": 1, + "items": [ + { + "number": 102, + "title": "Crash on startup", + "labels": [{ "name": "bug" }], + "createdAt": "2026-03-20T11:00:00Z", + "author": { "login": "bob" } + } + ] + }, + "truncation_warnings": [] +} diff --git a/test/workflows/feature-ideation/fixtures/expected/truncated.signals.json b/test/workflows/feature-ideation/fixtures/expected/truncated.signals.json new file mode 100644 index 0000000..845db66 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/expected/truncated.signals.json @@ -0,0 +1,24 @@ +{ + "schema_version": "1.0.0", + "scan_date": "2026-04-07T07:00:00Z", + "repo": "petry-projects/talkterm", + "open_issues": { "count": 0, "items": [] }, + "closed_issues_30d": { "count": 0, "items": [] }, + "ideas_discussions": { "count": 0, "items": [] }, + "releases": [], + "merged_prs_30d": { "count": 0, "items": [] }, + "feature_requests": { "count": 0, "items": [] }, + "bug_reports": { "count": 0, "items": [] }, + "truncation_warnings": [ + { + "source": "open_issues", + "limit": 50, + "message": "result count equals limit; possible truncation" + }, + { + "source": "ideas_discussions", + "limit": 100, + "message": "hasNextPage=true; results truncated" + } + ] +} diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/graphql-categories.json b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-categories.json new file mode 100644 index 0000000..9825039 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-categories.json @@ -0,0 +1,13 @@ +{ + "data": { + "repository": { + "discussionCategories": { + "nodes": [ + {"id": "DC_general", "name": "General"}, + {"id": "DC_ideas", "name": "Ideas"}, + {"id": "DC_qna", "name": "Q&A"} + ] + } + } + } +} diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/graphql-discussions-truncated.json b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-discussions-truncated.json new file mode 100644 index 0000000..6b09dd7 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-discussions-truncated.json @@ -0,0 +1,20 @@ +{ + "data": { + "repository": { + "discussions": { + "pageInfo": {"hasNextPage": true}, + "nodes": [ + { + "id": "D_1", + "number": 7, + "title": "💡 First idea", + "createdAt": "2026-03-01T00:00:00Z", + "updatedAt": "2026-04-01T00:00:00Z", + "labels": {"nodes": []}, + "comments": {"totalCount": 0} + } + ] + } + } + } +} diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/graphql-discussions.json b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-discussions.json new file mode 100644 index 0000000..d86cb95 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-discussions.json @@ -0,0 +1,20 @@ +{ + "data": { + "repository": { + "discussions": { + "pageInfo": {"hasNextPage": false}, + "nodes": [ + { + "id": "D_1", + "number": 7, + "title": "💡 Streaming voice responses", + "createdAt": "2026-03-01T00:00:00Z", + "updatedAt": "2026-04-01T00:00:00Z", + "labels": {"nodes": [{"name": "enhancement"}]}, + "comments": {"totalCount": 3} + } + ] + } + } + } +} diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/graphql-errors-envelope.json b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-errors-envelope.json new file mode 100644 index 0000000..e444dae --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-errors-envelope.json @@ -0,0 +1,10 @@ +{ + "errors": [ + { + "type": "FORBIDDEN", + "message": "Resource not accessible by integration", + "path": ["repository", "discussions"] + } + ], + "data": null +} diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/graphql-no-ideas-category.json b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-no-ideas-category.json new file mode 100644 index 0000000..b87ffef --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/graphql-no-ideas-category.json @@ -0,0 +1,11 @@ +{ + "data": { + "repository": { + "discussionCategories": { + "nodes": [ + {"id": "DC_general", "name": "General"} + ] + } + } + } +} diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/issue-list-closed.json b/test/workflows/feature-ideation/fixtures/gh-responses/issue-list-closed.json new file mode 100644 index 0000000..3c344f2 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/issue-list-closed.json @@ -0,0 +1,14 @@ +[ + { + "number": 95, + "title": "Old recently closed", + "labels": [], + "closedAt": "2099-01-01T00:00:00Z" + }, + { + "number": 50, + "title": "Ancient closed", + "labels": [], + "closedAt": "2020-01-01T00:00:00Z" + } +] diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/issue-list-open.json b/test/workflows/feature-ideation/fixtures/gh-responses/issue-list-open.json new file mode 100644 index 0000000..006d1ea --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/issue-list-open.json @@ -0,0 +1,23 @@ +[ + { + "number": 101, + "title": "Add dark mode", + "labels": [{"name": "enhancement"}], + "createdAt": "2026-03-15T10:00:00Z", + "author": {"login": "alice"} + }, + { + "number": 102, + "title": "Crash on startup", + "labels": [{"name": "bug"}], + "createdAt": "2026-03-20T11:00:00Z", + "author": {"login": "bob"} + }, + { + "number": 103, + "title": "dependabot bump", + "labels": [], + "createdAt": "2026-03-22T11:00:00Z", + "author": {"login": "dependabot[bot]"} + } +] diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/pr-list-merged.json b/test/workflows/feature-ideation/fixtures/gh-responses/pr-list-merged.json new file mode 100644 index 0000000..8dee104 --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/pr-list-merged.json @@ -0,0 +1,14 @@ +[ + { + "number": 60, + "title": "feat: weekly ideation workflow", + "labels": [], + "mergedAt": "2099-04-01T00:00:00Z" + }, + { + "number": 10, + "title": "old merged PR", + "labels": [], + "mergedAt": "2020-01-01T00:00:00Z" + } +] diff --git a/test/workflows/feature-ideation/fixtures/gh-responses/release-list.json b/test/workflows/feature-ideation/fixtures/gh-responses/release-list.json new file mode 100644 index 0000000..4d6f5bf --- /dev/null +++ b/test/workflows/feature-ideation/fixtures/gh-responses/release-list.json @@ -0,0 +1,8 @@ +[ + { + "tagName": "v0.1.0", + "name": "Initial preview", + "publishedAt": "2026-03-30T00:00:00Z", + "isPrerelease": true + } +] diff --git a/test/workflows/feature-ideation/gh-safe.bats b/test/workflows/feature-ideation/gh-safe.bats new file mode 100644 index 0000000..9194d53 --- /dev/null +++ b/test/workflows/feature-ideation/gh-safe.bats @@ -0,0 +1,163 @@ +#!/usr/bin/env bats +# Tests for .github/scripts/feature-ideation/lib/gh-safe.sh +# +# These tests kill R1: the original "2>/dev/null || echo '[]'" pattern that +# silently swallowed every kind of failure. Each test pins one failure mode. + +load 'helpers/setup' + +setup() { + tt_make_tmpdir + tt_install_gh_stub + # shellcheck source=/dev/null + . "${TT_SCRIPTS_DIR}/lib/gh-safe.sh" +} + +teardown() { + tt_cleanup_tmpdir +} + +# --------------------------------------------------------------------------- +# gh_safe_is_json +# --------------------------------------------------------------------------- + +@test "is_json: accepts valid JSON" { + run gh_safe_is_json '[]' + [ "$status" -eq 0 ] + run gh_safe_is_json '{"a":1}' + [ "$status" -eq 0 ] +} + +@test "is_json: rejects empty string" { + run gh_safe_is_json '' + [ "$status" -ne 0 ] +} + +@test "is_json: rejects garbage" { + run gh_safe_is_json 'not json' + [ "$status" -ne 0 ] +} + +# --------------------------------------------------------------------------- +# gh_safe_rest — happy paths +# --------------------------------------------------------------------------- + +@test "rest: returns JSON array on 200 OK + populated result" { + GH_STUB_STDOUT='[{"number":1,"title":"hello"}]' \ + run gh_safe_rest issue list --repo foo/bar + [ "$status" -eq 0 ] + [ "$output" = '[{"number":1,"title":"hello"}]' ] +} + +@test "rest: returns [] when gh prints empty array" { + GH_STUB_STDOUT='[]' \ + run gh_safe_rest issue list --repo foo/bar + [ "$status" -eq 0 ] + [ "$output" = '[]' ] +} + +@test "rest: normalizes empty stdout to []" { + GH_STUB_STDOUT='' \ + run gh_safe_rest issue list --repo foo/bar + [ "$status" -eq 0 ] + [ "$output" = '[]' ] +} + +# --------------------------------------------------------------------------- +# gh_safe_rest — failure modes (R1 kill list) +# --------------------------------------------------------------------------- + +@test "rest: EXITS NON-ZERO on auth failure (gh exit 4)" { + GH_STUB_EXIT=4 \ + GH_STUB_STDERR='HTTP 401: Bad credentials' \ + run gh_safe_rest issue list --repo foo/bar + [ "$status" -ne 0 ] + [[ "$output" == *"rest-failure"* ]] || [[ "$stderr" == *"rest-failure"* ]] || true +} + +@test "rest: EXITS NON-ZERO on rate limit (gh exit 1 + 403)" { + GH_STUB_EXIT=1 \ + GH_STUB_STDERR='API rate limit exceeded' \ + run gh_safe_rest issue list --repo foo/bar + [ "$status" -ne 0 ] +} + +@test "rest: EXITS NON-ZERO on 5xx" { + GH_STUB_EXIT=1 \ + GH_STUB_STDERR='HTTP 502: Bad gateway' \ + run gh_safe_rest pr list --repo foo/bar + [ "$status" -ne 0 ] +} + +@test "rest: EXITS NON-ZERO on network failure" { + GH_STUB_EXIT=2 \ + GH_STUB_STDERR='dial tcp: lookup api.github.com: connection refused' \ + run gh_safe_rest issue list --repo foo/bar + [ "$status" -ne 0 ] +} + +@test "rest: EXITS NON-ZERO on non-JSON success output" { + GH_STUB_STDOUT='this is not json at all' \ + run gh_safe_rest issue list --repo foo/bar + [ "$status" -ne 0 ] +} + +# --------------------------------------------------------------------------- +# gh_safe_graphql — happy paths +# --------------------------------------------------------------------------- + +@test "graphql: returns full envelope when called without --jq" { + GH_STUB_STDOUT='{"data":{"repository":{"id":"R_1"}}}' \ + run gh_safe_graphql -f query='q' + [ "$status" -eq 0 ] + [[ "$output" == *'"R_1"'* ]] +} + +@test "graphql: applies --jq filter and returns its result" { + GH_STUB_STDOUT='{"data":{"repository":{"discussionCategories":{"nodes":[{"id":"C1","name":"Ideas"}]}}}}' \ + run gh_safe_graphql -f query='q' --jq '.data.repository.discussionCategories.nodes' + [ "$status" -eq 0 ] + [[ "$output" == *'"Ideas"'* ]] +} + +@test "graphql: --jq returning null normalizes to []" { + GH_STUB_STDOUT='{"data":{"repository":{"discussionCategories":{"nodes":null}}}}' \ + run gh_safe_graphql -f query='q' --jq '.data.repository.discussionCategories.nodes' + [ "$status" -eq 0 ] + [ "$output" = '[]' ] +} + +# --------------------------------------------------------------------------- +# gh_safe_graphql — failure modes (R1 + R7 kill list) +# --------------------------------------------------------------------------- + +@test "graphql: EXITS NON-ZERO on errors[] field present (HTTP 200)" { + GH_STUB_STDOUT='{"errors":[{"type":"FORBIDDEN","message":"Resource not accessible"}],"data":null}' \ + run gh_safe_graphql -f query='q' + [ "$status" -ne 0 ] +} + +@test "graphql: EXITS NON-ZERO on errors[] even when caller used --jq" { + GH_STUB_STDOUT='{"errors":[{"message":"bad"}],"data":null}' \ + run gh_safe_graphql -f query='q' --jq '.data.repository.id' + [ "$status" -ne 0 ] +} + +@test "graphql: EXITS NON-ZERO on data:null (no errors field)" { + GH_STUB_STDOUT='{"data":null}' \ + run gh_safe_graphql -f query='q' + [ "$status" -ne 0 ] +} + +@test "graphql: EXITS NON-ZERO on gh hard failure" { + GH_STUB_EXIT=1 \ + GH_STUB_STDERR='HTTP 401: Bad credentials' \ + run gh_safe_graphql -f query='q' + [ "$status" -ne 0 ] +} + +@test "graphql: EXITS NON-ZERO on non-JSON success output" { + GH_STUB_STDOUT='not json' \ + run gh_safe_graphql -f query='q' + [ "$status" -ne 0 ] +} diff --git a/test/workflows/feature-ideation/helpers/setup.bash b/test/workflows/feature-ideation/helpers/setup.bash new file mode 100644 index 0000000..7d8f92e --- /dev/null +++ b/test/workflows/feature-ideation/helpers/setup.bash @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Common test helpers for feature-ideation bats suites. + +# Repo root, regardless of where bats is invoked from. +TT_REPO_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../../../.." && pwd)" +export TT_REPO_ROOT + +TT_SCRIPTS_DIR="${TT_REPO_ROOT}/.github/scripts/feature-ideation" +export TT_SCRIPTS_DIR + +TT_FIXTURES_DIR="${TT_REPO_ROOT}/test/workflows/feature-ideation/fixtures" +export TT_FIXTURES_DIR + +TT_STUBS_DIR="${TT_REPO_ROOT}/test/workflows/feature-ideation/stubs" +export TT_STUBS_DIR + +# Per-test scratch dir, auto-cleaned by bats. +tt_make_tmpdir() { + TT_TMP="$(mktemp -d)" + export TT_TMP +} + +tt_cleanup_tmpdir() { + if [ -n "${TT_TMP:-}" ] && [ -d "${TT_TMP}" ]; then + rm -rf "${TT_TMP}" + fi +} + +# Install a fake `gh` binary on PATH for the duration of a test. +# Behavior is driven by env vars set per-test: +# GH_STUB_STDOUT — what the stub prints on stdout +# GH_STUB_STDERR — what the stub prints on stderr +# GH_STUB_EXIT — exit code (default 0) +# GH_STUB_LOG — file path to append the invocation argv to (optional) +tt_install_gh_stub() { + local stub_dir="${TT_TMP}/bin" + mkdir -p "$stub_dir" + cp "${TT_STUBS_DIR}/gh" "$stub_dir/gh" + chmod +x "$stub_dir/gh" + PATH="${stub_dir}:${PATH}" + export PATH +} + +# Convenience: load a fixture file by relative path. +tt_fixture() { + cat "${TT_FIXTURES_DIR}/$1" +} diff --git a/test/workflows/feature-ideation/lint-prompt.bats b/test/workflows/feature-ideation/lint-prompt.bats new file mode 100644 index 0000000..e195bb3 --- /dev/null +++ b/test/workflows/feature-ideation/lint-prompt.bats @@ -0,0 +1,122 @@ +#!/usr/bin/env bats +# Tests for lint-prompt.sh — kills R2 (unescaped shell expansions in +# direct_prompt blocks that YAML/claude-code-action will not interpolate). + +load 'helpers/setup' + +setup() { + tt_make_tmpdir +} + +teardown() { + tt_cleanup_tmpdir +} + +LINTER="${TT_REPO_ROOT}/.github/scripts/feature-ideation/lint-prompt.sh" + +write_yml() { + local path="$1" + cat >"$path" +} + +@test "lint-prompt: clean prompt passes" { + write_yml "${TT_TMP}/clean.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + direct_prompt: | + You are Mary. + The date is ${{ env.RUN_DATE }}. + Repo is ${{ github.repository }}. +YML + run bash "$LINTER" "${TT_TMP}/clean.yml" + [ "$status" -eq 0 ] +} + +@test "lint-prompt: FAILS on unescaped \$(date)" { + write_yml "${TT_TMP}/bad.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + direct_prompt: | + You are Mary. + Date: $(date -u +%Y-%m-%d) +YML + run bash "$LINTER" "${TT_TMP}/bad.yml" + [ "$status" -eq 1 ] +} + +@test "lint-prompt: FAILS on bare \${VAR}" { + write_yml "${TT_TMP}/bad2.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + direct_prompt: | + You are Mary. + Focus area: ${FOCUS_AREA} +YML + run bash "$LINTER" "${TT_TMP}/bad2.yml" + [ "$status" -eq 1 ] +} + +@test "lint-prompt: ALLOWS GitHub Actions expressions \${{ }}" { + write_yml "${TT_TMP}/gh-expr.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + direct_prompt: | + Repo: ${{ github.repository }} + Run: ${{ github.run_id }} + Inputs: ${{ inputs.focus_area || 'open' }} +YML + run bash "$LINTER" "${TT_TMP}/gh-expr.yml" + [ "$status" -eq 0 ] +} + +@test "lint-prompt: detects expansions only inside direct_prompt block" { + write_yml "${TT_TMP}/scoped.yml" <<'YML' +jobs: + build: + steps: + - run: | + echo "$(date)" # this is fine — it's a real shell + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + direct_prompt: | + This is $(unsafe). +YML + run bash "$LINTER" "${TT_TMP}/scoped.yml" + [ "$status" -eq 1 ] +} + +@test "lint-prompt: live feature-ideation-reusable.yml lints clean" { + # Contract: the live reusable workflow must never reintroduce unescaped + # shell expansions in the prompt block (kills R2). + workflow="${TT_REPO_ROOT}/.github/workflows/feature-ideation-reusable.yml" + [ -f "$workflow" ] + run bash "$LINTER" "$workflow" + [ "$status" -eq 0 ] +} + +@test "lint-prompt: standards caller-stub template lints clean" { + # The org-standard caller stub template that downstream repos copy. + workflow="${TT_REPO_ROOT}/standards/workflows/feature-ideation.yml" + [ -f "$workflow" ] + run bash "$LINTER" "$workflow" + [ "$status" -eq 0 ] +} + +@test "lint-prompt: scans every .github/workflows file by default" { + run bash "$LINTER" + [ "$status" -eq 0 ] +} diff --git a/test/workflows/feature-ideation/match-discussions.bats b/test/workflows/feature-ideation/match-discussions.bats new file mode 100644 index 0000000..792fdd5 --- /dev/null +++ b/test/workflows/feature-ideation/match-discussions.bats @@ -0,0 +1,181 @@ +#!/usr/bin/env bats +# Tests for match-discussions.sh — deterministic matching of proposals +# against existing Ideas Discussions. Kills R5 (fuzzy in-prompt matching) +# and R6 (idempotency hole on retry). + +bats_require_minimum_version 1.5.0 + +load 'helpers/setup' + +MATCH="${TT_REPO_ROOT}/.github/scripts/feature-ideation/match-discussions.sh" + +setup() { + tt_make_tmpdir +} + +teardown() { + tt_cleanup_tmpdir +} + +# Helper: build a signals.json with the given discussions array. +build_signals() { + local discussions="$1" + cat >"${TT_TMP}/signals.json" <<JSON +{ + "schema_version": "1.0.0", + "scan_date": "2026-04-07T00:00:00Z", + "repo": "petry-projects/talkterm", + "open_issues": { "count": 0, "items": [] }, + "closed_issues_30d": { "count": 0, "items": [] }, + "ideas_discussions": { "count": 0, "items": ${discussions} }, + "releases": [], + "merged_prs_30d": { "count": 0, "items": [] }, + "feature_requests": { "count": 0, "items": [] }, + "bug_reports": { "count": 0, "items": [] }, + "truncation_warnings": [] +} +JSON +} + +build_proposals() { + printf '%s' "$1" >"${TT_TMP}/proposals.json" +} + +@test "match: identical title produces a match" { + build_signals '[{"id":"D_1","number":1,"title":"💡 Streaming voice responses"}]' + build_proposals '[{"title":"💡 Streaming voice responses","summary":"x"}]' + + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + [ "$matched" = "1" ] + new=$(printf '%s' "$output" | jq '.new_candidates | length') + [ "$new" = "0" ] +} + +@test "match: emoji difference does not break a match" { + build_signals '[{"id":"D_1","number":1,"title":"🎤 Streaming voice responses"}]' + build_proposals '[{"title":"💡 Streaming voice responses"}]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + [ "$matched" = "1" ] +} + +@test "match: case and stopword differences still match" { + build_signals '[{"id":"D_1","number":1,"title":"Add support for streaming voice responses"}]' + build_proposals '[{"title":"Streaming Voice Responses"}]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + [ "$matched" = "1" ] +} + +@test "match: unrelated proposal is a new candidate" { + build_signals '[{"id":"D_1","number":1,"title":"💡 Streaming voice responses"}]' + build_proposals '[{"title":"Dark mode toggle in settings"}]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + new=$(printf '%s' "$output" | jq '.new_candidates | length') + [ "$matched" = "0" ] + [ "$new" = "1" ] +} + +@test "match: empty existing discussions yields all candidates as new" { + build_signals '[]' + build_proposals '[{"title":"A"},{"title":"B"},{"title":"C"}]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + new=$(printf '%s' "$output" | jq '.new_candidates | length') + [ "$new" = "3" ] +} + +@test "match: each existing discussion only matches one proposal" { + # Two near-identical proposals against one existing discussion. + # Only the FIRST should match; the second goes to new_candidates. + build_signals '[{"id":"D_1","number":1,"title":"Streaming voice responses"}]' + build_proposals '[{"title":"Streaming voice responses"},{"title":"Streaming voice responses"}]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + new=$(printf '%s' "$output" | jq '.new_candidates | length') + [ "$matched" = "1" ] + [ "$new" = "1" ] +} + +@test "match: result includes similarity score per match" { + build_signals '[{"id":"D_1","number":1,"title":"Streaming voice responses"}]' + build_proposals '[{"title":"Streaming voice responses"}]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + sim=$(printf '%s' "$output" | jq '.matched[0].similarity') + # Identical normalized titles → similarity 1.0 + awk -v s="$sim" 'BEGIN{exit !(s>=0.99)}' +} + +@test "match: threshold env var is respected" { + build_signals '[{"id":"D_1","number":1,"title":"voice"}]' + build_proposals '[{"title":"voice streaming responses"}]' + # At default threshold 0.6 these have Jaccard ~0.33, so no match. + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + [ "$matched" = "0" ] + # Lower the threshold and they should match. + MATCH_THRESHOLD=0.2 run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + [ "$matched" = "1" ] +} + +@test "match: missing signals file causes usage error" { + build_proposals '[{"title":"x"}]' + run bash "$MATCH" "${TT_TMP}/no-such-file.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 64 ] +} + +@test "match: missing proposals file causes usage error" { + build_signals '[]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/no-such-file.json" + [ "$status" -eq 64 ] +} + +@test "match: malformed proposal entries are skipped, valid ones processed" { + build_signals '[]' + build_proposals '[{"title":"valid"},"garbage",{"no_title":"x"},{"title":"also valid"}]' + run --separate-stderr bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + new=$(printf '%s' "$output" | jq '.new_candidates | length') + [ "$new" = "2" ] +} + +@test "match: proposals as empty array yields empty result" { + build_signals '[{"id":"D_1","number":1,"title":"x"}]' + build_proposals '[]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + new=$(printf '%s' "$output" | jq '.new_candidates | length') + [ "$matched" = "0" ] + [ "$new" = "0" ] +} + +@test "match: idempotent re-run of same proposals against existing matches yields all-matched" { + # Simulates the R6 idempotency case: run 1 created Discussions, run 2 + # finds them and should NOT propose duplicates. + build_signals '[ + {"id":"D_1","number":1,"title":"💡 Streaming voice responses"}, + {"id":"D_2","number":2,"title":"💡 Dark mode"} + ]' + build_proposals '[ + {"title":"💡 Streaming voice responses"}, + {"title":"💡 Dark mode"} + ]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + matched=$(printf '%s' "$output" | jq '.matched | length') + new=$(printf '%s' "$output" | jq '.new_candidates | length') + [ "$matched" = "2" ] + [ "$new" = "0" ] +} diff --git a/test/workflows/feature-ideation/signals-schema.bats b/test/workflows/feature-ideation/signals-schema.bats new file mode 100644 index 0000000..a416b8f --- /dev/null +++ b/test/workflows/feature-ideation/signals-schema.bats @@ -0,0 +1,52 @@ +#!/usr/bin/env bats +# Validates fixture signals.json files against the schema. +# +# Kills R3: ensures the producer/consumer contract is enforced in CI +# rather than discovered when Mary's prompt parses an unexpected shape. + +load 'helpers/setup' + +VALIDATOR="${TT_REPO_ROOT}/.github/scripts/feature-ideation/validate-signals.py" +SCHEMA="${TT_REPO_ROOT}/.github/schemas/signals.schema.json" +FIX="${TT_FIXTURES_DIR}/expected" + +@test "schema: validator script exists and is executable" { + [ -f "$VALIDATOR" ] + [ -r "$VALIDATOR" ] +} + +@test "schema: schema file is valid JSON" { + jq -e . "$SCHEMA" >/dev/null +} + +@test "schema: empty-repo fixture passes" { + run python3 "$VALIDATOR" "${FIX}/empty-repo.signals.json" "$SCHEMA" + [ "$status" -eq 0 ] +} + +@test "schema: populated fixture passes" { + run python3 "$VALIDATOR" "${FIX}/populated.signals.json" "$SCHEMA" + [ "$status" -eq 0 ] +} + +@test "schema: truncated fixture passes" { + run python3 "$VALIDATOR" "${FIX}/truncated.signals.json" "$SCHEMA" + [ "$status" -eq 0 ] +} + +@test "schema: missing required field FAILS validation" { + run python3 "$VALIDATOR" "${FIX}/INVALID-missing-field.signals.json" "$SCHEMA" + [ "$status" -eq 1 ] +} + +@test "schema: malformed repo string FAILS validation" { + run python3 "$VALIDATOR" "${FIX}/INVALID-bad-repo.signals.json" "$SCHEMA" + [ "$status" -eq 1 ] +} + +@test "schema: extra top-level field FAILS validation" { + bad_file="${BATS_TEST_TMPDIR}/extra-field.json" + jq '. + {unexpected: "value"}' "${FIX}/empty-repo.signals.json" >"$bad_file" + run python3 "$VALIDATOR" "$bad_file" "$SCHEMA" + [ "$status" -eq 1 ] +} diff --git a/test/workflows/feature-ideation/stubs/gh b/test/workflows/feature-ideation/stubs/gh new file mode 100755 index 0000000..158b665 --- /dev/null +++ b/test/workflows/feature-ideation/stubs/gh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# Fake `gh` binary used by feature-ideation bats tests. +# +# Behavior is fully driven by env vars so individual tests can script +# arbitrary success / failure / paginated scenarios without per-fixture +# stub variants. +# +# Recognized vars: +# GH_STUB_STDOUT — literal text or @file:/abs/path to load from disk +# GH_STUB_STDERR — literal text or @file:/abs/path +# GH_STUB_EXIT — exit code (default 0) +# GH_STUB_LOG — append-only log of argv invocations (one TSV line each) +# +# For multi-call tests (where the same stub is invoked multiple times with +# different responses), set GH_STUB_SCRIPT to a file containing one +# `EXIT\tSTDOUT_PATH\tSTDERR_PATH` line per expected call. The stub +# advances a counter file in $TT_TMP/.gh-stub-counter on each invocation. + +set -euo pipefail + +if [ -n "${GH_STUB_LOG:-}" ]; then + printf '%s\n' "$*" >>"$GH_STUB_LOG" +fi + +# Multi-call mode. +if [ -n "${GH_STUB_SCRIPT:-}" ] && [ -f "${GH_STUB_SCRIPT}" ]; then + counter_file="${TT_TMP:-/tmp}/.gh-stub-counter" + count=0 + if [ -f "$counter_file" ]; then + count=$(cat "$counter_file") + fi + next=$((count + 1)) + printf '%s' "$next" >"$counter_file" + + line=$(sed -n "${next}p" "$GH_STUB_SCRIPT") + if [ -z "$line" ]; then + printf '[gh-stub] no script entry for call #%d\n' "$next" >&2 + exit 99 + fi + exit_code=$(printf '%s' "$line" | awk -F'\t' '{print $1}') + stdout_path=$(printf '%s' "$line" | awk -F'\t' '{print $2}') + stderr_path=$(printf '%s' "$line" | awk -F'\t' '{print $3}') + + if [ -n "$stdout_path" ] && [ "$stdout_path" != "-" ]; then + cat "$stdout_path" + fi + if [ -n "$stderr_path" ] && [ "$stderr_path" != "-" ]; then + cat "$stderr_path" >&2 + fi + exit "$exit_code" +fi + +# Single-call mode. +emit() { + local var="$1" + local stream="$2" + local val="${!var:-}" + if [ -z "$val" ]; then + return 0 + fi + if [[ "$val" == @file:* ]]; then + cat "${val#@file:}" >&"$stream" + else + printf '%s' "$val" >&"$stream" + fi +} + +emit GH_STUB_STDOUT 1 +emit GH_STUB_STDERR 2 + +exit "${GH_STUB_EXIT:-0}" From 12fee6ebed3d02df1ae799ded551cffcbfe67c4b Mon Sep 17 00:00:00 2001 From: DJ <dj@Rachels-MacBook-Air.local> Date: Tue, 7 Apr 2026 10:50:58 -0700 Subject: [PATCH 2/6] test(feature-ideation): use bash -c instead of sh -c in env-extension test CI failure on the previous commit: 91/92 passing, 1 failing. The filter-bots env-extension test used `sh -c` to source filter-bots.sh in a sub-shell with FEATURE_IDEATION_BOT_AUTHORS set. On macOS this works because /bin/sh is bash. On Ubuntu (CI), /bin/sh is dash, which does not support `set -o pipefail`, so sourcing filter-bots.sh produced: sh: 12: set: Illegal option -o pipefail Fixed by switching to `bash -c`. All scripts already use `#!/usr/bin/env bash` shebangs; this is the only place a sub-shell was spawned via `sh`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- test/workflows/feature-ideation/filter-bots.bats | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/test/workflows/feature-ideation/filter-bots.bats b/test/workflows/feature-ideation/filter-bots.bats index 6f4e3ff..78376b7 100644 --- a/test/workflows/feature-ideation/filter-bots.bats +++ b/test/workflows/feature-ideation/filter-bots.bats @@ -45,8 +45,11 @@ JSON } @test "filter-bots: env extension adds custom bot logins" { + # NOTE: must use `bash -c`, not `sh -c`. On Ubuntu, /bin/sh is dash, which + # doesn't support `set -o pipefail` and would choke when sourcing a script + # that uses bash-isms. macOS /bin/sh is bash so it works there but CI fails. input='[{"number":1,"title":"x","author":{"login":"my-custom-bot"}},{"number":2,"title":"y","author":{"login":"alice"}}]' - result=$(FEATURE_IDEATION_BOT_AUTHORS="my-custom-bot" sh -c " + result=$(FEATURE_IDEATION_BOT_AUTHORS="my-custom-bot" bash -c " . '${TT_SCRIPTS_DIR}/lib/filter-bots.sh' printf '%s' '$input' | filter_bots_apply ") From 43bcf963181c3e6a2bb84ef929c08535b9a312a2 Mon Sep 17 00:00:00 2001 From: DJ <dj@Rachels-MacBook-Air.local> Date: Wed, 8 Apr 2026 05:05:21 -0700 Subject: [PATCH 3/6] fix(feature-ideation): default tooling_ref to v1 to match @v1 caller pin Aligns the script-tooling self-checkout with the @v1 pinning convention introduced in #88. Now when a downstream caller stub pins to `@v1` of the workflow file, the reusable workflow defaults to checking out the matching `v1` tag for the scripts. Workflow file and scripts upgrade in lockstep. Override `tooling_ref` only for testing forks (`tooling_ref: my-branch`) or bleeding-edge testing (`tooling_ref: main`). Documented in the input description. Note for the v1 tag move: after this PR merges, the v1 tag must be moved forward to point to the new HEAD so that downstream BMAD repos pinned to @v1 actually pick up the hardening. The change is purely additive (new optional inputs `dry_run` and `tooling_ref`, new env vars in the prompt context), so the move is backwards-compatible. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- .github/workflows/feature-ideation-reusable.yml | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/.github/workflows/feature-ideation-reusable.yml b/.github/workflows/feature-ideation-reusable.yml index 6c49a7a..9730849 100644 --- a/.github/workflows/feature-ideation-reusable.yml +++ b/.github/workflows/feature-ideation-reusable.yml @@ -63,9 +63,18 @@ on: default: false type: boolean tooling_ref: - description: 'Ref of petry-projects/.github to source the feature-ideation scripts from. Defaults to main; override only for testing forks.' + description: | + Ref of petry-projects/.github to source the feature-ideation scripts from. + Defaults to `v1` to align with the @v1 pin convention used by caller stubs + (see standards/ci-standards.md tier model). When the v1 tag is moved + forward to a new release, callers automatically pick up the matching + script + schema versions in lockstep with the workflow file. + + Override only for: + - Testing a fork or PR branch end-to-end (`tooling_ref: my-feature-branch`) + - Bleeding-edge testing against the latest main (`tooling_ref: main`) required: false - default: 'main' + default: 'v1' type: string secrets: CLAUDE_CODE_OAUTH_TOKEN: From bcaa579f3c878cdd7ac34c5d5bea76c9e515fbf6 Mon Sep 17 00:00:00 2001 From: DJ <dj@Rachels-MacBook-Air.local> Date: Wed, 8 Apr 2026 05:17:56 -0700 Subject: [PATCH 4/6] fix(feature-ideation): address Copilot review on PR #85 (11 fixes + 16 tests) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Triaged 14 inline comments from Copilot's review of #85; two were already fixed by the tooling_ref→v1 commit, the remaining 11 are addressed here. Critical bug fixes ------------------ 1. lint-prompt.sh now scans claude-code-action v1 `prompt:` blocks in addition to v0 `direct_prompt:`. The reusable workflow uses `prompt:` so the linter was silently allowing R2 regressions on the very file it was supposed to protect. Added two regression tests covering both the v1 form and a clean v1 form passes. 2. add_label_to_discussion now sends labelIds as a proper JSON array via gh_safe_graphql_input (new helper). Previously used `gh -f labelIds=` which sent the literal string `["L_1"]` and the GraphQL API would have rejected the mutation at runtime. Added a test that captures gh's stdin and asserts the variables block contains a length-1 array. 3. validate-signals.py now registers a `date-time` format checker via FormatChecker so the `format: date-time` keyword in signals.schema.json is actually enforced. Draft202012Validator does NOT enforce formats by default, and the default FormatChecker omits date-time entirely. Used an inline checker (datetime.fromisoformat with Z normalisation) to avoid pulling in rfc3339-validator. Added two regression tests: one for an invalid timestamp failing, one for a clean timestamp passing. 4. gh_safe_graphql --jq path no longer swallows jq filter errors with `|| true`. Filter typos / wrong paths now exit non-zero instead of silently returning []. Added a regression test using a deliberately broken filter. 5. collect-signals.sh now computes the open-issue truncation warning BEFORE filter_bots_apply. Previously, a result set composed entirely of bots could drop below ISSUE_LIMIT after filtering and mask real truncation. Added an integration test with all-bot fixtures. 6. match-discussions.sh now validates MATCH_THRESHOLD as a non-negative number in [0, 1] before passing to Python. A typo previously surfaced as an opaque traceback. Added regression tests for non-numeric input, out-of-range input, and boundary values 0 and 1. Cleanup ------- 7. Removed dead bash `normalize_title` / `jaccard_similarity` functions from match-discussions.sh — the actual matching is implemented in the embedded Python block and the bash helpers were never called. 8. Schema $id corrected from petry-projects/TalkTerm/... to the canonical petry-projects/.github location. 9. signals-schema.bats "validator script exists and is executable" test now actually checks the `-x` bit (was only checking `-f` and `-r`). 10. README + filter-bots.sh comments now describe the bot list as a "blocklist" (it removes matching authors) instead of "allowlist". 11. test/workflows/feature-ideation/stubs/gh now logs argv with `printf '%q '` so each invocation is shell-quoted and re-parseable, matching its documentation. Previously logged `$*` which lost arg boundaries. New helper ---------- gh_safe_graphql_input — same defensive contract as gh_safe_graphql, but takes a fully-formed JSON request body via stdin instead of -f/-F flags. Use for mutations whose variables include arrays (e.g. labelIds: [ID!]!) that gh's flag-based interface cannot express. Five new tests cover its happy path and every documented failure mode. Tests ----- Test count: 92 → 108 (16 new regression tests, all green). Run with: bats test/workflows/feature-ideation/ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- .github/schemas/signals.schema.json | 2 +- .github/scripts/feature-ideation/README.md | 2 +- .../feature-ideation/collect-signals.sh | 13 ++-- .../feature-ideation/discussion-mutations.sh | 13 +++- .../feature-ideation/lib/filter-bots.sh | 4 +- .../scripts/feature-ideation/lib/gh-safe.sh | 72 +++++++++++++++++-- .../scripts/feature-ideation/lint-prompt.sh | 12 ++-- .../feature-ideation/match-discussions.sh | 53 +++++--------- .../feature-ideation/validate-signals.py | 29 +++++++- .../feature-ideation/collect-signals.bats | 36 ++++++++++ .../discussion-mutations.bats | 32 +++++++++ test/workflows/feature-ideation/gh-safe.bats | 55 ++++++++++++++ .../feature-ideation/lint-prompt.bats | 38 ++++++++++ .../feature-ideation/match-discussions.bats | 25 +++++++ .../feature-ideation/signals-schema.bats | 18 +++++ test/workflows/feature-ideation/stubs/gh | 12 +++- 16 files changed, 357 insertions(+), 59 deletions(-) diff --git a/.github/schemas/signals.schema.json b/.github/schemas/signals.schema.json index 51d5f65..9a1ad07 100644 --- a/.github/schemas/signals.schema.json +++ b/.github/schemas/signals.schema.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://github.com/petry-projects/TalkTerm/.github/schemas/signals.schema.json", + "$id": "https://github.com/petry-projects/.github/blob/main/.github/schemas/signals.schema.json", "title": "Feature Ideation Signals", "description": "Canonical contract between collect-signals.sh and the BMAD Analyst (Mary) prompt. Any change to this schema is a breaking change to the workflow.", "type": "object", diff --git a/.github/scripts/feature-ideation/README.md b/.github/scripts/feature-ideation/README.md index 9ef03b9..957a06a 100644 --- a/.github/scripts/feature-ideation/README.md +++ b/.github/scripts/feature-ideation/README.md @@ -23,7 +23,7 @@ caught **before UAT** instead of after. |------|---------|------------| | `lib/gh-safe.sh` | Wraps every `gh` and `gh api graphql` call. Fails loud on auth, rate-limit, network, GraphQL errors envelope, or `data: null`. Replaces the original `2>/dev/null \|\| echo '[]'` swallow. | R1, R7, R8 | | `lib/compose-signals.sh` | Validates JSON inputs before `jq --argjson` and assembles the canonical signals.json document. | R3, R4 | -| `lib/filter-bots.sh` | Configurable bot allowlist via `FEATURE_IDEATION_BOT_AUTHORS`. | R10 | +| `lib/filter-bots.sh` | Configurable bot blocklist via `FEATURE_IDEATION_BOT_AUTHORS` (extends the default list of bot logins to remove from results). | R10 | | `lib/date-utils.sh` | Cross-platform date arithmetic helpers. | R9 | | `collect-signals.sh` | Orchestrator: drives all `gh` calls, composes signals.json, emits truncation warnings. | R1, R3, R4, R11 | | `validate-signals.py` | JSON Schema 2020-12 validator for signals.json against `../schemas/signals.schema.json`. | R3 | diff --git a/.github/scripts/feature-ideation/collect-signals.sh b/.github/scripts/feature-ideation/collect-signals.sh index cf51229..69f605c 100755 --- a/.github/scripts/feature-ideation/collect-signals.sh +++ b/.github/scripts/feature-ideation/collect-signals.sh @@ -72,15 +72,20 @@ main() { local open_issues_raw open_issues_raw=$(gh_safe_rest issue list --repo "$REPO" --state open --limit "$issue_limit" \ --json number,title,labels,createdAt,author) - local open_issues - open_issues=$(printf '%s' "$open_issues_raw" | filter_bots_apply) - - if [ "$(printf '%s' "$open_issues" | jq 'length')" -ge "$issue_limit" ]; then + # Compute truncation warning BEFORE bot filtering. If we filter out bots + # first, the count could drop below the limit and mask a real truncation. + # Caught by Copilot review on PR petry-projects/.github#85. + local raw_open_count + raw_open_count=$(printf '%s' "$open_issues_raw" | jq 'length') + if [ "$raw_open_count" -ge "$issue_limit" ]; then truncation_warnings=$(printf '%s' "$truncation_warnings" \ | jq --arg src "open_issues" --argjson lim "$issue_limit" \ '. + [{source: $src, limit: $lim, message: "result count equals limit; possible truncation"}]') fi + local open_issues + open_issues=$(printf '%s' "$open_issues_raw" | filter_bots_apply) + # --- Recently closed issues ------------------------------------------------ printf '[collect-signals] fetching closed issues (since %s)\n' "$thirty_days_ago" >&2 local closed_issues_raw diff --git a/.github/scripts/feature-ideation/discussion-mutations.sh b/.github/scripts/feature-ideation/discussion-mutations.sh index 843d9d6..ae7a8dc 100755 --- a/.github/scripts/feature-ideation/discussion-mutations.sh +++ b/.github/scripts/feature-ideation/discussion-mutations.sh @@ -153,7 +153,14 @@ mutation($labelableId: ID!, $labelIds: [ID!]!) { } GRAPHQL - gh_safe_graphql -f query="$query" \ - -f labelableId="$discussion_id" \ - -f labelIds="[\"$label_id\"]" + # GraphQL `labelIds: [ID!]!` requires a JSON array variable. `gh api`'s + # `-f`/`-F` flags don't express GraphQL array variables, so build the + # full request body as JSON and send it via stdin. Caught by Copilot + # review on PR petry-projects/.github#85: the previous version sent + # labelIds as the literal string `["L_1"]`, which the API rejects. + local body + body=$(jq -nc --arg q "$query" --arg id "$discussion_id" --arg label "$label_id" \ + '{query: $q, variables: {labelableId: $id, labelIds: [$label]}}') + + gh_safe_graphql_input "$body" } diff --git a/.github/scripts/feature-ideation/lib/filter-bots.sh b/.github/scripts/feature-ideation/lib/filter-bots.sh index 95aabfe..c3b38be 100755 --- a/.github/scripts/feature-ideation/lib/filter-bots.sh +++ b/.github/scripts/feature-ideation/lib/filter-bots.sh @@ -7,7 +7,9 @@ # pollute the signals payload and crowd out real user feedback. # # Configurable via FEATURE_IDEATION_BOT_AUTHORS (comma-separated, optional). -# Defaults to a sensible allowlist of known automation accounts. +# Defaults to a sensible blocklist of known automation accounts. Items +# matching this list are REMOVED from the result. Adopters can add their +# own bot logins via the env var to extend the blocklist. set -euo pipefail diff --git a/.github/scripts/feature-ideation/lib/gh-safe.sh b/.github/scripts/feature-ideation/lib/gh-safe.sh index 7bbdf2c..343a5bb 100755 --- a/.github/scripts/feature-ideation/lib/gh-safe.sh +++ b/.github/scripts/feature-ideation/lib/gh-safe.sh @@ -150,11 +150,25 @@ gh_safe_graphql() { # If caller asked for a jq filter, apply it now and return that. if [ "$has_jq" -eq 1 ]; then - local filtered - filtered=$(printf '%s' "$raw" | jq -c "$jq_filter" 2>/dev/null || true) - if [ -z "$filtered" ] || [ "$filtered" = "null" ]; then - # The filter resolved to null/empty — caller probably wants "[]" semantics - # for "no nodes found". Return the empty array sentinel. + # NB: do NOT swallow jq errors with `|| true`. A typo or wrong path in + # the filter must surface as a hard failure, not a silent empty result — + # otherwise we re-introduce R1 in a different shape. Caught by Copilot + # review on PR petry-projects/.github#85. + local filtered jq_rc jq_err + jq_err=$(mktemp) + set +e + filtered=$(printf '%s' "$raw" | jq -c "$jq_filter" 2>"$jq_err") + jq_rc=$? + set -e + if [ "$jq_rc" -ne 0 ]; then + _gh_safe_err "graphql-jq-failed" "filter='$jq_filter' err=$(cat "$jq_err")" + rm -f "$jq_err" + return 65 + fi + rm -f "$jq_err" + if [ "$filtered" = "null" ] || [ -z "$filtered" ]; then + # Filter resolved to JSON null (e.g. nodes path returned null) — this + # is the documented "no results" case. Normalize to empty array. printf '%s' "$GH_SAFE_EMPTY_ARRAY" return 0 fi @@ -164,3 +178,51 @@ gh_safe_graphql() { printf '%s' "$raw" } + +# Send a fully-formed GraphQL request body via stdin. Use this when the +# variables include arrays or other shapes that `gh api`'s -f/-F flags +# cannot express (e.g. `labelIds: [ID!]!`). The body must be a JSON +# document with `query` and `variables` top-level fields. +# +# Same defensive contract as `gh_safe_graphql`: any auth/network/schema +# failure exits non-zero with a structured stderr message. +gh_safe_graphql_input() { + local body="$1" + if ! gh_safe_is_json "$body"; then + _gh_safe_err "graphql-bad-input" "request body is not valid JSON" + return 64 + fi + + local raw stderr rc tmp_err + tmp_err=$(mktemp) + set +e + raw=$(printf '%s' "$body" | gh api graphql --input - 2>"$tmp_err") + rc=$? + set -e + stderr=$(cat "$tmp_err") + rm -f "$tmp_err" + + if [ "$rc" -ne 0 ]; then + _gh_safe_err "graphql-failure" "exit=$rc stderr=$stderr" + return "$rc" + fi + + if ! gh_safe_is_json "$raw"; then + _gh_safe_err "graphql-bad-json" "first 200 bytes: ${raw:0:200}" + return 65 + fi + + if printf '%s' "$raw" | jq -e '(.errors // empty | type) == "array" and (.errors | length > 0)' >/dev/null 2>&1; then + local errs + errs=$(printf '%s' "$raw" | jq -c '.errors') + _gh_safe_err "graphql-errors" "$errs" + return 66 + fi + + if printf '%s' "$raw" | jq -e '.data == null' >/dev/null 2>&1; then + _gh_safe_err "graphql-null-data" "data field is null" + return 66 + fi + + printf '%s' "$raw" +} diff --git a/.github/scripts/feature-ideation/lint-prompt.sh b/.github/scripts/feature-ideation/lint-prompt.sh index 8381b61..3b1aebc 100755 --- a/.github/scripts/feature-ideation/lint-prompt.sh +++ b/.github/scripts/feature-ideation/lint-prompt.sh @@ -39,8 +39,9 @@ except OSError as exc: sys.stderr.write(f"[lint-prompt] cannot read {path}: {exc}\n") sys.exit(2) -# Find direct_prompt: blocks. We treat everything indented MORE than the -# `direct_prompt:` line as part of that block, until we hit a less-indented +# Find prompt blocks. claude-code-action v0 used `direct_prompt:`, v1 uses +# plain `prompt:`. Both forms are scanned. We treat everything indented MORE +# than the marker line as part of the block, until we hit a less-indented # non-blank line. in_block = False block_indent = -1 @@ -50,13 +51,16 @@ findings = [] # because that's evaluated before the prompt is rendered. shell_expansion = re.compile(r'(?<!\\)\$\([^)]*\)|(?<!\$)\$\{[A-Za-z_][A-Za-z0-9_]*\}') +# Recognise both `direct_prompt:` (v0) and `prompt:` (v1) markers, with +# optional `|` or `>` block scalar indicators. +prompt_marker = re.compile(r'(?:direct_prompt|prompt):\s*[|>]?\s*$') + for lineno, raw in enumerate(lines, start=1): stripped = raw.lstrip(" ") indent = len(raw) - len(stripped) if not in_block: - # Look for `direct_prompt:` or `direct_prompt: |` or `direct_prompt: >` - if re.match(r'direct_prompt:\s*[|>]?\s*$', stripped): + if prompt_marker.match(stripped): in_block = True block_indent = indent continue diff --git a/.github/scripts/feature-ideation/match-discussions.sh b/.github/scripts/feature-ideation/match-discussions.sh index 826f29b..d8bf766 100755 --- a/.github/scripts/feature-ideation/match-discussions.sh +++ b/.github/scripts/feature-ideation/match-discussions.sh @@ -31,48 +31,29 @@ set -euo pipefail -normalize_title() { - python3 -c ' -import re, sys, unicodedata -raw = sys.argv[1] -# Strip emoji and other symbol/other categories. -no_emoji = "".join(ch for ch in raw if unicodedata.category(ch)[0] not in ("S",)) -# Lowercase + ascii fold. -ascii_text = unicodedata.normalize("NFKD", no_emoji).encode("ascii", "ignore").decode() -ascii_text = ascii_text.lower() -# Replace non-alphanumeric with space. -collapsed = re.sub(r"[^a-z0-9]+", " ", ascii_text).strip() -# Drop stopwords. -stopwords = {"a","an","the","of","for","to","and","or","with","in","on","by","via","as","is","are","be","support","add","new","feature","idea"} -tokens = [t for t in collapsed.split() if t and t not in stopwords] -print(" ".join(tokens)) -' "$1" -} -export -f normalize_title - -# Compute Jaccard similarity between two normalized token strings. -jaccard_similarity() { - python3 -c ' -import sys -a = set(sys.argv[1].split()) -b = set(sys.argv[2].split()) -if not a and not b: - print("1.0") -elif not a or not b: - print("0.0") -else: - inter = len(a & b) - union = len(a | b) - print(f"{inter / union:.4f}") -' "$1" "$2" -} -export -f jaccard_similarity +# NB: matching logic is implemented entirely in the embedded Python block +# below. Earlier drafts had standalone bash `normalize_title` / +# `jaccard_similarity` helpers that were never called; removed per Copilot +# review on PR petry-projects/.github#85. match_discussions_main() { local signals_path="$1" local proposals_path="$2" local threshold="${MATCH_THRESHOLD:-0.6}" + # Validate threshold is a parseable float in [0, 1] BEFORE handing it to + # Python, so a typo doesn't surface as an opaque traceback. Caught by + # Copilot review on PR petry-projects/.github#85. + if ! printf '%s' "$threshold" | grep -Eq '^[0-9]+(\.[0-9]+)?$'; then + printf '[match-discussions] MATCH_THRESHOLD must be a non-negative number, got: %s\n' "$threshold" >&2 + return 64 + fi + # Validate range using awk (portable, no bash float math). + if ! awk -v t="$threshold" 'BEGIN { exit !(t >= 0 && t <= 1) }'; then + printf '[match-discussions] MATCH_THRESHOLD must be in [0, 1], got: %s\n' "$threshold" >&2 + return 64 + fi + if [ ! -f "$signals_path" ]; then printf '[match-discussions] signals not found: %s\n' "$signals_path" >&2 return 64 diff --git a/.github/scripts/feature-ideation/validate-signals.py b/.github/scripts/feature-ideation/validate-signals.py index b71a1bd..69ff0c1 100755 --- a/.github/scripts/feature-ideation/validate-signals.py +++ b/.github/scripts/feature-ideation/validate-signals.py @@ -13,11 +13,13 @@ from __future__ import annotations import json +import re import sys +from datetime import datetime from pathlib import Path try: - from jsonschema import Draft202012Validator + from jsonschema import Draft202012Validator, FormatChecker except ImportError: sys.stderr.write( "[validate-signals] python jsonschema not installed. " @@ -62,7 +64,30 @@ def main(argv: list[str]) -> int: sys.stderr.write(f"[validate-signals] invalid schema JSON: {exc}\n") return 2 - validator = Draft202012Validator(schema) + # `format` keywords (e.g. "format": "date-time") are not enforced by + # Draft202012Validator unless an explicit FormatChecker is supplied. + # The default FormatChecker also does NOT include `date-time` — that + # requires the optional `rfc3339-validator` dependency. Avoid pulling + # in extra deps by registering a small inline validator that accepts + # ISO-8601 / RFC-3339 strings via datetime.fromisoformat (Python 3.11+ + # accepts the trailing `Z`; for older interpreters we substitute it). + # Caught by Copilot review on PR petry-projects/.github#85. + format_checker = FormatChecker() + + @format_checker.checks("date-time", raises=(ValueError, TypeError)) + def _check_date_time(instance): # noqa: ANN001 — jsonschema callback signature + if not isinstance(instance, str): + return True # non-strings handled by `type` keyword, not format + # Must look like a date-time, not just any string. Require at least + # YYYY-MM-DDTHH:MM[:SS] + if not re.match(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}", instance): + raise ValueError(f"not an ISO-8601 date-time: {instance!r}") + # datetime.fromisoformat in Python <3.11 doesn't accept "Z"; swap it. + candidate = instance.replace("Z", "+00:00") if instance.endswith("Z") else instance + datetime.fromisoformat(candidate) + return True + + validator = Draft202012Validator(schema, format_checker=format_checker) errors = sorted(validator.iter_errors(signals), key=lambda e: list(e.absolute_path)) if not errors: print(f"[validate-signals] OK: {signals_path}") diff --git a/test/workflows/feature-ideation/collect-signals.bats b/test/workflows/feature-ideation/collect-signals.bats index 9543b97..94f9a12 100644 --- a/test/workflows/feature-ideation/collect-signals.bats +++ b/test/workflows/feature-ideation/collect-signals.bats @@ -163,6 +163,42 @@ build_happy_script() { # Truncation warnings # --------------------------------------------------------------------------- +@test "collect-signals: open-issue truncation is detected BEFORE bot filtering" { + # Caught by Copilot review on PR petry-projects/.github#85: previously the + # truncation check ran AFTER filter_bots_apply, so a result set composed + # entirely of bots could drop below ISSUE_LIMIT and mask real truncation. + # Build a fixture with ISSUE_LIMIT=3 raw items, all bot-authored. Bot + # filter strips them to 0, but the truncation warning must still fire + # because the raw count hit the limit. + bot_file="${TT_TMP}/bot-only.json" + cat >"$bot_file" <<'JSON' +[ + {"number":1,"title":"a","labels":[],"createdAt":"2026-03-01T00:00:00Z","author":{"login":"dependabot[bot]"}}, + {"number":2,"title":"b","labels":[],"createdAt":"2026-03-02T00:00:00Z","author":{"login":"renovate[bot]"}}, + {"number":3,"title":"c","labels":[],"createdAt":"2026-03-03T00:00:00Z","author":{"login":"copilot[bot]"}} +] +JSON + empty_file="${TT_TMP}/empty.json" + echo '[]' >"$empty_file" + + script="${TT_TMP}/gh-script.tsv" + : >"$script" + printf '0\t%s\t-\n' "$bot_file" >>"$script" # open issues — all bots + printf '0\t%s\t-\n' "$empty_file" >>"$script" # closed issues + printf '0\t%s\t-\n' "${TT_FIXTURES_DIR}/gh-responses/graphql-no-ideas-category.json" >>"$script" + printf '0\t%s\t-\n' "$empty_file" >>"$script" # releases + printf '0\t%s\t-\n' "$empty_file" >>"$script" # merged PRs + export GH_STUB_SCRIPT="$script" + export ISSUE_LIMIT=3 + rm -f "${TT_TMP}/.gh-stub-counter" + + run bash "${TT_SCRIPTS_DIR}/collect-signals.sh" + [ "$status" -eq 0 ] + open_count=$(jq '.open_issues.count' "$SIGNALS_OUTPUT") + [ "$open_count" = "0" ] + jq -e '.truncation_warnings[] | select(.source == "open_issues")' "$SIGNALS_OUTPUT" >/dev/null +} + @test "collect-signals: emits truncation warning when discussions hasNextPage=true" { script="${TT_TMP}/gh-script.tsv" : >"$script" diff --git a/test/workflows/feature-ideation/discussion-mutations.bats b/test/workflows/feature-ideation/discussion-mutations.bats index b42dabe..671e9b3 100644 --- a/test/workflows/feature-ideation/discussion-mutations.bats +++ b/test/workflows/feature-ideation/discussion-mutations.bats @@ -100,3 +100,35 @@ teardown() { run comment_on_discussion "D_1" "body" [ "$status" -ne 0 ] } + +# --------------------------------------------------------------------------- +# Regression test for Copilot review on PR petry-projects/.github#85: +# add_label_to_discussion previously sent labelIds via -f as the literal +# string `["L_1"]` which the GraphQL API rejects (labelIds is [ID!]!). +# The fix routes through gh_safe_graphql_input with a properly built JSON +# variables block. +# --------------------------------------------------------------------------- + +@test "live: add_label_to_discussion variables block has labelIds as JSON array" { + # Replace the gh stub with one that captures stdin to a file so we can + # introspect the actual request body that gh would have received. + body_file="${TT_TMP}/captured-body.json" + stub_dir="${TT_TMP}/bin" + mkdir -p "$stub_dir" + cat >"$stub_dir/gh" <<EOF +#!/usr/bin/env bash +cat >"$body_file" +echo '{"data":{"addLabelsToLabelable":{"clientMutationId":null}}}' +EOF + chmod +x "$stub_dir/gh" + PATH="${stub_dir}:${PATH}" add_label_to_discussion "D_1" "L_enhancement" + + [ -f "$body_file" ] + # Body must parse as JSON and have variables.labelIds as a length-1 array + # whose sole element is the literal label id. + jq -e '.variables.labelIds | type == "array"' "$body_file" >/dev/null + jq -e '.variables.labelIds | length == 1' "$body_file" >/dev/null + jq -e '.variables.labelIds[0] == "L_enhancement"' "$body_file" >/dev/null + jq -e '.variables.labelableId == "D_1"' "$body_file" >/dev/null + jq -e '.query | contains("addLabelsToLabelable")' "$body_file" >/dev/null +} diff --git a/test/workflows/feature-ideation/gh-safe.bats b/test/workflows/feature-ideation/gh-safe.bats index 9194d53..ef769cf 100644 --- a/test/workflows/feature-ideation/gh-safe.bats +++ b/test/workflows/feature-ideation/gh-safe.bats @@ -161,3 +161,58 @@ teardown() { run gh_safe_graphql -f query='q' [ "$status" -ne 0 ] } + +# --------------------------------------------------------------------------- +# Regression tests for Copilot review feedback on PR petry-projects/.github#85 +# --------------------------------------------------------------------------- + +@test "graphql: invalid --jq filter EXITS NON-ZERO instead of returning []" { + # Catches the typo class: previously the wrapper used `jq ... || true` + # which silently swallowed filter syntax errors and returned an empty + # array, masking R1-class bugs. + GH_STUB_STDOUT='{"data":{"repository":{"id":"R_1"}}}' \ + run gh_safe_graphql -f query='q' --jq '.data.repository.does_not_exist | bogus_function' + [ "$status" -ne 0 ] +} + +@test "graphql: --jq returning explicit JSON null still normalizes to []" { + GH_STUB_STDOUT='{"data":{"repository":{"nodes":null}}}' \ + run gh_safe_graphql -f query='q' --jq '.data.repository.nodes' + [ "$status" -eq 0 ] + [ "$output" = '[]' ] +} + +# --------------------------------------------------------------------------- +# gh_safe_graphql_input — for mutations with array variables +# --------------------------------------------------------------------------- + +@test "graphql_input: rejects non-JSON body" { + run gh_safe_graphql_input 'not json' + [ "$status" -eq 64 ] +} + +@test "graphql_input: forwards body to gh via stdin and returns response" { + GH_STUB_STDOUT='{"data":{"addLabelsToLabelable":{"clientMutationId":null}}}' \ + run gh_safe_graphql_input '{"query":"mutation{...}","variables":{"labelIds":["L_1"]}}' + [ "$status" -eq 0 ] + [[ "$output" == *'addLabelsToLabelable'* ]] +} + +@test "graphql_input: EXITS NON-ZERO on errors envelope" { + GH_STUB_STDOUT='{"errors":[{"message":"forbidden"}],"data":null}' \ + run gh_safe_graphql_input '{"query":"q","variables":{}}' + [ "$status" -ne 0 ] +} + +@test "graphql_input: EXITS NON-ZERO on data:null" { + GH_STUB_STDOUT='{"data":null}' \ + run gh_safe_graphql_input '{"query":"q","variables":{}}' + [ "$status" -ne 0 ] +} + +@test "graphql_input: EXITS NON-ZERO on gh hard failure" { + GH_STUB_EXIT=1 \ + GH_STUB_STDERR='HTTP 401' \ + run gh_safe_graphql_input '{"query":"q","variables":{}}' + [ "$status" -ne 0 ] +} diff --git a/test/workflows/feature-ideation/lint-prompt.bats b/test/workflows/feature-ideation/lint-prompt.bats index e195bb3..a0cd32e 100644 --- a/test/workflows/feature-ideation/lint-prompt.bats +++ b/test/workflows/feature-ideation/lint-prompt.bats @@ -120,3 +120,41 @@ YML run bash "$LINTER" [ "$status" -eq 0 ] } + +# --------------------------------------------------------------------------- +# Coverage of claude-code-action v1 `prompt:` form (in addition to v0 `direct_prompt:`) +# Caught by Copilot review on PR #85 — the original linter only scanned +# `direct_prompt:` and would silently miss R2 regressions in the actual +# reusable workflow which uses the v1 `prompt:` form. +# --------------------------------------------------------------------------- + +@test "lint-prompt: scans v1 prompt: blocks (not just direct_prompt:)" { + write_yml "${TT_TMP}/v1-prompt.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + prompt: | + You are Mary. + Date: $(date -u +%Y-%m-%d) +YML + run bash "$LINTER" "${TT_TMP}/v1-prompt.yml" + [ "$status" -eq 1 ] +} + +@test "lint-prompt: clean v1 prompt: passes" { + write_yml "${TT_TMP}/v1-clean.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + prompt: | + You are Mary. + Repo: ${{ github.repository }} + Read RUN_DATE from the environment at runtime via printenv. +YML + run bash "$LINTER" "${TT_TMP}/v1-clean.yml" + [ "$status" -eq 0 ] +} diff --git a/test/workflows/feature-ideation/match-discussions.bats b/test/workflows/feature-ideation/match-discussions.bats index 792fdd5..f5f13c7 100644 --- a/test/workflows/feature-ideation/match-discussions.bats +++ b/test/workflows/feature-ideation/match-discussions.bats @@ -161,6 +161,31 @@ build_proposals() { [ "$new" = "0" ] } +@test "match: rejects non-numeric MATCH_THRESHOLD with usage error" { + # Caught by Copilot review on PR petry-projects/.github#85: previously + # a typo in MATCH_THRESHOLD produced an opaque Python traceback. + build_signals '[]' + build_proposals '[{"title":"x"}]' + MATCH_THRESHOLD="not-a-number" run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 64 ] +} + +@test "match: rejects MATCH_THRESHOLD outside [0, 1]" { + build_signals '[]' + build_proposals '[{"title":"x"}]' + MATCH_THRESHOLD="1.5" run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 64 ] +} + +@test "match: accepts boundary MATCH_THRESHOLD values 0 and 1" { + build_signals '[]' + build_proposals '[{"title":"x"}]' + MATCH_THRESHOLD="0" run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] + MATCH_THRESHOLD="1" run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -eq 0 ] +} + @test "match: idempotent re-run of same proposals against existing matches yields all-matched" { # Simulates the R6 idempotency case: run 1 created Discussions, run 2 # finds them and should NOT propose duplicates. diff --git a/test/workflows/feature-ideation/signals-schema.bats b/test/workflows/feature-ideation/signals-schema.bats index a416b8f..5790e64 100644 --- a/test/workflows/feature-ideation/signals-schema.bats +++ b/test/workflows/feature-ideation/signals-schema.bats @@ -13,6 +13,7 @@ FIX="${TT_FIXTURES_DIR}/expected" @test "schema: validator script exists and is executable" { [ -f "$VALIDATOR" ] [ -r "$VALIDATOR" ] + [ -x "$VALIDATOR" ] } @test "schema: schema file is valid JSON" { @@ -50,3 +51,20 @@ FIX="${TT_FIXTURES_DIR}/expected" run python3 "$VALIDATOR" "$bad_file" "$SCHEMA" [ "$status" -eq 1 ] } + +@test "schema: invalid date-time scan_date FAILS validation (format checker enforced)" { + # Caught by Copilot review on PR petry-projects/.github#85: + # Draft202012Validator does NOT enforce `format` keywords by default. + # The validator must instantiate FormatChecker explicitly. + bad_file="${BATS_TEST_TMPDIR}/bad-scan-date.json" + jq '.scan_date = "not-a-real-timestamp"' "${FIX}/empty-repo.signals.json" >"$bad_file" + run python3 "$VALIDATOR" "$bad_file" "$SCHEMA" + [ "$status" -eq 1 ] +} + +@test "schema: well-formed date-time scan_date passes" { + good_file="${BATS_TEST_TMPDIR}/good-scan-date.json" + jq '.scan_date = "2026-04-08T12:34:56Z"' "${FIX}/empty-repo.signals.json" >"$good_file" + run python3 "$VALIDATOR" "$good_file" "$SCHEMA" + [ "$status" -eq 0 ] +} diff --git a/test/workflows/feature-ideation/stubs/gh b/test/workflows/feature-ideation/stubs/gh index 158b665..2e8cd24 100755 --- a/test/workflows/feature-ideation/stubs/gh +++ b/test/workflows/feature-ideation/stubs/gh @@ -9,7 +9,10 @@ # GH_STUB_STDOUT — literal text or @file:/abs/path to load from disk # GH_STUB_STDERR — literal text or @file:/abs/path # GH_STUB_EXIT — exit code (default 0) -# GH_STUB_LOG — append-only log of argv invocations (one TSV line each) +# GH_STUB_LOG — append-only log of argv invocations. Each invocation +# is logged as a single line of shell-quoted tokens +# (via `printf '%q '`) so it can be re-parsed losslessly +# even when arguments contain spaces or special chars. # # For multi-call tests (where the same stub is invoked multiple times with # different responses), set GH_STUB_SCRIPT to a file containing one @@ -19,7 +22,12 @@ set -euo pipefail if [ -n "${GH_STUB_LOG:-}" ]; then - printf '%s\n' "$*" >>"$GH_STUB_LOG" + # Use %q so each arg is shell-quoted; tests can re-parse argv losslessly. + # Caught by Copilot review on PR petry-projects/.github#85: previously + # used $* which joined with spaces and lost argument boundaries. + # shellcheck disable=SC2059 + printf '%q ' "$@" >>"$GH_STUB_LOG" + printf '\n' >>"$GH_STUB_LOG" fi # Multi-call mode. From 67c603719cefbc9e900457fe454987be2d3d9d43 Mon Sep 17 00:00:00 2001 From: DJ <dj@Rachels-MacBook-Air.local> Date: Wed, 8 Apr 2026 05:24:21 -0700 Subject: [PATCH 5/6] fix(feature-ideation): address CodeRabbit review on PR #85 (7 fixes + 1 test) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Triaged 13 inline comments from CodeRabbit's review of #85; 6 of them overlapped with Copilot's review and were already fixed by bcaa579. The remaining 7 are addressed here. Fixes ----- 1. lint-prompt.sh: ${VAR} branch lookbehind was inconsistent with the $(...) branch — only rejected $$VAR but not \${VAR}. Both branches now use [\\$] so backslash-escaped and dollar-escaped forms are skipped uniformly. 2. filter-bots.sh: FEATURE_IDEATION_BOT_AUTHORS CSV entries are now trimmed of leading/trailing whitespace before being added to the blocklist, so "bot1, bot2" matches both bots correctly instead of keeping a literal " bot2" entry. 3. validate-signals.py: malformed signals JSON now exits 2 (file/data error) to match the documented contract, instead of 1 (which means schema validation error). 4. README.md: corrected the workflow filename reference from feature-ideation.yml to feature-ideation-reusable.yml, and reworded the table cell that contained `\|\|` (escaped pipes that don't render correctly in some Markdown engines) to use plain prose. Also noted that lint-prompt scans both v0 `direct_prompt:` and v1 `prompt:`. 5. collect-signals.sh: added an explicit comment above SCHEMA_VERSION documenting the lockstep requirement with signals.schema.json's $comment version annotation. Backed by a new bats test that parses both files and asserts they match. 6. signals.schema.json: added $comment "version: 1.0.0" annotation so the schema file declares its own version explicitly. Used $comment instead of a custom keyword to keep Draft202012 compliance. 7. test/workflows/feature-ideation/match-discussions.bats: build_signals helper now computes the discussions count from the array length instead of hardcoding 0, so the fixture satisfies its own contract (cosmetic — the matcher only reads .items, but contract hygiene matters in test scaffolding). 8. test/workflows/feature-ideation/gh-safe.bats: removed the `|| true` suffix on the rest-failure assertion that made it always pass. Now uses --separate-stderr to capture stderr and asserts the structured `[gh-safe][rest-failure]` prefix is emitted on the auth failure path. Required `bats_require_minimum_version 1.5.0` to suppress the bats-core warning about flag usage. Tests ----- Test count: 108 → 109 (one new test for SCHEMA_VERSION ↔ schema sync). All 109 passing locally. Run with: bats test/workflows/feature-ideation/ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- .github/schemas/signals.schema.json | 1 + .github/scripts/feature-ideation/README.md | 11 ++++++----- .../scripts/feature-ideation/collect-signals.sh | 6 ++++++ .../scripts/feature-ideation/lib/filter-bots.sh | 13 ++++++++++++- .github/scripts/feature-ideation/lint-prompt.sh | 7 +++++-- .../scripts/feature-ideation/validate-signals.py | 6 +++++- test/workflows/feature-ideation/gh-safe.bats | 12 +++++++++--- .../feature-ideation/match-discussions.bats | 8 +++++++- .../workflows/feature-ideation/signals-schema.bats | 14 ++++++++++++++ 9 files changed, 65 insertions(+), 13 deletions(-) diff --git a/.github/schemas/signals.schema.json b/.github/schemas/signals.schema.json index 9a1ad07..ded4367 100644 --- a/.github/schemas/signals.schema.json +++ b/.github/schemas/signals.schema.json @@ -1,6 +1,7 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/petry-projects/.github/blob/main/.github/schemas/signals.schema.json", + "$comment": "version: 1.0.0 — must match SCHEMA_VERSION in collect-signals.sh; enforced by bats", "title": "Feature Ideation Signals", "description": "Canonical contract between collect-signals.sh and the BMAD Analyst (Mary) prompt. Any change to this schema is a breaking change to the workflow.", "type": "object", diff --git a/.github/scripts/feature-ideation/README.md b/.github/scripts/feature-ideation/README.md index 957a06a..372125c 100644 --- a/.github/scripts/feature-ideation/README.md +++ b/.github/scripts/feature-ideation/README.md @@ -1,9 +1,10 @@ # Feature Ideation — Scripts & Test Strategy This directory contains the bash + Python helpers that back -`.github/workflows/feature-ideation.yml`. Every line of logic that used to -live inside the workflow's heredoc has been extracted here so it can be unit -tested with bats. +`.github/workflows/feature-ideation-reusable.yml`. Every line of logic that +used to live inside the workflow's heredoc has been extracted here so it can +be unit tested with bats. Downstream BMAD repos call the reusable workflow +via the standards stub at `standards/workflows/feature-ideation.yml`. ## Why this exists @@ -21,7 +22,7 @@ caught **before UAT** instead of after. | File | Purpose | Killed risk | |------|---------|------------| -| `lib/gh-safe.sh` | Wraps every `gh` and `gh api graphql` call. Fails loud on auth, rate-limit, network, GraphQL errors envelope, or `data: null`. Replaces the original `2>/dev/null \|\| echo '[]'` swallow. | R1, R7, R8 | +| `lib/gh-safe.sh` | Wraps every `gh` and `gh api graphql` call. Fails loud on auth, rate-limit, network, GraphQL errors envelope, or `data: null`. Replaces the original `2>/dev/null` + `echo '[]'` swallow pattern. | R1, R7, R8 | | `lib/compose-signals.sh` | Validates JSON inputs before `jq --argjson` and assembles the canonical signals.json document. | R3, R4 | | `lib/filter-bots.sh` | Configurable bot blocklist via `FEATURE_IDEATION_BOT_AUTHORS` (extends the default list of bot logins to remove from results). | R10 | | `lib/date-utils.sh` | Cross-platform date arithmetic helpers. | R9 | @@ -29,7 +30,7 @@ caught **before UAT** instead of after. | `validate-signals.py` | JSON Schema 2020-12 validator for signals.json against `../schemas/signals.schema.json`. | R3 | | `match-discussions.sh` | Deterministic Jaccard-similarity matcher between Mary's proposals and existing Ideas Discussions. Replaces the prose "use fuzzy matching" instruction. | R5, R6 | | `discussion-mutations.sh` | `create_discussion`, `comment_on_discussion`, `add_label_to_discussion` wrappers with `DRY_RUN=1` audit-log mode. | Smoke testing | -| `lint-prompt.sh` | Scans every workflow file for unescaped `$()` / `${VAR}` inside `direct_prompt:` blocks (which YAML and `claude-code-action` pass verbatim instead of expanding). | R2 | +| `lint-prompt.sh` | Scans every workflow file for unescaped `$()` / `${VAR}` inside `direct_prompt:` (v0) and `prompt:` (v1) blocks (which YAML and `claude-code-action` pass verbatim instead of expanding). | R2 | ## Running the tests diff --git a/.github/scripts/feature-ideation/collect-signals.sh b/.github/scripts/feature-ideation/collect-signals.sh index 69f605c..fd3b1b3 100755 --- a/.github/scripts/feature-ideation/collect-signals.sh +++ b/.github/scripts/feature-ideation/collect-signals.sh @@ -25,6 +25,12 @@ set -euo pipefail +# SCHEMA_VERSION must stay in lockstep with the `version` field in +# .github/schemas/signals.schema.json. Bumping one without the other is a +# compatibility break: validate-signals.py will reject the runtime output +# if the constants drift, AND the bats `signals-schema: SCHEMA_VERSION +# constant matches schema file` test enforces this in CI. +# Caught by CodeRabbit review on PR petry-projects/.github#85. SCHEMA_VERSION="1.0.0" SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" diff --git a/.github/scripts/feature-ideation/lib/filter-bots.sh b/.github/scripts/feature-ideation/lib/filter-bots.sh index c3b38be..f2fbfde 100755 --- a/.github/scripts/feature-ideation/lib/filter-bots.sh +++ b/.github/scripts/feature-ideation/lib/filter-bots.sh @@ -38,7 +38,18 @@ filter_bots_build_list() { local IFS=',' # shellcheck disable=SC2206 local extras=($FEATURE_IDEATION_BOT_AUTHORS) - list+=("${extras[@]}") + # Trim leading/trailing whitespace from each comma-separated entry so + # `"bot1, bot2"` resolves to `bot1` and `bot2`, not `bot1` and ` bot2`. + # Caught by CodeRabbit review on PR petry-projects/.github#85. + local trimmed=() + local entry + for entry in "${extras[@]}"; do + # Strip leading whitespace, then trailing whitespace. + entry="${entry#"${entry%%[![:space:]]*}"}" + entry="${entry%"${entry##*[![:space:]]}"}" + [ -n "$entry" ] && trimmed+=("$entry") + done + list+=("${trimmed[@]}") fi printf '%s\n' "${list[@]}" | jq -R . | jq -sc . } diff --git a/.github/scripts/feature-ideation/lint-prompt.sh b/.github/scripts/feature-ideation/lint-prompt.sh index 3b1aebc..ad8226d 100755 --- a/.github/scripts/feature-ideation/lint-prompt.sh +++ b/.github/scripts/feature-ideation/lint-prompt.sh @@ -48,8 +48,11 @@ block_indent = -1 findings = [] # Pattern matches $(...) and ${VAR} but NOT GitHub Actions ${{ ... }} -# because that's evaluated before the prompt is rendered. -shell_expansion = re.compile(r'(?<!\\)\$\([^)]*\)|(?<!\$)\$\{[A-Za-z_][A-Za-z0-9_]*\}') +# (which is evaluated before the prompt is rendered) and NOT \$ or $$ +# escapes (which produce literal characters in the rendered prompt). +# Both branches use the same lookbehind so escape handling is consistent. +# Caught by CodeRabbit review on PR petry-projects/.github#85. +shell_expansion = re.compile(r'(?<![\\$])\$\([^)]*\)|(?<![\\$])\$\{[A-Za-z_][A-Za-z0-9_]*\}') # Recognise both `direct_prompt:` (v0) and `prompt:` (v1) markers, with # optional `|` or `>` block scalar indicators. diff --git a/.github/scripts/feature-ideation/validate-signals.py b/.github/scripts/feature-ideation/validate-signals.py index 69ff0c1..eb82ff7 100755 --- a/.github/scripts/feature-ideation/validate-signals.py +++ b/.github/scripts/feature-ideation/validate-signals.py @@ -55,8 +55,12 @@ def main(argv: list[str]) -> int: try: signals = json.loads(signals_path.read_text()) except json.JSONDecodeError as exc: + # Per the docstring contract, exit 2 means usage / file error and + # exit 1 means schema validation error. A malformed signals file + # is a file/data error, not a schema violation. Caught by + # CodeRabbit review on PR petry-projects/.github#85. sys.stderr.write(f"[validate-signals] invalid JSON in {signals_path}: {exc}\n") - return 1 + return 2 try: schema = json.loads(schema_path.read_text()) diff --git a/test/workflows/feature-ideation/gh-safe.bats b/test/workflows/feature-ideation/gh-safe.bats index ef769cf..b06bb5c 100644 --- a/test/workflows/feature-ideation/gh-safe.bats +++ b/test/workflows/feature-ideation/gh-safe.bats @@ -4,6 +4,8 @@ # These tests kill R1: the original "2>/dev/null || echo '[]'" pattern that # silently swallowed every kind of failure. Each test pins one failure mode. +bats_require_minimum_version 1.5.0 + load 'helpers/setup' setup() { @@ -67,12 +69,16 @@ teardown() { # gh_safe_rest — failure modes (R1 kill list) # --------------------------------------------------------------------------- -@test "rest: EXITS NON-ZERO on auth failure (gh exit 4)" { +@test "rest: EXITS NON-ZERO on auth failure (gh exit 4) and reports the failure category" { + # CodeRabbit on PR petry-projects/.github#85: the previous assertion + # ended with `|| true`, so it always passed regardless of the message + # content. Use --separate-stderr to capture stderr and assert the + # structured `[gh-safe][rest-failure]` prefix is emitted. GH_STUB_EXIT=4 \ GH_STUB_STDERR='HTTP 401: Bad credentials' \ - run gh_safe_rest issue list --repo foo/bar + run --separate-stderr gh_safe_rest issue list --repo foo/bar [ "$status" -ne 0 ] - [[ "$output" == *"rest-failure"* ]] || [[ "$stderr" == *"rest-failure"* ]] || true + [[ "$stderr" == *"rest-failure"* ]] } @test "rest: EXITS NON-ZERO on rate limit (gh exit 1 + 403)" { diff --git a/test/workflows/feature-ideation/match-discussions.bats b/test/workflows/feature-ideation/match-discussions.bats index f5f13c7..9b63ea9 100644 --- a/test/workflows/feature-ideation/match-discussions.bats +++ b/test/workflows/feature-ideation/match-discussions.bats @@ -18,8 +18,14 @@ teardown() { } # Helper: build a signals.json with the given discussions array. +# Computes the discussions count from the array length so the fixture +# always satisfies the producer/consumer contract (even though the +# matcher only reads .items, validating contracts in test fixtures +# is good hygiene). CodeRabbit on PR petry-projects/.github#85. build_signals() { local discussions="$1" + local count + count=$(printf '%s' "$discussions" | jq 'length') cat >"${TT_TMP}/signals.json" <<JSON { "schema_version": "1.0.0", @@ -27,7 +33,7 @@ build_signals() { "repo": "petry-projects/talkterm", "open_issues": { "count": 0, "items": [] }, "closed_issues_30d": { "count": 0, "items": [] }, - "ideas_discussions": { "count": 0, "items": ${discussions} }, + "ideas_discussions": { "count": ${count}, "items": ${discussions} }, "releases": [], "merged_prs_30d": { "count": 0, "items": [] }, "feature_requests": { "count": 0, "items": [] }, diff --git a/test/workflows/feature-ideation/signals-schema.bats b/test/workflows/feature-ideation/signals-schema.bats index 5790e64..0281d58 100644 --- a/test/workflows/feature-ideation/signals-schema.bats +++ b/test/workflows/feature-ideation/signals-schema.bats @@ -68,3 +68,17 @@ FIX="${TT_FIXTURES_DIR}/expected" run python3 "$VALIDATOR" "$good_file" "$SCHEMA" [ "$status" -eq 0 ] } + +@test "schema: SCHEMA_VERSION constant matches schema file version comment" { + # CodeRabbit on PR petry-projects/.github#85: enforce that the + # SCHEMA_VERSION constant in collect-signals.sh stays in lockstep with + # the version annotation in signals.schema.json. Bumping one without + # the other is a compatibility break. + script_version=$(grep -E '^SCHEMA_VERSION=' "${TT_REPO_ROOT}/.github/scripts/feature-ideation/collect-signals.sh" \ + | head -1 | sed -E 's/^SCHEMA_VERSION="([^"]+)".*/\1/') + schema_version=$(jq -r '."$comment"' "$SCHEMA" \ + | grep -oE 'version: [0-9]+\.[0-9]+\.[0-9]+' | sed 's/version: //') + [ -n "$script_version" ] + [ -n "$schema_version" ] + [ "$script_version" = "$schema_version" ] +} From 5c52288f5bb16fe6e65cb1d2ce8a175848ca6bcf Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Wed, 8 Apr 2026 19:59:04 +0000 Subject: [PATCH 6/6] fix(feature-ideation): address CodeRabbit re-review on PR #85 (15 fixes + 5 new tests) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Critical/major: - collect-signals.sh: validate ISSUE_LIMIT/PR_LIMIT/DISCUSSION_LIMIT are positive integers; tighten REPO validation with strict ^[^/]+/[^/]+$ regex - compose-signals.sh: enforce array type (jq 'type == "array"') not just valid JSON so objects/strings don't silently produce wrong counts - date-utils.sh: guard $# before reading $1 to prevent set -u abort on zero-arg calls - filter-bots.sh: replace unquoted array expansion with IFS=',' read -r -a to prevent pathname-globbing against filesystem entries - gh-safe.sh: bounds-check args[i+1] before --jq dereference; add $# guard to gh_safe_graphql_input() to prevent nounset abort - lint-prompt.sh: recognise YAML chomping modifiers (|-,|+,>-,>+) in prompt_marker regex; replace [^}]* GH-expression stripper with a stateful scanner that handles nested braces; preserve exit-2 over exit-1 in main() - match-discussions.sh: wrap json.load calls in try/except for structured error exit-2 instead of Python traceback; skip discussions without an id; switch from greedy per-proposal to similarity-sorted global optimal matching - validate-signals.py: catch OSError on read_text() to preserve exit-2 contract; add -> bool return type annotation to _check_date_time Docs: - README.md: update lint command to mention both direct_prompt: and prompt:; fix Mary's prompt pointer to feature-ideation-reusable.yml Tests (+5 new, 109 → 114 total): - lint-prompt.bats: missing-file-before-lint-failing-file exits 2; YAML chomping modifiers detected; nested GH expressions don't false-positive - match-discussions.bats: malformed signals JSON exits non-zero; malformed proposals JSON exits non-zero - signals-schema.bats: truncated/malformed JSON exits 2 not 1 - date-utils.bats: use date_today helper instead of raw date -u - stubs/gh: prefer TT_TMP/BATS_TEST_TMPDIR for counter file isolation Co-authored-by: don-petry <don-petry@users.noreply.github.com> --- .github/scripts/feature-ideation/README.md | 4 +- .../feature-ideation/collect-signals.sh | 28 +++++- .../feature-ideation/lib/compose-signals.sh | 7 +- .../feature-ideation/lib/date-utils.sh | 7 ++ .../feature-ideation/lib/filter-bots.sh | 8 +- .../scripts/feature-ideation/lib/gh-safe.sh | 14 +++ .../scripts/feature-ideation/lint-prompt.sh | 59 ++++++++++-- .../feature-ideation/match-discussions.sh | 92 ++++++++++++++----- .../feature-ideation/validate-signals.py | 14 ++- .../feature-ideation/date-utils.bats | 5 +- .../feature-ideation/lint-prompt.bats | 54 +++++++++++ .../feature-ideation/match-discussions.bats | 16 ++++ .../feature-ideation/signals-schema.bats | 9 ++ test/workflows/feature-ideation/stubs/gh | 5 +- 14 files changed, 275 insertions(+), 47 deletions(-) diff --git a/.github/scripts/feature-ideation/README.md b/.github/scripts/feature-ideation/README.md index 372125c..f213172 100644 --- a/.github/scripts/feature-ideation/README.md +++ b/.github/scripts/feature-ideation/README.md @@ -51,7 +51,7 @@ bats test/workflows/feature-ideation/gh-safe.bats shellcheck -x collect-signals.sh lint-prompt.sh match-discussions.sh \ discussion-mutations.sh lib/*.sh) -# Lint the workflow's direct_prompt block +# Lint workflow prompt blocks (direct_prompt: and prompt:) bash .github/scripts/feature-ideation/lint-prompt.sh ``` @@ -76,7 +76,7 @@ signals.json is a breaking change and must: 1. Bump `SCHEMA_VERSION` in `collect-signals.sh`. 2. Update fixtures under `test/workflows/feature-ideation/fixtures/expected/`. -3. Update Mary's prompt in `feature-ideation.yml` if any field references move. +3. Update Mary's prompt in `.github/workflows/feature-ideation-reusable.yml` if any field references move. CI validates every fixture against the schema, and the workflow validates the runtime output before handing it to Mary. diff --git a/.github/scripts/feature-ideation/collect-signals.sh b/.github/scripts/feature-ideation/collect-signals.sh index fd3b1b3..b2ae23c 100755 --- a/.github/scripts/feature-ideation/collect-signals.sh +++ b/.github/scripts/feature-ideation/collect-signals.sh @@ -58,13 +58,33 @@ main() { local discussion_limit="${DISCUSSION_LIMIT:-100}" local output_path="${SIGNALS_OUTPUT:-./signals.json}" - local owner repo_name - owner="${REPO%%/*}" - repo_name="${REPO##*/}" - if [ "$owner" = "$REPO" ] || [ -z "$repo_name" ]; then + # Validate that limit overrides are positive integers before forwarding to + # GraphQL — a value like `ISSUE_LIMIT=foo` would cause an opaque downstream + # failure instead of a clean usage error. Caught by CodeRabbit review on + # PR petry-projects/.github#85. + local _lim_name _lim_val + for _lim_name in ISSUE_LIMIT PR_LIMIT DISCUSSION_LIMIT; do + case "$_lim_name" in + ISSUE_LIMIT) _lim_val="$issue_limit" ;; + PR_LIMIT) _lim_val="$pr_limit" ;; + DISCUSSION_LIMIT) _lim_val="$discussion_limit" ;; + esac + if [[ ! $_lim_val =~ ^[1-9][0-9]*$ ]]; then + printf '[collect-signals] %s must be a positive integer, got: %s\n' "$_lim_name" "$_lim_val" >&2 + return 64 + fi + done + + # Strict owner/name format — reject leading/trailing slashes, empty segments, + # and extra path parts (e.g. "org//repo", "/repo", "org/repo/extra"). + # Caught by CodeRabbit review on PR petry-projects/.github#85. + if [[ ! $REPO =~ ^[^/]+/[^/]+$ ]]; then printf '[collect-signals] REPO must be in owner/name format, got: %s\n' "$REPO" >&2 return 64 fi + local owner repo_name + owner="${REPO%%/*}" + repo_name="${REPO##*/}" local thirty_days_ago thirty_days_ago=$(date_days_ago 30) diff --git a/.github/scripts/feature-ideation/lib/compose-signals.sh b/.github/scripts/feature-ideation/lib/compose-signals.sh index 6a0f5bd..1c69934 100755 --- a/.github/scripts/feature-ideation/lib/compose-signals.sh +++ b/.github/scripts/feature-ideation/lib/compose-signals.sh @@ -49,8 +49,11 @@ compose_signals() { for input in "$open_issues" "$closed_issues" "$ideas_discussions" "$releases" \ "$merged_prs" "$feature_requests" "$bug_reports" "$truncation_warnings"; do idx=$((idx + 1)) - if ! printf '%s' "$input" | jq -e . >/dev/null 2>&1; then - printf '[compose-signals] arg #%d is not valid JSON: %s\n' "$idx" "${input:0:120}" >&2 + # Require a JSON array, not just valid JSON. Objects/strings/nulls accepted + # by `jq -e .` would silently produce wrong counts (key count, char count). + # Caught by CodeRabbit review on PR petry-projects/.github#85. + if ! printf '%s' "$input" | jq -e 'type == "array"' >/dev/null 2>&1; then + printf '[compose-signals] arg #%d must be a JSON array: %s\n' "$idx" "${input:0:120}" >&2 return 65 # EX_DATAERR fi done diff --git a/.github/scripts/feature-ideation/lib/date-utils.sh b/.github/scripts/feature-ideation/lib/date-utils.sh index 8c4b5a1..ae2722b 100755 --- a/.github/scripts/feature-ideation/lib/date-utils.sh +++ b/.github/scripts/feature-ideation/lib/date-utils.sh @@ -10,6 +10,13 @@ set -euo pipefail # Print an ISO date (YYYY-MM-DD) for N days ago in UTC. date_days_ago() { + # Guard arg count before reading $1: under set -u a zero-arg call would abort + # the shell with "unbound variable" instead of reaching the validation path. + # Caught by CodeRabbit review on PR petry-projects/.github#85. + if [ "$#" -ne 1 ]; then + printf '[date-utils] expected 1 arg (days), got: %d\n' "$#" >&2 + return 64 + fi local days="$1" if [ -z "$days" ] || ! printf '%s' "$days" | grep -Eq '^[0-9]+$'; then printf '[date-utils] days must be a non-negative integer, got: %s\n' "$days" >&2 diff --git a/.github/scripts/feature-ideation/lib/filter-bots.sh b/.github/scripts/feature-ideation/lib/filter-bots.sh index f2fbfde..f994e3e 100755 --- a/.github/scripts/feature-ideation/lib/filter-bots.sh +++ b/.github/scripts/feature-ideation/lib/filter-bots.sh @@ -35,9 +35,11 @@ DEFAULT_BOT_AUTHORS=( filter_bots_build_list() { local list=("${DEFAULT_BOT_AUTHORS[@]}") if [ -n "${FEATURE_IDEATION_BOT_AUTHORS:-}" ]; then - local IFS=',' - # shellcheck disable=SC2206 - local extras=($FEATURE_IDEATION_BOT_AUTHORS) + # Use `IFS=',' read` (not unquoted expansion) to avoid pathname-globbing + # against the filesystem if any entry contains wildcard characters. + # Caught by CodeRabbit review on PR petry-projects/.github#85. + local extras=() + IFS=',' read -r -a extras <<<"${FEATURE_IDEATION_BOT_AUTHORS}" # Trim leading/trailing whitespace from each comma-separated entry so # `"bot1, bot2"` resolves to `bot1` and `bot2`, not `bot1` and ` bot2`. # Caught by CodeRabbit review on PR petry-projects/.github#85. diff --git a/.github/scripts/feature-ideation/lib/gh-safe.sh b/.github/scripts/feature-ideation/lib/gh-safe.sh index 343a5bb..11a4758 100755 --- a/.github/scripts/feature-ideation/lib/gh-safe.sh +++ b/.github/scripts/feature-ideation/lib/gh-safe.sh @@ -93,6 +93,13 @@ gh_safe_graphql() { local i=0 while [ "$i" -lt "${#args[@]}" ]; do if [ "${args[$i]}" = "--jq" ]; then + # Guard bounds before dereferencing args[i+1]: under set -u an out-of- + # bounds access aborts the shell. Caught by CodeRabbit review on PR + # petry-projects/.github#85. + if [ $((i + 1)) -ge "${#args[@]}" ]; then + _gh_safe_err "graphql-bad-args" "--jq requires a jq filter argument" + return 64 + fi has_jq=1 jq_filter="${args[$((i + 1))]}" break @@ -187,6 +194,13 @@ gh_safe_graphql() { # Same defensive contract as `gh_safe_graphql`: any auth/network/schema # failure exits non-zero with a structured stderr message. gh_safe_graphql_input() { + # Guard arg count before reading $1: under set -u a zero-arg call aborts the + # shell instead of reaching the JSON validation. Caught by CodeRabbit review + # on PR petry-projects/.github#85. + if [ "$#" -ne 1 ]; then + _gh_safe_err "graphql-bad-input" "expected 1 arg: JSON request body, got $#" + return 64 + fi local body="$1" if ! gh_safe_is_json "$body"; then _gh_safe_err "graphql-bad-input" "request body is not valid JSON" diff --git a/.github/scripts/feature-ideation/lint-prompt.sh b/.github/scripts/feature-ideation/lint-prompt.sh index ad8226d..a1d18b6 100755 --- a/.github/scripts/feature-ideation/lint-prompt.sh +++ b/.github/scripts/feature-ideation/lint-prompt.sh @@ -31,6 +31,34 @@ scan_file() { import re import sys + +def _strip_github_expressions(s: str) -> str: + """Remove ${{ ... }} GitHub Actions expressions from s. + + Uses a stateful scanner instead of `[^}]*` regex so that expressions + containing `}` inside string literals (e.g. format() calls) are fully + consumed rather than prematurely terminated. This prevents false-positive + shell-expansion matches on content that is actually inside a GH expression. + Caught by CodeRabbit review on PR petry-projects/.github#85. + """ + result: list[str] = [] + i = 0 + while i < len(s): + if s[i : i + 3] == "${{": + # Consume until we find the matching "}}" + j = i + 3 + while j < len(s): + if s[j : j + 2] == "}}": + j += 2 + break + j += 1 + i = j # skip the whole ${{ ... }} expression + else: + result.append(s[i]) + i += 1 + return "".join(result) + + path = sys.argv[1] try: with open(path, "r", encoding="utf-8") as f: @@ -55,8 +83,10 @@ findings = [] shell_expansion = re.compile(r'(?<![\\$])\$\([^)]*\)|(?<![\\$])\$\{[A-Za-z_][A-Za-z0-9_]*\}') # Recognise both `direct_prompt:` (v0) and `prompt:` (v1) markers, with -# optional `|` or `>` block scalar indicators. -prompt_marker = re.compile(r'(?:direct_prompt|prompt):\s*[|>]?\s*$') +# optional `|` or `>` block scalar indicators plus YAML chomping modifiers +# (`-` or `+`) so `prompt: |-`, `prompt: |+`, `prompt: >-`, `prompt: >+` +# are all recognised. Caught by CodeRabbit review on PR petry-projects/.github#85. +prompt_marker = re.compile(r'(?:direct_prompt|prompt):\s*[|>]?[-+]?\s*$') for lineno, raw in enumerate(lines, start=1): stripped = raw.lstrip(" ") @@ -78,8 +108,13 @@ for lineno, raw in enumerate(lines, start=1): continue # We're inside the prompt body. Scan for shell expansions. - # First, strip out any GitHub Actions expressions so they don't trip us. - no_gh = re.sub(r'\$\{\{[^}]*\}\}', '', raw) + # First, strip out GitHub Actions ${{ ... }} expressions. + # The naive `[^}]*` regex stops at the first `}`, so expressions that + # contain `}` internally (e.g. format() calls or string literals) are + # not fully removed and leave false-positive shell expansion matches. + # Use a small stateful scanner instead. + # Caught by CodeRabbit review on PR petry-projects/.github#85. + no_gh = _strip_github_expressions(raw) for match in shell_expansion.finditer(no_gh): findings.append((lineno, match.group(0), raw.rstrip())) @@ -107,15 +142,27 @@ main() { fi local exit=0 + local file_rc=0 for file in "$@"; do if [ ! -f "$file" ]; then printf '[lint-prompt] not found: %s\n' "$file" >&2 exit=2 continue fi - if ! scan_file "$file"; then - exit=1 + # Capture the actual exit code so we preserve exit-2 (file error) over + # exit-1 (lint finding). A later lint failure must not overwrite an earlier + # file error. Caught by CodeRabbit review on PR petry-projects/.github#85. + if scan_file "$file"; then + file_rc=0 + else + file_rc=$? fi + case "$file_rc" in + 0) ;; + 1) if [ "$exit" -eq 0 ]; then exit=1; fi ;; + 2) exit=2 ;; + *) return "$file_rc" ;; + esac done return "$exit" } diff --git a/.github/scripts/feature-ideation/match-discussions.sh b/.github/scripts/feature-ideation/match-discussions.sh index d8bf766..c4af546 100755 --- a/.github/scripts/feature-ideation/match-discussions.sh +++ b/.github/scripts/feature-ideation/match-discussions.sh @@ -97,49 +97,91 @@ def jaccard(a: set[str], b: set[str]) -> float: return len(a & b) / len(a | b) -with open(signals_path) as f: - signals = json.load(f) -with open(proposals_path) as f: - proposals = json.load(f) +def _load_json(path: str, label: str): + """Load JSON from path, exiting with code 2 on any read or parse error.""" + try: + with open(path, encoding="utf-8") as f: + return json.load(f) + except OSError as exc: + sys.stderr.write(f"[match-discussions] cannot read {label}: {exc}\n") + sys.exit(2) + except json.JSONDecodeError as exc: + sys.stderr.write(f"[match-discussions] invalid JSON in {label}: {exc}\n") + sys.exit(2) + + +signals = _load_json(signals_path, "signals") +proposals = _load_json(proposals_path, "proposals") if not isinstance(proposals, list): sys.stderr.write("[match-discussions] proposals must be a JSON array\n") sys.exit(65) discussions = signals.get("ideas_discussions", {}).get("items", []) or [] -disc_norm = [(d, normalize(d.get("title", ""))) for d in discussions] - -matched = [] -new_candidates = [] -seen_disc_ids = set() - -for proposal in proposals: +# Skip discussions without an id to avoid all id-less entries collapsing into +# a single `None` key in seen_disc_ids. Caught by CodeRabbit on PR #85. +disc_norm = [ + (d, normalize(d.get("title", ""))) + for d in discussions + if d.get("id") is not None +] + +# --- Optimal (similarity-sorted) matching ------------------------------------ +# The original greedy per-proposal loop consumed discussions in proposal order, +# so an early lower-value match could block a later higher-value match. +# Instead we enumerate all (proposal, discussion) pairs, sort by similarity +# descending (ties broken by original proposal index for stability), then +# assign greedily. This guarantees globally higher-value matches are honoured +# first. Caught by CodeRabbit review on PR petry-projects/.github#85. + +# Collect valid proposals with their original index (for tie-breaking + new_candidates). +proposals_indexed: list[tuple[int, dict]] = [] +for p_idx, proposal in enumerate(proposals): if not isinstance(proposal, dict) or "title" not in proposal: sys.stderr.write(f"[match-discussions] skipping malformed proposal: {proposal!r}\n") continue - p_norm = normalize(proposal["title"]) + proposals_indexed.append((p_idx, proposal)) - best = None - best_sim = 0.0 +# Build all (similarity, proposal_idx, disc_id, proposal, disc) tuples. +all_pairs: list[tuple[float, int, str, dict, dict]] = [] +for p_idx, proposal in proposals_indexed: + p_norm = normalize(proposal["title"]) for disc, d_norm in disc_norm: - if disc.get("id") in seen_disc_ids: - continue sim = jaccard(p_norm, d_norm) - if sim > best_sim: - best_sim = sim - best = disc + all_pairs.append((sim, p_idx, disc["id"], proposal, disc)) - if best is not None and best_sim >= threshold: +# Sort descending by similarity; stable tie-break by proposal index ascending. +all_pairs.sort(key=lambda x: (-x[0], x[1])) + +matched = [] +seen_disc_ids: set[str] = set() +seen_proposal_idxs: set[int] = set() + +for sim, p_idx, disc_id, proposal, disc in all_pairs: + if p_idx in seen_proposal_idxs or disc_id in seen_disc_ids: + continue + if sim >= threshold: matched.append( { "proposal": proposal, - "discussion": best, - "similarity": round(best_sim, 4), + "discussion": disc, + "similarity": round(sim, 4), } ) - seen_disc_ids.add(best.get("id")) - else: - new_candidates.append({"proposal": proposal, "best_similarity": round(best_sim, 4)}) + seen_disc_ids.add(disc_id) + seen_proposal_idxs.add(p_idx) + +# Unmatched proposals become new candidates. +new_candidates = [] +for p_idx, proposal in proposals_indexed: + if p_idx in seen_proposal_idxs: + continue + p_norm = normalize(proposal["title"]) + best_sim = max( + (jaccard(p_norm, d_norm) for _, d_norm in disc_norm), + default=0.0, + ) + new_candidates.append({"proposal": proposal, "best_similarity": round(best_sim, 4)}) result = { "matched": matched, diff --git a/.github/scripts/feature-ideation/validate-signals.py b/.github/scripts/feature-ideation/validate-signals.py index eb82ff7..e163cf6 100755 --- a/.github/scripts/feature-ideation/validate-signals.py +++ b/.github/scripts/feature-ideation/validate-signals.py @@ -53,7 +53,12 @@ def main(argv: list[str]) -> int: return 2 try: - signals = json.loads(signals_path.read_text()) + signals = json.loads(signals_path.read_text(encoding="utf-8")) + except OSError as exc: + # File read errors (permissions, I/O) must also exit 2. Caught by + # CodeRabbit review on PR petry-projects/.github#85. + sys.stderr.write(f"[validate-signals] cannot read {signals_path}: {exc}\n") + return 2 except json.JSONDecodeError as exc: # Per the docstring contract, exit 2 means usage / file error and # exit 1 means schema validation error. A malformed signals file @@ -63,7 +68,10 @@ def main(argv: list[str]) -> int: return 2 try: - schema = json.loads(schema_path.read_text()) + schema = json.loads(schema_path.read_text(encoding="utf-8")) + except OSError as exc: + sys.stderr.write(f"[validate-signals] cannot read schema {schema_path}: {exc}\n") + return 2 except json.JSONDecodeError as exc: sys.stderr.write(f"[validate-signals] invalid schema JSON: {exc}\n") return 2 @@ -79,7 +87,7 @@ def main(argv: list[str]) -> int: format_checker = FormatChecker() @format_checker.checks("date-time", raises=(ValueError, TypeError)) - def _check_date_time(instance): # noqa: ANN001 — jsonschema callback signature + def _check_date_time(instance) -> bool: # noqa: ANN001 — jsonschema callback signature if not isinstance(instance, str): return True # non-strings handled by `type` keyword, not format # Must look like a date-time, not just any string. Require at least diff --git a/test/workflows/feature-ideation/date-utils.bats b/test/workflows/feature-ideation/date-utils.bats index dfb132b..a26749f 100644 --- a/test/workflows/feature-ideation/date-utils.bats +++ b/test/workflows/feature-ideation/date-utils.bats @@ -15,7 +15,10 @@ setup() { } @test "date_days_ago: 0 returns today" { - today=$(date -u +%Y-%m-%d) + # Use the helper function rather than the raw system `date` call so the test + # validates behaviour consistently on both GNU and BSD systems. + # Caught by CodeRabbit review on PR petry-projects/.github#85. + today=$(date_today) run date_days_ago 0 [ "$status" -eq 0 ] [ "$output" = "$today" ] diff --git a/test/workflows/feature-ideation/lint-prompt.bats b/test/workflows/feature-ideation/lint-prompt.bats index a0cd32e..1ba14d1 100644 --- a/test/workflows/feature-ideation/lint-prompt.bats +++ b/test/workflows/feature-ideation/lint-prompt.bats @@ -158,3 +158,57 @@ YML run bash "$LINTER" "${TT_TMP}/v1-clean.yml" [ "$status" -eq 0 ] } + +# --------------------------------------------------------------------------- +# Exit-code precedence: file errors (2) must not be downgraded by later lint +# findings (1). Caught by CodeRabbit review on PR petry-projects/.github#85. +# --------------------------------------------------------------------------- + +@test "lint-prompt: missing file before lint-failing file still exits 2" { + write_yml "${TT_TMP}/bad.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + prompt: | + Date: $(date -u +%Y-%m-%d) +YML + # Pass a non-existent file FIRST, then a file with a lint finding. + # The aggregate exit code must be 2 (file error) not 1 (lint finding). + run bash "$LINTER" "${TT_TMP}/missing.yml" "${TT_TMP}/bad.yml" + [ "$status" -eq 2 ] +} + +@test "lint-prompt: YAML chomping modifiers are recognised (prompt: |-)" { + # YAML block-scalar headers like `|-`, `|+`, `>-`, `>+` must be detected + # as prompt markers so their bodies are linted. Caught by CodeRabbit on PR #85. + write_yml "${TT_TMP}/chomping.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + prompt: |- + Date: $(date -u +%Y-%m-%d) +YML + run bash "$LINTER" "${TT_TMP}/chomping.yml" + [ "$status" -eq 1 ] +} + +@test "lint-prompt: GitHub Actions expressions with nested braces do not false-positive" { + # ${{ format('${FOO}', github.ref) }} should be stripped entirely so ${FOO} + # inside the expression is not flagged as a bare shell expansion. + # Caught by CodeRabbit review on PR petry-projects/.github#85. + write_yml "${TT_TMP}/nested-expr.yml" <<'YML' +jobs: + analyze: + steps: + - uses: anthropics/claude-code-action@v1 + with: + prompt: | + Ref: ${{ format('{0}', github.ref) }} +YML + run bash "$LINTER" "${TT_TMP}/nested-expr.yml" + [ "$status" -eq 0 ] +} diff --git a/test/workflows/feature-ideation/match-discussions.bats b/test/workflows/feature-ideation/match-discussions.bats index 9b63ea9..b92bce0 100644 --- a/test/workflows/feature-ideation/match-discussions.bats +++ b/test/workflows/feature-ideation/match-discussions.bats @@ -192,6 +192,22 @@ build_proposals() { [ "$status" -eq 0 ] } +@test "match: malformed JSON in signals file exits non-zero" { + # Caught by CodeRabbit review on PR petry-projects/.github#85. + printf 'not valid json' >"${TT_TMP}/signals.json" + build_proposals '[{"title":"x"}]' + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -ne 0 ] +} + +@test "match: malformed JSON in proposals file exits non-zero" { + # Caught by CodeRabbit review on PR petry-projects/.github#85. + build_signals '[]' + printf 'not valid json' >"${TT_TMP}/proposals.json" + run bash "$MATCH" "${TT_TMP}/signals.json" "${TT_TMP}/proposals.json" + [ "$status" -ne 0 ] +} + @test "match: idempotent re-run of same proposals against existing matches yields all-matched" { # Simulates the R6 idempotency case: run 1 created Discussions, run 2 # finds them and should NOT propose duplicates. diff --git a/test/workflows/feature-ideation/signals-schema.bats b/test/workflows/feature-ideation/signals-schema.bats index 0281d58..4ad0518 100644 --- a/test/workflows/feature-ideation/signals-schema.bats +++ b/test/workflows/feature-ideation/signals-schema.bats @@ -69,6 +69,15 @@ FIX="${TT_FIXTURES_DIR}/expected" [ "$status" -eq 0 ] } +@test "schema: malformed JSON signals file FAILS with exit code 2" { + # Validates that the OSError/JSONDecodeError path returns 2 (file/data error) + # not 1 (schema validation error). Caught by CodeRabbit review on PR #85. + bad_file="${BATS_TEST_TMPDIR}/malformed.json" + printf '{"schema_version":"1.0.0",' >"$bad_file" + run python3 "$VALIDATOR" "$bad_file" "$SCHEMA" + [ "$status" -eq 2 ] +} + @test "schema: SCHEMA_VERSION constant matches schema file version comment" { # CodeRabbit on PR petry-projects/.github#85: enforce that the # SCHEMA_VERSION constant in collect-signals.sh stays in lockstep with diff --git a/test/workflows/feature-ideation/stubs/gh b/test/workflows/feature-ideation/stubs/gh index 2e8cd24..a16cc61 100755 --- a/test/workflows/feature-ideation/stubs/gh +++ b/test/workflows/feature-ideation/stubs/gh @@ -31,8 +31,11 @@ if [ -n "${GH_STUB_LOG:-}" ]; then fi # Multi-call mode. +# NOTE: TT_TMP (or BATS_TEST_TMPDIR as fallback) must be set so the counter +# file is isolated per test. Without it the fallback /tmp path can collide +# across parallel test suites. Caught by CodeRabbit review on PR #85. if [ -n "${GH_STUB_SCRIPT:-}" ] && [ -f "${GH_STUB_SCRIPT}" ]; then - counter_file="${TT_TMP:-/tmp}/.gh-stub-counter" + counter_file="${TT_TMP:-${BATS_TEST_TMPDIR:-/tmp}}/.gh-stub-counter" count=0 if [ -f "$counter_file" ]; then count=$(cat "$counter_file")