feat(ci): add skill format check workflow to ensure SKILL.md compliance#134
Closed
williamfzc wants to merge 5 commits intolarksuite:mainfrom
Closed
feat(ci): add skill format check workflow to ensure SKILL.md compliance#134williamfzc wants to merge 5 commits intolarksuite:mainfrom
williamfzc wants to merge 5 commits intolarksuite:mainfrom
Conversation
williamfzc
added a commit
to williamfzc/cli
that referenced
this pull request
Mar 31, 2026
- use PR larksuite#134 to test skill path behaviors - use PR larksuite#57 to test multi-path CCM resolution - use PR larksuite#11 to test track renames cross domains
williamfzc
added a commit
to williamfzc/cli
that referenced
this pull request
Mar 31, 2026
- Replaced `area/` prefix with `domain/` for PR labeling to align with existing GitHub labels - Renamed internal constants and variables from `area` to `domain` (e.g. `PATH_TO_AREA_MAP` to `PATH_TO_DOMAIN_MAP`) - Updated `samples.json` test data to use new `domain/` format and `expected_domains` key - Added `scripts/pr-labels/test.js` runner script for continuous validation of labeling logic against PR samples - Corrected expected size label for PR larksuite#134 test sample
fengzhangchi-bytedance
pushed a commit
that referenced
this pull request
Apr 1, 2026
…heck (#148) * feat(ci): add PR size label pipeline * chore(ci): make PR label sync non-blocking * feat(ci): add dry-run mode for PR label sync * feat(ci): add PR label dry-run samples * test(ci): update PR label samples with real historical merged PRs Replaced synthetic or open PR samples with actual merged/closed PRs from the repository to provide a more accurate reflection of the size label categorization. Added 4 samples each for sizes S, M, and L covering docs, fixes, ci, and features. * feat(ci): add high-level area tags for PRs Based on user feedback, fine-grained domain labels (like `domain/base`) are too detailed for the early stages. This change adds support for applying `area/*` tags to indicate which important top-level modules a PR touches. Currently tracked areas: - `area/shortcuts` - `area/skills` - `area/cmd` Minor modules like docs, ci, and tests are intentionally excluded to keep tags focused on critical architectural components. * refactor(ci): extract pr-label-sync logic to a dedicated directory To avoid polluting the root `scripts/` directory, moved `sync_pr_labels.js` and `sync_pr_labels.samples.json` into a new `scripts/sync-pr-labels/` folder. Added a dedicated README to document its usage and behavior. Updated `.github/workflows/pr-labels.yml` to reflect the new path. * refactor(ci): rename pr label script directory for simplicity Renamed `scripts/sync-pr-labels/` to `scripts/pr-labels/` to keep directory names concise. Updated internal references and GitHub workflow files to point to the new path. * ci: add GitHub Actions workflow to check skill format * test(ci): update sample json to include expected_areas Added `expected_areas` lists to each sample in `samples.json` to reflect the newly added `area/*` high-level module tagging logic. Allows testing to accurately check both `size/*` and `area/*` outputs. * refactor(scripts): move skill format check to isolated directory and add README * test(scripts): add positive and negative tests for skill format check * fix(scripts): revert skill changes and downgrade version/metadata checks to warnings * fix(scripts): completely remove version check and skip lark-shared * refactor(ci): improve pr-labels script readability and maintainability - Reorganized code into logical sections with clear comments - Encapsulated GitHub API interactions into a reusable `GitHubClient` class - Extracted and centralized classification logic into a pure `evaluateRules` function - Replaced magic numbers with named constants (`THRESHOLD_L`, `THRESHOLD_XL`) - Fixed `ROOT` path resolution logic - Simplified conditional statements and control flow * ci: fix setup-node version in pr-labels workflow * tmp * refactor(ci): replace generic area labels with business-specific ones - Add PATH_TO_AREA_MAP to map shortcuts/skills paths to business areas (im, vc, ccm, base, mail, calendar, task, contact) - Replace importantAreas with businessAreas throughout the codebase - Remove area/shortcuts, area/skills, area/cmd generic labels - Now generates specific labels like area/im, area/vc, area/ccm, etc. - Update samples.json expected_areas to match new behavior Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): address PR review feedback for label scripts and workflows - Add `edited` event to PR labels workflow to trigger on title changes - Add security warning comment in pr-labels.yml workflow - Update pr-labels README with latest business area labels - Exclude `skills/lark-*` paths from low risk doc classification - Handle renamed files properly in PR path classification - Fix YAML frontmatter extraction to handle CRLF line endings - Use precise regex for YAML key validation instead of substring match - Fix exit code checking logic in skill-format-check test script - Translate Chinese comments in skill-format-check to English * fix(skill-format-check): address CodeRabbit review feedback - Fix frontmatter closing delimiter detection to strictly match '---' using regex, preventing invalid closing tags like '----' from passing. - Improve test fixture reliability by failing tests immediately if fixture preparation fails, avoiding false positives. * fix: address review comments from PR 148 - ci: warn when PR label sync fails in job summary - test(skill-format-check): capture validator output for negative tests - fix(skill-format-check): catch errors when reading SKILL.md to avoid hard crashes * fix: add error handling for directory enumeration in skill-format-check - refactor: use `fs.readdirSync` with `{ withFileTypes: true }` to avoid extra stat calls - fix: catch and report errors gracefully during skills directory enumeration instead of crashing * docs(skill-format-check): clarify `metadata` requirement in README test(pr-labels): add edge case samples for skills paths, CCM multi-paths, and renames * test(pr-labels): add real PR edge case samples - use PR #134 to test skill path behaviors - use PR #57 to test multi-path CCM resolution - use PR #11 to test track renames cross domains * refactor(ci): migrate pr labels from area to domain prefix - Replaced `area/` prefix with `domain/` for PR labeling to align with existing GitHub labels - Renamed internal constants and variables from `area` to `domain` (e.g. `PATH_TO_AREA_MAP` to `PATH_TO_DOMAIN_MAP`) - Updated `samples.json` test data to use new `domain/` format and `expected_domains` key - Added `scripts/pr-labels/test.js` runner script for continuous validation of labeling logic against PR samples - Corrected expected size label for PR #134 test sample * test: use execFileSync instead of execSync in pr-labels test script * fix: resolve target path against process.cwd() instead of __dirname in skill-format-check * docs: correct label prefix in PR label workflow README - Updated README.md to reflect the new `domain/` label prefix instead of `area/` * fix(ci): fix dry-run console output formatting and enforce auth in tests - Removed duplicate domain array interpolation in printDryRunResult - Added process.env.GITHUB_TOKEN guard in test.js to prevent ambiguous failures from API rate limits * fix(ci): ensure PR labels can be applied reliably - Added `issues: write` permission to pr-labels workflow, which is strictly required by the GitHub REST API to modify labels on pull requests - Reordered script execution in `index.js` to apply/remove labels on the PR *before* attempting to sync repository-level label definitions (colors/descriptions). The definition sync is now a trailing best-effort step with error catching so transient repo-level API failures don't abort the critical path. * fix(ci): fix edge cases in pr-label index script - Added missing `skills/lark-task/` to `PATH_TO_DOMAIN_MAP` to properly detect task domain modifications - Updated GitHub REST API error checking in `syncLabelDefinition` to reliably match `error.status === 422` rather than loosely checking substring - Moved token presence check in `main()` to happen before `resolveContext` to avoid triggering unauthenticated 401 API limits when GITHUB_TOKEN is omitted locally * test(ci): clean up PR label test samples - Removed duplicate PR entries (#11 and #57) to reduce redundant API calls during testing - Renamed sample test cases to correctly reflect their expected labels (e.g. `size-l-skill-format-check` -> `size-m-skill-format-check`) * fix(ci): bootstrap new labels before applying to PRs - Prior changes correctly made full label sync best-effort, but broke the flow for brand new domains - GitHub API returns a 422 error if you attempt to attach a label to an Issue/PR that does not exist in the repository - Added a targeted bootstrap loop to create/sync specifically the labels in `toAdd` before attempting `client.addLabels()` - Left the remaining global label synchronization as a best-effort trailing action * test(ci): automate PR label regression testing - Added a dedicated GitHub Actions workflow (`pr-labels-test.yml`) to automatically run `test.js` against `samples.json` whenever the labeling logic is updated - Documented local testing instructions in `scripts/pr-labels/README.md` --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
tuxedomm
pushed a commit
that referenced
this pull request
Apr 3, 2026
…heck (#148) * feat(ci): add PR size label pipeline * chore(ci): make PR label sync non-blocking * feat(ci): add dry-run mode for PR label sync * feat(ci): add PR label dry-run samples * test(ci): update PR label samples with real historical merged PRs Replaced synthetic or open PR samples with actual merged/closed PRs from the repository to provide a more accurate reflection of the size label categorization. Added 4 samples each for sizes S, M, and L covering docs, fixes, ci, and features. * feat(ci): add high-level area tags for PRs Based on user feedback, fine-grained domain labels (like `domain/base`) are too detailed for the early stages. This change adds support for applying `area/*` tags to indicate which important top-level modules a PR touches. Currently tracked areas: - `area/shortcuts` - `area/skills` - `area/cmd` Minor modules like docs, ci, and tests are intentionally excluded to keep tags focused on critical architectural components. * refactor(ci): extract pr-label-sync logic to a dedicated directory To avoid polluting the root `scripts/` directory, moved `sync_pr_labels.js` and `sync_pr_labels.samples.json` into a new `scripts/sync-pr-labels/` folder. Added a dedicated README to document its usage and behavior. Updated `.github/workflows/pr-labels.yml` to reflect the new path. * refactor(ci): rename pr label script directory for simplicity Renamed `scripts/sync-pr-labels/` to `scripts/pr-labels/` to keep directory names concise. Updated internal references and GitHub workflow files to point to the new path. * ci: add GitHub Actions workflow to check skill format * test(ci): update sample json to include expected_areas Added `expected_areas` lists to each sample in `samples.json` to reflect the newly added `area/*` high-level module tagging logic. Allows testing to accurately check both `size/*` and `area/*` outputs. * refactor(scripts): move skill format check to isolated directory and add README * test(scripts): add positive and negative tests for skill format check * fix(scripts): revert skill changes and downgrade version/metadata checks to warnings * fix(scripts): completely remove version check and skip lark-shared * refactor(ci): improve pr-labels script readability and maintainability - Reorganized code into logical sections with clear comments - Encapsulated GitHub API interactions into a reusable `GitHubClient` class - Extracted and centralized classification logic into a pure `evaluateRules` function - Replaced magic numbers with named constants (`THRESHOLD_L`, `THRESHOLD_XL`) - Fixed `ROOT` path resolution logic - Simplified conditional statements and control flow * ci: fix setup-node version in pr-labels workflow * tmp * refactor(ci): replace generic area labels with business-specific ones - Add PATH_TO_AREA_MAP to map shortcuts/skills paths to business areas (im, vc, ccm, base, mail, calendar, task, contact) - Replace importantAreas with businessAreas throughout the codebase - Remove area/shortcuts, area/skills, area/cmd generic labels - Now generates specific labels like area/im, area/vc, area/ccm, etc. - Update samples.json expected_areas to match new behavior Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): address PR review feedback for label scripts and workflows - Add `edited` event to PR labels workflow to trigger on title changes - Add security warning comment in pr-labels.yml workflow - Update pr-labels README with latest business area labels - Exclude `skills/lark-*` paths from low risk doc classification - Handle renamed files properly in PR path classification - Fix YAML frontmatter extraction to handle CRLF line endings - Use precise regex for YAML key validation instead of substring match - Fix exit code checking logic in skill-format-check test script - Translate Chinese comments in skill-format-check to English * fix(skill-format-check): address CodeRabbit review feedback - Fix frontmatter closing delimiter detection to strictly match '---' using regex, preventing invalid closing tags like '----' from passing. - Improve test fixture reliability by failing tests immediately if fixture preparation fails, avoiding false positives. * fix: address review comments from PR 148 - ci: warn when PR label sync fails in job summary - test(skill-format-check): capture validator output for negative tests - fix(skill-format-check): catch errors when reading SKILL.md to avoid hard crashes * fix: add error handling for directory enumeration in skill-format-check - refactor: use `fs.readdirSync` with `{ withFileTypes: true }` to avoid extra stat calls - fix: catch and report errors gracefully during skills directory enumeration instead of crashing * docs(skill-format-check): clarify `metadata` requirement in README test(pr-labels): add edge case samples for skills paths, CCM multi-paths, and renames * test(pr-labels): add real PR edge case samples - use PR #134 to test skill path behaviors - use PR #57 to test multi-path CCM resolution - use PR #11 to test track renames cross domains * refactor(ci): migrate pr labels from area to domain prefix - Replaced `area/` prefix with `domain/` for PR labeling to align with existing GitHub labels - Renamed internal constants and variables from `area` to `domain` (e.g. `PATH_TO_AREA_MAP` to `PATH_TO_DOMAIN_MAP`) - Updated `samples.json` test data to use new `domain/` format and `expected_domains` key - Added `scripts/pr-labels/test.js` runner script for continuous validation of labeling logic against PR samples - Corrected expected size label for PR #134 test sample * test: use execFileSync instead of execSync in pr-labels test script * fix: resolve target path against process.cwd() instead of __dirname in skill-format-check * docs: correct label prefix in PR label workflow README - Updated README.md to reflect the new `domain/` label prefix instead of `area/` * fix(ci): fix dry-run console output formatting and enforce auth in tests - Removed duplicate domain array interpolation in printDryRunResult - Added process.env.GITHUB_TOKEN guard in test.js to prevent ambiguous failures from API rate limits * fix(ci): ensure PR labels can be applied reliably - Added `issues: write` permission to pr-labels workflow, which is strictly required by the GitHub REST API to modify labels on pull requests - Reordered script execution in `index.js` to apply/remove labels on the PR *before* attempting to sync repository-level label definitions (colors/descriptions). The definition sync is now a trailing best-effort step with error catching so transient repo-level API failures don't abort the critical path. * fix(ci): fix edge cases in pr-label index script - Added missing `skills/lark-task/` to `PATH_TO_DOMAIN_MAP` to properly detect task domain modifications - Updated GitHub REST API error checking in `syncLabelDefinition` to reliably match `error.status === 422` rather than loosely checking substring - Moved token presence check in `main()` to happen before `resolveContext` to avoid triggering unauthenticated 401 API limits when GITHUB_TOKEN is omitted locally * test(ci): clean up PR label test samples - Removed duplicate PR entries (#11 and #57) to reduce redundant API calls during testing - Renamed sample test cases to correctly reflect their expected labels (e.g. `size-l-skill-format-check` -> `size-m-skill-format-check`) * fix(ci): bootstrap new labels before applying to PRs - Prior changes correctly made full label sync best-effort, but broke the flow for brand new domains - GitHub API returns a 422 error if you attempt to attach a label to an Issue/PR that does not exist in the repository - Added a targeted bootstrap loop to create/sync specifically the labels in `toAdd` before attempting `client.addLabels()` - Left the remaining global label synchronization as a best-effort trailing action * test(ci): automate PR label regression testing - Added a dedicated GitHub Actions workflow (`pr-labels-test.yml`) to automatically run `test.js` against `samples.json` whenever the labeling logic is updated - Documented local testing instructions in `scripts/pr-labels/README.md` --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR introduces an automated formatting validation mechanism for all
SKILL.mdfiles in the repository.It ensures that the metadata structures conform to the standards set by
skill-template.md.Changes
scripts/skill-format-check/index.jsvalidationscript. It uses pure Node.js (
fs,path) without introducing any 3rd-party dependencies, keepingpackage.jsonclean..github/workflows/skill-format-check.ymlto automatically triggervalidation on pushes and pull requests when the
skills/directory or the checking script itself is modified.test.sh) with 6 fixtures (3 positive, 3 negative) toquickly verify the format checker's logic.
lark-whiteboard,lark- shared), missingversionandmetadatafields in YAML frontmatter have been downgraded from blockingerrors to terminal warnings. Missing
nameordescriptionwill still block the PR.-fail the CI.
Testing
To run the format check tests locally: