diff --git a/.editorconfig b/.editorconfig index de4966e..204f58d 100644 --- a/.editorconfig +++ b/.editorconfig @@ -6,6 +6,7 @@ root = true # All files [*] charset = utf-8 +end_of_line = lf indent_style = space indent_size = 4 insert_final_newline = true diff --git a/.github/agents/repo-consistency-agent.md b/.github/agents/repo-consistency-agent.md index d3cf95d..3f289af 100644 --- a/.github/agents/repo-consistency-agent.md +++ b/.github/agents/repo-consistency-agent.md @@ -81,6 +81,34 @@ The agent reviews the following areas for consistency with the template: - `quality/` (auto-generated) - **Definition Files**: `definition.yaml` files for document generation +### Tracking Template Evolution + +To ensure downstream projects benefit from recent template improvements, review recent pull requests +merged into the template repository: + +1. **List Recent PRs**: Retrieve recently merged PRs from `demaconsulting/TemplateDotNetTool` + - Review the last 10-20 PRs to identify template improvements + +2. **Identify Propagatable Changes**: For each PR, determine if changes should apply to downstream + projects: + - Focus on structural changes (workflows, agents, configurations) over content-specific changes + - Note changes to `.github/`, linting configurations, project patterns, and documentation + structure + +3. **Check Downstream Application**: Verify if identified changes exist in the downstream project: + - Check if similar files/patterns exist in downstream + - Compare file contents between template and downstream project + - Look for similar PR titles or commit messages in downstream repository history + +4. **Recommend Missing Updates**: For changes not yet applied, include them in the consistency + review with: + - Description of the template change (reference PR number) + - Explanation of benefits for the downstream project + - Specific files or patterns that need updating + +This technique ensures downstream projects don't miss important template improvements and helps +maintain long-term consistency. + ### Review Process 1. **Identify Differences**: Compare downstream repository structure with template diff --git a/.github/agents/requirements-agent.md b/.github/agents/requirements-agent.md index 292d24a..8babb7e 100644 --- a/.github/agents/requirements-agent.md +++ b/.github/agents/requirements-agent.md @@ -49,6 +49,23 @@ Follow the `requirements.yaml` structure: - Linked to appropriate test(s) - Enforced via: `dotnet reqstream --requirements requirements.yaml --tests "test-results/**/*.trx" --enforce` +### Test Source Filters + +Test links in `requirements.yaml` can include a source filter prefix to restrict which test results count as +evidence. This is critical for platform and framework requirements - **never remove these filters**. + +- `windows@TestName` - proves the test passed on a Windows platform +- `ubuntu@TestName` - proves the test passed on a Linux (Ubuntu) platform +- `net8.0@TestName` - proves the test passed under the .NET 8 target framework +- `net9.0@TestName` - proves the test passed under the .NET 9 target framework +- `net10.0@TestName` - proves the test passed under the .NET 10 target framework +- `dotnet8.x@TestName` - proves the self-validation test ran on a machine with .NET 8.x runtime +- `dotnet9.x@TestName` - proves the self-validation test ran on a machine with .NET 9.x runtime +- `dotnet10.x@TestName` - proves the self-validation test ran on a machine with .NET 10.x runtime + +Without the source filter, a test result from any platform/framework satisfies the requirement. Removing a +filter invalidates the evidence for platform/framework requirements. + ## Defer To - **Software Developer Agent**: For implementing self-validation tests diff --git a/.github/agents/test-developer.md b/.github/agents/test-developer.md index 585a925..02a40c9 100644 --- a/.github/agents/test-developer.md +++ b/.github/agents/test-developer.md @@ -64,6 +64,23 @@ public void ClassName_MethodUnderTest_Scenario_ExpectedBehavior() - Failure-testing and error handling scenarios - Verifying internal behavior beyond requirement scope +### Test Source Filters + +Test links in `requirements.yaml` can include a source filter prefix to restrict which test results count as +evidence. These filters are critical for platform and framework requirements - **do not remove them**. + +- `windows@TestName` - proves the test passed on a Windows platform +- `ubuntu@TestName` - proves the test passed on a Linux (Ubuntu) platform +- `net8.0@TestName` - proves the test passed under the .NET 8 target framework +- `net9.0@TestName` - proves the test passed under the .NET 9 target framework +- `net10.0@TestName` - proves the test passed under the .NET 10 target framework +- `dotnet8.x@TestName` - proves the self-validation test ran on a machine with .NET 8.x runtime +- `dotnet9.x@TestName` - proves the self-validation test ran on a machine with .NET 9.x runtime +- `dotnet10.x@TestName` - proves the self-validation test ran on a machine with .NET 10.x runtime + +Removing a source filter means a test result from any environment can satisfy the requirement, which invalidates +the evidence-based proof that the tool works on a specific platform or framework. + ### VersionMark-Specific - **NOT self-validation tests** - those are handled by Software Developer Agent @@ -71,6 +88,41 @@ public void ClassName_MethodUnderTest_Scenario_ExpectedBehavior() - Use MSTest V4 testing framework - Follow existing naming conventions in the test suite +### MSTest V4 Best Practices + +Common anti-patterns to avoid (not exhaustive): + +1. **Avoid Assertions in Catch Blocks (MSTEST0058)** - Instead of wrapping code in try/catch and asserting in the + catch block, use `Assert.ThrowsExactly()`: + + ```csharp + var ex = Assert.ThrowsExactly(() => SomeWork()); + Assert.Contains("Some message", ex.Message); + ``` + +2. **Avoid using Assert.IsTrue / Assert.IsFalse for equality checks** - Use `Assert.AreEqual` / + `Assert.AreNotEqual` instead, as it provides better failure messages: + + ```csharp + // ❌ Bad: Assert.IsTrue(result == expected); + // ✅ Good: Assert.AreEqual(expected, result); + ``` + +3. **Avoid non-public test classes and methods** - Test classes and `[TestMethod]` methods must be `public` or + they will be silently ignored: + + ```csharp + // ❌ Bad: internal class MyTests + // ✅ Good: public class MyTests + ``` + +4. **Avoid Assert.IsTrue(collection.Count == N)** - Use `Assert.HasCount` for count assertions: + + ```csharp + // ❌ Bad: Assert.IsTrue(collection.Count == 3); + // ✅ Good: Assert.HasCount(3, collection); + ``` + ## Defer To - **Requirements Agent**: For test strategy and coverage requirements diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 50f6773..1f4b90f 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -45,7 +45,7 @@ Please run the following checks before submitting: - [ ] **Spell checker passes**: `cspell "**/*.{md,cs}"` - [ ] **Markdown linter passes**: `markdownlint "**/*.md"` -- [ ] **YAML linter passes**: `yamllint '**/*.{yml,yaml}'` +- [ ] **YAML linter passes**: `yamllint .` ### Testing diff --git a/.gitignore b/.gitignore index 63f68b6..ace443f 100644 --- a/.gitignore +++ b/.gitignore @@ -90,6 +90,13 @@ __pycache__/ docs/**/*.html docs/**/*.pdf !docs/template/** +docs/requirements/requirements.md +docs/justifications/justifications.md +docs/tracematrix/tracematrix.md +docs/quality/codeql-quality.md +docs/quality/sonar-quality.md +docs/buildnotes.md +docs/buildnotes/versions.md # Test results TestResults/ diff --git a/.markdownlint-cli2.jsonc b/.markdownlint-cli2.jsonc index 8d25b55..a46ee1a 100644 --- a/.markdownlint-cli2.jsonc +++ b/.markdownlint-cli2.jsonc @@ -4,6 +4,7 @@ "MD003": { "style": "atx" }, "MD007": { "indent": 2 }, "MD013": { "line_length": 120 }, + "MD025": false, "MD033": false, "MD041": false }, diff --git a/AGENTS.md b/AGENTS.md index 0cc4d9d..722fbb2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,6 +43,23 @@ VersionMark is a tool that: - Enforced in CI: `dotnet reqstream --requirements requirements.yaml --tests "test-results/**/*.trx" --enforce` - When adding features: add requirement + link to test +## Test Source Filters + +Test links in `requirements.yaml` can include a source filter prefix to restrict which test results count as +evidence. This is critical for platform and framework requirements - **do not remove these filters**. + +- `windows@TestName` - proves the test passed on a Windows platform +- `ubuntu@TestName` - proves the test passed on a Linux (Ubuntu) platform +- `net8.0@TestName` - proves the test passed under the .NET 8 target framework +- `net9.0@TestName` - proves the test passed under the .NET 9 target framework +- `net10.0@TestName` - proves the test passed under the .NET 10 target framework +- `dotnet8.x@TestName` - proves the self-validation test ran on a machine with .NET 8.x runtime +- `dotnet9.x@TestName` - proves the self-validation test ran on a machine with .NET 9.x runtime +- `dotnet10.x@TestName` - proves the self-validation test ran on a machine with .NET 10.x runtime + +Without the source filter, a test result from any platform/framework satisfies the requirement. Adding the filter +ensures the CI evidence comes specifically from the required environment. + ## Testing - **Test Naming**: `VersionMark_MethodUnderTest_Scenario` for self-validation tests diff --git a/docs/guide/guide.md b/docs/guide/guide.md index 515126b..c37d51f 100644 --- a/docs/guide/guide.md +++ b/docs/guide/guide.md @@ -1,10 +1,27 @@ -# VersionMark User Guide +# Introduction VersionMark is a tool for capturing and publishing tool version information across CI/CD environments. It helps track which versions of build tools, compilers, and dependencies are used in different jobs and environments. -## Installation +## Purpose + +VersionMark provides a standardized way to capture, track, and document tool versions used +across different CI/CD jobs and environments. This ensures transparency, reproducibility, and +helps teams understand which tool versions are used in their build and deployment pipelines. + +## Scope + +This user guide covers: + +- Installing and configuring VersionMark +- Capturing tool versions in CI/CD environments +- Publishing version information to documentation +- Integrating with CI/CD systems like GitHub Actions +- Configuration file format and options +- Troubleshooting and best practices + +# Installation Install the tool globally using the .NET CLI: @@ -12,9 +29,9 @@ Install the tool globally using the .NET CLI: dotnet tool install -g DemaConsulting.VersionMark ``` -## Quick Start +# Quick Start -### Step 1: Create Configuration File +## Step 1: Create Configuration File Create a `.versionmark.yaml` file in your repository root: @@ -33,7 +50,7 @@ tools: regex: 'gcc \(.*\) (?\d+\.\d+\.\d+)' ``` -### Step 2: Capture Tool Versions in CI/CD +## Step 2: Capture Tool Versions in CI/CD In each CI/CD job, capture tool versions with a unique job identifier: @@ -43,7 +60,7 @@ versionmark --capture --job-id "windows-net8" This creates a JSON file (e.g., `versionmark-windows-net8.json`) containing the captured versions. -### Step 3: Publish Versions to Documentation +## Step 3: Publish Versions to Documentation After all jobs complete, publish the captured versions: @@ -53,9 +70,9 @@ versionmark --publish --report versions.md This generates a markdown file consolidating versions from all jobs. -## Command-Line Reference +# Command-Line Reference -### Global Options +## Global Options These options are available for all commands: @@ -66,7 +83,7 @@ These options are available for all commands: | `--silent` | Suppress console output | | `--log ` | Write output to log file | -### Capture Command +## Capture Command Capture tool versions from the current environment: @@ -74,7 +91,7 @@ Capture tool versions from the current environment: versionmark --capture --job-id [options] [-- tool1 tool2 ...] ``` -#### Options +### Options | Option | Description | | ------------------------- | ----------------------------------------------------------------------------------- | @@ -85,7 +102,7 @@ versionmark --capture --job-id [options] [-- tool1 tool2 ...] | `-- ` | List of tool names to capture. If not specified, all tools defined in the | | | configuration file will be captured. | -#### Example +### Example ```bash # Capture all tools defined in config @@ -98,7 +115,7 @@ versionmark --capture --job-id "linux-gcc" -- gcc make cmake versionmark --capture --job-id "macos" --output versions/macos.json ``` -### Publish Command +## Publish Command Publish captured versions to markdown documentation: @@ -106,7 +123,7 @@ Publish captured versions to markdown documentation: versionmark --publish --report [options] [-- pattern1 pattern2 ...] ``` -#### Publish Options +### Publish Options | Option | Description | | ------------------------ | ---------------------------------------------------------------- | @@ -115,7 +132,7 @@ versionmark --publish --report [options] [-- pattern1 pattern2 ...] | `--report-depth ` | Heading depth for markdown output (default: 2, min: 1, max: 6) | | `-- ` | Glob patterns for JSON files (default: `versionmark-*.json`) | -#### Publish Examples +### Publish Examples ```bash # Basic publish with default patterns (versionmark-*.json) @@ -131,7 +148,7 @@ versionmark --publish --report versions.md --report-depth 3 versionmark --publish --report docs/versions.md --report-depth 1 -- versionmark-*.json ``` -#### Glob Patterns +### Glob Patterns The publish command uses glob patterns to find JSON files. Multiple patterns can be specified after the `--` separator: @@ -149,11 +166,11 @@ Common glob pattern syntax: - `[abc]` - Matches one character from the set - `{a,b}` - Matches either pattern a or b -## Configuration File Format +# Configuration File Format The `.versionmark.yaml` file defines which tools to capture and how to extract version information. -### Basic Configuration +## Basic Configuration ```yaml tools: @@ -170,7 +187,7 @@ tools: regex: 'Python (?\d+\.\d+\.\d+)' ``` -### Configuration Properties +## Configuration Properties Each tool entry in the `tools` dictionary supports the following properties: @@ -185,7 +202,7 @@ Each tool entry in the `tools` dictionary supports the following properties: | `regex-linux` | No | Regex override for Linux | | `regex-macos` | No | Regex override for macOS | -### OS-Specific Overrides +## OS-Specific Overrides You can provide platform-specific commands and regex patterns for tools that have different behavior on different operating systems: @@ -213,7 +230,7 @@ The tool uses OS-specific overrides when running on the corresponding platform. is specified for the current platform, it falls back to the default `command` and `regex` values. -### Regular Expression Tips +## Regular Expression Tips The regex must contain a named 'version' capture group using .NET syntax `(?...)` that captures the version number. Examples: @@ -223,17 +240,16 @@ captures the version number. Examples: - **Multiline output**: `(?m)version (?\d+\.\d+\.\d+)` - Uses multiline mode - **Build metadata**: `(?\d+\.\d+\.\d+[-+][a-zA-Z0-9.]+)` - Captures `1.2.3-beta.1` -## Output Formats +# Output Formats -### Capture Output (JSON) +## Capture Output (JSON) When you run the capture command, VersionMark creates a JSON file with the following structure: ```json { - "job-id": "windows-net8", - "timestamp": "2024-03-15T10:30:00Z", - "versions": { + "JobId": "windows-net8", + "Versions": { "dotnet": "8.0.100", "node": "20.11.0", "gcc": "13.2.0", @@ -242,54 +258,53 @@ When you run the capture command, VersionMark creates a JSON file with the follo } ``` -#### JSON Structure +### JSON Structure -- **job-id**: The unique identifier provided via `--job-id` -- **timestamp**: ISO 8601 timestamp of when versions were captured -- **versions**: Object mapping tool names to their captured versions +- **JobId**: The unique identifier provided via `--job-id` +- **Versions**: Object mapping tool names to their captured versions -### Publish Output (Markdown) +## Publish Output (Markdown) The publish command generates a markdown file with a bulleted list of tool versions: -#### Example 1: All Jobs Use Same Version +### Example 1: All Jobs Use Same Version ```markdown ## Tool Versions -- **dotnet**: 8.0.100 (All jobs) -- **node**: 20.11.0 (All jobs) +- **dotnet**: 8.0.100 +- **node**: 20.11.0 ``` -When all jobs capture the same version of a tool, it's displayed as "(All jobs)". +When all jobs capture the same version of a tool, only the version is displayed without job +identifiers. -#### Example 2: Different Versions Across Jobs +### Example 2: Different Versions Across Jobs ```markdown ## Tool Versions -- **dotnet**: 8.0.100 (All jobs) -- **gcc**: 11.4.0 (windows-net8, windows-net9), 13.2.0 (linux-net8, -linux-net9) -- **node**: 20.11.0 (windows-net8, linux-net8), 21.0.0 (windows-net9, -linux-net9) +- **dotnet**: 8.0.100 +- **gcc**: 11.4.0 (windows-net8, windows-net9) +- **gcc**: 13.2.0 (linux-net8, linux-net9) +- **node**: 20.11.0 (windows-net8, linux-net8) +- **node**: 21.0.0 (windows-net9, linux-net9) ``` -When a tool has different versions across jobs, each version is listed separately with the -jobs that use it shown in subscripts. Job IDs within each subscript are listed in +When a tool has different versions across jobs, each version is listed as a separate bullet +with the jobs that use it shown in parentheses. Job IDs within each group are listed in alphabetical order. -#### Output Format Details +### Output Format Details - **Heading**: Controlled by `--report-depth` parameter (default: `##` for depth 2) - **Tool Order**: Tools are listed in alphabetical order (case-insensitive) - **Version Order**: When multiple versions exist, they are sorted alphabetically - **Job IDs**: Within each version group, job IDs are sorted alphabetically -- **Subscripts**: Job IDs use HTML `` tags for compact display -## CI/CD Integration +# CI/CD Integration -### GitHub Actions Example +## GitHub Actions Example Here's a complete example of using VersionMark in a GitHub Actions workflow: @@ -370,7 +385,7 @@ jobs: git push ``` -### Key Integration Points +## Key Integration Points 1. **Install VersionMark**: Install the tool in each job that needs to capture versions 2. **Capture per Job**: Run `versionmark capture` with a unique `--job-id` in each job @@ -378,9 +393,9 @@ jobs: 4. **Download Artifacts**: In the publish job, download all captured JSON files 5. **Publish**: Run `versionmark publish` to generate consolidated documentation -## Common Workflows +# Common Workflows -### Workflow 1: Track Build Environment Versions +## Workflow 1: Track Build Environment Versions Use VersionMark to document which tool versions are used in your build environment: @@ -396,7 +411,7 @@ tools: regex: '(?\d+\.\d+\.\d+\.\d+)' ``` -### Workflow 2: Compare Versions Across Platforms +## Workflow 2: Compare Versions Across Platforms Track how tool versions differ between Windows, Linux, and macOS: @@ -412,7 +427,7 @@ tools: regex: 'clang version (?\d+\.\d+\.\d+)' ``` -### Workflow 3: Monitor Dependency Versions +## Workflow 3: Monitor Dependency Versions Track versions of runtime dependencies and tools: @@ -432,9 +447,9 @@ tools: regex: 'Terraform v(?\d+\.\d+\.\d+)' ``` -## Troubleshooting +# Troubleshooting -### Tool Not Found +## Tool Not Found If a tool command fails because the tool is not installed: @@ -442,7 +457,7 @@ If a tool command fails because the tool is not installed: - The capture will continue for other tools - The published output will note which tools failed to capture -### Version Not Matched +## Version Not Matched If the regex doesn't match the command output: @@ -451,7 +466,7 @@ If the regex doesn't match the command output: - Use online regex testers to validate your pattern - Remember to escape special regex characters -### OS-Specific Issues +## OS-Specific Issues If a tool behaves differently on different platforms: @@ -459,7 +474,7 @@ If a tool behaves differently on different platforms: - Test on each platform to ensure the commands work - Consider using platform-specific tools in separate configurations -### No JSON Files Found +## No JSON Files Found If the publish command reports "No JSON files found": @@ -468,7 +483,7 @@ If the publish command reports "No JSON files found": - Use `-- versionmark-*.json` explicitly if files don't match the default pattern - Check that capture jobs successfully created JSON files before publishing -### Invalid JSON Files +## Invalid JSON Files If a JSON file cannot be parsed during publish: @@ -477,7 +492,7 @@ If a JSON file cannot be parsed during publish: - Verify the file contains required fields: `JobId` and `Versions` - Re-run the capture command if the file is corrupted -## Best Practices +# Best Practices 1. **Use Descriptive Job IDs**: Make job-ids descriptive (e.g., `windows-net8-release` instead of `job1`) diff --git a/requirements.yaml b/requirements.yaml index 7777802..4f96adf 100644 --- a/requirements.yaml +++ b/requirements.yaml @@ -11,6 +11,21 @@ # Platform-specific requirements require test results from CI/CD runs across # the full OS and runtime matrix. # +# Test links can include a source filter prefix (e.g. "windows@", "ubuntu@", "net8.0@", +# "dotnet8.x@") to restrict which test results count as evidence for a requirement. This +# is critical for platform and framework requirements - removing these filters invalidates +# the evidence-based proof. +# +# Source filter prefixes: +# windows@TestName - proves the test passed on a Windows platform +# ubuntu@TestName - proves the test passed on a Linux (Ubuntu) platform +# net8.0@TestName - proves the test passed under the .NET 8 target framework +# net9.0@TestName - proves the test passed under the .NET 9 target framework +# net10.0@TestName - proves the test passed under the .NET 10 target framework +# dotnet8.x@TestName - proves the self-validation test ran with .NET 8.x runtime +# dotnet9.x@TestName - proves the self-validation test ran with .NET 9.x runtime +# dotnet10.x@TestName - proves the self-validation test ran with .NET 10.x runtime +# --- sections: - title: VersionMark Requirements