[Feature Request]: Automate Static Validation documentation

[eriqua]

Description

We should improve the documentation of the Pester tests we run.

A standard Wiki page listing them is probably not the best choice as the list can soon get stale.

Discuss extending the pipeline generating the Wiki from docs with a function retrieving the test description from the global tests file and printing the list to an automatically generated Wiki page.

E.g. from the It:

It "[<moduleFolderName>] outputs' description should start with a capital letter and contain text ending with a dot." -TestCases $deploymentFolderTestCases {
...
It '[<moduleFolderName>] All parameters in parameters files exist in template file (deploy.json)' -TestCases $deploymentFolderTestCases {
...
It '[<moduleFolderName>] All required parameters in template file (deploy.json) should exist in parameters files' -TestCases $deploymentFolderTestCases {

generate and publish

This is the list of Pester tests:
- Outputs' description should start with a capital letter and contain text ending with a dot.
- All parameters in parameters files exist in template file (deploy.json)
- All required parameters in template file (deploy.json) should exist in parameters files

[AlexanderSehr]

I agree with the idea. This could be an update to one of the docs - probably a separate markdown document to not run into conflicts. The grouping could be done via the 'Describe' & 'Context' containers - while we should proably do a sweep to make sure all test descriptions make sense.

[AlexanderSehr]

Can be done using the following working draft of a function

function Get-TestsAsMarkdown {

    [CmdletBinding()]
    param (
        [Parameter(Mandatory)]
        [string] $TestFilePath
    )

    $content = Get-Content $TestFilePath

    $relevantContent = [System.Collections.ArrayList]@()
    foreach ($line in $content) {
        if ($line -match "^\s*Describe '(.*?)'.*$") {
            $formatted = $Matches[1]
            $formatted = (($formatted -split '\s*\[<.+?>\]\s*') -join ' ').Trim()
            $relevantContent += "# Describe $formatted"
        } elseif ( $line -match "^\s*Context '(.*?)'.*$" ) {
            $formatted = $Matches[1]
            $formatted = (($formatted -split '\s*\[<.+?>\]\s*') -join ' ').Trim()
            $relevantContent += "## Context $formatted"
        } elseif ( $line -match "^\s*It '(.*?)'.*$" ) {
            $formatted = $Matches[1]
            $formatted = (($formatted -split '\s*\[<.+?>\]\s*') -join ' ').Trim()
            $relevantContent += "1. $formatted"
        }
    }

    return $relevantContent
}
$relevantContent = Get-TestsAsMarkdown -TestFilePath '.\utilities\pipelines\staticValidation\module.tests.ps1'
$relevantContent

Output

# Describe File/folder tests
## Context General module folder tests
1. Module should have a GitHub workflow
1. Module should have an Azure DevOps pipeline
1. Module should contain a [deploy.json/deploy.bicep] file
1. Module should contain a [readme.md] file
1. Module should contain a [.test] folder
1. Module should contain a [version.json] file
## Context .test folder
1. folder should contain one or more test files
1. JSON test files in the .test folder should be valid json
# Describe Readme tests
## Context Readme content tests
1. Readme.md file should not be empty
1. Readme.md file should contain these sections in order: Navigation, Resource Types, Parameters, Outputs, Cross-referenced modules, Deployment examples
1. Resources section should contain all resources from the template file
1. Resources section should not contain more resources than the template file
1. Parameters section should contain a table for each existing parameter category in the following order: Required, Conditional, Optional, Generated
1. parameter tables should provide columns in the following order: Parameter Name, Type, Default Value, Allowed Values, Description. Each column should be present unless empty for all the rows.
1. Parameters section should contain all parameters from the template file
1. Outputs section should contain a table with these column names in order: Output Name, Type
1. Output section should contain all outputs defined in the template file
1. Dependencies section should contain all cross-references defined in the template file
1. Set-ModuleReadMe script should not apply any updates
# Describe Parameter file tests
## Context Deployment test file tests
1. Bicep test deployment name should contain [-test-]
1. Bicep test deployment should have parameter [serviceShort]
1. JSON test deployment name should contain [-test-]
1. JSON test deployment should have parameter [serviceShort]
## Context Parameter file token tests
1. [Tokens] Parameter file should not contain the plain value for token guid
# Describe Deployment template tests
## Context Deployment template tests
1. the template file should not be empty
1. Template schema version should be the latest
1. Template schema should use HTTPS reference
1. All apiVersion properties should be set to a static, hard-coded value
1. the template file should contain required elements: schema, contentVersion, resources
1. If delete lock is implemented, the template should have a lock parameter with the default value of [
1. Parameter names should be camel-cased (no dashes or underscores and must start with lower-case letter)
1. Variable names should be camel-cased (no dashes or underscores and must start with lower-case letter)
1. Output names should be camel-cased (no dashes or underscores and must start with lower-case letter)
1. CUA ID deployment should be present in the template
1. Location output should be returned for resources that use it
1. Resource Group output should exist for resources that are deployed into a resource group scope
1. Resource name output should exist
1. Resource ID output should exist
1. All parameters in parameters files exist in template file (deploy.json)
1. All required parameters in template file (deploy.json) should exist in parameters files
1. All non-required parameters in template file should not have description that start with "Required."
## Context Parameter file token tests
1. [Tokens] Parameter file should not contain the plain value for token
1. In used resource type should use one of the recent API version(s). Currently using

Rendered

Describe File/folder tests

Context General module folder tests

1. Module should have a GitHub workflow
2. Module should have an Azure DevOps pipeline
3. Module should contain a [deploy.json/deploy.bicep] file
4. Module should contain a [readme.md] file
5. Module should contain a [.test] folder
6. Module should contain a [version.json] file

Context .test folder

1. folder should contain one or more test files
2. JSON test files in the .test folder should be valid json

Describe Readme tests

Context Readme content tests

1. Readme.md file should not be empty
2. Readme.md file should contain these sections in order: Navigation, Resource Types, Parameters, Outputs, Cross-referenced modules, Deployment examples
3. Resources section should contain all resources from the template file
4. Resources section should not contain more resources than the template file
5. Parameters section should contain a table for each existing parameter category in the following order: Required, Conditional, Optional, Generated
6. parameter tables should provide columns in the following order: Parameter Name, Type, Default Value, Allowed Values, Description. Each column should be present unless empty for all the rows.
7. Parameters section should contain all parameters from the template file
8. Outputs section should contain a table with these column names in order: Output Name, Type
9. Output section should contain all outputs defined in the template file
10. Dependencies section should contain all cross-references defined in the template file
11. Set-ModuleReadMe script should not apply any updates

Describe Parameter file tests

Context Deployment test file tests

1. Bicep test deployment name should contain [-test-]
2. Bicep test deployment should have parameter [serviceShort]
3. JSON test deployment name should contain [-test-]
4. JSON test deployment should have parameter [serviceShort]

Context Parameter file token tests

1. [Tokens] Parameter file should not contain the plain value for token guid

Describe Deployment template tests

Context Deployment template tests

1. the template file should not be empty
2. Template schema version should be the latest
3. Template schema should use HTTPS reference
4. All apiVersion properties should be set to a static, hard-coded value
5. the template file should contain required elements: schema, contentVersion, resources
6. If delete lock is implemented, the template should have a lock parameter with the default value of [
7. Parameter names should be camel-cased (no dashes or underscores and must start with lower-case letter)
8. Variable names should be camel-cased (no dashes or underscores and must start with lower-case letter)
9. Output names should be camel-cased (no dashes or underscores and must start with lower-case letter)
10. CUA ID deployment should be present in the template
11. Location output should be returned for resources that use it
12. Resource Group output should exist for resources that are deployed into a resource group scope
13. Resource name output should exist
14. Resource ID output should exist
15. All parameters in parameters files exist in template file (deploy.json)
16. All required parameters in template file (deploy.json) should exist in parameters files
17. All non-required parameters in template file should not have description that start with "Required."

Context Parameter file token tests

1. [Tokens] Parameter file should not contain the plain value for token
2. In used resource type should use one of the recent API version(s). Currently using