[Feature Request]: Testing improvements proposal

[itpropro]

Description

I would like to suggest some improvements to test new modules and module changes with PSRule.
This would have multiple benefits:

1. Parameter files can be written in bicep and can be stored in a more intuitive .test folder.
2. We can directly target .bicep test files with ps-rule.yaml files
3. We can perform static analysis on the templates as well as on deployed resources in Azure with the in-flight analysis and can then compare the results to be the same

Questions/Tasks for testing with PSRule:

1. Do we need a custom ruleset? For parameter sets that have full coverage, we can use the PSRule for Azure module, but for minimal parameter sets, these would throw a lot of errors like missing tags, managed identities, etc.
2. Do we need multiple runs in the pipeline (for minimal and full parameter sets)
3. Do we want to automatically deploy and test resources to Azure?
4. How to migrate the existing parameter.json files to a proper testing model?

I would like to hear your thoughts on this!

[AlexanderSehr]

Hey @itpropro,
great suggesstion and it couldn't be more current - in a way. A few thoughts:

• We actually do have a Pull Request open by @BernieWhite (Bicep updates for validation and clean up #161) - in which he implemented a PoC for PSRule (next to other things) - and it's our longest running PR yet ;) I should probably add that it's long lasting nature is just rooted in CARML's current structure that requires a rework to fit PSRule
• That's were we actually have some news for upcoming, planned changes (that will play into our hand here):
  • For some time, we've been working on several PoCs to make the CARML module more 'independent' - that is, to allow a module to deploy its own dependencies. This however has also several implications for the pipelines, so it took some time to get through it all. If you're interested, the issue we used for tracking was this one: Testing PoC: Modules should be more self-contained for testing #329
  • Long story short, we plan to implement the listed variant 5, for which we have a PoC in this PR PoC: Variant 5 for 'Self-contained' module dependency deployment  #1575 and track in this issue [Feature Request]: Enable new dependency approach from a pipeline perspective #1583.
  • Essentially, we'll rework the current parameter.json files to parameter.bicep files which deploy not only the module, but further also the required dependencies. This has the added benefit that we get closer to test format expected by the Public Bicep Registry to which we plan to contribute - ideally on scale
  • Linked to this, it will also make sense to rename the .parameters folder to something else like .deploymentTests or simply .tests (though the latter may be a bit misleading as it's not including static tests). For the we also have an issue here [Feature Request] BREAKING change: Rename .parameters folder to .test #1577. This will likly be implemented directly for the next release 0.7 - while the general dependency rework may take a bit longer and we happen iteratively.
• My hope is that, once the new dependency approach is implemented, not only the onboarding of CARML and its costs (for the dependency resources) gets's improved, but that we can also introduce PSRule as an additional testing framework to make sure our modules follow the high coding standard we aim for.

[AlexanderSehr]

I know this does not answer all your questions just yet - but I hope things get clearer once we get closer to the first ported modules. And then ideally it will also become clearer to me. Iterative imrovements for the win.

I haven't worked with PSRule myself yet, but heard great things about it. So we definitely want to go there.

[itpropro]

Thanks for the detailed information on this topic!
A few things just from the top of my head:

• I don't really like depending on Pester, as it is a PowerShell focused unit test framework not maintained by Microsoft
• PSRule would perfectly fit for our usecase, as it is made (especially the Azure ruleset) for testing Infrastructure as Code (IaC) before or after deployment to Azure, is maintained by Microsoft and has ready to go GitHub Actions.
• I think the test folder should just be called .test which is convention in the most languages/OSS products, independent of the kind of tests contained there. You can later on still reflect different types with a specific naming or folder structure
• Reading through all the referenced issues, I also think we should focus on .bicep files and try to drop support for .json files asap. The bicep compiler can always be used to build .bicep files and the CARML project itself so far doesn't even host .json files for deployment (which should definitely stay this way)

I am really looking forward to help shape the future of bicep and the CARML library. I think we should definitely have a solid and flexible testing solution ready by the end of July when the first non-Azure module (Kubernetes) will be released to preview. After that, I can only image with which kind of crazy resource providers people will come up with 😀

[AlexanderSehr]

A couple comments from my side:

• We're currently using Pester 'only' for our Unit Tests (i.e., static validation) to make sure that e.g. the ReadMe is correctly formatted, the module structure is as expected and e.g. parameters match up. For these purposes Pester is very well suited and a commonly used Framework when using PowerShell as the primary language.
• PSRule is instead great for anything template releated, for which I definitely on board to introduce it
• Regarding the folder name, .test would be fine too. The only reason I'd prefer e.g. .deploymentTests over .test would be to avoid confusion with the static tests that are part of the centralized global.module.tests.ps1 file. A folder named .test may give the impression that all module related tests are contained in these folders. Not the strongest argument, but at least something to consider :D
• Regarding the Bicep vs ARM-JSON point I agree and there are some very big fans for the same in the community. The only thing I'd like to add here is a related topic, which are the deploymentExamples in the ReadMe. Currently we copy each ARM-JSON-Parameter file as a deployment example into the README and further convert it to a 'bicep' format to make it easier to go the bicep-native route. I'd like to keep this feature/content to not exclude those that want to still use JSON. This means I'd adjust the logic currently implemented in the Set-ModuleReadMe.ps1 script and flip it on its head (i.e. convert bicep to ARM json deployment examples).

[MariusStorhaug]

Folder name for test is now .test.

[rahalan]

Blocked, because #1624 needs to be implemented plus all modules need to be converted to the new testing approach. After that, this item can be continued.

[itpropro]

@rahalan, @MrMCake #1624 is merged, how do we proceed for converting the test files?

[AlexanderSehr]

Hey @itpropro, we're currently working on the implementation that we'll start merging once the upcoming September release 0.7 is out.

[itpropro]

Sounds great, is there any way to contribute @MrMCake ?