From e21f09c00c009e672ec1b38e45f25d639d45c8a6 Mon Sep 17 00:00:00 2001 From: Ruben Romero Montes Date: Tue, 2 Sep 2025 09:11:12 +0200 Subject: [PATCH] docs: add github templates for PR Signed-off-by: Ruben Romero Montes --- .github/config_template.md | 38 ++++++++++++++++++++++++++++++++ .github/pull_request_template.md | 26 ++++++++++++++++++++++ .github/report_template.md | 26 ++++++++++++++++++++++ .github/spec_template.md | 24 ++++++++++++++++++++ README.md | 31 +++++++++++++++++++------- 5 files changed, 137 insertions(+), 8 deletions(-) create mode 100644 .github/config_template.md create mode 100644 .github/pull_request_template.md create mode 100644 .github/report_template.md create mode 100644 .github/spec_template.md diff --git a/.github/config_template.md b/.github/config_template.md new file mode 100644 index 0000000..16bcdb4 --- /dev/null +++ b/.github/config_template.md @@ -0,0 +1,38 @@ +# Configuration Update PR Template + +## Configuration Change + +### What configuration is being updated? + + +### Configuration Type +- [ ] **Tasks** - Task definitions and metadata +- [ ] **Models** - Model information and capabilities +- [ ] **Guardrails** - Operational constraints and policies +- [ ] **Policies** - Policy definitions with embedded thresholds +- [ ] **Metrics** - Metric definitions and types + +--- + +## Change Details + +### What was changed? + + +### Why was this change needed? + + +--- + +## Validation + +### Configuration Validation +- [ ] All modified files pass `evalguard config validate` +- [ ] Schema compliance verified against `schemas/v1/` +- [ ] Configuration follows project structure guidelines + +--- + +## Additional Notes + + diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..840f0c0 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,26 @@ +# Pull Request Template + +## Type of Change + +Please select the type of change this PR represents and use the appropriate template: + +- [ ] **Report Contribution** - Use [Report Contribution PR Template](.github/report_template.md) +- [ ] **Configuration Update** - Use [Configuration Template](.github/config_template.md) +- [ ] **Specification change** - Use [Specification PR Template](.github/spec_template.md) +- [ ] **Other** - Please describe below + +--- + +## Quick Description + + + +--- + +## Type-Specific Template + +Please copy the content from the appropriate template above and paste it below, then fill it out completely: + +```markdown + +``` diff --git a/.github/report_template.md b/.github/report_template.md new file mode 100644 index 0000000..b689a24 --- /dev/null +++ b/.github/report_template.md @@ -0,0 +1,26 @@ +# Report Contribution PR Template + +## Report Information + +### Model Details +- **Model Organization**: +- **Model Name**: +- **Model Version**: +- **Report Format**: + +## Checklist + +- [ ] I have read and followed the [Contributing Guidelines](README.md#contributing) +- [ ] Report follows the required directory structure +- [ ] Report format is supported or discussed in an issue +- [ ] I have used `evalguard lm-eval gen` to generate missing configuration files +- [ ] All generated configuration files have been reviewed and enhanced +- [ ] Configuration files pass validation: `evalguard config validate` +- [ ] No placeholder text remains in configurations +- [ ] I have enhanced configurations with meaningful descriptions + +--- + +## Additional Notes + + diff --git a/.github/spec_template.md b/.github/spec_template.md new file mode 100644 index 0000000..35dfe8e --- /dev/null +++ b/.github/spec_template.md @@ -0,0 +1,24 @@ +# Specification Change PR Template + +## Change Summary + +### What specification change is being proposed? + + +### Why is this change needed? + + +--- + +## Technical Details + +### Change Type +- [ ] **API Schema** - REST API schema modification +- [ ] **Config Schema** - Configuration schema + +--- + +## Additional Notes + + + diff --git a/README.md b/README.md index 7603e3c..65c5434 100644 --- a/README.md +++ b/README.md @@ -102,17 +102,18 @@ EvalGuard thrives on community contributions! We welcome contributions of evalua #### Report Format Reports must follow the directory structure: ``` -reports///lm-eval/report.json +reports////report.json ``` **Example**: `reports/meta-llama/Llama-3.1-8B-Instruct/lm-eval/report.json` #### Supported Formats -- **lm-evaluation-harness**: Currently supported with full CLI integration +- **lm-evaluation-harness** (`lm-eval`): Currently supported with full CLI integration - **Other formats**: If you have evaluation reports in other formats, please create an issue to discuss integration #### Adding Missing Data -When adding a new report, you can use the CLI to automatically generate missing tasks, models, and metrics: +When adding a new report, you can use the CLI to automatically generate missing tasks, models, and metrics but bear +in mind that some fields will not be filled like friendly name or description so take some time to review what is missing: ```bash # Generate missing configuration from a single report @@ -124,6 +125,17 @@ evalguard lm-eval gen -d reports/ This will create the necessary configuration files in the appropriate `config/` subdirectories. +#### Ready to Contribute? +**🚀 Create a Pull Request** using our [Report Contribution Template](.github/report_template.md) to submit your evaluation reports! + +The template will guide you through: +- Proper report structure and format +- Configuration generation and validation +- Quality assurance steps +- Required checklist items + +Simply copy the template content when creating your PR to ensure all requirements are met. + ### Contributing Configuration Files #### Adding New Configurations @@ -151,13 +163,16 @@ evalguard config validate -t guardrails #### Schema Compliance All configuration files must follow the schemas defined in `schemas/v1/`. See [SPECIFICATION.md](SPECIFICATION.md) for detailed schema definitions and examples. -### Contribution Guidelines - -1. **Evaluation Reports**: Submit reports following the directory structure `reports///lm-eval/report.json`. Currently, only [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) reports are supported. For other evaluation formats, please create an issue to discuss integration. +#### Ready to Contribute Configurations? +**⚙️ Create a Pull Request** using our [Configuration Update Template](.github/config_template.md) to submit your configuration changes! -2. **Configuration Generation**: When contributing new reports, utilize the CLI tool to automatically generate missing tasks, models, and metrics: `evalguard lm-eval gen` +The template will guide you through: +- Proper configuration file structure +- Validation and schema compliance +- Quality assurance for configuration updates +- Required checklist items -3. **Configuration Validation**: Ensure all configuration files adhere to the API specification and validate them using: `evalguard config validate` +Simply copy the template content when creating your PR to ensure all requirements are met. ---