docs: rewrite test documentation with structured developer guide#256
docs: rewrite test documentation with structured developer guide#256Gokul-social wants to merge 2 commits intoopenml:mainfrom
Conversation
Addresses openml#169. Replaces informal benchmark notes with a structured guide covering the four existing test categories (migration, integration, direct database, direct function), performance tradeoffs, mocking philosophy, and running instructions.
WalkthroughThis pull request updates documentation files with two main changes. The 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs). Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
docs/contributing/tests.md (1)
26-37: Document the known migration-test auth limitation explicitly.Add a short caveat that some auth-failure paths (e.g., Code 103) cannot be validated through
php_apiparity checks when legacy responses are XML.Suggested doc patch
### 1) Migration tests Migration tests compare Python API responses against the legacy PHP API for equivalent endpoints. These tests live under `tests/routers/openml/migration/`. Characteristics: @@ - Focus on compatibility guarantees during migration. + +Known limitation: + +- Some authentication failure paths (for example Code 103 on endpoints where legacy PHP returns XML) + cannot be reliably asserted via migration parity tests; cover those in Python unit/integration tests.Based on learnings: Migration tests for endpoints relying on
php_apicannot validate authentication failure paths (Code 103) when legacy PHP returns XML instead of JSON.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/contributing/tests.md` around lines 26 - 37, Add a short caveat to the "Migration tests" section that explicitly notes migration tests using the php_api fixture cannot validate certain authentication-failure paths (specifically "Code 103") when the legacy PHP API returns XML instead of JSON; reference the "Migration tests" heading and mention php_api and Code 103 so readers know the limitation and that parity checks won’t catch those auth-failure responses when formats differ.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@docs/contributing/tests.md`:
- Around line 26-37: Add a short caveat to the "Migration tests" section that
explicitly notes migration tests using the php_api fixture cannot validate
certain authentication-failure paths (specifically "Code 103") when the legacy
PHP API returns XML instead of JSON; reference the "Migration tests" heading and
mention php_api and Code 103 so readers know the limitation and that parity
checks won’t catch those auth-failure responses when formats differ.
There was a problem hiding this comment.
Pull request overview
This PR rewrites the contributor-facing test documentation to be a structured guide describing the repository’s current test layers (migration, integration, database-level, and direct function tests), and updates the MkDocs navigation label to better match the project/site name.
Changes:
- Replaced informal notes in
docs/contributing/tests.mdwith a structured testing strategy guide (fixtures, test categories, tradeoffs, and how to run tests). - Updated
mkdocs.ymlnav label from “Intro” to “OpenML Server” for the index page.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| mkdocs.yml | Renames the top-level nav entry for index.md to “OpenML Server” (aligns with site naming). |
| docs/contributing/tests.md | Full rewrite into a structured developer guide covering fixtures, test layers, tradeoffs, and common commands. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
This PR rewrites docs/contributing/tests.md to provide structured developer documentation for writing tests, addressing issue #169.
The previous version of tests.md consisted of informal engineering notes, benchmark measurements, and scattered implementation details. While useful internally, it was not structured as contributor-facing documentation.
This update converts the content into a clear testing guide aligned with the current repository structure and actual test implementation.