feat: Contract Testing & Integration Test CI#6
Conversation
Add contract testing suite to validate SDK models against real API responses, preventing API/SDK structural mismatches before releases. Contract Testing: - 19 contract tests covering all response types - JSON fixtures for health, query, blocked, plan, and policy responses - Tests validate datetime parsing with nanoseconds - Edge case coverage for null vs missing fields CI Integration: - New integration.yml workflow for live testing - Contract tests on every PR - Integration tests against staging (on merge to main) - Demo script validation - Community stack E2E tests (manual trigger) Infrastructure: - tests/fixtures/ directory with recorded API responses - load_json_fixture() helper in conftest.py - scripts/record_fixtures.py for capturing live responses - Version bump to 0.2.0 Fixes #5
| name: Contract Tests | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.11' | ||
|
|
||
| - name: Install dependencies | ||
| run: pip install -e ".[dev]" | ||
|
|
||
| - name: Run contract tests | ||
| run: pytest tests/test_contract.py -v --no-cov | ||
|
|
||
| integration-tests: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 4 months ago
To resolve the problem, add an explicit permissions section at the root of the workflow file (.github/workflows/integration.yml). This ensures all jobs in the workflow are run with the minimum necessary privileges on the GITHUB_TOKEN. The best practice is to set permissions: { contents: read } unless specific jobs need something else (e.g., writing pull requests or issues). In this workflow, all jobs only need to read code, so adding the following near the top of the file is appropriate:
permissions:
contents: readThis line should be inserted after the name: Integration Tests and before the on: block (typically at line 2). No further changes are needed, as none of the jobs appear to need broader permissions.
| @@ -1,4 +1,6 @@ | ||
| name: Integration Tests | ||
| permissions: | ||
| contents: read | ||
|
|
||
| on: | ||
| push: |
| name: Integration Tests | ||
| runs-on: ubuntu-latest | ||
| # Only run on main branch or manual dispatch with secrets configured | ||
| if: github.event_name == 'workflow_dispatch' || (github.ref == 'refs/heads/main' && github.event_name == 'push') | ||
| needs: contract-tests | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.11' | ||
|
|
||
| - name: Install dependencies | ||
| run: pip install -e ".[dev,all]" | ||
|
|
||
| - name: Run integration tests | ||
| env: | ||
| RUN_INTEGRATION_TESTS: '1' | ||
| AXONFLOW_LICENSE_KEY: ${{ secrets.AXONFLOW_LICENSE_KEY }} | ||
| run: pytest tests/test_integration.py -v --no-cov | ||
| continue-on-error: true # Don't fail build if staging is down | ||
|
|
||
| demo-scripts: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 4 months ago
To fix the problem, add a permissions key at the root level of the workflow file (just below the name: and before the on: block). This root-level permissions block will apply to all jobs, unless a job-level override is specified. The majority of jobs shown do not perform any actions that require write access, so the minimum permission of contents: read is appropriate, as suggested by CodeQL. This grants read-only access to repository contents, which is sufficient for typical test and validation workflows.
- Add the following block near the top of the file:
permissions: contents: read - This should go after the
name: Integration Testsline (line 1) and before theon:block (line 3). - No extra imports or dependencies are required.
| @@ -1,4 +1,6 @@ | ||
| name: Integration Tests | ||
| permissions: | ||
| contents: read | ||
|
|
||
| on: | ||
| push: |
| name: Demo Scripts Validation | ||
| runs-on: ubuntu-latest | ||
| needs: contract-tests | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.11' | ||
|
|
||
| - name: Install dependencies | ||
| run: pip install -e ".[dev,all]" | ||
|
|
||
| - name: Validate quickstart.py syntax | ||
| run: python -m py_compile examples/quickstart.py | ||
|
|
||
| - name: Validate gateway_mode.py syntax | ||
| run: python -m py_compile examples/gateway_mode.py | ||
|
|
||
| - name: Validate openai_integration.py syntax | ||
| run: python -m py_compile examples/openai_integration.py | ||
|
|
||
| - name: Run quickstart (dry-run mode) | ||
| run: | | ||
| python -c " | ||
| import asyncio | ||
| from examples.quickstart import main | ||
| # Verify module imports correctly | ||
| print('quickstart.py imports successfully') | ||
| " | ||
|
|
||
| - name: Run gateway_mode (dry-run mode) | ||
| run: | | ||
| python -c " | ||
| import asyncio | ||
| from examples.gateway_mode import main, blocked_example | ||
| # Verify module imports correctly | ||
| print('gateway_mode.py imports successfully') | ||
| " | ||
|
|
||
| community-stack-tests: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 4 months ago
To fix this issue, we should add a permissions block to the demo-scripts job in .github/workflows/integration.yml that restricts the GITHUB_TOKEN permissions to the minimum required for the steps listed. Since all the steps only require reading repository contents (for actions/checkout) and do not write to GitHub, the minimal permission of contents: read is sufficient.
This addition should be made at the same indentation level as runs-on (i.e., a job-level key, not inside steps). No additional methods/imports/definitions are needed.
| @@ -62,6 +62,8 @@ | ||
|
|
||
| demo-scripts: | ||
| name: Demo Scripts Validation | ||
| permissions: | ||
| contents: read | ||
| runs-on: ubuntu-latest | ||
| needs: contract-tests | ||
| steps: |
| name: Community Stack E2E | ||
| runs-on: ubuntu-latest | ||
| if: github.event_name == 'workflow_dispatch' | ||
| needs: [contract-tests, demo-scripts] | ||
| services: | ||
| agent: | ||
| image: ghcr.io/getaxonflow/axonflow-agent:latest | ||
| ports: | ||
| - 8080:8080 | ||
| env: | ||
| AXONFLOW_MODE: community | ||
| AXONFLOW_DEBUG: 'true' | ||
| options: >- | ||
| --health-cmd "wget --spider -q http://localhost:8080/health || exit 1" | ||
| --health-interval 10s | ||
| --health-timeout 5s | ||
| --health-retries 5 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.11' | ||
|
|
||
| - name: Install dependencies | ||
| run: pip install -e ".[dev,all]" | ||
|
|
||
| - name: Wait for agent to be ready | ||
| run: | | ||
| for i in {1..30}; do | ||
| if curl -s http://localhost:8080/health | grep -q healthy; then | ||
| echo "Agent is ready!" | ||
| exit 0 | ||
| fi | ||
| echo "Waiting for agent... ($i/30)" | ||
| sleep 2 | ||
| done | ||
| echo "Agent failed to start" | ||
| exit 1 | ||
|
|
||
| - name: Run SDK against community stack | ||
| env: | ||
| AXONFLOW_AGENT_URL: 'http://localhost:8080' | ||
| AXONFLOW_CLIENT_ID: 'test-client' | ||
| AXONFLOW_CLIENT_SECRET: 'test-secret' | ||
| RUN_INTEGRATION_TESTS: '1' | ||
| run: | | ||
| # Run integration tests against local community stack | ||
| pytest tests/test_integration.py -v --no-cov | ||
|
|
||
| - name: Run demo scripts against community stack | ||
| env: | ||
| AXONFLOW_AGENT_URL: 'http://localhost:8080' | ||
| AXONFLOW_CLIENT_ID: 'test-client' | ||
| AXONFLOW_CLIENT_SECRET: 'test-secret' | ||
| run: | | ||
| # Run quickstart demo | ||
| python examples/quickstart.py || echo "Quickstart completed (may fail without LLM)" |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 4 months ago
The best way to fix the problem is to add a permissions key at the top level of the workflow (recommended), or individually to any jobs that require a unique set of permissions. For this workflow, there is no evidence of any requirement for write access -- the workflow only runs tests and scripts using repository code and secrets. Therefore, the global default permissions: contents: read should be set at the top level, immediately after the name line, to ensure all jobs run with the minimum required permissions, following the principle of least privilege.
Steps:
- Add a line such as
permissions:followed bycontents: readas mapping to the root level (after thename). - No further code or import changes are required.
| @@ -1,4 +1,6 @@ | ||
| name: Integration Tests | ||
| permissions: | ||
| contents: read | ||
|
|
||
| on: | ||
| push: |
Workflow: - Fix inputs reference for push/PR events (use github.event.inputs) Tests: - Remove duplicate load_fixture, use conftest.load_json_fixture - Add blocked policy context test and fixture - Add audit response contract test and fixture - Remove unused HTTPXMock import from conftest - Fix type annotation consistency Coverage increased from 19 to 21 contract tests.
Summary
Implements contract testing and CI integration for the Python SDK, addressing issue #5.
Problem
The SDK experienced three consecutive bug cycles in one week:
generate_planNoneRoot cause: Test suites relied on mocked responses that didn't reflect actual Agent API structures.
Solution
load_json_fixture()helper and recording scriptChanges
tests/test_contract.pytests/fixtures/*.json.github/workflows/integration.ymltests/conftest.pyscripts/record_fixtures.pypyproject.tomlCHANGELOG.mdTest Coverage
Contract Tests (19 tests)
CI Workflow Jobs
Test Plan
ruff check .)Fixes #5