Skip to content

feat: Contract Testing & Integration Test CI#6

Merged
saurabhjain1592 merged 3 commits intomainfrom
feature/contract-testing-issue-5
Dec 15, 2025
Merged

feat: Contract Testing & Integration Test CI#6
saurabhjain1592 merged 3 commits intomainfrom
feature/contract-testing-issue-5

Conversation

@saurabhjain1592
Copy link
Copy Markdown
Member

Summary

Implements contract testing and CI integration for the Python SDK, addressing issue #5.

Problem

The SDK experienced three consecutive bug cycles in one week:

  • Datetime parsing failures (nanoseconds unhandled)
  • Missing authorization headers in generate_plan
  • Policy name extraction returning None

Root cause: Test suites relied on mocked responses that didn't reflect actual Agent API structures.

Solution

  • Contract Testing Suite - 19 tests validating SDK models against real API response fixtures
  • JSON Fixtures - Recorded API responses for health, query, blocked, plan, and policy scenarios
  • CI Integration Workflow - GitHub Actions for contract tests, integration tests, and demo validation
  • Fixture Infrastructure - load_json_fixture() helper and recording script

Changes

File Change
tests/test_contract.py 19 contract tests for all response types
tests/fixtures/*.json 6 JSON fixtures from API contract
.github/workflows/integration.yml CI workflow for integration testing
tests/conftest.py Fixture loading utilities
scripts/record_fixtures.py Script to record fixtures from staging
pyproject.toml Version bump to 0.2.0
CHANGELOG.md Release notes for 0.2.0

Test Coverage

Contract Tests (19 tests)

  • Health response structure validation
  • Successful query response parsing
  • Blocked query (PII) response parsing
  • Plan generation response parsing
  • Gateway Mode policy context parsing
  • Datetime with nanoseconds handling
  • Rate limit info parsing
  • Connector list parsing
  • Edge cases (empty, minimal, unknown fields)

CI Workflow Jobs

  1. contract-tests - Runs on every PR
  2. integration-tests - Runs on merge to main (with staging secrets)
  3. demo-scripts - Validates all example scripts
  4. community-stack-tests - E2E with Docker Agent (manual trigger)

Test Plan

  • All 143 existing tests pass
  • 19 new contract tests pass
  • Demo scripts compile successfully
  • Linting passes (ruff check .)
  • CI workflow runs successfully

Fixes #5

Add contract testing suite to validate SDK models against real API
responses, preventing API/SDK structural mismatches before releases.

Contract Testing:
- 19 contract tests covering all response types
- JSON fixtures for health, query, blocked, plan, and policy responses
- Tests validate datetime parsing with nanoseconds
- Edge case coverage for null vs missing fields

CI Integration:
- New integration.yml workflow for live testing
- Contract tests on every PR
- Integration tests against staging (on merge to main)
- Demo script validation
- Community stack E2E tests (manual trigger)

Infrastructure:
- tests/fixtures/ directory with recorded API responses
- load_json_fixture() helper in conftest.py
- scripts/record_fixtures.py for capturing live responses
- Version bump to 0.2.0

Fixes #5
Comment on lines +22 to +38
name: Contract Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: pip install -e ".[dev]"

- name: Run contract tests
run: pytest tests/test_contract.py -v --no-cov

integration-tests:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 4 months ago

To resolve the problem, add an explicit permissions section at the root of the workflow file (.github/workflows/integration.yml). This ensures all jobs in the workflow are run with the minimum necessary privileges on the GITHUB_TOKEN. The best practice is to set permissions: { contents: read } unless specific jobs need something else (e.g., writing pull requests or issues). In this workflow, all jobs only need to read code, so adding the following near the top of the file is appropriate:

permissions:
  contents: read

This line should be inserted after the name: Integration Tests and before the on: block (typically at line 2). No further changes are needed, as none of the jobs appear to need broader permissions.


Suggested changeset 1
.github/workflows/integration.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -1,4 +1,6 @@
 name: Integration Tests
+permissions:
+  contents: read
 
 on:
   push:
EOF
@@ -1,4 +1,6 @@
name: Integration Tests
permissions:
contents: read

on:
push:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +39 to +62
name: Integration Tests
runs-on: ubuntu-latest
# Only run on main branch or manual dispatch with secrets configured
if: github.event_name == 'workflow_dispatch' || (github.ref == 'refs/heads/main' && github.event_name == 'push')
needs: contract-tests
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: pip install -e ".[dev,all]"

- name: Run integration tests
env:
RUN_INTEGRATION_TESTS: '1'
AXONFLOW_LICENSE_KEY: ${{ secrets.AXONFLOW_LICENSE_KEY }}
run: pytest tests/test_integration.py -v --no-cov
continue-on-error: true # Don't fail build if staging is down

demo-scripts:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 4 months ago

To fix the problem, add a permissions key at the root level of the workflow file (just below the name: and before the on: block). This root-level permissions block will apply to all jobs, unless a job-level override is specified. The majority of jobs shown do not perform any actions that require write access, so the minimum permission of contents: read is appropriate, as suggested by CodeQL. This grants read-only access to repository contents, which is sufficient for typical test and validation workflows.

  • Add the following block near the top of the file:
    permissions:
      contents: read
    
  • This should go after the name: Integration Tests line (line 1) and before the on: block (line 3).
  • No extra imports or dependencies are required.

Suggested changeset 1
.github/workflows/integration.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -1,4 +1,6 @@
 name: Integration Tests
+permissions:
+  contents: read
 
 on:
   push:
EOF
@@ -1,4 +1,6 @@
name: Integration Tests
permissions:
contents: read

on:
push:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +63 to +104
name: Demo Scripts Validation
runs-on: ubuntu-latest
needs: contract-tests
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: pip install -e ".[dev,all]"

- name: Validate quickstart.py syntax
run: python -m py_compile examples/quickstart.py

- name: Validate gateway_mode.py syntax
run: python -m py_compile examples/gateway_mode.py

- name: Validate openai_integration.py syntax
run: python -m py_compile examples/openai_integration.py

- name: Run quickstart (dry-run mode)
run: |
python -c "
import asyncio
from examples.quickstart import main
# Verify module imports correctly
print('quickstart.py imports successfully')
"

- name: Run gateway_mode (dry-run mode)
run: |
python -c "
import asyncio
from examples.gateway_mode import main, blocked_example
# Verify module imports correctly
print('gateway_mode.py imports successfully')
"

community-stack-tests:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 4 months ago

To fix this issue, we should add a permissions block to the demo-scripts job in .github/workflows/integration.yml that restricts the GITHUB_TOKEN permissions to the minimum required for the steps listed. Since all the steps only require reading repository contents (for actions/checkout) and do not write to GitHub, the minimal permission of contents: read is sufficient.

This addition should be made at the same indentation level as runs-on (i.e., a job-level key, not inside steps). No additional methods/imports/definitions are needed.


Suggested changeset 1
.github/workflows/integration.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -62,6 +62,8 @@
 
   demo-scripts:
     name: Demo Scripts Validation
+    permissions:
+      contents: read
     runs-on: ubuntu-latest
     needs: contract-tests
     steps:
EOF
@@ -62,6 +62,8 @@

demo-scripts:
name: Demo Scripts Validation
permissions:
contents: read
runs-on: ubuntu-latest
needs: contract-tests
steps:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +105 to +163
name: Community Stack E2E
runs-on: ubuntu-latest
if: github.event_name == 'workflow_dispatch'
needs: [contract-tests, demo-scripts]
services:
agent:
image: ghcr.io/getaxonflow/axonflow-agent:latest
ports:
- 8080:8080
env:
AXONFLOW_MODE: community
AXONFLOW_DEBUG: 'true'
options: >-
--health-cmd "wget --spider -q http://localhost:8080/health || exit 1"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: pip install -e ".[dev,all]"

- name: Wait for agent to be ready
run: |
for i in {1..30}; do
if curl -s http://localhost:8080/health | grep -q healthy; then
echo "Agent is ready!"
exit 0
fi
echo "Waiting for agent... ($i/30)"
sleep 2
done
echo "Agent failed to start"
exit 1

- name: Run SDK against community stack
env:
AXONFLOW_AGENT_URL: 'http://localhost:8080'
AXONFLOW_CLIENT_ID: 'test-client'
AXONFLOW_CLIENT_SECRET: 'test-secret'
RUN_INTEGRATION_TESTS: '1'
run: |
# Run integration tests against local community stack
pytest tests/test_integration.py -v --no-cov

- name: Run demo scripts against community stack
env:
AXONFLOW_AGENT_URL: 'http://localhost:8080'
AXONFLOW_CLIENT_ID: 'test-client'
AXONFLOW_CLIENT_SECRET: 'test-secret'
run: |
# Run quickstart demo
python examples/quickstart.py || echo "Quickstart completed (may fail without LLM)"

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 4 months ago

The best way to fix the problem is to add a permissions key at the top level of the workflow (recommended), or individually to any jobs that require a unique set of permissions. For this workflow, there is no evidence of any requirement for write access -- the workflow only runs tests and scripts using repository code and secrets. Therefore, the global default permissions: contents: read should be set at the top level, immediately after the name line, to ensure all jobs run with the minimum required permissions, following the principle of least privilege.

Steps:

  • Add a line such as permissions: followed by contents: read as mapping to the root level (after the name).
  • No further code or import changes are required.

Suggested changeset 1
.github/workflows/integration.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -1,4 +1,6 @@
 name: Integration Tests
+permissions:
+  contents: read
 
 on:
   push:
EOF
@@ -1,4 +1,6 @@
name: Integration Tests
permissions:
contents: read

on:
push:
Copilot is powered by AI and may make mistakes. Always verify output.
Workflow:
- Fix inputs reference for push/PR events (use github.event.inputs)

Tests:
- Remove duplicate load_fixture, use conftest.load_json_fixture
- Add blocked policy context test and fixture
- Add audit response contract test and fixture
- Remove unused HTTPXMock import from conftest
- Fix type annotation consistency

Coverage increased from 19 to 21 contract tests.
@saurabhjain1592 saurabhjain1592 merged commit caab5ee into main Dec 15, 2025
13 checks passed
@saurabhjain1592 saurabhjain1592 deleted the feature/contract-testing-issue-5 branch December 15, 2025 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SDK Quality: Contract Testing & Integration Tests

2 participants