[Q1-P0] Add integration tests against real production API

[karlwaldman]

Problem

We shipped SDK v1.4.1 with 100% timeout rate on historical queries. Unit tests passed because they used mocked responses.

Root Cause: No integration tests calling real API endpoints

Impact

• Severity: P0 - 100% failure rate for core feature
• Detection: Customer report (not CI/CD)
• Time to fix: 2 hours emergency fix
• Could have been: Hundreds of users affected

Solution

Add integration test suite that calls real production API:

# tests/integration/test_real_api_historical.py

@pytest.mark.integration
def test_1_year_query_against_production():
    """Test real 1-year historical query."""
    client = OilPriceAPI(api_key=os.getenv('TEST_API_KEY'))

    start_time = time.time()
    historical = client.historical.get(
        commodity='WTI_USD',
        start_date='2024-01-01',
        end_date='2024-12-31',
        interval='daily'
    )
    duration = time.time() - start_time

    # Would have caught the timeout bug
    assert duration < 120, f"Query took {duration}s, exceeds timeout"
    assert len(historical.data) > 300

Test Cases to Add

1. Endpoint existence tests

   • Verify all 4 historical endpoints exist (past_day, past_week, past_month, past_year)
   • Verify they return expected data format

2. Query completion tests

   • 1 day query completes successfully
   • 1 week query completes successfully
   • 1 month query completes successfully
   • 1 year query completes successfully

3. Timeout behavior tests

   • Queries complete within calculated timeout
   • TimeoutError raised if query exceeds timeout

4. Data validation tests

   • Correct number of records returned
   • Data format matches expectations
   • Pagination works correctly

Implementation

# New directory structure
tests/
  integration/
    __init__.py
    conftest.py          # Shared fixtures
    test_historical.py   # Historical endpoint tests
    test_prices.py       # Current price tests
    test_alerts.py       # Alert endpoint tests

CI Integration

# .github/workflows/test.yml
jobs:
  integration-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run integration tests
        env:
          TEST_API_KEY: ${{ secrets.TEST_API_KEY }}
        run: |
          pytest tests/integration/ --integration -v

Acceptance Criteria

• Integration test suite created in tests/integration/
• Tests cover all historical endpoints
• Tests validate timeout behavior
• Tests run in CI before every release
• Test API key configured in GitHub Secrets
• Tests pass consistently

Estimated Effort

Time: 2-3 hours
Complexity: Low (straightforward API calls)
Blocker: Need test API key with Production Boost access

Success Metrics

• Integration tests catch issues that unit tests miss
• Zero P0 bugs ship to production due to SDK issues
• Confidence in releases increases

Related Issues

• Historical timeout bug (idan@comity.ai)
• QA Assessment: sdks/python/QA_ASSESSMENT_HISTORICAL_TIMEOUT_ISSUE.md

Notes

This would have caught the historical timeout bug in CI before it reached customers. Must be completed before next SDK release.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

_{You can configure auto-planning by selecting labels in the issue_enrichment configuration.}

🔗 Similar Issues

Possible Duplicates

• ISSUE-14

Related Issues

• [Q1-P0] Add integration tests against real production API #14
• [Q1-P0] Add performance baseline tests for historical queries #15

👤 Suggested Assignees

• karlwaldman

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!