HTTP Collection API (#7)

.github/workflows/test.yml

@@ -22,6 +22,9 @@ jobs:
       with:
         python-version: "3.12"
 
+    - name: Start Firestore Emulator
+      run: docker compose up -d --wait
+
     - name: Install uv
       uses: astral-sh/setup-uv@v4
       with:
@@ -44,6 +47,9 @@ jobs:
         uv run mypy src/eventkit
 
     - name: Run tests
+      env:
+        FIRESTORE_EMULATOR_HOST: localhost:8080
+        GCP_PROJECT_ID: test-project
       run: |
         uv run pytest --cov=src/eventkit --cov-report=term-missing --cov-report=xml
 
@@ -52,3 +58,7 @@ jobs:
       with:
         file: ./coverage.xml
         fail_ci_if_error: false
+
+    - name: Stop Firestore Emulator
+      if: always()
+      run: docker compose down

CLAUDE.md

@@ -8,23 +8,35 @@ For detailed architecture and implementation patterns, see `specs/core-pipeline/
 
 ```bash
 # Setup
-uv sync --all-extras        # Install all dependencies with lockfile
+uv sync --all-extras              # Install all dependencies with lockfile
+pre-commit install                # Set up git hooks (one-time)
 
-# Development
-pytest                        # Run all tests
-pytest --cov                  # Run tests with coverage
-mypy src/eventkit             # Type check
-ruff check src/               # Lint
-ruff format src/              # Format
-ruff format --check src/      # Check formatting without changing
+# Local Development
+docker compose up -d --wait       # Start Firestore emulator (with healthcheck)
+docker compose down               # Stop emulator
+
+export FIRESTORE_EMULATOR_HOST="localhost:8080"
+export GCP_PROJECT_ID="test-project"
+uv run uvicorn eventkit.api.app:app --reload --port 8000
+
+# Testing
+pytest                            # Run all tests
+pytest --cov                      # Run tests with coverage
+pytest -k test_name               # Run specific test
+pytest tests/unit/api/            # Run specific test directory
+
+# Code Quality
+mypy src/eventkit                 # Type check
+ruff check src/ tests/            # Lint
+ruff format src/ tests/           # Format
+ruff format --check src/ tests/   # Check formatting without changing
 
 # Update dependencies
-uv lock                       # Update lockfile
-uv sync                       # Sync after lockfile update
+uv lock                           # Update lockfile
+uv sync                           # Sync after lockfile update
 
-# Pre-commit (optional)
-pre-commit install            # Set up git hooks
-pre-commit run --all-files    # Run all hooks manually
+# Pre-commit
+pre-commit run --all-files        # Run all hooks manually
 
 # Publishing
 python -m build                                      # Build distribution
@@ -44,6 +56,24 @@ Keep first line under 72 characters.
 
 ## Critical Patterns
 
+### Queue-Agnostic Processor
+
+The `Processor` doesn't know about queues - it only has `process_event()`:
+
+```python
+class Processor:
+    async def process_event(self, raw_event: RawEvent) -> None:
+        # Adapt → Sequence → Buffer
+        ...
+```
+
+Queues call the processor:
+- **DirectQueue**: Calls `process_event()` immediately (inline)
+- **AsyncQueue**: Workers call `process_event()` from `asyncio.Queue`
+- **PubSubQueue**: Subscribers call `process_event()` from Pub/Sub
+
+Factory pattern (`create_queue()`) selects queue based on `EVENTKIT_QUEUE_MODE`.
+
 ### Two-Phase Event Model
 
 1. **RawEvent** (flexible): Accept any JSON at `/collect` endpoint
@@ -55,20 +85,58 @@ Keep first line under 72 characters.
 
 ### Never Reject at Edge
 
-- Invalid events → route to error store (dead letter queue)
+- **Always return 202 Accepted** (even for invalid events)
+- Invalid events → route to `ErrorStore` (dead letter queue)
 - Don't raise exceptions in collection endpoint
-- Accept everything, validate downstream
+- Accept everything, validate downstream in `Processor`
+
+### API & Dependency Injection
+
+Use FastAPI `Depends()` with singleton factories:
+
+```python
+@lru_cache
+def get_queue(settings: Settings = Depends(get_settings)) -> EventQueue:
+    # Wire: adapter → sequencer → buffer → processor → queue
+    return create_queue(processor, settings)
+
+@router.post("/collect/{stream}")
+async def collect(
+    request: Request,
+    stream: str = "default",
+    queue: EventQueue = Depends(get_queue),
+):
+    await queue.enqueue(RawEvent(payload=await request.json(), stream=stream))
+    return JSONResponse({"status": "accepted"}, status_code=202)
+```
+
+Lifespan manager handles queue lifecycle:
+
+```python
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    queue = get_queue()
+    await queue.start()  # Start workers, buffer flusher
+    yield
+    await queue.stop()   # Drain queue, flush buffers
+```
 
 ### Protocols Over ABCs
 
-Use `Protocol` for interfaces:
+Use `Protocol` for interfaces (structural typing):
 
 ```python
 class EventStore(Protocol):
-    async def write(self, events: list[TypedEvent]) -> None: ...
+    async def store_batch(self, events: list[TypedEvent]) -> None: ...
+    async def health_check(self) -> None: ...
+
+class EventQueue(Protocol):
+    async def enqueue(self, event: RawEvent) -> None: ...
+    async def start(self) -> None: ...
+    async def stop(self) -> None: ...
 ```
 
-Not abstract base classes.
+Not abstract base classes. Enables duck typing and easier testing.
 
 ### Stream Routing & Sequencing
 
@@ -91,3 +159,21 @@ Events route to named streams, then sequenced by identity hash:
 - **Error handling**: Return `AdapterResult`, don't raise in hot path
 - **Comments**: Why, not what. Avoid obvious comments
 - **Protocols over ABCs**: Duck typing for interfaces
+- **No version references**: Don't add "v0.1.0", "future", etc. in code/specs
+- **Prefix settings**: Use `EVENTKIT_*` for all environment variables
+- **Testing**: Every commit should include tests (unit + integration where applicable)
+- **Docker Compose**: Use same setup locally and in CI (`docker compose up -d --wait`)
+
+## Health Checks
+
+- `/health` - **Liveness**: Returns 200 if process is running (no dependencies)
+- `/ready` - **Readiness**: Returns 200 if dependencies (Firestore) are healthy, 503 if not
+
+Used by Kubernetes/load balancers to determine traffic routing.
+
+## Documentation
+
+See `LOCAL_DEV.md` for detailed local development instructions including:
+- Setting up Firestore emulator with Docker Compose
+- Running the FastAPI server
+- Manual testing with curl

LOCAL_DEV.md

@@ -0,0 +1,93 @@
+# Local Development Guide
+
+## Quick Start
+
+### Prerequisites
+
+- Docker & Docker Compose
+- Python 3.12+
+- [uv](https://docs.astral.sh/uv/) (recommended) or pip
+
+### 1. Start Firestore Emulator
+
+```bash
+docker compose up -d
+```
+
+This starts the Firestore emulator on `localhost:8080`.
+
+### 2. Install Dependencies
+
+```bash
+uv sync
+```
+
+### 3. Run the API Server
+
+```bash
+export FIRESTORE_EMULATOR_HOST="localhost:8080"
+export GCP_PROJECT_ID="test-project"
+
+uv run uvicorn eventkit.api.app:app --reload --port 8000
+```
+
+The API will be available at `http://localhost:8000`.
+
+### 4. Test the API
+
+**Health Check:**
+```bash
+curl http://localhost:8000/health
+# {"status": "ok"}
+```
+
+**Send an Event:**
+```bash
+curl -X POST http://localhost:8000/collect \
+  -H "Content-Type: application/json" \
+  -d '{"type": "track", "event": "button_click", "userId": "user_123"}'
+# {"status": "accepted"}
+```
+
+---
+
+## Running Tests
+
+```bash
+# Start emulator
+docker compose up -d
+
+# Run tests
+export FIRESTORE_EMULATOR_HOST="localhost:8080"
+export GCP_PROJECT_ID="test-project"
+uv run pytest --cov=src/eventkit
+
+# Stop emulator
+docker compose down
+```
+
+---
+
+## Configuration
+
+See `src/eventkit/config.py` for all available settings.
+
+**Key Settings:**
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `GCP_PROJECT_ID` | *required* | GCP project ID |
+| `FIRESTORE_EMULATOR_HOST` | - | Firestore emulator address |
+| `EVENTKIT_QUEUE_MODE` | `"direct"` | Queue mode: `direct`, `async`, `pubsub` |
+| `EVENTKIT_BUFFER_SIZE` | `100` | Events per partition before flush |
+
+---
+
+## Stopping Services
+
+```bash
+# Stop API server: Ctrl+C
+
+# Stop Firestore emulator
+docker compose down
+```

README.md

@@ -272,37 +272,23 @@ settings = Settings(
 
 ## Development
 
-**Setup:**
-```bash
-# Install dependencies
-uv sync --all-extras
-
-# Install pre-commit hooks (one-time setup)
-uv run pre-commit install
-```
-
-**Run unit tests (fast, no Docker):**
-```bash
-uv run pytest
-```
+See [LOCAL_DEV.md](LOCAL_DEV.md) for detailed local development instructions.
 
-**Run integration tests (requires Docker):**
+**Quick Start:**
 ```bash
 # Start Firestore emulator
 docker compose up -d
 
-# Run integration tests
-export FIRESTORE_EMULATOR_HOST=localhost:8080
-uv run pytest -m integration
+# Install dependencies
+uv sync
 
-# Stop emulator
-docker compose down
-```
+# Run API server
+export FIRESTORE_EMULATOR_HOST="localhost:8080"
+export GCP_PROJECT_ID="test-project"
+uv run uvicorn eventkit.api.app:app --reload
 
-**Run all tests:**
-```bash
-export FIRESTORE_EMULATOR_HOST=localhost:8080
-uv run pytest -m ""
+# Run tests
+uv run pytest --cov=src/eventkit
 ```
 
 **Type check:**

pyproject.toml

@@ -56,11 +56,6 @@ Issues = "https://github.com/prosdev/eventkit/issues"
 [tool.setuptools.packages.find]
 where = ["src"]
 
-[tool.pytest.ini_options]
-testpaths = ["tests"]
-asyncio_mode = "auto"
-addopts = "--cov=eventkit --cov-report=term-missing"
-
 [tool.ruff]
 line-length = 100
 target-version = "py312"

specs/core-pipeline/plan.md

@@ -709,7 +709,7 @@ app = FastAPI(lifespan=lifespan)
 | `README.md` | Installation and usage docs | - |
 | `CHANGELOG.md` | Version history | - |
 | `examples/basic_usage.py` | Quick start example | - |
-| `examples/docker-compose.yml` | Firestore emulator setup | - |
+| `examples/docker compose.yml` | Firestore emulator setup | - |
 | `.github/workflows/test.yml` | CI pipeline | - |
 | `.github/workflows/publish.yml` | CD pipeline (PyPI) | - |
 
@@ -824,7 +824,7 @@ eventkit/
 ├── examples/
 │   ├── basic_usage.py
 │   ├── cloud_run_deployment.py
-│   └── docker-compose.yml
+│   └── docker compose.yml
 ├── specs/
 │   └── core-pipeline/
 │       ├── spec.md              # User stories (this file)
@@ -989,7 +989,7 @@ spec:
 ### Local Development
 
 ```yaml
-# docker-compose.yml
+# docker compose.yml
 services:
   firestore:
     image: gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators