Templates marketplace, report overhaul, and agentic chat upgrades

echo/.vscode/sessions.json

@@ -1,68 +1,68 @@
 {
     "$schema": "https://cdn.statically.io/gh/nguyenngoclongdev/cdn/main/schema/v11/terminal-keeper.json",
-    "theme": "tribe",
     "active": "default",
     "keepExistingTerminals": false,
     "sessions": {
         "default": [
             {
                 "autoExecuteCommands": true,
-                "name": "server",
-                "icon": "server",
                 "commands": [
                     "cd server",
                     "./run.sh"
-                ]
+                ],
+                "icon": "server",
+                "name": "server"
             },
             [
                 {
                     "autoExecuteCommands": true,
-                    "name": "workers",
-                    "icon": "gear",
                     "commands": [
                         "cd server",
                         "./run-worker.sh"
-                    ]
+                    ],
+                    "icon": "gear",
+                    "name": "workers"
                 },
                 {
                     "autoExecuteCommands": true,
-                    "name": "workers-cpu",
-                    "icon": "gear",
                     "commands": [
                         "cd server",
                         "./run-worker-cpu.sh"
-                    ]
+                    ],
+                    "icon": "gear",
+                    "name": "workers-cpu"
                 },
                 {
                     "autoExecuteCommands": true,
-                    "name": "scheduler",
-                    "icon": "clock",
                     "commands": [
                         "cd server",
                         "./run-scheduler.sh"
-                    ]
+                    ],
+                    "icon": "clock",
+                    "name": "scheduler"
                 }
             ],
             [
                 {
                     "autoExecuteCommands": true,
-                    "name": "admin-dashboard",
-                    "icon": "browser",
                     "commands": [
                         "cd frontend",
                         "pnpm run dev"
-                    ]
+                    ],
+                    "icon": "browser",
+                    "name": "admin-dashboard"
                 },
                 {
                     "autoExecuteCommands": true,
-                    "name": "participant-portal",
-                    "icon": "browser",
                     "commands": [
                         "cd frontend",
                         "pnpm run participant:dev"
-                    ]
+                    ],
+                    "icon": "browser",
+                    "name": "participant-portal"
                 }
             ]
         ]
-    }
+    },
+    "theme": "tribe"
 }

echo/AGENTS.md

@@ -158,35 +158,94 @@ bash echo/server/scripts/agentic/latest_runs.sh --chat-id <chat_uuid> --limit 1
 | `frontend/src/config.ts` | Frontend feature flags |
 | `server/dembrane/settings.py` | Backend configuration |
 | `docs/frontend_translations.md` | Translation workflow |
+| `docs/branching_and_releases.md` | Branching strategy, release process, hotfixes |
+| `docs/database_migrations.md` | Directus data migration steps |
 
 ## Code Style
 
 - Frontend: TypeScript, React, Mantine UI
 - Backend: Python 3.11+, FastAPI, Pydantic
 - Use existing patterns in the codebase as reference
 
-## Dev Notes
+## Branching Strategy & Deployment
 
-### Recent Changes (testing branch)
-- Copy guide enforcement: "context limit" → "selection too large"
-- Translations updated for all 6 languages
-- Suggestions use faster model (`TEXT_FAST` instead of `MULTI_MODAL_PRO`)
-- Stream status shows inline under "Thinking..." instead of toast
-- Webhooks (conversation-level notifications)
+See [docs/branching_and_releases.md](docs/branching_and_releases.md) for the full guide including hotfix process, release checklist, and ASCII diagrams.
 
-### Tech Debt / Known Issues
+Quick reference:
+- **Feature flow**: branch off `main` → (optional) merge to `testing` → PR to `main` → auto-deploys to Echo Next
+- **Releases**: tagged from `main` every ~2 weeks → auto-deploys to production
+- **Hotfixes**: branch off release tag → fix → new release → cherry-pick into main
+- **Project management**: Linear (`ECHO-xxx` tickets, two-week cycles)
+- **GitOps**: `dembrane/echo-gitops` (Terraform + Helm + Argo CD)
+
+## Architecture Notes
+
+### High-Level Stack
+
+```
+Frontend (React/Vite/Mantine)  →  Backend API (FastAPI)  →  Directus (headless CMS/DB)
+                                       ↕                          ↕
+                               Dramatiq Workers           PostgreSQL
+                               (gevent + standard)
+                                       ↕
+                                    Redis (pub/sub, task broker, caching)
+                                       ↕
+                               Agent Service (LangGraph, port 8001)
+```
+
+- **Directus** is the data layer — all collections (projects, conversations, reports, etc.) live there
+- **FastAPI** handles API routes, SSE streaming, and orchestration
+- **Dramatiq** handles background work: transcription, summarization, report generation
+- **Redis** is used for task brokering, pub/sub (SSE progress), and caching
+- **LiteLLM** routes all LLM calls with automatic failover between deployments
+
+### Report Generation Pipeline
+
+Report generation runs **synchronously** in Dramatiq network-queue workers (no asyncio — this was a deliberate choice after recurring event-loop corruption bugs):
+
+1. Fetch conversations for the project
+2. Fan-out summarization of individual conversations via `dramatiq.group()`
+3. Poll Redis for group completion
+4. Refetch conversations with summaries
+5. Fetch full transcripts via `gevent.pool.Pool` (concurrent I/O)
+6. Build prompt with token budget management
+7. Call LLM via `router_completion()` (sync litellm, uses `MULTI_MODAL_PRO`)
+
+Key files:
+- `server/dembrane/report_generation.py` — main pipeline
+- `server/dembrane/report_events.py` — Redis pub/sub for real-time SSE progress
+- `server/prompt_templates/system_report.{lang}.jinja` — per-language prompt templates (written IN the target language)
+
+### BFF Pattern
+
+Backend For Frontend endpoints under `/bff/` aggregate data the frontend needs in a single call. This is the preferred pattern over having the frontend make multiple Directus SDK calls directly.
+
+Example: `/bff/projects/home` bundles pinned projects, paginated project list, search results, and admin info into one response.
+
+### Transcription Pipeline
+
+Two-step process:
+1. **AssemblyAI** (`universal-3-pro`) for raw speech-to-text — supports en, es, pt, fr, de, it. Dutch ("nl") requires `universal-2` fallback.
+2. **Gemini correction** — fixes transcripts, normalizes hotwords, PII redaction, adds recording feedback
+
+Production uses webhook mode (`ASSEMBLYAI_WEBHOOK_URL`); polling is only a fallback path.
+
+### Agent Service
+
+The `agent/` directory contains the agentic chat service (LangGraph-based). It runs as a separate FastAPI service on port 8001. Agentic chat streams via `POST /api/agentic/runs/{run_id}/stream` — no Dramatiq dispatch. See `agent/README.md`.
+
+## Tech Debt / Known Issues
 - Some mypy errors in `llm_router.py` and `settings.py` (pre-existing, non-blocking)
 
-## Deployment Process
+## Deployment Checklist
 
-### Merging to Main (for echo-next environment)
+### Before Merging to Main
 
-1. **Compare branches**: `git log main..testing --oneline`
-2. **Check for new env vars**: Look for new `Field()` definitions in `settings.py` and new exports in `config.ts`
-3. **Update deployment env vars** if needed (see checklist below)
-4. **Push Directus schema** if there were database changes
-5. **Create PR**: `testing` → `main`
-6. **Deploy** after merge
+1. **Check for new env vars**: Look for new `Field()` definitions in `settings.py` and new exports in `config.ts`
+2. **Update deployment env vars** if needed (see checklist below)
+3. **Push Directus schema** if there were database changes (see `docs/database_migrations.md`)
+4. **Create PR** from feature branch to `main`
+5. After merge → auto-deploys to Echo Next
 
 ### Environment Variables Checklist
 
@@ -225,4 +284,4 @@ LLM__MULTI_MODAL_FAST_2__GCP_SA_JSON=${GCP_SA_JSON}
 LLM__MULTI_MODAL_FAST_2__VERTEX_LOCATION=europe-west1
 ```
 
-Model groups: `TEXT_FAST`, `MULTI_MODAL_PRO`, `MULTI_MODAL_FAST`
+Model groups: `MULTI_MODAL_PRO` (Gemini 2.5 Pro — chat, reports, transcript correction), `MULTI_MODAL_FAST` (Gemini 2.5 Flash — suggestions, verification, lightweight tasks), `TEXT_FAST` (Azure GPT-4.1 — being deprecated)

echo/agent/README.md

@@ -12,8 +12,9 @@ Isolated CopilotKit/LangGraph runtime for Agentic Chat.
 
 ```bash
 cd echo/agent
-cp .env.sample .env
-# set GEMINI_API_KEY in .env
+# configure Vertex auth via one of:
+# - VERTEX_PROJECT + VERTEX_LOCATION + VERTEX_CREDENTIALS / GCP_SA_JSON
+# - Application Default Credentials (ADC)
 uv sync
 uv run uvicorn main:app --host 0.0.0.0 --port 8001 --reload
 ```
@@ -35,3 +36,5 @@ docker run --rm -p 8001:8001 --env-file .env echo-agent:local
 
 - This service is intentionally scoped to one purpose: agentic chat execution.
 - Auth, persistence, and notifications should be owned by `echo/server` gateway routes.
+- Default model is Vertex Anthropic Claude Opus 4.6 via `LLM_MODEL=claude-opus-4-6`.
+- Default Vertex location is `europe-west1` because that is the working Europe region for the current project setup.