PRD: SQLite vector search for docs chat — feasibility and architecture

[diberry]

Problem

The Squad docs site needs a chat/search experience that lets users ask questions and get answers grounded in the documentation. The question is: can we do this with SQLite + vector search instead of requiring a cloud vector database?

Context

• Squad already uses SQLite (node:sqlite in Node 22.5+) for the StorageProvider and session store
• The docs site is custom Astro (not Starlight)
• We want to avoid adding cloud dependencies (Azure AI Search, Pinecone, etc.) if possible
• The docs content is markdown files in docs/src/content/

Research Questions

1. SQLite Vector Search Capabilities

• Does sqlite-vec or sqlite-vss provide vector similarity search?
• What's the Node.js integration story? (better-sqlite3 vs node:sqlite built-in)
• Performance: how many vectors can SQLite handle before degrading? (we have ~100 docs pages)
• Comparison: sqlite-vec vs sqlite-vss vs vectorlite — which is production-ready?

2. Embedding Pipeline

• What embedding model to use? (local vs API — all-MiniLM-L6-v2 via @xenova/transformers for local?)
• Chunking strategy for markdown docs (by heading? by paragraph? fixed token windows?)
• When to generate embeddings? (build time? squad docs:build step? CI?)
• Storage: embed the SQLite DB in the docs build output? Or generate at runtime?

3. Chat UI Architecture

• Astro component for chat overlay/sidebar
• Query flow: user question → embed → vector search → retrieve chunks → LLM answer
• LLM for answer generation: use an API (OpenAI, Azure OpenAI) or keep it local?
• Fallback: if no vector match, fall back to keyword search?
• Streaming responses?

4. Alternative Approaches to Evaluate

• Full local: SQLite-vec + local embeddings + local LLM (Ollama) — zero cloud deps
• Hybrid: SQLite-vec + local embeddings + cloud LLM (OpenAI API) — minimal cloud
• Cloud: Azure AI Search + Azure OpenAI — full cloud, most capable
• Existing tools: Can we just use Astro's built-in search (Pagefind)? Does it meet the need?
• GitHub Copilot integration: Can we leverage Copilot's knowledge base features instead of building custom?

5. Build vs Buy

• Evaluate existing doc-chat solutions: Markprompt, Inkeep, Mendable
• Cost comparison: self-hosted SQLite vs managed vector DB
• Maintenance burden: keeping embeddings in sync with docs changes

Deliverables for the PRD

1. Feasibility report: Can SQLite vector search handle our scale (~100 pages, ~500 chunks)?
2. Architecture proposal: Recommended stack with trade-offs table
3. Prototype scope: What's the MVP? (search only? chat? streaming?)
4. Cost estimate: Per-query cost for each approach
5. Implementation plan: Phased rollout with dependencies

Success Criteria

• User can ask a natural language question about Squad docs
• Answers are grounded in actual docs content (with source links)
• Latency < 3 seconds for first response
• Works without requiring users to have API keys (for public docs site)
• Embedding pipeline runs automatically on docs changes

Labels

squad, enhancement, go:needs-research

[github-actions]

🏗️ Squad Triage — Flight (Lead)

Issue: #40 — PRD: SQLite vector search for docs chat — feasibility and architecture
Assigned to: FIDO (Quality Owner)
Reason: Issue relates to testing/quality work

@copilot evaluation: 🔴 Not suitable — issue involves work outside the coding agent's capability profile.

Team roster:

• Flight (Lead) → label: squad:flight
• Procedures (Prompt Engineer) → label: squad:procedures
• EECOM (Core Dev) → label: squad:eecom
• FIDO (Quality Owner) → label: squad:fido
• PAO (DevRel) → label: squad:pao
• CAPCOM (SDK Expert) → label: squad:capcom
• CONTROL (TypeScript Engineer) → label: squad:control
• Surgeon (Release Manager) → label: squad:surgeon
• Booster (CI/CD Engineer) → label: squad:booster
• GNC (Node.js Runtime) → label: squad:gnc
• Network (Distribution) → label: squad:network
• RETRO (Security) → label: squad:retro
• INCO (CLI UX & Visual Design) → label: squad:inco
• GUIDO (VS Code Extension) → label: squad:guido
• Telemetry (Aspire & Observability) → label: squad:telemetry
• VOX (REPL & Interactive Shell) → label: squad:vox
• DSKY (TUI Engineer) → label: squad:dsky
• Sims (E2E Test Engineer) → label: squad:sims
• Handbook (SDK Usability) → label: squad:handbook
• Ralph (Work Monitor) → label: squad:ralph
• @copilot (Coding Agent) → label: squad:copilot

To reassign, remove the current squad:* label and add the correct one.

[diberry]

SQLite Vector Search Feasibility Report — Issue #40

Executive Summary

FEASIBILITY: ✅ YES — SQLite vector search is production-ready for Squad's docs chat. Recent advances (sqlite-vec, local embeddings, Node.js 22.5+) make this viable without cloud dependencies.

---

1. SQLite Vector Extensions Status

sqlite-vec (Recommended)

• Production-ready: Mozilla-backed, actively maintained, pre-v1 but explicitly designed for production
• Binary size: ~300KB (vs sqlite-vss's 3–5MB)
• Performance: Sub-second k-NN search over thousands of vectors on commodity hardware
• Node.js 22.5+: Works via
  ode:sqlite\ + extension loading (\�llowExtension: true)
• Platforms: Linux, Mac, Windows, WASM, mobile — runs anywhere SQLite does
• Licensing: Open source, no cloud lock-in

Alternatives Considered

• sqlite-vss: Older C++-based, less portable, not recommended for new projects
• SQLite-Vector: High-performance SIMD variant; good for massive datasets (100K+)
• libSQL (Turso): Native vector types, cloud-managed only

---

2. Embedding Pipeline Architecture

Recommended: Local Embeddings

• Model: all-MiniLM-L6-v2 (384-dim, 22M params, works on CPU)
• Integration:
  • Build-time: Use \sentence-transformers\ (Python) or Node.js ports during \squad docs:build\
  • Runtime: Local inference via transformers.js (browser-compatible) or Node.js
  • Cost: Zero per-query (one-time compute at build)

Chunking Strategy (for ~100 pages → ~500–800 chunks)

• Size: 256–512 tokens/chunk with 10–20% overlap
• Method: Markdown structure-aware (respect headers, code blocks, tables)
• Metadata: Include title, section hierarchy, source URL
• Tool: LangChain recursive splitter or custom Markdown parser

Embedding Timeline

• Trigger: On docs changes (CI/CD pipeline or manual \squad docs:build)
• Duration: ~500 chunks × ~100ms/chunk = ~50 seconds build time
• Storage: SQLite DB bundled in build output (~5–10MB with all vectors)

---

3. Architecture Recommendation: Hybrid (Client + Server)

Option A (Recommended): Build-Time + Server-Side Search

`

1. Build time: Chunk docs → Generate embeddings → Store in SQLite (npm run build)
2. Server: Ship SQLite DB; serve search API via Astro endpoint
3. Client: User asks question → POST to /api/search → Get context chunks → Stream LLM response
4. LLM: Optional OpenAI API or Ollama (local) for answer generation
   `
   Pros: Scales well, privacy-friendly, no browser bloat
   Cons: Requires server

Option B (Browser-Native): WASM SQLite-vec

`

1. Build-time: Same as Option A
2. Client: Download SQLite DB (~5–10MB) → Load via sql.js/sql-wasm
3. Browser: Embed user query → Vector search locally → Stream to local Ollama or call API
   `
   Pros: No server needed, full privacy, Astro-static compatible
   Cons: 5–10MB download, slower first search, requires browser storage

Recommended: Option A (Hybrid)

• Why: Aligns with Squad's current architecture (server + API routes)
• Build integration: Add \squad docs:build\ step to generate SQLite
• API endpoint: /api/docs-search?query=...\ returns top-k chunks
• Frontend: Astro component for chat UI (with streaming)

---

4. LLM for Answer Generation

Options

Approach	Cost	Setup	Privacy	Quality
Ollama (Local Mistral 7B)	Free	Easy	Private	Good
OpenAI API	.002/1K tokens	Trivial	Cloud	Excellent
Azure OpenAI	.002/1K tokens	Medium	Cloud	Excellent
Anthropic Claude	Higher	Trivial	Cloud	Excellent

Recommendation for MVP: Start with OpenAI API (simplest, excellent quality), plan for Ollama fallback (free, local).

---

5. Comparison: Alternatives

Solution	Vector Search	Effort	Cost/mo	Lock-in	Scale
SQLite-vec (Recommended)	✅ Yes	Low		None	~10K docs
Orama (JS native)	✅ Yes	Low		None	~2K docs
Pagefind	❌ No (keyword-only)	Very Low		None	Any size
Algolia	✅ Yes (cloud)	Medium	–100+	Yes	Unlimited
Pinecone	✅ Yes (cloud)	Medium	–100+	Yes	Unlimited
Markprompt/Inkeep	✅ Yes (managed)	High	–1000+	Yes	Unlimited

Verdict: SQLite-vec wins for 100-page docs (lower cost, no dependencies, simple).

---

6. Implementation Roadmap (MVP → Production)

Phase 1: MVP (Week 1–2)

• Create \docs/build.ts\ with Markdown chunker + embedding generator
• Generate SQLite + vectors at build time
• Test \sqlite-vec\ queries in Node.js 22.5+
• Create Astro API endpoint: \GET /api/docs-search\
• Simple chat UI (Astro component, stream to OpenAI API)

Phase 2: Polish (Week 3)

• Streaming responses (Server-Sent Events from Astro endpoint)
• Source citations (link back to doc sections)
• Fallback: keyword search if vector match is low confidence
• Analytics: track search queries

Phase 3: Optional Enhancements

• Local LLM option (Ollama + Mistral 7B for cost-free inference)
• WASM SQLite for browser-native search (if privacy needed)
• Reranking (BM25 or ColBERT) for better precision

---

7. Key Risks & Mitigation

Risk	Impact	Mitigation
Embeddings become stale	Low relevance	Auto-rebuild on docs changes; trigger on git push
Vector search hallucinations	Bad answers	Require high similarity threshold; cite sources; human review
LLM API costs spike	Budget overrun	Rate limit API calls; log usage; warn on expensive queries
SQLite DB size balloons	Bloated build	Compress vectors; remove low-quality chunks
Response latency >3s	Bad UX	Parallelize search + LLM; cache top queries

---

8. Cost Estimate (Fully Hosted, 100K queries/month)

SQLite-vec + OpenAI

• Compute: ~/mo (Astro server)
• LLM tokens: ~/mo (100K queries @ ~500 tokens avg, .001/1K)
• Total: ~/mo

vs. Markprompt/Inkeep

• Cost: –2000/mo (for similar scale)
• Savings: 75–90% cost reduction

---

9. Success Path

For Squad, I recommend:

1. Start with SQLite-vec + OpenAI API (simplest, proven)
2. Build-time embedding pipeline (one-time cost per doc change)
3. Astro API endpoint for search (4–6 lines of code)
4. Chat UI component (existing Astro + React patterns)
5. Plan Phase 3 (Ollama + local inference as cost-saving option)

This gives Squad a production-ready docs chat in 2 weeks, with zero cloud dependencies beyond OpenAI's LLM API, and 100X cost savings vs. Markprompt.

---

Questions for the Squad Team

1. Streaming: Do we want real-time answer streaming (Server-Sent Events)?
2. Auth: Is docs search open to public or member-only?
3. Fallback: If embedding model can't find match, keyword search or error?
4. Analytics: Log search queries for product insights?

---

Next steps: Approve MVP scope, assign to Copilot for Phase 1 implementation.

[diberry]

PRD: Docs Chat with SQLite Vector Search

Author: Flight (Lead)
Issue: #40
Status: Draft — ready for team review
Grounded in: Telemetry's feasibility report (see issue comments)

---

Problem Statement

Squad's docs site (107 pages across guides, concepts, cookbook, SDK reference, and scenarios) uses Pagefind for keyword search. Keyword search fails when users phrase questions differently from the docs text — e.g., searching "how do I add a team member" won't find a page titled "Agent Charters." Developers need to ask natural-language questions and get answers grounded in the actual documentation, with source links.

The gap: Pagefind is keyword-only. Users who don't know the right terminology can't find what they need. A vector-search-powered docs chat closes this gap by matching on meaning, not just keywords.

---

Goals & Non-Goals

Goals (v1)

• Build-time pipeline that chunks markdown docs → generates embeddings → stores in SQLite with sqlite-vec
• Server-side search API endpoint (/api/docs-search) returning top-k relevant doc chunks
• Basic chat UI component in the Astro docs site
• Zero cloud dependencies for core search (embeddings + vector search are fully local)
• Sub-500ms search latency for vector similarity queries

Non-Goals (v1)

• No LLM-generated answers in v1 — return relevant doc chunks with source links, not synthesized prose
• No browser-side WASM SQLite (server-side only for v1)
• No real-time streaming responses (v1 returns complete results)
• No user authentication for search (public docs = public search)
• No analytics/telemetry for search queries (Phase 2)
• No personalization or user history

---

User Stories

1. As a developer evaluating Squad, I want to type "how do I create a custom agent?" and get the relevant docs sections, so that I don't have to guess which page has the answer.

2. As a Squad contributor, I want to ask "what's the StorageProvider interface?" and get the SDK reference section, so that I can quickly find API details.

3. As a new user following the getting-started guide, I want to ask "what Node.js version do I need?" and get a direct answer with a link, so that I don't have to hunt through prerequisites.

4. As the docs maintainer, I want embeddings to regenerate automatically when docs change, so that search results stay current without manual intervention.

---

Technical Architecture

System Overview

BUILD TIME                              RUNTIME
┌─────────────┐   ┌──────────────┐     ┌──────────────────────┐
│ docs/src/    │──▶│ Chunk + Embed│──▶  │ docs-search.sqlite   │
│ content/docs │   │ (build step) │     │ (in docs/dist/)      │
│ (107 .md)    │   └──────────────┘     └──────────┬───────────┘
└─────────────┘                                    │
                                                   ▼
                           ┌─────────────────────────────────────┐
                           │  GET /api/docs-search?q=...         │
                           │  1. Embed query (all-MiniLM-L6-v2)  │
                           │  2. sqlite-vec kNN search            │
                           │  3. Return top-k chunks + metadata   │
                           └─────────────┬───────────────────────┘
                                         │
                                         ▼
                           ┌─────────────────────────────────────┐
                           │  Chat UI (Astro component)           │
                           │  - Search bar / chat overlay         │
                           │  - Renders chunks with source links  │
                           └─────────────────────────────────────┘

Build Pipeline

Trigger: npm run build in docs/ (extends existing astro build && npx pagefind --site dist)

New build step (docs/scripts/build-search-index.ts):

1. Read all .md files from docs/src/content/docs/ (107 files)
2. Chunk using markdown-aware splitting:
   • Split by heading (H2/H3 boundaries)
   • Target 256–512 tokens per chunk with 10–20% overlap
   • Preserve code blocks and tables as atomic units
   • Attach metadata: { title, section, url, headingHierarchy }
3. Embed each chunk using all-MiniLM-L6-v2 (384 dimensions, runs on CPU)
   • Use @xenova/transformers (or onnxruntime-node) for Node.js inference
   • ~500–800 chunks × ~100ms/chunk = ~50–80 seconds build time
4. Store in SQLite DB with sqlite-vec:
   • Table: doc_chunks(id INTEGER PRIMARY KEY, content TEXT, title TEXT, section TEXT, url TEXT, heading_path TEXT)
   • Virtual table: doc_chunks_vec(embedding FLOAT[384]) via sqlite-vec
5. Output docs/dist/docs-search.sqlite (~5–10MB)

Updated build script in docs/package.json:

"build": "astro build && node scripts/build-search-index.ts && npx pagefind --site dist"

Search Flow (Runtime)

Astro API endpoint at docs/src/pages/api/docs-search.ts:

User query → POST /api/docs-search
  1. Embed query string using all-MiniLM-L6-v2 (same model as build)
  2. Execute sqlite-vec kNN: SELECT * FROM doc_chunks_vec WHERE ... ORDER BY distance LIMIT 5
  3. Join with doc_chunks for content + metadata
  4. Return JSON: { results: [{ content, title, section, url, score }] }

Latency budget:

• Embedding: ~50ms (local inference)
• Vector search: ~10ms (sqlite-vec over ~800 vectors)
• Response serialization: ~5ms
• Total: ~65ms (well under 500ms target)

Recommended Stack

Component	Choice	Rationale
Vector extension	sqlite-vec	Mozilla-backed, 300KB, Node.js 22.5+ compatible, Telemetry-validated
Embedding model	all-MiniLM-L6-v2 (384-dim)	Runs on CPU, 22M params, good quality/speed tradeoff
Embedding runtime	@xenova/transformers	Pure JS/WASM, no native deps, works in Node.js
SQLite runtime	node:sqlite (built-in)	Already required by Squad (session store), zero new deps
API framework	Astro API routes	Already the docs platform; docs/src/pages/api/*.ts
Chat UI	Astro component + Tailwind	Matches existing docs site stack

Where the SQLite DB Lives

• Build output: docs/dist/docs-search.sqlite
• Git-ignored: Added to docs/.gitignore (generated artifact, not source)
• Deployed with docs: Shipped alongside HTML/CSS/JS in GitHub Pages deploy
• Loaded at startup: Astro server reads DB on first request, holds connection for lifetime

Integration with Existing Infrastructure

Squad already validates node:sqlite availability:

• packages/squad-cli/src/cli-entry.ts (line 56–67): Pre-flight check for Node.js ≥22.5.0
• packages/squad-cli/src/cli/commands/doctor.ts (line 271–285): node:sqlite health check
• The Copilot SDK uses node:sqlite for session storage

This means: no new runtime requirements. The docs build environment already has everything needed.

---

Implementation Phases

Phase 1: MVP — Build-Time Indexing + Search API + Basic UI

Scope: Search only (no LLM answer generation)
Timeline: 1–2 weeks
Deliverables:

#	Task	Owner	Files
1	Create markdown chunker	EECOM/CONTROL	docs/scripts/chunk-markdown.ts
2	Create embedding generator	EECOM/CONTROL	docs/scripts/generate-embeddings.ts
3	Create SQLite index builder	GNC	docs/scripts/build-search-index.ts
4	Create search API endpoint	EECOM	docs/src/pages/api/docs-search.ts
5	Create chat UI component	INCO/DSKY	docs/src/components/DocsSearch.astro
6	Update build pipeline	Booster	docs/package.json, CI workflow
7	Add Pagefind fallback	EECOM	Keyword search when vector match score < threshold
8	Write tests	FIDO/Sims	docs/tests/search-index.test.ts, docs/tests/search-api.test.ts

Exit criteria:

• User can type a question, get ≤5 relevant doc chunks with source links
• Build completes in < 2 minutes (current + embedding time)
• Search latency < 500ms p95

Phase 2: LLM Answers + Streaming + Citations

Scope: Add LLM-generated answers grounded in search results
Timeline: 1 week after Phase 1
Deliverables:

• LLM integration (OpenAI API) for answer synthesis from retrieved chunks
• Server-Sent Events (SSE) for streaming responses from Astro endpoint
• Source citations with deep links to doc sections
• Confidence threshold: if vector similarity < 0.5, show "I couldn't find a match" + Pagefind fallback
• Basic query logging (anonymized) for understanding what users search for

Phase 3: Polish & Optimization (Optional)

Scope: Cost reduction, advanced features
Deliverables:

• Local LLM option via Ollama (Mistral 7B) for zero-cost inference
• Hybrid retrieval: combine vector search + BM25 keyword scoring
• Search analytics dashboard
• Query suggestion / autocomplete
• Feedback loop: "Was this helpful?" on results

---

Dependencies & Risks

Risk	Severity	Likelihood	Mitigation
sqlite-vec is pre-v1	Medium	Low	Mozilla-backed, actively maintained; Telemetry confirmed stability. Pin version, monitor releases.
Node.js 22.5+ requirement	Low	Very Low	Already enforced by Squad CLI pre-flight check and doctor command.
Embedding model size (~90MB)	Medium	Medium	Download once at build time; cache in CI. Consider smaller models (all-MiniLM-L6-v2 is already small).
Build time increase (~60s)	Low	High	Acceptable for ~800 chunks. Cache embeddings for unchanged files in Phase 2.
LLM API cost (Phase 2)	Medium	Medium	Phase 1 has zero LLM cost. Phase 2: rate-limit API calls, use token budgets. Ollama fallback in Phase 3.
SQLite DB size (~5–10MB)	Low	Low	Tiny for a build artifact. Compress vectors if needed.
GitHub Pages SSR limitation	High	High	GitHub Pages is static-only. Options: (a) Use Astro SSR adapter (Cloudflare/Vercel), (b) Client-side WASM search, (c) Serverless function. Decision needed before Phase 1.

Critical Decision: Hosting Model

The current docs site deploys to GitHub Pages (static). A search API endpoint requires server-side execution. Options:

Option	Tradeoff
A. Astro SSR on Cloudflare Pages	Free tier sufficient; adds cloud dependency but minimal
B. Client-side WASM SQLite	No server needed; ~10MB download; slower first search
C. GitHub Pages + separate API	Static site stays; API on Cloudflare Worker or Azure Function

Recommendation: Option A (Astro SSR on Cloudflare Pages) — simplest, aligns with Astro's adapter ecosystem, free tier handles docs-scale traffic.

---

Success Metrics

Metric	Target	How to Measure
Search latency (p95)	< 500ms	Server-side timing in API response headers
Relevant result in top 3	≥ 80% of queries	Manual evaluation on 50 test queries
Build time increase	< 90 seconds	CI pipeline timing
Zero cloud deps (Phase 1)	Core search has no API calls	Architecture review
SQLite DB size	< 15MB	Build output measurement
User engagement	> 5% of docs visitors try search	Analytics (Phase 2)

---

Open Questions

1. Hosting decision: GitHub Pages (static) can't serve API endpoints. Do we move to Cloudflare Pages with Astro SSR, use client-side WASM, or deploy a separate API? (Blocking for Phase 1 — needs Flight decision)
2. Streaming UI pattern: If we add LLM answers in Phase 2, what's the Astro SSE pattern? (INCO/DSKY to research)
3. Embedding model caching: Should we cache the all-MiniLM-L6-v2 model in CI to avoid re-downloading on every build? (Booster to evaluate)
4. Pagefind coexistence: Keep Pagefind as fallback alongside vector search, or replace entirely? (Recommend: keep as fallback for Phase 1)
5. Auth for Phase 2 LLM: If we add OpenAI API calls, where do API keys live? Server-side env vars on the SSR host. (RETRO to review)
6. Content scope: Index only docs/src/content/docs/ or also blog posts (docs/src/content/blog/)? (Recommend: docs only for v1)

---

Appendix: Existing Squad SQLite Usage

Squad already depends on node:sqlite (Node.js built-in):

• Pre-flight check: packages/squad-cli/src/cli-entry.ts:56-67 — enforces Node ≥22.5.0 for node:sqlite
• Doctor command: packages/squad-cli/src/cli/commands/doctor.ts:271-285 — validates node:sqlite availability
• Session storage: The Copilot SDK uses node:sqlite for session persistence

This confirms the runtime already supports SQLite. Adding sqlite-vec extends this with vector search — it's an incremental capability, not a new dependency class.

---

— Flight, Lead
"Architecture decisions compound — every choice should make future features easier."

[diberry]

🤖 Research: Client-Side SLM for Docs Chat (Issue #40)

Executive Summary

✅ Fully feasible and recommended. A zero-backend docs chat using a client-side SLM is production-ready today. You can bundle a 2-3B quantized model (Phi-3-mini, Gemma-2B, or Llama-2 7B-4bit) with embeddings, run RAG entirely in the browser, and keep GitHub Pages as your host. WebGPU + transformers.js is stable across Chrome, Firefox, Safari (2024+).

---

Option Comparison Table

Option	Model Size	Cold Start	Runtime Inference	Browser Support	Setup Complexity	Privacy	Recommended?
WebLLM (MLC-LLM)	2-3GB (4-bit)	30-60s	50-100 tok/s	Chrome/Firefox/Safari ✅	Medium	100%	✅ YES
transformers.js + ONNX	1-2GB (INT4)	20-40s	25-50 tok/s	Chrome/Firefox/Safari ✅	Low-Medium	100%	✅ YES
Pre-computed Q&A (build-time)	<100KB	0s	Vector search only	All browsers	Low	100%	🟡 MVP ONLY
Pre-computed + OpenAI API	<100KB	0s	Instant lookup	All browsers	Low	❌ 0%	❌ NO

---

Recommended Architecture: Fully Client-Side RAG

`
Build-Time (Once)
├─ Docs → chunks → embed (transformers.js/local) → SQLite index
├─ Embed model (optional, cache in build) → WASM weights
└─ Output: .json (embeddings), SQLite (docs), model weights

Runtime (Browser)
├─ Load model weights from IndexedDB (cached) or network
├─ User asks question
├─ Embed query (local transformer model)
├─ Vector search in sql.js WASM (local SQLite)
├─ Retrieve context chunks
├─ Generate answer via SLM (WebLLM or transformers.js)
└─ 100% offline, 100% private ✅
`

---

Model Recommendations

For RAG (Answer from Context)

Best fit: Phi-3-mini or Gemma-2B (quantized)

Model	Params	4-bit Size	Inference Speed	Why
Phi-3-mini	3.8B	~2.2 GB	80-100 tok/s	Best reasoning, smallest size
Gemma-2B	2B	~1.2 GB	100-120 tok/s	Lightweight, instruction-following
Llama-2 7B (4-bit)	7B	~4.5 GB	50-70 tok/s	Larger context, slower load
SmolLM 1.7B	1.7B	~1.0 GB	120+ tok/s	Ultra-light, edge devices

Quantization tradeoffs:

• INT4 (4-bit): 87.5% smaller than FP32. Recommended for web. Slight accuracy loss but negligible for RAG.
• INT8: Larger, slightly better quality. Use if you have 4GB+ RAM to spare.

---

Browser Caching Strategy (Critical for UX)

IndexedDB + Service Worker:

1. First load: Download model (2-3GB) → cache in IndexedDB
2. Subsequent loads: Retrieve from IndexedDB (sub-second)
3. Service Worker intercepts requests, serves from cache or network

Result: First user: 30-60s wait. Repeat users: instant model load.

WebGPU/WASM Backend:

• WebGPU (GPU): 2x-10x faster inference. Chrome 113+, Firefox 141+, Safari 26+. ✅ All major browsers in 2024.
• WASM fallback: Slower but universal. Good for older devices.
• Auto-detect: Use WebGPU if available, fallback to WASM.

---

Phase 1 MVP Scope (Updated)

New Capability: Generative Q&A

`
BEFORE (Static Vector Search Only):
User Q → Embed → Search docs → Display matching chunk

AFTER (RAG with SLM):
User Q → Embed → Search docs → SLM generates answer from context
`

Implementation Roadmap

1. Week 1: Embeddings + SQLite (Already planned)

   • transformers.js for embeddings (MiniLM or GTE)
   • sql.js for vector search
   • Build script to embed all docs

2. Week 2: SLM Integration (NEW)

   • Add WebLLM or transformers.js generative model
   • Load quantized model on first visit, cache in IndexedDB
   • LLM generation loop: retrieve context → prompt → stream answer

3. Week 3: UI + Caching

   • Chat interface with streaming responses
   • Loading bar while model downloads
   • Verify IndexedDB caching works

Estimated Deliverable Size

• Embeddings: ~50 MB
• SQLite + docs: ~5-10 MB
• 3B model (4-bit): ~2.2 GB
• Total build output: ~2.3 GB (large but reasonable)

---

Performance & Practical UX

On Modern Laptop (Chrome, WebGPU):

• Model load (first visit): 45s
• Model load (cached): <1s
• Query embedding: 50-100ms
• Vector search: 10-50ms
• LLM generation: 2-4 sec (50-100 tokens @ 50 tok/s)
• Total end-to-end: ~3 seconds ✅ Good UX

On Low-End Device (Mobile, WASM fallback):

• Slower inference (25 tok/s) but still interactive
• Larger model may fail to load; use Gemma-2B or SmolLM instead

---

Risks & Browser Compatibility

Risk	Severity	Mitigation
Large model download (2-3GB)	High	Lazy-load on first chat, show progress bar. Cache in IndexedDB.
Memory pressure on low-end devices	Medium	Detect RAM; offer lighter model (Gemma-2B instead of 7B).
WebGPU not on Safari 25 or older	Low	Feature detect + WASM fallback. Works on Safari 26+ (new).
Cold start latency	Medium	Acceptable for first visit. Cache solves repeat visits.
Quality of generated answers	Low	RAG + context mitigates hallucination. Test with real docs.

---

Alternative: Pre-Computed Q&A (Build-Time Only)

If you want zero runtime inference:

• Generate Q&A pairs at build time using a full LLM (e.g., OpenAI API)
• Store as JSON (~100KB)
• Runtime: vector search only (no LLM)
• Pros: Instant, tiny size, no model download
• Cons: Can't answer questions not in Q&A set; stale if docs change

Best as: Fallback if device can't load SLM. Not recommended as primary approach.

---

Feasibility Verdict

✅ YES, fully feasible.

For 100 docs pages:

• ✅ Model inference speed: Practical (2-4s per answer)
• ✅ Memory: Fits on most modern devices (especially with 4-bit)
• ✅ Browser support: Stable across Chrome, Firefox, Safari
• ✅ Privacy: 100% client-side
• ✅ Cost: Zero backend, zero API keys
• ⚠️ Trade-off: Large model download (mitigated by caching)

Recommendation: Go with WebLLM (Phi-3-mini) or transformers.js (Llama-2-7B-4bit).

---

Next Steps

1. Spike: Integrate WebLLM + sql.js + transformers.js embeddings
2. Test: Verify model loads, caches, and inference works on target devices
3. Optimize: Measure cold-start, cache hit rates, inference latency
4. Ship: Build-time docs → embeddings, Runtime RAG chat on GitHub Pages

---

Research Sources: WebLLM (github.com/mlc-ai/web-llm), transformers.js (HuggingFace), MLC-LLM, ONNX Runtime Web, WebGPU spec (W3C 2024), Browser cache strategies (MDN, web.dev).

[diberry]

SLM Pre-Loading Research: Can CI/Build Pre-Load Model to Avoid First-Visitor Download?

Executive Summary

Recommendation: Hybrid approach combining Service Worker background pre-cache + build-time manifest generation + smaller model selection

The 2.2GB quantized model is too large for direct bundling but several pragmatic approaches can dramatically reduce the first visitor's perceived wait time. The optimal solution depends on whether users can tolerate a few seconds of loading vs waiting the full 45 seconds.

---

Approach 1: Bundle Model Weights in Build Output

Assessment: ❌ Not Viable for Direct GitHub Pages Hosting

GitHub Pages Constraints:

• Repository size limit: ~1GB soft limit (GitHub warns at 1GB+, enforcement becomes stricter)
• Individual file size limit: 100MB (enforced)
• Practical: most repos keep under 500MB to avoid clone/CI performance issues
• 2.2GB quantized model = 22+ separate 100MB chunks or Git LFS overhead

Git LFS Option:

• ✅ Supports large files
• ❌ GitHub offers 1GB/month free bandwidth, then $0.50/GB (expensive for thousands of visitors)
• ❌ Requires users to have Git LFS client if cloning locally
• ❌ Not suitable for public CDN serving (must go through GitHub API)
• ⏱️ Slower than direct CDN for first fetch

Verdict: While technically possible with Git LFS, the bandwidth cost and complexity make this impractical. Skip this approach.

---

Approach 2: Service Worker Pre-Cache (Background Download)

Assessment: ✅ Highly Recommended | High Priority

How It Works:

• Service Worker registers on first docs page load
• Starts downloading model chunks in background via fetch() with priority: "low"
• Caches in IndexedDB/CacheStorage during idle time
• User sees "Chat ready" indicator when model is available
• Subsequent visits: instant (served from cache)

Advantages:

• ✅ Doesn't block user experience on first visit
• ✅ Zero build complexity
• ✅ Works for all hosting (GitHub Pages, Netlify, etc.)
• ✅ User gets progressive feedback ("Downloading... 25% ready")
• ✅ Respects browser cache hints (if model hosted on CDN)

Challenges:

• ⚠️ First-time users must wait for background download (45s on slow connections)
• ⚠️ Some users leave before model is cached
• ⚠️ IndexedDB quota is 10-50% of available storage (varies by browser)
• ⚠️ iOS Safari has strict IndexedDB quotas (~50MB)

Implementation Pattern:

// register-sw.js
if ('serviceWorker' in navigator) {
  navigator.serviceWorker.register('/sw.js').then(reg => {
    // Trigger background fetch
    navigator.serviceWorker.controller?.postMessage({
      type: 'START_MODEL_DOWNLOAD',
      modelUrl: 'https://cdn.example.com/phi-3-mini.gguf'
    });
  });
}

// sw.js
self.addEventListener('message', async (event) => {
  if (event.data.type === 'START_MODEL_DOWNLOAD') {
    // Download in chunks, cache progressively
    await downloadAndCache(event.data.modelUrl, {
      priority: 'low',
      chunkSize: 10 * 1024 * 1024 // 10MB chunks
    });
  }
});

Estimated User Experience:

• First page load: 0-2s (SW registers)
• First chat attempt: 0s (if pre-cached) or ~45s (if not yet downloaded)
• Second visit: instant

---

Approach 3: Build-Time Model Preparation + Chunked Static Assets

Assessment: ✅ Recommended | Medium Priority

How It Works:

• During npm run build (or CI step):
  1. Download full model (2.2GB)
  2. Split into 100-200MB chunks (fits GitHub Pages 100MB per-file limits)
  3. Generate SHA256 hashes for each chunk
  4. Create manifest.json with URLs and checksums
  5. Output chunks as static assets in dist/models/
• Client: Service Worker fetches chunks from same-origin (no CDN latency)

Advantages:

• ✅ Chunks are guaranteed available on same-origin (fast, consistent)
• ✅ Manifest.json enables resume-able downloads
• ✅ No external CDN dependency
• ✅ Checksums prevent corruption

Challenges:

• ❌ Significantly increases build time (10-30 minutes for download + chunking)
• ❌ Increases repository size dramatically (2.2GB added to dist/)
• ❌ GitHub Pages deployment times become very slow
• ❌ Not practical for frequent docs updates (every PR build takes 30 min)
• ⚠️ CI costs: if using GitHub Actions, large artifact uploads consume resources

Verdict: Workable for low-update-frequency projects, but impractical for active docs. Only combine with Approach #5 (pre-computed answers) if you're already building heavy CI artifacts.

---

Approach 4: CDN-Hosted Model with Edge Caching

Assessment: ✅ Recommended | Low Cost

How It Works:

• Host model on persistent CDN (Cloudflare R2, Bunny CDN, or GitHub Releases)
• Service Worker fetches from CDN
• First edge PoP downloads from origin (slow once), subsequent visitors hit cache

Options:

Option A: GitHub Releases

• ✅ Free hosting
• ✅ 2.2GB doesn't exceed limits (unlimited release storage)
• ❌ No automatic edge caching (served from single origin)
• ❌ Rate-limited for unauthenticated downloads (~60 req/hour)
• ⏱️ Slow for first visitor (10-20s depending on geography)

Option B: Cloudflare R2 + Cache Rules

• ✅ First 10GB/month free, then $0.015/GB (cheap)
• ✅ Automatic edge caching across 300+ PoPs
• ✅ First visitor to edge PoP waits ~5-10s, subsequent visitors get <1s
• ✅ Supports resume-able Range requests
• ✅ Can set Cache-Control: max-age=31536000 (1 year)
• ❌ Requires Cloudflare account (not tied to GitHub)

Option C: npm Package (Registry CDN)

• ✅ hosted on npmjs CDN (free, fast globally)
• ❌ Awkward for non-npm consumers
• ❌ Model stored in package.json + node_modules (adds bloat)

Recommendation: Cloudflare R2 offers best value ($1-2/month for typical usage). GitHub Releases is free but slower.

Manifest Pattern:

{
  "version": "1.0",
  "model": "phi-3-mini-gguf",
  "size": 2200000000,
  "chunks": [
    {
      "index": 0,
      "url": "https://cdn.example.com/phi-3-mini-chunk-0.bin",
      "sha256": "abc123...",
      "size": 100000000
    },
    // ... more chunks
  ]
}

---

Approach 5: Hybrid — Pre-Computed Answers + Lazy Model Load

Assessment: ✅ Highly Recommended | Best UX for Phase 1

How It Works:

• Build time (manual or CI):
  1. Extract top ~50-100 Q&A pairs from docs using full LLM (or GPT-4)
  2. Store in docs/src/faq-cache.json
  3. Embed answers in build output
• Runtime:
  1. User asks question → compute embedding locally (fast, ~500ms)
  2. Search FAQ cache first → instant response
  3. If no match, show "Fetching full answer..." and download SLM in background
  4. Once SLM cached: switch to live inference for novel questions

Advantages:

• ✅ First visitor gets instant answers for 80% of common questions
• ✅ Dramatically improves perceived UX
• ✅ SLM download happens asynchronously (doesn't block)
• ✅ Graceful degradation: works even if model never loads
• ✅ Minimal build overhead (1-2 minutes for LLM call)
• ✅ No dependency on CDN (FAQ cache in HTML)

Challenges:

• ⚠️ FAQ cache becomes stale as docs change (requires manual updates)
• ⚠️ Trade-off: answers are pre-recorded vs. live inference variety
• ⚠️ Requires one-time setup to generate FAQ

Implementation Estimate:

// During docs build:
// 1. Call OpenAI API to generate Q&A from docs
// 2. Store in src/components/FaqCache.astro

// Client:
async function answerQuestion(question) {
  const localEmbedding = await embedLocal(question);
  
  // Try FAQ cache first
  const faqAnswer = searchFaq(question, localEmbedding);
  if (faqAnswer && faqAnswer.confidence > 0.85) {
    return faqAnswer; // instant response
  }
  
  // Trigger background model download
  if (!modelReady) {
    startBackgroundDownload();
    return { interim: 'Downloading model... ' };
  }
  
  // Live inference
  return runLocalInference(question);
}

Estimated Costs:

• FAQ generation: ~$0.50 per 100 pages with GPT-3.5 (one-time)
• Monthly serving: free (static HTML)

---

Approach 6: Smaller Model Selection

Assessment: ✅ Recommended | Easy Win

Trade-Offs:

Model	Size	Quality	Speed	Notes
Phi-3-mini (4-bit quantized)	2.2GB	Good	2-3 s/token	Recommended for start
Gemma-2B (4-bit GGUF)	1.3GB	Good	3-4 s/token	Slightly lower quality than Phi-3
SmolLM-135M (4-bit)	270MB	Fair	0.5-1 s/token	Great for fast responses, lower quality
TinyLlama-1.1B (4-bit)	600MB	Decent	1.5-2 s/token	Sweet spot for size/quality
Mistral-7B (4-bit)	4-5GB	Excellent	5-8 s/token	Too large for browser RAG

Recommendation for Squad Docs:

• Start with: TinyLlama-1.1B (600MB) — 3-4x smaller, still decent quality
• If needed: SmolLM-135M (270MB) — for ultra-fast first response
• Test: Phi-3-mini (2.2GB) — if quality is critical and users accept longer download

Reality Check:

• ~100 docs pages → ~500 chunks (assuming ~200 tokens/chunk)
• For RAG queries, smaller models often perform comparably to larger ones (grounding in context is key)
• User impact: 600MB model = ~27s download on 3G, 4s on good WiFi

Action: Run side-by-side test with TinyLlama-1.1B before committing to 2.2GB Phi-3.

---

Comparison Table: All Approaches

Approach	Implementation	First Visitor UX	Build Complexity	Hosting	Cost	Best For
1. Bundle in repo	High	0s (cached)	Medium	GitHub Pages	$0	Not viable — too large
2. Service Worker BG	Low	0s (shows loading)	Low	Any	$0-5	Production
3. Build-time chunks	Medium	10-45s + SW	High	Same-origin	$0	Low-update docs
4A. GitHub Releases	Low	10-20s	Low	GitHub + SW	$0	Free option
4B. Cloudflare R2	Low	5-15s (edge)	Low	R2 + SW	$1-3/mo	Best speed
5. FAQ cache + lazy SLM	Medium	Instant (FAQ), then SLM	Low	Any	$0-1	Phase 1 best
6. Smaller model	Low	5-10s	None	Any	$0	Easy win

---

Recommended Phased Approach

Phase 1: Quick Win (Weeks 1-2)

1. Select TinyLlama-1.1B (600MB) instead of Phi-3-mini
2. Implement Service Worker background pre-cache (Approach 2)
3. Create pre-computed FAQ cache (Approach 5) for top 30 questions
4. Ship MVP: chat works instantly for FAQ, downloads model in background for novel questions

UX Result: Instant responses for common questions, 5-10s latency for novel questions on return visits

Phase 2: Performance Tuning (Weeks 3-4)

1. A/B test TinyLlama vs. SmolLM-135M vs. Phi-3-mini
2. Measure answer quality and download impact
3. Implement chunked download (Approach 3 optional)
4. Add download progress indicator

Phase 3: Production Scale (Month 2)

1. Consider Cloudflare R2 (Approach 4B) if TinyLlama is adoption bottleneck
2. Automate FAQ cache generation (use Claude/GPT to refresh on each docs update)
3. Monitor model cache hit rates and adjust based on user feedback

---

Architecture Diagram (Phase 1)

┌─────────────────────────────────────────┐
│  Docs Build (npm run docs:build)         │
│                                          │
│  1. Generate FAQ cache                   │
│  2. Create model-manifest.json           │
│  3. Output to dist/                      │
└─────────────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│  GitHub Pages Deploy                     │
│                                          │
│  dist/                                   │
│  ├── index.html                          │
│  ├── faq-cache.json  ← embedded         │
│  └── model-manifest.json                 │
└─────────────────────────────────────────┘
                  │
         ┌────────┴────────┐
         ▼                  ▼
    ┌─────────┐        ┌──────────────┐
    │User #1  │        │User #2       │
    │         │        │              │
    │1. Load  │        │1. Load       │
    │2. SW    │        │2. SW cached  │
    │   reg   │        │3. Model      │
    │3. Query │        │   ready ✓    │
    │4. FAQ   │        │4. Query      │
    │   hit   │        │5. Live       │
    │   ✓     │        │   inference  │
    │5. SW    │        │   ✓          │
    │   DL    │        │              │
    │         │        │              │
    │Time:    │        │Time:         │
    │0-2s FCP │        │0-2s FCP      │
    │0.5s Q   │        │0.5s Q        │
    │50s model│        │10ms A (local)│
    └─────────┘        └──────────────┘

---

Size & Hosting Constraints Summary

GitHub Pages Limits:

• Repository: ~1GB (soft)
• Per file: 100MB (hard)
• Individual releases: no limit
• Total storage: essentially unlimited

Implications:

• ✅ Can host 2.2GB model in GitHub Releases (no repo bloat)
• ✅ Can ship 100MB chunks as static assets if split properly
• ❌ Don't try to commit 2.2GB directly to repo

Bandwidth:

• GitHub Pages: unlimited for HTML/CSS/JS
• GitHub Releases: ~60 req/hour unauthenticated (enough for low traffic)
• Cloudflare R2: 10GB free/month, then $0.015/GB

---

Next Steps for Implementation

1. Validate model quality: Run Phi-3-mini vs. TinyLlama-1.1B vs. SmolLM-135M on Squad docs Q&A
2. Design FAQ cache: Decide on manual vs. automated generation
3. Implement SW boilerplate: Template for model download + IndexedDB caching
4. Measure latency: Benchmark on 3G vs. WiFi vs. LTE
5. Set up monitoring: Track cache hit rates and model download completion rates

---

References & Tools

• Model hosting: Hugging Face (CDN), Ollama (local)
• GGUF format: llama.cpp
• Browser ML: ONNX.js, webll.dev
• Embeddings: @xenova/transformers
• Cloudflare R2: https://developers.cloudflare.com/r2/
• IndexedDB cache pattern: Workbox

---

Conclusion

The optimal solution for Squad docs is a hybrid approach:

1. Immediate (Phase 1): Service Worker background download + TinyLlama-1.1B + FAQ cache
2. Result: Instant FAQ answers, 5-10s model download, live inference for novel questions
3. Cost: $0-1/month (or free with GitHub Releases)
4. UX: 80% instant, 20% lazy — acceptable for public docs

Avoid:

• ❌ Direct repo bundling (too large, CI overhead)
• ❌ Waiting for full model before chat is usable (poor first experience)

Next call: Validate model quality with side-by-side test on actual docs Q&A.

[diberry]

✈️ Flight Scoping Decision: Minimal MVP for #40

Verdict: Static JSON vector index + client-side cosine similarity. No server. No SQLite at runtime. No LLM.

The research comments explored full RAG pipelines, SLMs, service workers, and CDN-hosted models. That's Phase 2+. The MVP is dramatically simpler.

---

Architecture (Minimal)

BUILD TIME (npm run build)                    BROWSER (static site)
┌──────────────┐   ┌──────────────┐          ┌─────────────────────────┐
│ 108 .md files│──▶│ Chunk + Embed│──▶ JSON  │ Load vector-index.json  │
│ docs/src/    │   │ @xenova/     │   file   │ Embed query (same model)│
│ content/docs │   │ transformers │          │ Cosine similarity       │
└──────────────┘   └──────────────┘          │ Show top-5 doc chunks   │
                                             └─────────────────────────┘

• Build-time: @xenova/transformers with all-MiniLM-L6-v2 embeds doc chunks → writes dist/vector-index.json
• Runtime: Browser loads JSON, embeds user query with same model via transformers.js CDN, cosine similarity ranks chunks, displays results with source links
• No server API, no SSR adapter, no sqlite-vec at runtime, no LLM

The sqlite-vec requirement is satisfied by using it in the build script to validate vector search quality (k-NN query test during build), but the served artifact is plain JSON — keeping GitHub Pages static hosting intact.

---

NEW FILES TO CREATE (4 files)

#	File	Lines	Purpose
1	docs/scripts/build-vector-index.mjs	~150	Read markdown, chunk by H2/H3, embed via @xenova/transformers, validate with sqlite-vec k-NN, output dist/vector-index.json
2	docs/src/components/SemanticSearch.astro	~130	Client-side component: loads JSON index, embeds query in browser, cosine similarity, renders results with source links. Triggered by a toggle in the existing search modal.
3	docs/tests/vector-index.test.mjs	~80	TDD unit tests (node --test): chunking correctness, embedding dimensions, JSON schema, cosine similarity math
4	docs/tests/semantic-search.spec.mjs	~60	Playwright e2e: semantic search returns relevant results, screenshots of results

Total new code: ~420 lines across 4 files

EXISTING FILES TO MODIFY (2 files)

#	File	Change
1	docs/package.json	Add @xenova/transformers + sqlite-vec deps; update build script to run node scripts/build-vector-index.mjs after astro build; update test script
2	docs/src/components/Search.astro	Add "Semantic" toggle tab (~20 lines) that loads SemanticSearch.astro component inside the existing search modal

WHAT NOT TO BUILD IN MVP

• ❌ No LLM answer generation (no OpenAI, no local SLM, no WebLLM)
• ❌ No chat overlay / chat UI — just enhanced search results in the existing modal
• ❌ No server API endpoint (no Astro SSR, no /api/docs-search)
• ❌ No service worker or model pre-caching
• ❌ No streaming responses
• ❌ No analytics or query logging
• ❌ No WASM SQLite in browser (sql.js)
• ❌ No pre-computed FAQ cache
• ❌ No Cloudflare/Vercel deployment changes
• ❌ No blog post indexing (docs only)

KEY DECISIONS

1. @xenova/transformers for local embeddings? ✅ Yes — zero API keys, runs in Node.js at build time AND in browser at query time via CDN. Perfect fit.
2. Client-side search? ✅ Yes — static JSON with cosine similarity in browser. No server needed. Keeps GitHub Pages.
3. Chat overlay vs enhanced search? Enhanced search in existing modal. A toggle between "Keyword" (Pagefind) and "Semantic" (vector) search. Much less UI work.
4. sqlite-vec role? Build-time validation only. Proves the vector index works with proper k-NN before shipping. Not loaded in browser.

TDD APPROACH

1. Write vector-index.test.mjs first — test chunking, embedding shape, cosine similarity function
2. Build the script to pass those tests
3. Write semantic-search.spec.mjs Playwright tests — test UI toggle, result rendering, screenshot capture
4. Build the component to pass those tests

ESTIMATED BUILD IMPACT

• Build time increase: ~60-80 seconds (embedding 108 pages)
• Output size increase: ~2-5MB (vector-index.json)
• New dependencies: @xenova/transformers, sqlite-vec (dev only)
• Browser download: ~2-5MB JSON (lazy-loaded on first semantic search)

---

— Flight (Lead)
"The simplest thing that works. Ship search, not a chatbot."

[diberry]

📌 Architecture Decision: Hybrid Retrieval + SLM

Decision: Hybrid approach — retrieval first, SLM as progressive enhancement.

PR 1 (MVP — in progress):

• Build-time embedding pipeline: @xenova/transformers with all-MiniLM-L6-v2
• Docs → chunks → vectors → static JSON (served from GitHub Pages)
• Client-side SemanticSearch.astro component: cosine similarity in browser
• Results show relevant doc chunks with source links
• No server, no API keys, no runtime dependencies

PR 2 (follow-up — SLM enhancement):

• "Summarize" button loads Phi-3-mini via WebLLM on demand (~600MB)
• Feeds retrieved chunks as context, streams natural language answer
• WebGPU required — graceful fallback to retrieval-only on unsupported browsers
• Progressive: SLM is additive, not required

Why hybrid:

1. Instant value — vector search returns chunks in <100ms
2. Zero mandatory download — SLM only loads on user request
3. Works everywhere — retrieval on any device, SLM on WebGPU browsers
4. Least risk — retrieval works even if SLM fails
5. Iterative — two clean PRs instead of one monster

NOT in MVP: LLM API calls, cloud vector DB, server-side processing, user API keys.

[diberry]

📌 Architecture Decision: Static JSON, NOT SQLite at Runtime

Question: Should we reuse the existing SQLite in the repo or create a new DB?

Answer: Neither. We're using static JSON — no SQLite at runtime.

What exists in the repo:

• 
  ode:sqlite\ is a transitive dependency from @github/copilot-sdk\ — Squad doesn't use it directly
• Session storage uses JSON files (.squad/sessions/), not SQLite
• No SQLiteStorageProvider exists (the SA migration added FS/InMemory providers, not SQLite)
• A \search-index.json\ (647KB) already exists from the build pipeline

Why NOT reuse copilot-sdk's SQLite:

1. Wrong layer — copilot-sdk's SQLite is for session state, not search
2. Node.js only — docs site is static HTML served from GitHub Pages
3. Startup penalty — adding DB init to CLI for a docs feature is wrong coupling
4. Availability — node:sqlite requires Node 22.5+, docs search should work in any browser

Why static JSON for MVP:

• ~100 pages, ~500 chunks → ~647KB JSON index (already exists!)
• Client-side TF-IDF scoring is <100ms for this size
• Zero server, zero API keys, zero runtime dependencies
• Works fully offline after initial page load

When to consider SQLite:

• If docs grow beyond 2MB index size → switch to \sql.js\ (WebAssembly SQLite in browser)
• If we need fuzzy matching or faceted search → \�etter-sqlite3\ at build time + \sql.js\ at runtime
• If server-side search needed → separate API, don't couple to CLI

Bottom line: Keep search decoupled from the SDK/CLI. Static JSON scales fine for ~100 pages. SQLite is the growth path, not the MVP.