Skip to content

Remove OpenAI: replace vector search with full-text search#107

Draft
Copilot wants to merge 6 commits intomasterfrom
copilot/remove-openai-embeddings
Draft

Remove OpenAI: replace vector search with full-text search#107
Copilot wants to merge 6 commits intomasterfrom
copilot/remove-openai-embeddings

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 25, 2026

  • Persist SQLite index to disksearch.New("search.db") writes to a file. On restart, if the DB already exists and has data, Built() returns true and indexing is skipped entirely. First startup still builds normally.
  • Add synonym expansion at query time — Curated map of ~40 common Islamic terms (mercy↔forgiveness↔pardon, prayer↔salah, patience↔sabr, paradise↔jannah, etc.). expandSynonyms() automatically broadens FTS5 queries so conceptually related terms match without needing embeddings or LLM.
  • Update tests — 4 new tests: TestSynonymExpansion, TestExpandSynonymsDedup, TestPersistentIndex, TestInMemoryNotBuilt. All 14 tests pass.
  • Verify build and existing tests pass
  • Code review and security scan — No issues found.
Original prompt

This section details on the original issue you should resolve

<issue_title>Remove OpenAI</issue_title>
<issue_description>Embeddings were generated using openai and we need to find a way to remove them. It might require a rethink in terms of how we're doing. Full text search. Essentially we generated embeddings so that we could do semantic search on all of the texts and then we could hand it to the llm to do rag and summarisation. But I guess there's some problems with that. One is completely tied to openai because of the way the embeddings are generated. We're also two. The searches primarily about summarisation and answers, whereas maybe a lot of people just want to see the actual results, right? So I don't know whether it means it's probably a better idea to move to full text search in a SQL database or something like that, or whether there's a way to do this without opening our eye. Or I mean, I think the answer summarisation is kind of useful book. Maybe if the core app doesn't include it that would be good. I don't know that we necessarily need a specific solution, but we need some sort of plan</issue_description>

Comments on the Issue (you are @copilot in this section)


⚡ Quickly spin up Copilot coding agent tasks from anywhere on your macOS or Windows machine with Raycast.

Copilot AI and others added 2 commits March 25, 2026 14:06
…, use direct HTTP for LLM

- Replace chromem-go vector search with in-memory full-text search
- Replace go-openai SDK with direct HTTP calls for LLM (Fanar/Ollama)
- Remove OpenAI as LLM fallback (keep Fanar and Ollama only)
- Remove pre-built vector index (search/data/reminder.idx.gob.gz)
- Remove seeker.go (chromem-go helper no longer needed)
- Remove --index/--export/--import CLI flags (no longer needed)
- Remove checkpoint indexing logic (full-text indexing is fast)
- Remove sashabaranov/go-openai and philippgille/chromem-go dependencies
- Update README to reflect new search approach

Co-authored-by: asim <17530+asim@users.noreply.github.com>
Agent-Logs-Url: https://github.com/asim/reminder/sessions/d60a35d0-678a-4159-b9d2-6837a46eeffe
Copilot AI changed the title [WIP] Remove OpenAI embeddings for improved search functionality Remove OpenAI: replace vector search with full-text search Mar 25, 2026
Copilot AI requested a review from asim March 25, 2026 14:10
Copilot finished work on behalf of asim March 25, 2026 14:10
Copilot finished work on behalf of asim March 25, 2026 17:45
- search.New() accepts optional file path; reuses existing DB on restart
- Add synonym map for ~40 common Islamic terms (mercy/forgiveness/pardon, prayer/salah, etc.)
- Query() auto-expands search terms with synonyms before FTS5 MATCH
- Add Built() method so callers can skip re-indexing when DB already exists
- main.go uses search.db file, skips indexing on restart
- Add search.db to .gitignore
- 4 new tests: synonym expansion, dedup, persistence, in-memory flag

Co-authored-by: asim <17530+asim@users.noreply.github.com>
Agent-Logs-Url: https://github.com/asim/reminder/sessions/3b7ced44-15db-4bd9-9d63-69626d012379
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove OpenAI

2 participants