Right now the index and SQL table need to be built from scratch. While this works for experimenting offline, when this eventually becomes live we'll want to be able to fetch comments regularly without having to rebuild everything from scratch.
Incremental updates for the SQL table should be easy. Incremental updates for indices probably requires loading every token list file into memory, adding the new tokens, sorting it, and writing the whole file back out (inserting into sorted files has to take O(n) time unfortunately :/)
Right now the index and SQL table need to be built from scratch. While this works for experimenting offline, when this eventually becomes live we'll want to be able to fetch comments regularly without having to rebuild everything from scratch.
Incremental updates for the SQL table should be easy. Incremental updates for indices probably requires loading every token list file into memory, adding the new tokens, sorting it, and writing the whole file back out (inserting into sorted files has to take O(n) time unfortunately :/)