tanstack-db-sql-interpreter

Let app users write their own SQL for tanstack db queries.

Why?

Imagine you had a data stream that you want to visualise and even query on. The traditional approach of querying from a database will not perform for complex queries.

To improve performance, we need a database that supports incremental view maintenance (IVM):

• Pglite live
  • Additional complexity with OPFS, workers and headers
  • Large WASM, around 3mb gzip
• Turso
  • WASM import failed 2026-02-07.
  • Additional complexity with OPFS, workers and headers
• Tanstack db
  • 4.27mb, I assume treeshakable

Tanstack db would offer the least overhead and complexity as it is written in TypeScript.

However, app users cannot write SQL to create their own Tanstack db collections and queries. Therefore, an SQL interpreter is required for custom user created dashboards.

Design considerations

Core features

1. Parser grammar should not support destructive SQL - DELETE, ALTER
2. Parser should be reused for language server features:
   • Error correcting for auto suggestion.
   • Strict parsing for interpreting.
   • Incremental parsing if possible for longer SQL statements.
   • Parser should be written in TypeScript/JavaScript for smaller bundle sizes.
   • Parser candidates:
     • Lezer
     • Chevrotain
     • Antlr4 - no incremental parsing
     • Treesitter - compiles to WASM.
     • Ohm - doesn't have error correction
3. Once parsed, each AST/CST node should either chain methods (of the tanstack db query builder) or apply a function (eq(), and()...).

It may be easier to generate own parser from custom grammar.

• Lezer existing sql grammars:
  • Snowsql

Additional features - if core features are done

1. Language server should integrate with open source browser based code editors:
   • Provides language support for tanstack db in IDEs like VSCode.
   • CodeMirror
2. Create eslint plugin for linting.

Development

Vibe coding

Although some parts of the code are vibe coded. New packages should ideally be written by hand first to ensure proper architecture is layed out for LLMs to use as an example.

Structure

Library code is placed in ./packages:

• ./packages/sql-parser - produces typesafe SQL AST.
• ./packages/tanstack-db-sql-interpreter - interpreter for Tanstack db query builder.

Local

1. pnpm i
2. pnpm watch - Watch all tests and builds in dependency order.

To ensure caching works properly, you MUST only:

• Place all tests in test folder
• Place all source code in src folder
• Generate builds to dist folder

Make changes to nx.json if needed