feat: interpret migrations with yaegi instead of compiling

README.md

@@ -2,15 +2,27 @@
 
 A **Go-first** database migration tool with a Django-style workflow. Define your schema in YAML, generate type-safe Go migration files, and run them in-process — no Go toolchain required at runtime, no compiled binary to ship.
 
+> **How migrations run.** Generated migration files are real `.go` source —
+> your IDE, `gopls`, and `go vet` treat them as ordinary Go. At runtime
+> `makemigrations migrate` does **not** invoke `go build`. Instead it loads
+> each `.go` file with [yaegi](https://github.com/traefik/yaegi), an
+> embedded Go interpreter, and runs the migrations in the makemigrations
+> process. The language inside migration files is "Go-like" — yaegi
+> implements the Go spec but is not the official `gc` compiler, so a few
+> features (cgo, deep reflection, some generics edge cases) work
+> differently. For migrations generated by `makemigrations makemigrations`
+> this is invisible; for hand-edited migrations that import third-party
+> packages, see [Extending the yaegi Symbol Map](docs/extending-yaegi-symbols.md).
+
 ## ✨ Why Go Migrations?
 
-- 🔒 **Type-safe at edit time**: Migrations are real Go files — caught by your IDE and `go vet`
-- ⚡ **No build step**: Migrations are interpreted in-process via [yaegi](https://github.com/traefik/yaegi); no `go build`, no temporary binary, no GOWORK juggling
+- 🔒 **Type-safe at edit time**: Migrations are real Go files — caught by your IDE and `go vet` before they ever run
+- ⚡ **No build step at runtime**: yaegi interprets the `.go` files directly; no `go build`, no temporary binary, no GOWORK juggling
 - 🗄️ **Database-agnostic schema**: Write YAML once, deploy to PostgreSQL, MySQL, SQLite, or SQL Server
 - 🔀 **DAG-based ordering**: Migrations form a dependency graph so parallel branches merge cleanly
 - 🔄 **Auto change detection**: Diff YAML schemas, generate only what changed
 - ⚠️ **Safe destructive ops**: Field removals, table drops, and renames require explicit review
-- 🛠 **Optional fallback**: The generated `migrations/` directory is still a buildable Go module, so you can `go build` it for IDE checks or as an escape hatch
+- 🛠 **Optional standalone binary**: The generated `migrations/` directory is still a buildable Go module, so you can `go build` it for IDE type-checking or as an escape hatch
 
 ---
 

docs/Usage.md

@@ -409,7 +409,7 @@ Before applying anything, inspect the SQL that will be executed. There are two w
 
 ### Option 1 — dump_sql (schema preview, no migration state)
 
-`dump_sql` shows the CREATE TABLE statements that your YAML schema would generate, without building the migration binary:
+`dump_sql` shows the CREATE TABLE statements that your YAML schema would generate, without consulting the migration history at all:
 
 ```bash
 makemigrations dump_sql --database postgresql
@@ -477,9 +477,9 @@ Notice that:
 - `default: new_uuid` resolves to `gen_random_uuid()` (from `defaults`)
 - `default: default_role` resolves to `'member'`
 
-### Option 2 — showsql (migration binary SQL)
+### Option 2 — showsql (pending-migration SQL)
 
-After generating the migration file, `showsql` shows exactly what SQL the migration binary will execute when you run `up`. This includes only the pending migrations:
+After generating the migration file, `showsql` shows exactly what SQL `makemigrations migrate up` will execute. This includes only the pending migrations:
 
 ```bash
 makemigrations migrate showsql
@@ -500,7 +500,7 @@ Use `showsql` for final review before production deployments.
 
 ## Applying Migrations
 
-`makemigrations migrate` compiles the migration binary automatically and runs it:
+`makemigrations migrate` loads the migration `.go` files in-process via the [yaegi](https://github.com/traefik/yaegi) Go interpreter and runs them — no `go build`, no temporary binary:
 
 ```bash
 makemigrations migrate up

docs/commands/db-diff.md

@@ -5,9 +5,10 @@ from the migration DAG.
 
 ## What It Does
 
-1. **Reads migration files** from the migrations directory and queries the
-   compiled migration binary (`dag --format json`) to reconstruct the "expected"
-   schema state from the migration chain.
+1. **Reads migration files** from the migrations directory by loading them
+   in-process via the yaegi Go interpreter, then reconstructs the "expected"
+   schema state by replaying the migration DAG (the same path used by
+   `migrate dag --format json`).
 2. **Connects to the live database** and extracts the actual schema using the
    provider's `GetDatabaseSchema`.
 3. **Normalizes SQL-native type names** (e.g. `character varying` to `varchar`)

docs/commands/init.md

@@ -1,14 +1,14 @@
 # init Command
 
-The `init` command initialises a new makemigrations project. By default it sets up the **Go migration framework** — a compiled, type-safe migration binary that lives alongside your application. A legacy YAML-to-SQL workflow is available via the `--sql` flag.
+The `init` command initialises a new makemigrations project. By default it sets up the **Go migration framework** — type-safe migration `.go` files that the makemigrations CLI runs in-process via the [yaegi](https://github.com/traefik/yaegi) Go interpreter. A legacy YAML-to-SQL workflow is available via the `--sql` flag.
 
 ## Overview
 
 Running `makemigrations init` bootstraps everything needed to start writing Go-based migrations:
 
 - Creates the `migrations/` directory
-- Generates `migrations/main.go` — the entry point for the compiled migration binary
-- Generates `migrations/go.mod` — a dedicated module that imports `github.com/ocomsoft/makemigrations/migrate`
+- Generates `migrations/go.mod` — a dedicated module that imports `github.com/ocomsoft/makemigrations/migrate`. This is what gives your IDE / `gopls` type-checking on the migration files; it is **not** consulted at runtime by `makemigrations migrate`.
+- Generates `migrations/main.go` — an **optional** entry point for compiling the migrations directory into a self-contained binary (`go build -o migrate .`). `makemigrations migrate` does not invoke this `main()`; the file exists purely as a fallback for users who want a standalone binary (e.g. for shipping in a release artifact, or running on a host without makemigrations installed).
 - If an existing `migrations/.schema_snapshot.yaml` is found, generates `migrations/0001_initial.go` with `CreateTable` operations for every table already defined in that snapshot, and prints instructions for fake-applying it
 
 If no snapshot is found the command creates an empty setup and prints instructions for generating the first migration.
@@ -42,14 +42,14 @@ makemigrations init [flags]
 ```
 project/
 └── migrations/
-    ├── go.mod          # Dedicated module: <project>/migrations
-    ├── main.go         # Entry point for the compiled migrate binary
+    ├── go.mod          # Dedicated module: <project>/migrations  (used by IDE/gopls)
+    ├── main.go         # Optional standalone-binary entry point  (NOT used by `makemigrations migrate`)
     └── 0001_initial.go # Only created when .schema_snapshot.yaml is found
 ```
 
-### `migrations/main.go`
+### `migrations/main.go` (optional)
 
-The generated entry point reads database connection details from environment variables and runs the compiled CLI:
+`makemigrations migrate` interprets the `.go` files in this directory via yaegi and never invokes `main()`. The generated `main.go` exists so you can `go build` the directory into a self-contained binary if you want one. Its body reads database connection details from environment variables:
 
 ```go
 package main
@@ -87,7 +87,7 @@ require (
 )
 ```
 
-The module name is derived from the parent project's `go.mod`. The Go version is read from the nearest `go.work` or `go.mod` in the parent tree so the binary is always built with a locally-available toolchain.
+The module name is derived from the parent project's `go.mod`. The Go version is read from the nearest `go.work` or `go.mod` in the parent tree, ensuring `gopls` resolves the same toolchain the rest of your project uses (and that the optional standalone-binary build picks a locally-available toolchain).
 
 ### `migrations/0001_initial.go` (snapshot import)
 
@@ -146,9 +146,10 @@ makemigrations init
 # To generate your first migration:
 #   makemigrations makemigrations --name "initial"
 #
-# Then build and run:
-#   cd migrations && go mod tidy && go build -o migrate .
-#   ./migrate up
+# Then run:
+#   makemigrations migrate up
+#
+# Migrations are interpreted in-process — no Go toolchain required at runtime.
 ```
 
 Step-by-step after a fresh init:
@@ -157,7 +158,7 @@ Step-by-step after a fresh init:
 # 1. Generate your first migration
 makemigrations makemigrations --name "initial"
 
-# 2. Apply migrations to the database (build is handled automatically)
+# 2. Apply migrations (yaegi loads the .go files in-process; no go build)
 makemigrations migrate up
 ```
 
@@ -353,30 +354,25 @@ git add migrations/0001_initial.go   # if generated from snapshot
 git commit -m "chore: initialise Go migration framework"
 ```
 
-### Keep the Migrations Module Tidy
+### Keep the Migrations Module Tidy (optional)
 
-Run `go mod tidy` inside `migrations/` after every new migration file is added so the lock file stays up to date:
+`migrations/go.mod` is consulted by your IDE / `gopls`, not by `makemigrations migrate` (which uses yaegi and the symbol map shipped with the CLI). If you also use the optional standalone-binary path, run `go mod tidy` after adding new migrations to keep `go.sum` accurate:
 
 ```bash
 cd migrations && go mod tidy
 ```
 
-### Rebuild After Changes
-
-The migration binary must be recompiled whenever migration files are added or changed. `makemigrations migrate` handles this automatically, or do it manually:
-
-```bash
-cd migrations && go build -o migrate .
-```
+### No Rebuild Step Required
 
-See the [Manual Build Guide](../manual-migration-build.md) if you need to control `GOWORK` or `GOTOOLCHAIN` explicitly.
+`makemigrations migrate` reads the latest migration files on every invocation — there is no compile or rebuild step between generating a migration and applying it. If you want a self-contained binary as a fallback, see the [Manual Build Guide](../manual-migration-build.md).
 
 ---
 
 ## See Also
 
 - [migrate command](./migrate.md) — run `up`, `down`, `status`, `fake` etc. without manual builds
-- [Manual Build Guide](../manual-migration-build.md) — build the binary with explicit GOWORK/GOTOOLCHAIN
+- [Manual Build Guide](../manual-migration-build.md) — optional: compile `migrations/` into a standalone binary (GOWORK/GOTOOLCHAIN guidance)
+- [Extending the yaegi Symbol Map](../extending-yaegi-symbols.md) — let interpreted migrations import third-party packages
 - [makemigrations Command](./makemigrations.md) — Generate a new migration file
 - [migrate-to-go command](./migrate_to_go.md) — convert existing Goose SQL migrations to Go
 - [Configuration Guide](../configuration.md) — Full configuration reference

docs/commands/makemigrations.md

@@ -6,7 +6,7 @@ The `makemigrations` command is the **primary command** for generating Go-based
 
 The `makemigrations` command compares the desired schema (defined in YAML files) against the current schema (reconstructed by replaying all registered Go migration files) and generates a new `.go` migration file containing typed operations for each detected change.
 
-Unlike the SQL-mode commands, Go migrations are compiled into a standalone binary (`./migrations/migrate`) that manages migration state, runs `up`/`down` operations, and emits the DAG structure for introspection.
+Unlike the SQL-mode commands, Go migrations are real `.go` source files. They run via `makemigrations migrate up` (etc.), which loads them in-process with the [yaegi](https://github.com/traefik/yaegi) Go interpreter — no `go build`, no temporary binary, no Go toolchain at runtime. The same files can also be compiled into a self-contained binary as an optional fallback.
 
 ## Usage
 
@@ -42,14 +42,14 @@ The command scans the `migrations/` directory (as configured) for `*.go` files,
 
 When migration files exist, the command:
 
-1. Compiles all `*.go` files in the migrations directory into a temporary binary using `go build`.
-2. Executes `<binary> dag --format json` to retrieve `DAGOutput` — a JSON structure containing:
+1. Loads all `*.go` files in the migrations directory in-process via `internal/interp.LoadRegistry`. The yaegi Go interpreter runs each file's `init()`, registering its `Migration` into a fresh `*migrate.Registry`. No `go build` is invoked.
+2. Calls `migrate.BuildGraph(reg).ToDAGOutput()` to produce a `DAGOutput` value containing:
    - The full migration graph (names, dependencies, operations)
    - The reconstructed `SchemaState` (all tables, fields, and indexes after replaying every migration in topological order)
    - The list of leaf migrations (the "tips" of the graph that a new migration must depend on)
    - Whether the graph has branches (concurrent development)
 
-The temporary binary is discarded after the query.
+The registry and graph live only for the duration of this query; nothing is written to disk.
 
 ### Step 3 — Parse the YAML schema
 
@@ -406,7 +406,7 @@ makemigrations makemigrations --check
 makemigrations makemigrations --verbose
 
 # Output
-Building migration binary from migrations/...
+Loading migrations/ via yaegi...
 No changes detected.
 ```
 
@@ -416,7 +416,7 @@ No changes detected.
 
 ```bash
 # 1. Initialise the migrations directory
-makemigrations init-go
+makemigrations init
 
 # 2. Edit the schema
 vim schema/schema.yaml
@@ -425,11 +425,8 @@ vim schema/schema.yaml
 makemigrations makemigrations --name "initial"
 # Created migrations/0001_initial.go
 
-# 4. Build and run the migrations binary
-cd migrations && go mod tidy && go build -o migrate .
-
-# 5. Apply migrations
-./migrations/migrate up
+# 4. Apply (yaegi loads the .go file in-process — no go build)
+makemigrations migrate up
 ```
 
 ### Adding a New Table
@@ -444,11 +441,8 @@ makemigrations makemigrations --name "add_products"
 # 3. Review the generated file
 cat migrations/0002_add_products.go
 
-# 4. Rebuild the binary
-cd migrations && go build -o migrate .
-
-# 5. Apply
-./migrations/migrate up
+# 4. Apply
+makemigrations migrate up
 ```
 
 ### Altering an Existing Field
@@ -460,8 +454,8 @@ cd migrations && go build -o migrate .
 makemigrations makemigrations --name "expand_user_status"
 # Created migrations/0003_expand_user_status.go
 
-# 3. Build and apply
-cd migrations && go build -o migrate . && ./migrate up
+# 3. Apply
+makemigrations migrate up
 ```
 
 ## Branch and Merge Workflow
@@ -553,11 +547,8 @@ else
     echo "Generating migrations..."
     makemigrations makemigrations --verbose
 
-    echo "Rebuilding migration binary..."
-    cd migrations && go build -o migrate .
-
     echo "Applying migrations..."
-    ./migrate up
+    makemigrations migrate up
 
     echo "Done"
 fi
@@ -569,18 +560,17 @@ After initialisation and several generated migrations, the `migrations/` directo
 
 ```
 migrations/
-├── go.mod              # Module file: myproject/migrations
+├── go.mod              # Module file: myproject/migrations  (used by IDE/gopls)
 ├── go.sum
-├── main.go             # Entry point — runs the migrate app
+├── main.go             # Optional standalone-binary entry point  (NOT used by `makemigrations migrate`)
 ├── 0001_initial.go     # Auto-generated
 ├── 0002_add_products.go
-├── 0003_expand_user_status.go
-└── migrate             # Compiled binary (gitignored)
+└── 0003_expand_user_status.go
 ```
 
-### main.go
+### main.go (optional)
 
-`main.go` is generated once by `makemigrations init-go` and must not be deleted:
+`main.go` is generated once by `makemigrations init`. **It is not invoked by `makemigrations migrate`** — that command interprets the migration files in-process via yaegi. The file exists so you can `go build` the directory into a self-contained binary if you want one. Safe to delete if you only ever use `makemigrations migrate`:
 
 ```go
 package main
@@ -619,30 +609,29 @@ require (
 
 ## After Generating a Migration
 
-Every time a new migration file is generated you must rebuild the binary before applying:
+There is **no rebuild step**. `makemigrations migrate` re-reads the latest migration files on every invocation and interprets them in-process via yaegi:
 
 ```bash
-cd migrations && go mod tidy && go build -o migrate .
-./migrations/migrate up
+makemigrations migrate up
 ```
 
 To verify the migration was applied:
 
 ```bash
-./migrations/migrate status
+makemigrations migrate status
 ```
 
 To roll back the last migration:
 
 ```bash
-./migrations/migrate down
+makemigrations migrate down
 ```
 
 To view the full DAG:
 
 ```bash
-./migrations/migrate dag
-./migrations/migrate dag --format json
+makemigrations migrate dag
+makemigrations migrate dag --format json
 ```
 
 ## Configuration Integration
@@ -667,15 +656,18 @@ Error: parsing YAML schema: no schema files found
 ```
 Create `schema/schema.yaml` or check the search paths.
 
-**Build failure in migrations directory**
+**yaegi load failure in migrations directory**
 ```
-Error: querying migration DAG: building migration binary: ...
+Error: querying migration DAG: loading migrations: interpreting <file>: ...
 ```
-Run `cd migrations && go mod tidy && go build -o migrate .` manually to see the compiler error. Often caused by a missing `go.sum` entry after adding dependencies.
+yaegi failed to interpret a migration file. Common causes:
+- A typo or syntax error in a hand-edited migration. Run `cd migrations && go vet ./...` or open the file in your IDE — `gopls` will surface the same error with better context.
+- An `import` of a third-party package not in the yaegi symbol map. See [Extending the yaegi Symbol Map](../extending-yaegi-symbols.md).
+- A language feature yaegi does not support (cgo, certain reflection patterns). Either rewrite the migration to avoid it or use the optional standalone-binary path: `cd migrations && go mod tidy && go build -o migrate . && ./migrate up`.
 
 **Missing dependency**
 ```
-Error: querying migration DAG: running dag command: migration "0003_add_orders" depends on "0002_missing" which is not registered
+Error: querying migration DAG: migration "0003_add_orders" depends on "0002_missing" which is not registered
 ```
 A migration file references a dependency that does not exist. Check the `Dependencies` field in the affected migration file.