diff --git a/README.md b/README.md index cacc3d4..0a12eef 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,27 @@ A **Go-first** database migration tool with a Django-style workflow. Define your schema in YAML, generate type-safe Go migration files, and run them in-process — no Go toolchain required at runtime, no compiled binary to ship. +> **How migrations run.** Generated migration files are real `.go` source — +> your IDE, `gopls`, and `go vet` treat them as ordinary Go. At runtime +> `makemigrations migrate` does **not** invoke `go build`. Instead it loads +> each `.go` file with [yaegi](https://github.com/traefik/yaegi), an +> embedded Go interpreter, and runs the migrations in the makemigrations +> process. The language inside migration files is "Go-like" — yaegi +> implements the Go spec but is not the official `gc` compiler, so a few +> features (cgo, deep reflection, some generics edge cases) work +> differently. For migrations generated by `makemigrations makemigrations` +> this is invisible; for hand-edited migrations that import third-party +> packages, see [Extending the yaegi Symbol Map](docs/extending-yaegi-symbols.md). + ## ✨ Why Go Migrations? -- 🔒 **Type-safe at edit time**: Migrations are real Go files — caught by your IDE and `go vet` -- ⚡ **No build step**: Migrations are interpreted in-process via [yaegi](https://github.com/traefik/yaegi); no `go build`, no temporary binary, no GOWORK juggling +- 🔒 **Type-safe at edit time**: Migrations are real Go files — caught by your IDE and `go vet` before they ever run +- ⚡ **No build step at runtime**: yaegi interprets the `.go` files directly; no `go build`, no temporary binary, no GOWORK juggling - 🗄️ **Database-agnostic schema**: Write YAML once, deploy to PostgreSQL, MySQL, SQLite, or SQL Server - 🔀 **DAG-based ordering**: Migrations form a dependency graph so parallel branches merge cleanly - 🔄 **Auto change detection**: Diff YAML schemas, generate only what changed - ⚠️ **Safe destructive ops**: Field removals, table drops, and renames require explicit review -- 🛠 **Optional fallback**: The generated `migrations/` directory is still a buildable Go module, so you can `go build` it for IDE checks or as an escape hatch +- 🛠 **Optional standalone binary**: The generated `migrations/` directory is still a buildable Go module, so you can `go build` it for IDE type-checking or as an escape hatch --- diff --git a/docs/Usage.md b/docs/Usage.md index 557b35c..402a125 100644 --- a/docs/Usage.md +++ b/docs/Usage.md @@ -409,7 +409,7 @@ Before applying anything, inspect the SQL that will be executed. There are two w ### Option 1 — dump_sql (schema preview, no migration state) -`dump_sql` shows the CREATE TABLE statements that your YAML schema would generate, without building the migration binary: +`dump_sql` shows the CREATE TABLE statements that your YAML schema would generate, without consulting the migration history at all: ```bash makemigrations dump_sql --database postgresql @@ -477,9 +477,9 @@ Notice that: - `default: new_uuid` resolves to `gen_random_uuid()` (from `defaults`) - `default: default_role` resolves to `'member'` -### Option 2 — showsql (migration binary SQL) +### Option 2 — showsql (pending-migration SQL) -After generating the migration file, `showsql` shows exactly what SQL the migration binary will execute when you run `up`. This includes only the pending migrations: +After generating the migration file, `showsql` shows exactly what SQL `makemigrations migrate up` will execute. This includes only the pending migrations: ```bash makemigrations migrate showsql @@ -500,7 +500,7 @@ Use `showsql` for final review before production deployments. ## Applying Migrations -`makemigrations migrate` compiles the migration binary automatically and runs it: +`makemigrations migrate` loads the migration `.go` files in-process via the [yaegi](https://github.com/traefik/yaegi) Go interpreter and runs them — no `go build`, no temporary binary: ```bash makemigrations migrate up diff --git a/docs/architecture.md b/docs/architecture.md index f324b74..28743c2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,11 +2,23 @@ ## Overview -Makemigrations is a YAML-first database migration tool for Go that generates typed, compiled Go migration files from declarative schema definitions. It follows a Django-inspired workflow while remaining idiomatic Go: migrations are real Go source files that register themselves via `init()`, are compiled into a standalone binary, and executed without any external migration runner. +Makemigrations is a YAML-first database migration tool for Go that generates typed Go migration source files from declarative schema definitions. It follows a Django-inspired workflow while remaining idiomatic Go: migrations are real Go source files that register themselves via `init()` and are loaded into the makemigrations CLI process at runtime. + +> **Runtime model.** Migrations are real Go source files (your IDE, `gopls`, +> and `go vet` treat them as ordinary Go), but at runtime they are +> **interpreted** by [yaegi](https://github.com/traefik/yaegi) — an embedded +> Go interpreter — rather than compiled by `gc`. The resulting language is +> "Go-like": yaegi implements the Go specification, but cgo, deep +> reflection, and some generics edge cases differ from the official +> compiler. For migrations generated by `makemigrations makemigrations` +> (which only use the `migrate` package and stdlib) the difference is +> invisible. Hand-edited migrations that import third-party packages can +> register additional symbols — see +> [Extending the yaegi Symbol Map](extending-yaegi-symbols.md). The tool supports two distinct workflows: -- **Go migrations (primary)** — Generates `.go` migration files. The compiled binary is the single source of truth for migration state. No separate state file is required. +- **Go migrations (primary)** — Generates `.go` migration files. The CLI's in-process registry (loaded via yaegi) is the single source of truth for migration state. No separate state file is required. - **SQL migrations (legacy, opt-in via `--sql`)** — Generates Goose-compatible `.sql` files applied via `makemigrations goose up`. --- @@ -24,9 +36,8 @@ The tool supports two distinct workflows: │ migrations/0001_initial.go │ │ migrations/0002_add_users.go ... │ │ │ │ -│ go build -o migrations/migrate │ -│ │ │ -│ ./migrations/migrate up | down | status │ +│ makemigrations migrate up | down | status │ +│ (yaegi interprets the .go files in-process) │ └──────────────────────────────────────────────────────────────────┘ │ ┌──────────────────────────────────────────────────────────────────┐ @@ -76,12 +87,12 @@ The tool supports two distinct workflows: └──────────────────────────────────────────────────────────────────┘ │ ┌──────────────────────────────────────────────────────────────────┐ -│ Compiled Migrations Binary │ +│ Embedded Migration Runtime (migrate/) │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ -│ │ Runner │ │ Recorder │ │ App │ │ Cobra │ │ -│ │ Up/Down │ │ history │ │ (CLI in │ │ Commands │ │ -│ │ ShowSQL │ │ table │ │ binary) │ │ │ │ +│ │ Runner │ │ Recorder │ │ App │ │ yaegi │ │ +│ │ Up/Down │ │ history │ │ (CLI in │ │ loader │ │ +│ │ ShowSQL │ │ table │ │ process)│ │ (interp) │ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ └──────────────────────────────────────────────────────────────────┘ ``` @@ -97,21 +108,25 @@ The tool supports two distinct workflows: │ 2. makemigrations makemigrations ├── Parses schema/schema.yaml (internal/yaml) - ├── Runs compiled binary: ./migrations/migrate dag --format json - │ └── Binary emits DAGOutput JSON (graph + SchemaState) + ├── Loads existing migrations/*.go via yaegi (internal/interp.LoadRegistry) + │ └── Builds DAG and replays SchemaState in-process — no go build ├── Diffs YAML schema against reconstructed SchemaState │ └── internal/yaml: SchemaDiff └── Writes migrations/NNNN_.go (internal/codegen: GoGenerator) │ -3. Developer compiles - └── cd migrations && go mod tidy && go build -o migrate . - │ -4. Developer applies - ├── ./migrations/migrate up (apply pending) - ├── ./migrations/migrate down (rollback) - ├── ./migrations/migrate status (show applied/pending) - ├── ./migrations/migrate showsql (preview SQL without executing) - └── ./migrations/migrate dag (visualise DAG) +3. Developer applies + ├── makemigrations migrate up (apply pending) + ├── makemigrations migrate down (rollback) + ├── makemigrations migrate status (show applied/pending) + ├── makemigrations migrate showsql (preview SQL without executing) + └── makemigrations migrate dag (visualise DAG) + + In each case the CLI loads migrations/*.go via yaegi, then invokes + migrate.NewAppWithRegistry(...).Run(args) directly — no fork/exec. + +(Optional) The migrations/ directory is still a buildable Go module, so +`cd migrations && go build -o migrate .` produces a self-contained binary +if you want one for shipping or CI artifacts. ``` ### Branch Detection and Merge Migrations @@ -155,13 +170,13 @@ The SQL workflow is opt-in and retained for compatibility. It uses YAML snapshot Key differences from the Go workflow: -| Concern | Go migrations (primary) | SQL migrations (legacy) | -|----------------------|--------------------------------|------------------------------------| -| State storage | Compiled binary (DAG replay) | `.schema_snapshot.yaml` file | -| Migration format | `.go` (typed, compiled) | `.sql` (Goose format) | -| Execution | Compiled binary: `migrate up` | `makemigrations goose up` | -| Branch detection | Graph leaves, `--merge` flag | Not supported | -| VCS merge conflicts | None (binary is rebuilt) | Possible (snapshot file) | +| Concern | Go migrations (primary) | SQL migrations (legacy) | +|----------------------|--------------------------------------|------------------------------------| +| State storage | yaegi-loaded registry (DAG replay) | `.schema_snapshot.yaml` file | +| Migration format | `.go` (typed, interpreted at runtime)| `.sql` (Goose format) | +| Execution | `makemigrations migrate up` (in-proc)| `makemigrations goose up` | +| Branch detection | Graph leaves, `--merge` flag | Not supported | +| VCS merge conflicts | None (graph reloaded from .go files) | Possible (snapshot file) | --- @@ -187,7 +202,7 @@ Each command is in its own source file. The CLI is built with Cobra. ### 2. `migrate/` Package (Runtime Library) -This is the library imported by all generated migration files and by the compiled binary. It is self-contained and has no dependency on the `cmd/` or `internal/` packages. +This is the library imported by all generated migration files. It is self-contained and has no dependency on the `cmd/` or `internal/` packages — keeping the public API surface tight makes it cheap to maintain the yaegi symbol map (`migrate/symbols/`) that exposes it to interpreted migration code. #### 2.1 Type System (`migrate/types.go`) @@ -297,13 +312,14 @@ Graph.ReconstructState() Graph.ToDAGOutput() └── Produces DAGOutput (JSON-serialisable) including SchemaState - └── Emitted by `./migrations/migrate dag --format json` + └── Returned in-process by queryDAG (cmd/go_migrations.go) + └── Also emitted as JSON by `migrate dag --format json` for CLI inspection Graph.DetectBranches() └── Returns leaf groups when multiple leaves exist (concurrent branches) ``` -The `DAGOutput` JSON is the mechanism by which `makemigrations makemigrations` reads the current compiled schema state without parsing Go source files. +`DAGOutput` is how `makemigrations makemigrations` reads the current schema state. With yaegi, the round-trip is direct: the CLI calls `interp.LoadRegistry` → `BuildGraph` → `ToDAGOutput` in the same process. (Pre-yaegi, the same struct was serialised to JSON and read back from a child binary; the on-disk format survives so external tooling that scrapes `migrate dag --format json` still works.) #### 2.5 SchemaState (`migrate/state.go`) @@ -348,9 +364,12 @@ Methods: `EnsureTable`, `GetApplied`, `RecordApplied`, `RecordRolledBack`, `Fake #### 2.8 App (`migrate/app.go`) -`App` is the embedded Cobra CLI inside the compiled migration binary. It is constructed in each generated `migrations/main.go` and wires together Registry → Graph → Runner. +`App` is the embedded Cobra CLI for migration management. It is constructed in two places: -Commands exposed by the compiled binary: +1. **Inside the makemigrations CLI** (primary path) — `cmd/migrate.go`'s `ExecuteMigrate` calls `migrate.NewAppWithRegistry(cfg, reg)` after `internal/interp.LoadRegistry` has populated `reg` from the migration `.go` files via yaegi. The App runs entirely in the makemigrations process. +2. **Inside the optional standalone binary** — the `migrations/main.go` template generated by `makemigrations init` calls `migrate.NewApp(cfg)`, which uses the package-level `globalRegistry` populated by each migration file's `init()` at compile time. + +Commands exposed by `App`: | Subcommand | Description | |------------------|------------------------------------------------------| @@ -361,6 +380,12 @@ Commands exposed by the compiled binary: | `fake NAME` | Mark a migration applied without executing SQL | | `dag [--format ascii\|json]` | Visualise or export the migration graph | +#### 2.9 yaegi Symbol Map (`migrate/symbols/`) + +The yaegi interpreter loads packages by reflective lookup against a symbol map keyed by `/`. `migrate/symbols/migrate.go` is auto-generated by `yaegi extract github.com/ocomsoft/makemigrations/migrate` and registers every public identifier in the `migrate` package against the key `github.com/ocomsoft/makemigrations/migrate/migrate`. + +`migrate/symbols/symbols.go` declares the `Symbols` map and a `Register` function that lets external callers merge additional symbol maps for packages that interpreted migrations want to import. See [Extending the yaegi Symbol Map](extending-yaegi-symbols.md). + ### 3. Code Generation (`internal/codegen/`) #### 3.1 GoGenerator (`internal/codegen/go_generator.go`) @@ -381,7 +406,7 @@ func (g *GoGenerator) GenerateMigration( previousSchema *yaml.Schema, ) ([]byte, error) -// GenerateMainGo produces migrations/main.go (the binary entry point). +// GenerateMainGo produces migrations/main.go (optional standalone-binary entry point; not invoked by `makemigrations migrate`). func (g *GoGenerator) GenerateMainGo(databaseType string) ([]byte, error) // GenerateGoMod produces migrations/go.mod. @@ -556,13 +581,16 @@ Reverse-engineers a live database to a YAML schema file. Supports all 12 provide ### 1. Init() Registration Pattern -The fundamental pattern for Go migrations. Each generated `.go` file in the `migrations/` directory contains exactly one `init()` function that calls `m.Register()`. Because Go runs all `init()` functions before `main()`, the global registry is fully populated by the time the binary executes any command. +The fundamental pattern for Go migrations. Each generated `.go` file in the `migrations/` directory contains exactly one `init()` function that calls `m.Register()`. + +- **Under yaegi (primary path):** `internal/interp.LoadRegistry` evaluates each `.go` file via the interpreter, which runs `init()` and registers each `Migration` into a fresh per-load `*migrate.Registry` (the package-level `Register` symbol is shimmed at load time to write into that registry instead of the global). No `main()` is involved. +- **Under standalone-binary compilation (optional fallback):** the Go runtime runs all `init()`s before `main()`, populating the package-level `globalRegistry`. -This pattern requires no reflection, no file system scanning at runtime, and no external configuration. The compiled binary is self-contained. +Either way, the same source file works without modification — that's the point of yaegi: the migrations are valid Go, executable both ways. ### 2. DAG as Single Source of Truth -The `Graph` built from the `Registry` is the authoritative record of migration history and schema state. There is no separate snapshot file in the Go workflow. The `makemigrations` tool queries the binary via `dag --format json` to read the current state before generating a new migration. +The `Graph` built from the `Registry` is the authoritative record of migration history and schema state. There is no separate snapshot file in the Go workflow. `makemigrations makemigrations` reads the current state by calling `interp.LoadRegistry` + `BuildGraph` + `ToDAGOutput` directly — no external process, no JSON serialisation round-trip. ### 3. State Reconstruction by Replay @@ -593,9 +621,12 @@ Each CLI command is its own source file in `cmd/`. Each command is a Cobra `*cob │ ├── internal/yaml: Parse schema/schema.yaml → Schema │ - ├── os/exec: Run ./migrations/migrate dag --format json + ├── cmd/go_migrations.go: queryDAG(migrationsDir) │ │ - │ └── migrate/graph.go: ToDAGOutput() → JSON + │ ├── internal/interp: LoadRegistry(migrationsDir) + │ │ └── yaegi evaluates each .go file → *migrate.Registry + │ │ + │ └── migrate/graph.go: BuildGraph(reg).ToDAGOutput() │ └── Includes SchemaState (fully reconstructed) │ ├── internal/yaml: Convert SchemaState → Schema (for diffing) @@ -608,29 +639,37 @@ Each CLI command is its own source file in `cmd/`. Each command is a Cobra `*cob └── Write migrations/NNNN_.go ``` -### Migration Apply Flow (compiled binary) +### Migration Apply Flow (in-process via yaegi) ``` -1. Developer runs ./migrations/migrate up +1. Developer runs `makemigrations migrate up` │ -2. migrate/app.go: buildRunner() +2. cmd/migrate.go: ExecuteMigrate(migrationsDir, args) │ - ├── migrate/graph.go: BuildGraph(GlobalRegistry()) - │ └── Validates all dependencies exist, no cycles + ├── internal/interp.LoadRegistry(migrationsDir) + │ └── yaegi evaluates each .go file → *migrate.Registry │ - ├── migrate/recorder.go: GetApplied() → map[string]bool + ├── migrate.NewAppWithRegistry(cfg, reg) │ - └── migrate/runner.go: Up("") - │ - ├── graph.Linearize() → []*Migration (topological order) - │ - ├── Replay already-applied migrations → SchemaState + └── app.Run(args) │ - └── For each pending migration: - ├── op.ForwardSQL(provider) → SQL string - ├── db.Exec(SQL) - ├── op.Mutate(state) → update in-memory state - └── recorder.RecordApplied(name) + └── migrate/app.go: buildRunner() + │ + ├── migrate/graph.go: BuildGraph(reg) + │ └── Validates all dependencies exist, no cycles + │ + ├── migrate/recorder.go: GetApplied() → map[string]bool + │ + └── migrate/runner.go: Up("") + │ + ├── graph.Linearize() → []*Migration (topological order) + ├── Replay already-applied migrations → SchemaState + │ + └── For each pending migration: + ├── op.ForwardSQL(provider) → SQL string + ├── db.Exec(SQL) + ├── op.Mutate(state) → update in-memory state + └── recorder.RecordApplied(name) ``` ### Struct2Schema Flow @@ -672,11 +711,12 @@ Each CLI command is its own source file in `cmd/`. Each command is a Cobra `*cob ## Security Considerations -1. **No direct SQL execution by the generator** — `makemigrations makemigrations` only writes `.go` files. SQL is only executed when the developer explicitly runs the compiled binary. +1. **No direct SQL execution by the generator** — `makemigrations makemigrations` only writes `.go` files. SQL is only executed when the developer explicitly runs `makemigrations migrate up` (or the optional standalone binary). 2. **Input validation** — Strict YAML schema validation before any processing. 3. **SQL injection prevention** — Identifier quoting via `provider.QuoteName()` throughout all DDL generation. 4. **No credential storage** — Database credentials are passed via environment variables or DSN at runtime by the developer; never stored in generated files or config committed to VCS. -5. **Destructive operations are explicit** — `DropTable`, `DropField`, and `AlterField` (type changes) are always visible in the generated `.go` file and reviewed before the binary is compiled. +5. **Destructive operations are explicit** — `DropTable`, `DropField`, and `AlterField` (type changes) are always visible in the generated `.go` file and reviewed before the migrations are applied. +6. **Interpreted code runs in the makemigrations process** — yaegi-loaded migrations execute with the same OS-level privileges as the CLI itself. Treat the `migrations/` directory the way you treat any other source-controlled Go code: review changes before running them. --- @@ -712,7 +752,7 @@ Each CLI command is its own source file in `cmd/`. Each command is a Cobra `*cob ### Test Levels 1. **Unit tests** — Component-level; each package has `_test.go` files. Registry, Graph, SchemaState, and all Operation types have table-driven unit tests. -2. **Integration tests** — `integration_test.go` at the project root and `yaml_integration_test.go` exercise full pipelines: parse schema → diff → generate → compile → run. +2. **Integration tests** — `integration_test.go` at the project root and `yaml_integration_test.go` exercise full pipelines: parse schema → diff → generate → load via yaegi → run. 3. **Provider tests** — Each provider is tested for correct DDL output for all 10 operation types. 4. **End-to-end tests** — Full command execution via `go test ./...` invoking CLI commands and verifying file output. @@ -729,23 +769,26 @@ Each CLI command is its own source file in `cmd/`. Each command is a Cobra `*cob | `internal/codegen/merge_generator_test.go` | Merge migration generation | | `internal/codegen/squash_generator_test.go` | Squash migration generation | | `cmd/go_migrations_test.go` | End-to-end generation command | -| `integration_test.go` | Full parse → generate → compile pipeline | +| `internal/interp/loader_test.go` | yaegi loader correctness, isolation per-load | +| `integration_test.go` | Full parse → generate → load → run pipeline | --- ## Key Architectural Decisions -### 1. Compiled Binary as Source of Truth +### 1. .go Files as Source of Truth, Loaded via yaegi -**Decision:** The migration binary is rebuilt from `.go` source files and queried via `dag --format json` to determine current schema state. +**Decision:** The migration `.go` files in `migrations/` are the source of truth. The CLI loads them in-process via the [yaegi](https://github.com/traefik/yaegi) Go interpreter (`internal/interp.LoadRegistry`) to reconstruct the current schema state. No external compile step or fork/exec is required at runtime. -**Rationale:** Eliminates the `.schema_snapshot.yaml` file that caused merge conflicts in parallel development. The binary is always rebuilt from committed source, so the state is reproducible and VCS-friendly. The `DAGOutput` JSON is a stable, machine-readable interface between the generator and the binary. +**Rationale:** Eliminates the `.schema_snapshot.yaml` file that caused merge conflicts in parallel development. The state is derived deterministically from committed Go source on every invocation, so it's reproducible and VCS-friendly. Replacing the previous `go build` + `dag --format json` round-trip with in-process interpretation removes the runtime Go-toolchain dependency, the GOWORK/GOTOOLCHAIN dance, and the ~250 lines of build plumbing that came with it. The `migrations/` directory stays a buildable Go module so IDE tooling works and `go build` is still available as a fallback. + +**Tradeoff:** yaegi implements the Go spec but is not the official compiler; cgo, deep reflection, and some generics edge cases differ. The yaegi symbol map at `migrate/symbols/` defines exactly which host packages are reachable from interpreted migrations. Generated migrations only use the `migrate` package and stdlib (both fully covered), so the limitation only affects hand-edited migrations that import third-party code — addressed by the `symbols.Register` extension hook. ### 2. Init() Registration Over File Scanning -**Decision:** Migrations self-register via `init()` into a global registry rather than being discovered by scanning the file system. +**Decision:** Migrations self-register via `init()` into a `migrate.Registry` rather than being discovered by scanning the file system. -**Rationale:** Go-native; no reflection, no file system access, no special naming conventions beyond what the generator enforces. Compilation errors are caught at build time. The registry is populated before `main()` runs, so startup is deterministic. +**Rationale:** Idiomatic Go and works identically under both yaegi (the primary path: yaegi runs `init()` exactly as the compiler does, with the package-level `Register` shimmed at load time to write into a per-load registry) and `gc` compilation (the optional fallback path, where the package-level global is populated before `main()`). No reflection, no file-system globbing, no naming conventions beyond what the generator enforces. ### 3. Typed Operations Over Raw SQL @@ -800,7 +843,7 @@ github.com/ocomsoft/makemigrations │ ├── state.go SchemaState, TableState, mutation methods │ ├── runner.go Runner: Up, Down, Status, ShowSQL │ ├── recorder.go MigrationRecorder (makemigrations_history table) -│ ├── app.go App (Cobra CLI embedded in migration binary) +│ ├── app.go App (Cobra CLI invoked in-process by `makemigrations migrate` or by an optional standalone binary) │ ├── config.go Config for App (DSN, database type) │ ├── provider_bridge.go Wires providers.Provider into Runner │ └── dag_ascii.go ASCII DAG renderer @@ -832,10 +875,18 @@ github.com/ocomsoft/makemigrations │ ├── analyzer/ Schema semantic validation │ └── writer/ File writing utilities │ +├── internal/interp/ yaegi loader: evaluates migration .go files +│ ├── loader.go LoadRegistry(dir) → *migrate.Registry +│ └── loader_test.go +│ +├── migrate/symbols/ yaegi symbol map for the migrate package +│ ├── symbols.go Symbols map declaration + Register() hook +│ └── migrate.go Auto-generated by `yaegi extract` +│ └── migrations/ Generated per-project (not committed to this repo) - ├── go.mod Standalone module importing migrate/ - ├── main.go Binary entry point: NewApp(cfg).Run(os.Args[1:]) - ├── 0001_initial.go Generated migration file + ├── go.mod Standalone module — used by IDE/gopls + ├── main.go Optional standalone-binary entry point + ├── 0001_initial.go Generated migration file (loaded by yaegi at runtime) ├── 0002_add_users.go Generated migration file └── ... ``` diff --git a/docs/commands/db-diff.md b/docs/commands/db-diff.md index 2126b85..6ea5a36 100644 --- a/docs/commands/db-diff.md +++ b/docs/commands/db-diff.md @@ -5,9 +5,10 @@ from the migration DAG. ## What It Does -1. **Reads migration files** from the migrations directory and queries the - compiled migration binary (`dag --format json`) to reconstruct the "expected" - schema state from the migration chain. +1. **Reads migration files** from the migrations directory by loading them + in-process via the yaegi Go interpreter, then reconstructs the "expected" + schema state by replaying the migration DAG (the same path used by + `migrate dag --format json`). 2. **Connects to the live database** and extracts the actual schema using the provider's `GetDatabaseSchema`. 3. **Normalizes SQL-native type names** (e.g. `character varying` to `varchar`) diff --git a/docs/commands/init.md b/docs/commands/init.md index 9003052..cff3d08 100644 --- a/docs/commands/init.md +++ b/docs/commands/init.md @@ -1,14 +1,14 @@ # init Command -The `init` command initialises a new makemigrations project. By default it sets up the **Go migration framework** — a compiled, type-safe migration binary that lives alongside your application. A legacy YAML-to-SQL workflow is available via the `--sql` flag. +The `init` command initialises a new makemigrations project. By default it sets up the **Go migration framework** — type-safe migration `.go` files that the makemigrations CLI runs in-process via the [yaegi](https://github.com/traefik/yaegi) Go interpreter. A legacy YAML-to-SQL workflow is available via the `--sql` flag. ## Overview Running `makemigrations init` bootstraps everything needed to start writing Go-based migrations: - Creates the `migrations/` directory -- Generates `migrations/main.go` — the entry point for the compiled migration binary -- Generates `migrations/go.mod` — a dedicated module that imports `github.com/ocomsoft/makemigrations/migrate` +- Generates `migrations/go.mod` — a dedicated module that imports `github.com/ocomsoft/makemigrations/migrate`. This is what gives your IDE / `gopls` type-checking on the migration files; it is **not** consulted at runtime by `makemigrations migrate`. +- Generates `migrations/main.go` — an **optional** entry point for compiling the migrations directory into a self-contained binary (`go build -o migrate .`). `makemigrations migrate` does not invoke this `main()`; the file exists purely as a fallback for users who want a standalone binary (e.g. for shipping in a release artifact, or running on a host without makemigrations installed). - If an existing `migrations/.schema_snapshot.yaml` is found, generates `migrations/0001_initial.go` with `CreateTable` operations for every table already defined in that snapshot, and prints instructions for fake-applying it If no snapshot is found the command creates an empty setup and prints instructions for generating the first migration. @@ -42,14 +42,14 @@ makemigrations init [flags] ``` project/ └── migrations/ - ├── go.mod # Dedicated module: /migrations - ├── main.go # Entry point for the compiled migrate binary + ├── go.mod # Dedicated module: /migrations (used by IDE/gopls) + ├── main.go # Optional standalone-binary entry point (NOT used by `makemigrations migrate`) └── 0001_initial.go # Only created when .schema_snapshot.yaml is found ``` -### `migrations/main.go` +### `migrations/main.go` (optional) -The generated entry point reads database connection details from environment variables and runs the compiled CLI: +`makemigrations migrate` interprets the `.go` files in this directory via yaegi and never invokes `main()`. The generated `main.go` exists so you can `go build` the directory into a self-contained binary if you want one. Its body reads database connection details from environment variables: ```go package main @@ -87,7 +87,7 @@ require ( ) ``` -The module name is derived from the parent project's `go.mod`. The Go version is read from the nearest `go.work` or `go.mod` in the parent tree so the binary is always built with a locally-available toolchain. +The module name is derived from the parent project's `go.mod`. The Go version is read from the nearest `go.work` or `go.mod` in the parent tree, ensuring `gopls` resolves the same toolchain the rest of your project uses (and that the optional standalone-binary build picks a locally-available toolchain). ### `migrations/0001_initial.go` (snapshot import) @@ -146,9 +146,10 @@ makemigrations init # To generate your first migration: # makemigrations makemigrations --name "initial" # -# Then build and run: -# cd migrations && go mod tidy && go build -o migrate . -# ./migrate up +# Then run: +# makemigrations migrate up +# +# Migrations are interpreted in-process — no Go toolchain required at runtime. ``` Step-by-step after a fresh init: @@ -157,7 +158,7 @@ Step-by-step after a fresh init: # 1. Generate your first migration makemigrations makemigrations --name "initial" -# 2. Apply migrations to the database (build is handled automatically) +# 2. Apply migrations (yaegi loads the .go files in-process; no go build) makemigrations migrate up ``` @@ -353,30 +354,25 @@ git add migrations/0001_initial.go # if generated from snapshot git commit -m "chore: initialise Go migration framework" ``` -### Keep the Migrations Module Tidy +### Keep the Migrations Module Tidy (optional) -Run `go mod tidy` inside `migrations/` after every new migration file is added so the lock file stays up to date: +`migrations/go.mod` is consulted by your IDE / `gopls`, not by `makemigrations migrate` (which uses yaegi and the symbol map shipped with the CLI). If you also use the optional standalone-binary path, run `go mod tidy` after adding new migrations to keep `go.sum` accurate: ```bash cd migrations && go mod tidy ``` -### Rebuild After Changes - -The migration binary must be recompiled whenever migration files are added or changed. `makemigrations migrate` handles this automatically, or do it manually: - -```bash -cd migrations && go build -o migrate . -``` +### No Rebuild Step Required -See the [Manual Build Guide](../manual-migration-build.md) if you need to control `GOWORK` or `GOTOOLCHAIN` explicitly. +`makemigrations migrate` reads the latest migration files on every invocation — there is no compile or rebuild step between generating a migration and applying it. If you want a self-contained binary as a fallback, see the [Manual Build Guide](../manual-migration-build.md). --- ## See Also - [migrate command](./migrate.md) — run `up`, `down`, `status`, `fake` etc. without manual builds -- [Manual Build Guide](../manual-migration-build.md) — build the binary with explicit GOWORK/GOTOOLCHAIN +- [Manual Build Guide](../manual-migration-build.md) — optional: compile `migrations/` into a standalone binary (GOWORK/GOTOOLCHAIN guidance) +- [Extending the yaegi Symbol Map](../extending-yaegi-symbols.md) — let interpreted migrations import third-party packages - [makemigrations Command](./makemigrations.md) — Generate a new migration file - [migrate-to-go command](./migrate_to_go.md) — convert existing Goose SQL migrations to Go - [Configuration Guide](../configuration.md) — Full configuration reference diff --git a/docs/commands/makemigrations.md b/docs/commands/makemigrations.md index 8dedeb6..a6643ae 100644 --- a/docs/commands/makemigrations.md +++ b/docs/commands/makemigrations.md @@ -6,7 +6,7 @@ The `makemigrations` command is the **primary command** for generating Go-based The `makemigrations` command compares the desired schema (defined in YAML files) against the current schema (reconstructed by replaying all registered Go migration files) and generates a new `.go` migration file containing typed operations for each detected change. -Unlike the SQL-mode commands, Go migrations are compiled into a standalone binary (`./migrations/migrate`) that manages migration state, runs `up`/`down` operations, and emits the DAG structure for introspection. +Unlike the SQL-mode commands, Go migrations are real `.go` source files. They run via `makemigrations migrate up` (etc.), which loads them in-process with the [yaegi](https://github.com/traefik/yaegi) Go interpreter — no `go build`, no temporary binary, no Go toolchain at runtime. The same files can also be compiled into a self-contained binary as an optional fallback. ## Usage @@ -42,14 +42,14 @@ The command scans the `migrations/` directory (as configured) for `*.go` files, When migration files exist, the command: -1. Compiles all `*.go` files in the migrations directory into a temporary binary using `go build`. -2. Executes ` dag --format json` to retrieve `DAGOutput` — a JSON structure containing: +1. Loads all `*.go` files in the migrations directory in-process via `internal/interp.LoadRegistry`. The yaegi Go interpreter runs each file's `init()`, registering its `Migration` into a fresh `*migrate.Registry`. No `go build` is invoked. +2. Calls `migrate.BuildGraph(reg).ToDAGOutput()` to produce a `DAGOutput` value containing: - The full migration graph (names, dependencies, operations) - The reconstructed `SchemaState` (all tables, fields, and indexes after replaying every migration in topological order) - The list of leaf migrations (the "tips" of the graph that a new migration must depend on) - Whether the graph has branches (concurrent development) -The temporary binary is discarded after the query. +The registry and graph live only for the duration of this query; nothing is written to disk. ### Step 3 — Parse the YAML schema @@ -406,7 +406,7 @@ makemigrations makemigrations --check makemigrations makemigrations --verbose # Output -Building migration binary from migrations/... +Loading migrations/ via yaegi... No changes detected. ``` @@ -416,7 +416,7 @@ No changes detected. ```bash # 1. Initialise the migrations directory -makemigrations init-go +makemigrations init # 2. Edit the schema vim schema/schema.yaml @@ -425,11 +425,8 @@ vim schema/schema.yaml makemigrations makemigrations --name "initial" # Created migrations/0001_initial.go -# 4. Build and run the migrations binary -cd migrations && go mod tidy && go build -o migrate . - -# 5. Apply migrations -./migrations/migrate up +# 4. Apply (yaegi loads the .go file in-process — no go build) +makemigrations migrate up ``` ### Adding a New Table @@ -444,11 +441,8 @@ makemigrations makemigrations --name "add_products" # 3. Review the generated file cat migrations/0002_add_products.go -# 4. Rebuild the binary -cd migrations && go build -o migrate . - -# 5. Apply -./migrations/migrate up +# 4. Apply +makemigrations migrate up ``` ### Altering an Existing Field @@ -460,8 +454,8 @@ cd migrations && go build -o migrate . makemigrations makemigrations --name "expand_user_status" # Created migrations/0003_expand_user_status.go -# 3. Build and apply -cd migrations && go build -o migrate . && ./migrate up +# 3. Apply +makemigrations migrate up ``` ## Branch and Merge Workflow @@ -553,11 +547,8 @@ else echo "Generating migrations..." makemigrations makemigrations --verbose - echo "Rebuilding migration binary..." - cd migrations && go build -o migrate . - echo "Applying migrations..." - ./migrate up + makemigrations migrate up echo "Done" fi @@ -569,18 +560,17 @@ After initialisation and several generated migrations, the `migrations/` directo ``` migrations/ -├── go.mod # Module file: myproject/migrations +├── go.mod # Module file: myproject/migrations (used by IDE/gopls) ├── go.sum -├── main.go # Entry point — runs the migrate app +├── main.go # Optional standalone-binary entry point (NOT used by `makemigrations migrate`) ├── 0001_initial.go # Auto-generated ├── 0002_add_products.go -├── 0003_expand_user_status.go -└── migrate # Compiled binary (gitignored) +└── 0003_expand_user_status.go ``` -### main.go +### main.go (optional) -`main.go` is generated once by `makemigrations init-go` and must not be deleted: +`main.go` is generated once by `makemigrations init`. **It is not invoked by `makemigrations migrate`** — that command interprets the migration files in-process via yaegi. The file exists so you can `go build` the directory into a self-contained binary if you want one. Safe to delete if you only ever use `makemigrations migrate`: ```go package main @@ -619,30 +609,29 @@ require ( ## After Generating a Migration -Every time a new migration file is generated you must rebuild the binary before applying: +There is **no rebuild step**. `makemigrations migrate` re-reads the latest migration files on every invocation and interprets them in-process via yaegi: ```bash -cd migrations && go mod tidy && go build -o migrate . -./migrations/migrate up +makemigrations migrate up ``` To verify the migration was applied: ```bash -./migrations/migrate status +makemigrations migrate status ``` To roll back the last migration: ```bash -./migrations/migrate down +makemigrations migrate down ``` To view the full DAG: ```bash -./migrations/migrate dag -./migrations/migrate dag --format json +makemigrations migrate dag +makemigrations migrate dag --format json ``` ## Configuration Integration @@ -667,15 +656,18 @@ Error: parsing YAML schema: no schema files found ``` Create `schema/schema.yaml` or check the search paths. -**Build failure in migrations directory** +**yaegi load failure in migrations directory** ``` -Error: querying migration DAG: building migration binary: ... +Error: querying migration DAG: loading migrations: interpreting : ... ``` -Run `cd migrations && go mod tidy && go build -o migrate .` manually to see the compiler error. Often caused by a missing `go.sum` entry after adding dependencies. +yaegi failed to interpret a migration file. Common causes: +- A typo or syntax error in a hand-edited migration. Run `cd migrations && go vet ./...` or open the file in your IDE — `gopls` will surface the same error with better context. +- An `import` of a third-party package not in the yaegi symbol map. See [Extending the yaegi Symbol Map](../extending-yaegi-symbols.md). +- A language feature yaegi does not support (cgo, certain reflection patterns). Either rewrite the migration to avoid it or use the optional standalone-binary path: `cd migrations && go mod tidy && go build -o migrate . && ./migrate up`. **Missing dependency** ``` -Error: querying migration DAG: running dag command: migration "0003_add_orders" depends on "0002_missing" which is not registered +Error: querying migration DAG: migration "0003_add_orders" depends on "0002_missing" which is not registered ``` A migration file references a dependency that does not exist. Check the `Dependencies` field in the affected migration file. diff --git a/docs/installation.md b/docs/installation.md index 5ec8ebb..c4f877c 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -4,10 +4,16 @@ This guide covers installing and setting up makemigrations for Go-based database ## Prerequisites -- **Go 1.24 or later** — Required for building and running makemigrations +- **Go 1.24 or later** — Required to install/build the makemigrations CLI itself; **not** required at runtime when applying migrations (yaegi interprets them in-process — see [How migrations run](#how-migrations-run)) - **Git** — For cloning the repository -- **CGO_ENABLED=1** — Required only when using SQLite; PostgreSQL and MySQL do not need CGO -- **Database** — One of the supported databases: PostgreSQL, MySQL, or SQLite +- **CGO_ENABLED=1** — Required only when using SQLite; PostgreSQL, MySQL, and SQL Server do not need CGO +- **Database** — One of the supported databases: PostgreSQL, MySQL, SQLite, or SQL Server + +## How migrations run + +`makemigrations migrate` does **not** invoke `go build`. Each migration `.go` file in your project is loaded by the embedded [yaegi](https://github.com/traefik/yaegi) Go interpreter and executed in the makemigrations CLI process. The migration files themselves are valid Go — your IDE, `gopls`, and `go vet` treat them as ordinary source — but the runtime is "Go-like": yaegi implements the Go specification, with documented differences for cgo, deep reflection, and some generics edge cases. For migrations generated by `makemigrations makemigrations` this is invisible. See [Extending the yaegi Symbol Map](extending-yaegi-symbols.md) if you hand-edit migrations to import third-party packages. + +The optional fallback path still exists: the generated `migrations/` directory is a buildable Go module, so you can `go build -o migrate .` it into a standalone binary if you prefer (see [Manual Build Guide](manual-migration-build.md)). ## Installation Methods @@ -54,7 +60,7 @@ makemigrations init --help ## Quickstart: Go Migration Workflow -This is the primary workflow. Migrations are generated as Go source files, compiled into a standalone binary, and applied directly against your database. +This is the primary workflow. Migrations are generated as Go source files and applied in-process via the yaegi Go interpreter — no compile step required at runtime. ### 1. Install @@ -103,26 +109,28 @@ makemigrations makemigrations --name "initial" # Creates migrations/0001_initial.go ``` -### 5. Build the Migrations Binary +### 5. Apply Migrations ```bash -cd migrations && go mod tidy && go build -o migrate . +makemigrations migrate up ``` -### 6. Apply Migrations +This loads every `.go` file in `migrations/` via the yaegi interpreter and runs them in-process. No `go build` step. + +### 6. Ongoing Workflow + +After each change to `schema.yaml`, regenerate and apply: ```bash -./migrations/migrate up +makemigrations makemigrations +makemigrations migrate up ``` -### 7. Ongoing Workflow - -After each change to `schema.yaml`, regenerate, rebuild, and apply: +(Optional) If you want a self-contained migration binary — for shipping in a release artifact, debugging a yaegi-specific issue, or running on a host where you don't want makemigrations installed — the `migrations/` directory remains a buildable Go module: ```bash -makemigrations makemigrations -cd migrations && go build -o migrate . -./migrations/migrate up +cd migrations && go mod tidy && go build -o migrate . +./migrate up ``` ### Project Layout After Init @@ -239,12 +247,14 @@ yamllint schema/schema.yaml ### SQLite: CGO Errors -SQLite requires CGO. Rebuild the migrations binary with CGO enabled: +SQLite requires CGO at the time the makemigrations CLI binary is built. If you installed makemigrations via `go install` and SQLite errors appear at runtime, rebuild with CGO enabled: ```bash -CGO_ENABLED=1 go build -o migrate . +CGO_ENABLED=1 go install github.com/ocomsoft/makemigrations@latest ``` +If you use the optional standalone-binary path (`cd migrations && go build -o migrate .`), the same applies to that build. + ## Legacy SQL Workflow For projects using the older YAML-to-SQL+Goose workflow, initialise with: diff --git a/docs/manual-migration-build.md b/docs/manual-migration-build.md index faddbad..38b4b74 100644 --- a/docs/manual-migration-build.md +++ b/docs/manual-migration-build.md @@ -1,8 +1,24 @@ -# Manual Migration Binary Build Guide +# Manual Migration Binary Build Guide (Optional) -This guide explains how to build and run the compiled migrations binary by hand — useful for CI/CD pipelines, Docker builds, debugging, or any situation where you can't use `makemigrations migrate`. +> **You almost certainly do not need this.** The default workflow is +> `makemigrations migrate `, which loads the migration `.go` files +> in-process via the [yaegi](https://github.com/traefik/yaegi) Go interpreter +> and runs them without ever invoking `go build`. There is no compile or +> rebuild step in the day-to-day developer loop. -> **For day-to-day use** prefer `makemigrations migrate ` which handles all of this automatically. +This guide covers the **optional** standalone-binary path: compiling the +`migrations/` directory into a self-contained binary using `go build`. Reasons +you might want this: + +- **Shipping a release artifact** that runs migrations on a host where you don't + want to install the makemigrations CLI. +- **Avoiding yaegi entirely** — for example to debug a yaegi-specific + interpretation issue, or because your migrations import a third-party + package that you don't want to register in the symbol map. +- **A locked-down CI environment** that already has Go but not makemigrations. + +The migration `.go` files are valid Go source either way: yaegi-interpreted +and `gc`-compiled paths produce the same on-disk schema state. --- @@ -75,7 +91,7 @@ EOF GOWORK=/tmp/migrations.work go build -o migrations/migrate ./migrations/ ``` -> `makemigrations migrate` does all of this automatically by detecting the local replace directive in parent `go.mod` files. +> Note: `makemigrations migrate` (the default, yaegi path) does **not** need any of this — it neither compiles the migrations directory nor reads `go.mod` at runtime. The setup above only matters if you specifically want a standalone binary. --- @@ -97,9 +113,9 @@ GOWORK=off go build -o migrations/migrate ./migrations/ --- -## Running the binary +## Running the standalone binary -The binary reads database connection details from environment variables wired up in `migrations/main.go`. The **generated** `main.go` only reads `DB_TYPE` and `DATABASE_URL`: +Once compiled, the standalone binary reads database connection details from environment variables wired up in `migrations/main.go`. The **generated** `main.go` only reads `DB_TYPE` and `DATABASE_URL`: ```bash export DATABASE_URL="postgresql://user:pass@localhost/mydb" @@ -132,24 +148,28 @@ app := m.NewApp(m.Config{ ## CI/CD example (GitHub Actions) +The recommended approach — install makemigrations and use yaegi: + ```yaml - name: Apply database migrations env: DATABASE_URL: ${{ secrets.DATABASE_URL }} run: | - cd migrations - GOWORK=off go mod download - GOWORK=off go build -o migrate . - ./migrate up + go install github.com/ocomsoft/makemigrations@latest + makemigrations migrate up ``` -Or, if `makemigrations` is installed as a tool: +Or, if you specifically want the standalone-binary path: ```yaml - name: Apply database migrations env: DATABASE_URL: ${{ secrets.DATABASE_URL }} - run: makemigrations migrate up + run: | + cd migrations + GOWORK=off go mod download + GOWORK=off go build -o migrate . + ./migrate up ``` --- diff --git a/docs/migrations.md b/docs/migrations.md index 886eee4..a05cf7e 100644 --- a/docs/migrations.md +++ b/docs/migrations.md @@ -8,11 +8,24 @@ Migrations are auto-generated by `makemigrations makemigrations`, but you will o - Write a migration entirely by hand (complex DDL, RunSQL, data seeding) - Understand why a particular operation was generated +> **Runtime model.** Migration files are valid Go source — your IDE, +> `gopls`, and `go vet` treat them as ordinary Go. At runtime +> `makemigrations migrate` does **not** invoke `go build`; it loads each +> file with [yaegi](https://github.com/traefik/yaegi), an embedded Go +> interpreter, and runs the migrations in the makemigrations process. +> The dialect inside migration files is "Go-like": yaegi implements the +> Go specification, but cgo, deep reflection, and some generics edge +> cases differ from the official compiler. For migrations that only use +> the `migrate` package and stdlib (everything `makemigrations` generates, +> plus most hand-edits) the difference is invisible. If your `RunSQL` +> body imports a third-party package, see +> [Extending the yaegi Symbol Map](extending-yaegi-symbols.md). + --- ## Anatomy of a Migration File -Every migration is a regular Go source file in `package main`. It registers itself with the global migration registry using `func init()`, which Go calls automatically when the binary starts. +Every migration is a regular Go source file in `package main`. It registers itself with a migration registry using `func init()` — yaegi runs the `init()` automatically when the file is loaded by `makemigrations migrate`, exactly as the Go runtime would when compiled into a standalone binary. ```go package main @@ -747,8 +760,7 @@ func init() { Before running `up`, preview exactly what SQL will execute: ```bash -cd migrations && go build -o migrate . -./migrate showsql +makemigrations migrate showsql ``` Output: