Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion docs/concepts/models/model_kinds.md
Original file line number Diff line number Diff line change
Expand Up @@ -1552,4 +1552,31 @@ Depending on the target engine, models of the `INCREMENTAL_BY_PARTITION` kind ar
| BigQuery | DELETE by partitioning key, then INSERT |
| Redshift | DELETE by partitioning key, then INSERT |
| Postgres | DELETE by partitioning key, then INSERT |
| DuckDB | DELETE by partitioning key, then INSERT |
| DuckDB | DELETE by partitioning key, then INSERT |

## INCREMENTAL_UNMANAGED

The `INCREMENTAL_UNMANAGED` model kind exists to support append-only tables. It's "unmanaged" in the sense that SQLMesh doesnt try to manage how the data is loaded. SQLMesh will just run your query on the configured cadence and append whatever it gets into the table.

!!! question "Should you use this model kind?"

Some patterns for data management, such as Data Vault, may rely on append-only tables. In this situation, `INCREMENTAL_UNMANAGED` is the correct type to use.

In most other situations, you probably want `INCREMENTAL_BY_TIME_RANGE` or `INCREMENTAL_BY_UNIQUE_KEY` because they give you much more control over how the data is loaded.

Usage of the `INCREMENTAL_UNMANAGED` model kind is straightforward:

```sql linenums="1" hl_lines="3"
MODEL (
name db.events,
kind INCREMENTAL_UNMANAGED,
);
```

Since it's unmanaged, it doesnt support the `batch_size` and `batch_concurrency` properties to control how data is loaded like the other incremental model types do.

!!! warning "Only full restatements supported"

Similar to `INCREMENTAL_BY_PARTITION`, attempting to [restate](../plans.md#restatement-plans) an `INCREMENTAL_UNMANAGED` model will trigger a full restatement. That is, the model will be rebuilt from scratch rather than from a time slice you specify.

This is because an append-only table is inherently non-idempotent. Restating `INCREMENTAL_UNMANAGED` models may lead to data loss and should be performed with care.
1 change: 1 addition & 0 deletions sqlmesh/core/dialect.py
Original file line number Diff line number Diff line change
Expand Up @@ -594,6 +594,7 @@ def parse(self: Parser) -> t.Optional[exp.Expression]:
ModelKindName.INCREMENTAL_BY_TIME_RANGE,
ModelKindName.INCREMENTAL_BY_UNIQUE_KEY,
ModelKindName.INCREMENTAL_BY_PARTITION,
ModelKindName.INCREMENTAL_UNMANAGED,
ModelKindName.SEED,
ModelKindName.VIEW,
ModelKindName.SCD_TYPE_2,
Expand Down
36 changes: 36 additions & 0 deletions tests/core/test_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -3003,6 +3003,42 @@ def test_incremental_unmanaged_validation():
model.validate_definition()


def test_incremental_unmanaged():
expr = d.parse(
"""
MODEL (
name foo,
kind INCREMENTAL_UNMANAGED
);

SELECT x.a AS a FROM test.x AS x
"""
)

model = load_sql_based_model(expressions=expr)

assert isinstance(model.kind, IncrementalUnmanagedKind)
assert not model.kind.insert_overwrite

expr = d.parse(
"""
MODEL (
name foo,
kind INCREMENTAL_UNMANAGED (
insert_overwrite true
),
partitioned_by a
);

SELECT x.a AS a FROM test.x AS x
"""
)

model = load_sql_based_model(expressions=expr)
assert isinstance(model.kind, IncrementalUnmanagedKind)
assert model.kind.insert_overwrite


def test_custom_interval_unit():
assert (
load_sql_based_model(
Expand Down