Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -642,6 +642,7 @@ To add/modify JSON schemas:
| `dj.dataExplorer.autoRefresh` | boolean | `false` | Auto-refresh Data Explorer on file switch |
| `dj.lightdashProjectPath` | string | — | Custom path to dbt project for Lightdash |
| `dj.lightdashProfilesPath` | string | — | Custom path to dbt profiles for Lightdash |
| `dj.lightdash.defaultPartitionColumnCaseSensitive` | boolean | `false` | Auto-emit `meta.dimension.case_sensitive: true` on partition columns in generated YAML (stops Lightdash from wrapping partition columns in `UPPER()`, preserving Trino predicate pushdown). Requires `DJ: Sync to SQL and YML` to apply. |
| `dj.materialization.defaultIncrementalStrategy` | string | `overwrite_existing_partitions` | Default incremental strategy: `append`, `delete+insert`, `merge`, or `overwrite_existing_partitions` |
| `dj.autoGenerateTests` | object | — | (Experimental) Auto-generate tests on models |

Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Change Log

## 1.3.5

- **`unique_key` no longer emitted for `overwrite_existing_partitions`** — this strategy requires a custom dbt macro in your project (typically `get_incremental_overwrite_existing_partitions_sql`); the DJ extension does not ship it and dbt-trino does not provide it natively. If your project does not define the macro, switch to `{ "type": "delete+insert" }` — it auto-derives `unique_key` from partition columns.
- **`dj.lightdash.defaultPartitionColumnCaseSensitive`** (default: `false`) — when `true`, partition columns in generated YAML get `meta.dimension.case_sensitive: true`. This stops Lightdash from wrapping them in `UPPER()` in queries, preserving Trino predicate pushdown on partitioned tables. Per-model and per-column `lightdash.case_sensitive` overrides in `.model.json` continue to apply.
- **Aggregation Validator Enhancements** — Validation issues are now flagged as Warnings rather than errors, allowing you to generate SQL and iterate even if columns are un-aggregated. The validator now ignores constant values (e.g., 0, null, 'foo') and Jinja/dbt macros that it cannot introspect. Specific messages added to guide on partition-column alignment for window functions, replacing generic aggregation errors.

## 1.3.4

- Partition columns automatically emit `case_sensitive: true` in YAML meta so Lightdash does not wrap them in `UPPER()`, preserving predicate pushdown
Expand Down
48 changes: 30 additions & 18 deletions docs/SETTINGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,25 @@ Complete guide to configuring the DJ VS Code extension.

## Quick Reference

| Setting | Purpose | Takes Effect |
| --------------------------- | ---------------------------------- | --------------- |
| `pythonVenvPath` | Python virtual environment path | Next command ⚡ |
| `trinoPath` | Trino CLI executable location | Next query ⚡ |
| `dbtProjectNames` | Filter which dbt projects to load | Refresh 🔄 |
| `dbtMacroPath` | Custom path for extension macros | Refresh 🔄 |
| `dbtGenericTestsPath` | Custom path for generic test files | Refresh 🔄 |
| `airflowGenerateDags` | Enable Airflow DAG generation | Refresh 🔄 |
| `airflowTargetVersion` | Target Airflow version | Refresh 🔄 |
| `airflowDagsPath` | Custom path for Airflow DAGs | Refresh 🔄 |
| `lightdashProjectPath` | Custom Lightdash project path | Next preview ⚡ |
| `lightdashProfilesPath` | Custom Lightdash profiles path | Next preview ⚡ |
| `aiHintTag` | Tag for AI-generated hints | Next sync 🔄 |
| `codingAgent` | Coding agent integration | Refresh 🔄 |
| `autoGenerateTests` | Auto-generate row count tests | Varies 🔄 |
| `columnLineage.autoRefresh` | Auto-refresh column lineage | File switch ✅ |
| `dataExplorer.autoRefresh` | Auto-refresh data explorer | File switch ✅ |
| `logLevel` | Extension logging level | Immediate ✅ |
| Setting | Purpose | Takes Effect |
| ----------------------------------------------- | ---------------------------------------------------------------- | --------------- |
| `pythonVenvPath` | Python virtual environment path | Next command ⚡ |
| `trinoPath` | Trino CLI executable location | Next query ⚡ |
| `dbtProjectNames` | Filter which dbt projects to load | Refresh 🔄 |
| `dbtMacroPath` | Custom path for extension macros | Refresh 🔄 |
| `dbtGenericTestsPath` | Custom path for generic test files | Refresh 🔄 |
| `airflowGenerateDags` | Enable Airflow DAG generation | Refresh 🔄 |
| `airflowTargetVersion` | Target Airflow version | Refresh 🔄 |
| `airflowDagsPath` | Custom path for Airflow DAGs | Refresh 🔄 |
| `lightdashProjectPath` | Custom Lightdash project path | Next preview ⚡ |
| `lightdashProfilesPath` | Custom Lightdash profiles path | Next preview ⚡ |
| `lightdash.defaultPartitionColumnCaseSensitive` | Set default `case_sensitive` value for partition columns in YAML | Next sync 🔄 |
| `aiHintTag` | Tag for AI-generated hints | Next sync 🔄 |
| `codingAgent` | Coding agent integration | Refresh 🔄 |
| `autoGenerateTests` | Auto-generate row count tests | Varies 🔄 |
| `columnLineage.autoRefresh` | Auto-refresh column lineage | File switch ✅ |
| `dataExplorer.autoRefresh` | Auto-refresh data explorer | File switch ✅ |
| `logLevel` | Extension logging level | Immediate ✅ |

**Legend:** ✅ Immediate | ⚡ Next command/action | 🔄 Requires `DJ: Refresh Projects` or sync

Expand Down Expand Up @@ -117,6 +118,16 @@ Complete guide to configuring the DJ VS Code extension.
- Both optional (extension auto-detects by default)
- See [Lightdash Configuration Guide](setup/lightdash-configuration.md)

**`dj.lightdash.defaultPartitionColumnCaseSensitive`** - Auto-emit `case_sensitive: true` on partition columns in generated YAML (default: `false`)

```json
{ "dj.lightdash.defaultPartitionColumnCaseSensitive": true }
```

- When `true`, every generated partition column in `.yml` files gets `meta.dimension.case_sensitive: true`. This stops Lightdash from wrapping the column in `UPPER()` in queries, preserving Trino predicate pushdown on partitioned tables.
- When `false` (the default), partition columns are emitted without the auto-injected `case_sensitive` flag. Per-model and per-column `lightdash.case_sensitive` overrides in `.model.json` continue to work in either mode.
- Takes effect on next `DJ: Sync to SQL and YML`.

---

### AI & Coding Agents
Expand Down Expand Up @@ -230,6 +241,7 @@ Run this command (`Cmd/Ctrl+Shift+P` → `DJ: Refresh Projects`) after changing:
### 🔄 Requires `DJ: Sync to SQL and YML`

- `aiHintTag` - Recompiles models with updated tags
- `lightdash.defaultPartitionColumnCaseSensitive` - Re-emits partition column YAML meta

---

Expand Down
2 changes: 1 addition & 1 deletion docs/models/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -433,7 +433,7 @@ Set `materialization.strategy.type` to one of the following (or rely on the exte
| `append` | Inserts new rows with no de-duplication. Fastest. | Upstream must guarantee no duplicates in the new slice. |
| `delete+insert` | Partition-safe upsert. Safe default. | `unique_key` auto-derived from partition columns when omitted. Works on Delta Lake, Hive, and Iceberg. |
| `merge` | Row-level upsert on `unique_key`. | **dbt-trino requires Iceberg format** on the target table. On Delta Lake / Hive use `delete+insert` instead. |
| `overwrite_existing_partitions` | Drops and rewrites only the partitions present in the new slice. | **Requires a custom dbt macro in your project** (e.g. `get_incremental_overwrite_existing_partitions_sql`). The DJ extension does NOT ship this macro and dbt-trino does NOT provide it natively. If your project does not define it, use `delete+insert` with a partition column as `unique_key` \u2014 it produces equivalent behavior for daily/monthly partitioned models. |
| `overwrite_existing_partitions` | Drops and rewrites only the partitions present in the new slice. `unique_key` is not applicable, the consumer macro derives the partition list from the new slice itself, so the schema rejects `unique_key` on this strategy. | **Requires a custom dbt macro in your project** (e.g. `get_incremental_overwrite_existing_partitions_sql`). The DJ extension does NOT ship this macro and dbt-trino does NOT provide it natively. If your project does not define it, use `delete+insert` with a partition column as `unique_key` \u2014 it produces equivalent behavior for daily/monthly partitioned models. |

### Data Quality

Expand Down
13 changes: 13 additions & 0 deletions docs/setup/lightdash-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,19 @@ You can override the auto-detection by setting these VS Code configuration optio
- **Description**: Custom path to the dbt profiles directory (relative to workspace root)
- **Default**: Uses the project directory or workspace root if not specified

#### `dj.lightdash.defaultPartitionColumnCaseSensitive`

- **Type**: `boolean`
- **Default**: `false`
- **Description**: When `true`, the extension automatically emits `meta.dimension.case_sensitive: true` on every partition column in generated YAML. This stops Lightdash from wrapping the column in `UPPER()` in queries, preserving Trino predicate pushdown on partitioned tables.
- **When to enable**: Lightdash projects that query partitioned tables and notice slow scans because partition predicates are wrapped in `UPPER()`.
- **Per-model override**: `lightdash.case_sensitive` on a specific model or column in `.model.json` continues to take precedence in either mode.
- **Takes effect**: Run `DJ: Sync to SQL and YML` (`Cmd+Shift+P`) to regenerate YAML with the new value.

```json
{ "dj.lightdash.defaultPartitionColumnCaseSensitive": true }
```

## Examples

### Example 1: dbt project in subdirectory
Expand Down
33 changes: 16 additions & 17 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 7 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"type": "git",
"url": "https://github.com/Workday/vscode-dbt-json.git"
},
"version": "1.3.4",
"version": "1.3.5",
"workspaces": [
"web"
],
Expand Down Expand Up @@ -321,6 +321,11 @@
"type": "string",
"description": "Custom path to the dbt profiles directory for Lightdash preview (relative to workspace root). If not set, will use the project directory or workspace root. Takes effect on next command execution."
},
"dj.lightdash.defaultPartitionColumnCaseSensitive": {
"type": "boolean",
"default": false,
"description": "Automatically set case_sensitive: true on partition columns in generated YAML. Requires 'DJ: Sync to SQL and YML' to take effect."
},
"dj.columnLineage.autoRefresh": {
"type": "boolean",
"default": true,
Expand Down Expand Up @@ -784,4 +789,4 @@
"typescript": "^5.4.2",
"typescript-eslint": "^8.50.1"
}
}
}
15 changes: 2 additions & 13 deletions schemas/model.incremental_strategy.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -76,26 +76,15 @@
}
},
{
"description": "Overwrite existing partitions: drop and rewrite only the partitions present in the new slice. REQUIRES a custom dbt macro in your project (e.g. get_incremental_overwrite_existing_partitions_sql). The DJ extension does NOT ship this macro and dbt-trino does NOT provide it natively. If your project does not define this macro, use 'delete+insert' with a partition column as unique_key — it produces equivalent behavior for daily/monthly partitioned models.",
"markdownDescription": "**`overwrite_existing_partitions`** — drop & rewrite only partitions in the new slice.\n\n> **WARNING — CUSTOM MACRO REQUIRED.** This strategy is NOT shipped by the DJ extension and is NOT part of vanilla dbt-trino. Your dbt project must define the dispatch macro (typically `get_incremental_overwrite_existing_partitions_sql`) for it to compile.\n>\n> **Don't have the macro?** Use `{ \"type\": \"delete+insert\" }` instead — if a partition column exists, DJ auto-fills `unique_key` and the behavior is equivalent for partition-aligned daily/monthly incrementals.",
"description": "Overwrite existing partitions: drop and rewrite only the partitions present in the new slice. REQUIRES a custom dbt macro in your project (e.g. get_incremental_overwrite_existing_partitions_sql). The DJ extension does NOT ship this macro and dbt-trino does NOT provide it natively. The macro derives the partition list from the new slice itself, so unique_key is not applicable for this strategy and is rejected by the schema. If your project does not define this macro, use 'delete+insert' with a partition column as unique_key — it produces equivalent behavior for daily/monthly partitioned models.",
"markdownDescription": "**`overwrite_existing_partitions`** — drop & rewrite only partitions in the new slice.\n\n> **WARNING — CUSTOM MACRO REQUIRED.** This strategy is NOT shipped by the DJ extension and is NOT part of vanilla dbt-trino. Your dbt project must define the dispatch macro (typically `get_incremental_overwrite_existing_partitions_sql`) for it to compile.\n>\n> **`unique_key` is not applicable** — the macro derives the partition list from the new slice itself, so this strategy ignores `unique_key` and the schema rejects it.\n>\n> **Don't have the macro?** Use `{ \"type\": \"delete+insert\" }` instead — if a partition column exists, DJ auto-fills `unique_key` and the behavior is equivalent for partition-aligned daily/monthly incrementals.",
"type": "object",
"required": ["type"],
"additionalProperties": false,
"properties": {
"type": {
"const": "overwrite_existing_partitions",
"description": "Overwrite only the partitions in the new slice. REQUIRES a custom macro in your dbt project — prefer 'delete+insert' if you do not have one."
},
"unique_key": {
"description": "Optional override for the unique key(s). Defaults to the model's partition column when omitted.",
"anyOf": [
{ "type": "string" },
{
"type": "array",
"minItems": 1,
"items": { "type": "string" }
}
]
}
}
}
Expand Down
22 changes: 22 additions & 0 deletions src/services/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ export function getDjConfig(): CoderConfig {
trinoPath: config.get('trinoPath', undefined),
lightdashProjectPath: config.get('lightdashProjectPath', undefined),
lightdashProfilesPath: config.get('lightdashProfilesPath', undefined),
lightdashDefaultPartitionColumnCaseSensitive: config.get(
'lightdash.defaultPartitionColumnCaseSensitive',
false,
),

// Settings with defaults from package.json.
// All fallbacks (extension, webview, generator) route through the shared
Expand Down Expand Up @@ -264,6 +268,12 @@ export function getSettingReloadRequirement(
actionCommand: 'dj.command.jsonSync',
description: "Requires 'DJ: Sync to SQL and YML' to take effect",
},
'lightdash.defaultPartitionColumnCaseSensitive': {
requiresAction: true,
action: 'compile',
actionCommand: 'dj.command.jsonSync',
description: "Requires 'DJ: Sync to SQL and YML' to take effect",
},

// Complex settings
autoGenerateTests: {
Expand Down Expand Up @@ -761,6 +771,18 @@ export function registerConfigurationChangeHandler(
);
}

// lightdash.defaultPartitionColumnCaseSensitive - notify about compile requirement
if (
event.affectsConfiguration(
'dj.lightdash.defaultPartitionColumnCaseSensitive',
)
) {
void vscode.window.setStatusBarMessage(
"DJ: Partition column case_sensitive updated - run 'DJ: Sync to SQL and YML' to apply",
5000,
);
}

// Lightdash paths - simple notifications
if (event.affectsConfiguration('dj.lightdashProjectPath')) {
void vscode.window.setStatusBarMessage(
Expand Down
Loading
Loading