Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions docs/guides/linter.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Linter guide

![Linter](linter_example.png)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get people excited to see this image in their own terminal


Linting is a powerful tool for improving code quality and consistency. It enables you to automatically validate model definition, ensuring they adhere to your team's best practices.

When a SQLMesh command is executed and the project is loaded, each model's code is checked for compliance with a set of rules you choose.
Expand Down Expand Up @@ -68,10 +70,10 @@ Here are all of SQLMesh's built-in linting rules:

| Name | Check type | Explanation |
| -------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ |
| ambiguousorinvalidcolumn | Correctness | SQLMesh found duplicate columns or was unable to determine whether a column is duplicated or not |
| invalidselectstarexpansion | Correctness | The query's top-level selection may be `SELECT *`, but only if SQLMesh can expand the `SELECT *` into individual columns |
| noselectstar | Stylistic | The query's top-level selection may not be `SELECT *`, even if SQLMesh can expand the `SELECT *` into individual columns |

| `ambiguousorinvalidcolumn` | Correctness | SQLMesh found duplicate columns or was unable to determine whether a column is duplicated or not |
| `invalidselectstarexpansion` | Correctness | The query's top-level selection may be `SELECT *`, but only if SQLMesh can expand the `SELECT *` into individual columns |
| `noselectstar` | Stylistic | The query's top-level selection may not be `SELECT *`, even if SQLMesh can expand the `SELECT *` into individual columns |
| `nomissingaudits` | Governance | SQLMesh did not find any `audits` in the model's configuration to test data quality. |

### User-defined rules

Expand Down Expand Up @@ -211,7 +213,7 @@ MODEL(

Linting rule violations raise an error by default, preventing the project from running until the violation is addressed.

You may specify that a rule's violation should not error and only log a warning by specifying it in the `warning_rules` key instead of the `rules` key.
You may specify that a rule's violation should not error and only log a warning by specifying it in the `warn_rules` key instead of the `rules` key.

=== "YAML"

Expand All @@ -221,7 +223,7 @@ You may specify that a rule's violation should not error and only log a warning
# error if `ambiguousorinvalidcolumn` rule violated
rules: ["ambiguousorinvalidcolumn"]
# but only warn if "invalidselectstarexpansion" is violated
warning_rules: ["invalidselectstarexpansion"]
warn_rules: ["invalidselectstarexpansion"]
```

=== "Python"
Expand All @@ -235,9 +237,9 @@ You may specify that a rule's violation should not error and only log a warning
# error if `ambiguousorinvalidcolumn` rule violated
rules=["ambiguousorinvalidcolumn"],
# but only warn if "invalidselectstarexpansion" is violated
warning_rules=["invalidselectstarexpansion"],
warn_rules=["invalidselectstarexpansion"],
)
)
```

SQLMesh will raise an error if the same rule is included in more than one of the `rules`, `warning_rules`, and `ignored_rules` keys since they should be mutually exclusive.
SQLMesh will raise an error if the same rule is included in more than one of the `rules`, `warn_rules`, and `ignored_rules` keys since they should be mutually exclusive.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed this. warning_rules doesn't exist

Binary file added docs/guides/linter_example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 7 additions & 0 deletions sqlmesh/core/linter/rules/builtin.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,11 @@ def check_model(self, model: Model) -> t.Optional[RuleViolation]:
return self.violation(violation_msg)


class NoMissingAudits(Rule):
"""Model `audits` must be configured to test data quality."""

def check_model(self, model: Model) -> t.Optional[RuleViolation]:
return self.violation() if not model.audits else None


BUILTIN_RULES = RuleSet(subclasses(__name__, Rule, (Rule,)))
13 changes: 10 additions & 3 deletions tests/core/test_context.py
Original file line number Diff line number Diff line change
Expand Up @@ -1676,7 +1676,9 @@ def assert_cached_violations_exist(cache: OptimizedQueryCache, model: Model):
# Case: Ensure NoSelectStar only raises for top-level SELECTs, new model shouldn't raise
# and thus should also be cached
model2 = load_sql_based_model(
d.parse("MODEL (name test2); SELECT col FROM (SELECT * FROM tbl)")
d.parse(
"MODEL (name test2, audits (at_least_one(column := col))); SELECT col FROM (SELECT * FROM tbl)"
)
)
ctx.upsert_model(model2)

Expand Down Expand Up @@ -1739,7 +1741,7 @@ def assert_cached_violations_exist(cache: OptimizedQueryCache, model: Model):
create_temp_file(
tmp_path,
pathlib.Path(pathlib.Path("models"), "test2.sql"),
"MODEL(name test2, ignored_rules ['noselectstar']); SELECT * FROM (SELECT 1 AS col);",
"MODEL(name test2, audits (at_least_one(column := col)), ignored_rules ['noselectstar']); SELECT * FROM (SELECT 1 AS col);",
)

ctx.load()
Expand Down Expand Up @@ -1777,7 +1779,12 @@ def model4_entrypoint(context, **kwargs):
with pytest.raises(LinterError, match=config_err):
sushi_context.upsert_model(python_model)

@model(name="memory.sushi.model5", columns={"col": "int"}, owner="test")
@model(
name="memory.sushi.model5",
columns={"col": "int"},
owner="test",
audits=[("at_least_one", {"column": "col"})],
)
def model5_entrypoint(context, **kwargs):
yield pd.DataFrame({"col": []})

Expand Down
26 changes: 26 additions & 0 deletions tests/core/test_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,32 @@ def test_model_qualification(tmp_path: Path):
)


@use_terminal_console
def test_model_missing_audits(tmp_path: Path):
with patch.object(get_console(), "log_warning") as mock_logger:
expressions = d.parse(
"""
MODEL (
name db.table,
kind FULL,
);

SELECT a
"""
)

ctx = Context(
config=Config(linter=LinterConfig(enabled=True, warn_rules=["nomissingaudits"])),
paths=tmp_path,
)
ctx.upsert_model(load_sql_based_model(expressions))

assert (
"""Model `audits` must be configured to test data quality."""
in mock_logger.call_args[0][0]
)


@pytest.mark.parametrize(
"partition_by_input, partition_by_output, output_dialect, expected_exception",
[
Expand Down