Version 1.3.0

[ueshin]

Features

• Support python model through run command API, currently supported materializations are table and incremental. (dbt-labs/dbt-spark#377, #126)
• Enable Pandas and Pandas-on-Spark DataFrames for dbt python models (dbt-labs/dbt-spark#469, #181)
• Support job cluster in notebook submission method (dbt-labs/dbt-spark#467, #194)
  • In all_purpose_cluster submission method, a config http_path can be specified in Python model config to switch the cluster where Python model runs.

    def model(dbt, _):
        dbt.config(
            materialized='table',
            http_path='...'
        )
        ...

• Use builtin timestampadd and timestampdiff functions for dateadd/datediff macros if available (#185)
• Implement testing for a test for various Python models (#189)
• Implement testing for type_boolean in Databricks (dbt-labs/dbt-spark#471, #188)
• Add a macro to support COPY INTO (#190)

Under the hood

• Apply "Initial refactoring of incremental materialization" (#148)
  • Now dbt-databricks uses adapter.get_incremental_strategy_macro instead of dbt_spark_get_incremental_sql macro to dispatch the incremental strategy macro. The overwritten dbt_spark_get_incremental_sql macro will not work anymore.
• Better interface for python submission (dbt-labs/dbt-spark#452, #178)

---

This discussion was created from the release Version 1.3.0.