Case mismatch between model and yaml resulting in error (dbt-databricks 1.11 a) #1215
Description
Describe the bug
When there is a casing mismatch between the model's column and the yaml, dbt-databricks 1.11 is error.
Steps To Reproduce
Project setup
# dbt_project.yml
name: analytics
profile: all
version: "1.0.0"
models:
analytics:
+materialized: table
# models/foo.yaml
models:
- name: foo
columns:
- name: account_id
description: The account identifier.{{ config(materialized='table', persist_docs={"columns": true}) }}
select 1 as `Account_ID`Using dbt-databricks 1.11a
$ dbt --debug run
02:17:45 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108e70d90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108eac190>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108e70ad0>]}
02:17:45 Running with dbt=1.11.0-b3
02:17:45 running dbt with arguments {'profiles_dir': '/Users/jeremy/.dbt', 'write_json': 'True', 'log_format': 'default', 'static_parser': 'True', 'introspect': 'True', 'cache_selected_only': 'False', 'version_check': 'True', 'invocation_command': 'dbt --debug run', 'target_path': 'None', 'empty': 'False', 'log_path': '/Users/jeremy/git/dbt-basic/logs', 'printer_width': '80', 'quiet': 'False', 'debug': 'True', 'use_colors': 'True', 'log_cache_events': 'False', 'no_print': 'None', 'partial_parse': 'True', 'use_experimental_parser': 'False', 'send_anonymous_usage_stats': 'True', 'indirect_selection': 'eager', 'warn_error': 'None', 'fail_fast': 'False', 'warn_error_options': 'WarnErrorOptionsV2(error=[], warn=[], silence=[])'}
02:17:46 Spark adapter: Setting pyhive.hive logging to ERROR
02:17:46 Spark adapter: Setting thrift.transport logging to ERROR
02:17:46 Spark adapter: Setting thrift.protocol logging to ERROR
02:17:46 Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x11080ffd0>]}
02:17:46 Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1088dc050>]}
02:17:46 Registered adapter: databricks=1.11.0-a2
02:17:46 checksum: e63134dccc1251cfb572caf0e4aa952f030c125ee3634f3bf2c0a2e1bb6ae349, vars: {}, profile: , target: , version: 1.11.0b3
02:17:46 Unable to do partial parsing because of a version mismatch
02:17:46 Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1081982d0>]}
02:17:47 Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x110cd7950>]}
02:17:47 Wrote artifact WritableManifest to /Users/jeremy/git/dbt-basic/target/manifest.json
02:17:47 Wrote artifact SemanticManifest to /Users/jeremy/git/dbt-basic/target/semantic_manifest.json
02:17:47 Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x110e1c850>]}
02:17:47 Found 1 model, 706 macros
02:17:47 Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x110a7aad0>]}
02:17:47
02:17:47 Concurrency: 1 threads (target='db')
02:17:47
02:17:47 Databricks adapter: DatabricksDBTConnection(session-id=None, name=master) - Creating connection
02:17:47 Acquiring new databricks connection 'master'
02:17:47 Databricks adapter: DatabricksDBTConnection(session-id=None, name=list_dev) - Creating connection
02:17:47 Acquiring new databricks connection 'list_dev'
02:17:47 Using databricks connection "list_dev"
02:17:47 On list_dev: /* {"app": "dbt", "dbt_version": "1.11.0b3", "dbt_databricks_version": "1.11.0a2", "databricks_sql_connector_version": "4.1.3", "profile_name": "all", "target_name": "db", "connection_name": "list_dev"} */
show databases
02:17:47 Opening a new connection, currently in state init
02:17:49 Databricks adapter: Connection(session-id=01f0a7da-d0a9-192f-9abd-fde7afb57704) - Created
02:17:49 SQL status: OK in 2.210 seconds
02:17:49 Databricks adapter: Cursor(session-id=01f0a7da-d0a9-192f-9abd-fde7afb57704, command-id=01f0a7da-d0d0-1baf-9175-3268a8e0cfc9) - Closing
02:17:49 On list_dev: Close
02:17:49 Databricks adapter: Connection(session-id=01f0a7da-d0a9-192f-9abd-fde7afb57704) - Closing
02:17:50 Databricks adapter: DatabricksDBTConnection(session-id=None, name=list_dev_dbt_jyeo_prod) - Creating connection
02:17:50 Acquiring new databricks connection 'list_dev_dbt_jyeo_prod'
02:17:50 Using databricks connection "list_dev_dbt_jyeo_prod"
02:17:50 On list_dev_dbt_jyeo_prod: /* {"app": "dbt", "dbt_version": "1.11.0b3", "dbt_databricks_version": "1.11.0a2", "databricks_sql_connector_version": "4.1.3", "profile_name": "all", "target_name": "db", "connection_name": "list_dev_dbt_jyeo_prod"} */
SELECT
table_name,
if(table_type IN ('EXTERNAL', 'MANAGED', 'MANAGED_SHALLOW_CLONE', 'EXTERNAL_SHALLOW_CLONE'), 'table', lower(table_type)) AS table_type,
lower(data_source_format) AS file_format,
table_owner,
if(
table_type IN (
'EXTERNAL',
'MANAGED',
'MANAGED_SHALLOW_CLONE',
'EXTERNAL_SHALLOW_CLONE'
),
lower(table_type),
NULL
) AS databricks_table_type
FROM `system`.`information_schema`.`tables`
WHERE table_catalog = 'dev'
AND table_schema = 'dbt_jyeo_prod'
02:17:50 Opening a new connection, currently in state init
02:17:51 Databricks adapter: Connection(session-id=01f0a7da-d1a9-1d7a-bc21-06923fc08c81) - Created
02:17:51 SQL status: OK in 1.500 seconds
02:17:51 Databricks adapter: Cursor(session-id=01f0a7da-d1a9-1d7a-bc21-06923fc08c81, command-id=01f0a7da-d1cf-15c7-87d6-bc6a038acf72) - Closing
02:17:51 On list_dev_dbt_jyeo_prod: Close
02:17:51 Databricks adapter: Connection(session-id=01f0a7da-d1a9-1d7a-bc21-06923fc08c81) - Closing
02:17:51 Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x110ea1650>]}
02:17:51 Began running node model.analytics.foo
02:17:51 1 of 1 START sql table model dbt_jyeo_prod.foo ................................. [RUN]
02:17:51 Databricks adapter: DatabricksDBTConnection(session-id=None, name=model.analytics.foo) - Creating connection
02:17:51 Acquiring new databricks connection 'model.analytics.foo'
02:17:51 Began compiling node model.analytics.foo
02:17:51 Writing injected SQL for node "model.analytics.foo"
02:17:52 Began executing node model.analytics.foo
02:17:52 MATERIALIZING TABLE
02:17:52 [WARNING]: Use revamped materializations based on separating create and insert. This allows more performant column comments, as well as new column features.
You may opt into the new behavior sooner by setting `flags.use_materialization_v2` to `True` in `dbt_project.yml`.
Visit https://docs.getdbt.com/reference/global-configs/behavior-changes for more information.
02:17:52 Sending event: {'category': 'dbt', 'action': 'BehaviorChangeEvent', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x110b853d0>]}
02:17:52 Writing runtime sql for node "model.analytics.foo"
02:17:52 Using databricks connection "model.analytics.foo"
02:17:52 On model.analytics.foo: /* {"app": "dbt", "dbt_version": "1.11.0b3", "dbt_databricks_version": "1.11.0a2", "databricks_sql_connector_version": "4.1.3", "profile_name": "all", "target_name": "db", "node_id": "model.analytics.foo"} */
create or replace table `dev`.`dbt_jyeo_prod`.`foo`
using delta
as
select 1 as `Account_ID`
02:17:52 Opening a new connection, currently in state init
02:17:53 Databricks adapter: Connection(session-id=01f0a7da-d2c3-1ca3-9df2-f964fd5eef4e) - Created
02:17:57 SQL status: OK in 5.150 seconds
02:17:57 Databricks adapter: Cursor(session-id=01f0a7da-d2c3-1ca3-9df2-f964fd5eef4e, command-id=01f0a7da-d2f6-13d8-98f9-d083d8fe1b34) - Closing
02:17:57 Applying tags to relation None
02:17:57 Using databricks connection "model.analytics.foo"
02:17:57 On model.analytics.foo: /* {"app": "dbt", "dbt_version": "1.11.0b3", "dbt_databricks_version": "1.11.0a2", "databricks_sql_connector_version": "4.1.3", "profile_name": "all", "target_name": "db", "node_id": "model.analytics.foo"} */
DESCRIBE TABLE EXTENDED `dev`.`dbt_jyeo_prod`.`foo` AS JSON
02:17:57 SQL status: OK in 0.390 seconds
02:17:57 Databricks adapter: Cursor(session-id=01f0a7da-d2c3-1ca3-9df2-f964fd5eef4e, command-id=01f0a7da-d570-1e59-9ed6-df4ed6a61e2d) - Closing
02:17:57 On model.analytics.foo: Close
02:17:57 Databricks adapter: Connection(session-id=01f0a7da-d2c3-1ca3-9df2-f964fd5eef4e) - Closing
02:17:57 Unhandled error while executing target/run/analytics/models/foo.sql
'Account_ID'
02:17:57 Traceback (most recent call last):
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/task/base.py", line 363, in safe_run
result = self.compile_and_execute(manifest, ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/task/base.py", line 310, in compile_and_execute
result = self.run(ctx.node, manifest)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/task/base.py", line 410, in run
return self.execute(compiled_node, manifest)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/task/run.py", line 336, in execute
return self._execute_model(hook_ctx, context_config, model, context, materialization_macro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/task/run.py", line 296, in _execute_model
result = MacroGenerator(
^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/clients/jinja.py", line 82, in __call__
return self.call_macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 395, in call_macro
return macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 770, in __call__
return self._invoke(arguments, autoescape)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 784, in _invoke
rv = self._func(*arguments)
^^^^^^^^^^^^^^^^^^^^^^
File "<template>", line 203, in macro
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/sandbox.py", line 401, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 303, in call
return __obj(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/clients/jinja.py", line 82, in __call__
return self.call_macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 395, in call_macro
return macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 770, in __call__
return self._invoke(arguments, autoescape)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 784, in _invoke
rv = self._func(*arguments)
^^^^^^^^^^^^^^^^^^^^^^
File "<template>", line 26, in macro
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/sandbox.py", line 401, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 303, in call
return __obj(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/clients/jinja.py", line 82, in __call__
return self.call_macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 395, in call_macro
return macro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 770, in __call__
return self._invoke(arguments, autoescape)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 784, in _invoke
rv = self._func(*arguments)
^^^^^^^^^^^^^^^^^^^^^^
File "<template>", line 43, in macro
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/sandbox.py", line 401, in call
return __context.call(__obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/jinja2/runtime.py", line 303, in call
return __obj(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/lib/python3.11/site-packages/dbt/adapters/databricks/impl.py", line 795, in get_persist_doc_columns
return_columns[name] = columns[name]
~~~~~~~^^^^^^
KeyError: 'Account_ID'
02:17:57 Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '49a354af-89b3-4770-8c7b-776271a1ae44', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x110bb3c10>]}
02:17:57 1 of 1 ERROR creating sql table model dbt_jyeo_prod.foo ........................ [ERROR in 5.91s]
02:17:57 Finished running node model.analytics.foo
02:17:57 Marking all children of 'model.analytics.foo' to be skipped because of status 'error'. Reason: 'Account_ID'.
02:17:57 Databricks adapter: DatabricksDBTConnection(session-id=None, name=master) - Creating connection
02:17:57 Acquiring new databricks connection 'master'
02:17:57
02:17:57 Finished running 1 table model in 0 hours 0 minutes and 10.19 seconds (10.19s).
02:17:57 Command end result
02:17:57 Wrote artifact WritableManifest to /Users/jeremy/git/dbt-basic/target/manifest.json
02:17:57 Wrote artifact SemanticManifest to /Users/jeremy/git/dbt-basic/target/semantic_manifest.json
02:17:57 Wrote artifact RunExecutionResult to /Users/jeremy/git/dbt-basic/target/run_results.json
02:17:57
02:17:57 Completed with 1 error, 0 partial successes, and 0 warnings:
02:17:57
02:17:57 Failure in model foo (models/foo.sql)
02:17:57 'Account_ID'
02:17:57
02:17:57 compiled code at target/compiled/analytics/models/foo.sql
02:17:57
02:17:57 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 TOTAL=1
02:17:57 Resource report: {"command_name": "run", "command_success": false, "command_wall_clock_time": 12.022592, "process_in_blocks": "0", "process_kernel_time": 0.282299, "process_mem_max_rss": "255590400", "process_out_blocks": "0", "process_user_time": 2.277789}
02:17:57 Command `dbt run` failed at 15:17:57.936201 after 12.02 seconds
02:17:57 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108c05390>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108ecb050>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x106617e50>]}
02:17:57 Flushing usage events
02:17:59 An error was encountered while trying to flush usage eventsUsing dbt-databricks 1.10
$ dbt --debug run
02:19:07 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10c385150>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10c384e90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10c370ad0>]}
02:19:07 Running with dbt=1.10.13
02:19:07 running dbt with arguments {'no_print': 'None', 'indirect_selection': 'eager', 'log_format': 'default', 'quiet': 'False', 'write_json': 'True', 'profiles_dir': '/Users/jeremy/.dbt', 'empty': 'False', 'use_colors': 'True', 'static_parser': 'True', 'send_anonymous_usage_stats': 'True', 'warn_error': 'None', 'fail_fast': 'False', 'cache_selected_only': 'False', 'partial_parse': 'True', 'use_experimental_parser': 'False', 'log_cache_events': 'False', 'introspect': 'True', 'invocation_command': 'dbt --debug run', 'warn_error_options': 'WarnErrorOptionsV2(error=[], warn=[], silence=[])', 'debug': 'True', 'printer_width': '80', 'version_check': 'True', 'target_path': 'None', 'log_path': '/Users/jeremy/git/dbt-basic/logs'}
02:19:07 Spark adapter: Setting pyhive.hive logging to ERROR
02:19:07 Spark adapter: Setting thrift.transport logging to ERROR
02:19:07 Spark adapter: Setting thrift.protocol logging to ERROR
/Users/jeremy/git/dbt-basic/venv_dbt_1.10.latest/lib/python3.11/site-packages/pydantic/_internal/_config.py:383: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'validate_by_name'
warnings.warn(message, UserWarning)
02:19:07 Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x11d6e8350>]}
02:19:07 Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10aa77510>]}
02:19:07 Registered adapter: databricks=1.10.12
02:19:08 checksum: 9f0c81e2574b4ff463f3d16d080df83c6982fc0372c9feeacae0504ac9ea3ffe, vars: {}, profile: , target: , version: 1.10.13
02:19:08 Unable to do partial parsing because of a version mismatch
02:19:08 Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10a96a250>]}
02:19:08 Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x11f820710>]}
02:19:08 Wrote artifact WritableManifest to /Users/jeremy/git/dbt-basic/target/manifest.json
02:19:08 Wrote artifact SemanticManifest to /Users/jeremy/git/dbt-basic/target/semantic_manifest.json
02:19:08 Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x11f962150>]}
02:19:08 Found 1 model, 686 macros
02:19:08 Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x11dbab550>]}
02:19:08
02:19:08 Concurrency: 1 threads (target='db')
02:19:08
02:19:08 Databricks adapter: DatabricksDBTConnection(session-id=None, name=master) - Creating connection
02:19:08 Acquiring new databricks connection 'master'
02:19:08 Databricks adapter: DatabricksDBTConnection(session-id=None, name=list_dev) - Creating connection
02:19:08 Acquiring new databricks connection 'list_dev'
02:19:08 Using databricks connection "list_dev"
02:19:08 On list_dev: /* {"app": "dbt", "dbt_version": "1.10.13", "dbt_databricks_version": "1.10.12", "databricks_sql_connector_version": "4.0.5", "profile_name": "all", "target_name": "db", "connection_name": "list_dev"} */
show databases
02:19:08 Opening a new connection, currently in state init
02:19:10 Databricks adapter: Connection(session-id=01f0a7db-00fa-197a-a647-ccd79597f3d6) - Created
02:19:10 SQL status: OK in 2.160 seconds
02:19:10 Databricks adapter: Cursor(session-id=01f0a7db-00fa-197a-a647-ccd79597f3d6, command-id=01f0a7db-011f-163e-ac7b-0d9ce29a9302) - Closing
02:19:10 On list_dev: Close
02:19:10 Databricks adapter: Connection(session-id=01f0a7db-00fa-197a-a647-ccd79597f3d6) - Closing
02:19:11 Databricks adapter: DatabricksDBTConnection(session-id=None, name=list_dev_dbt_jyeo_prod) - Creating connection
02:19:11 Acquiring new databricks connection 'list_dev_dbt_jyeo_prod'
02:19:11 Using databricks connection "list_dev_dbt_jyeo_prod"
02:19:11 On list_dev_dbt_jyeo_prod: /* {"app": "dbt", "dbt_version": "1.10.13", "dbt_databricks_version": "1.10.12", "databricks_sql_connector_version": "4.0.5", "profile_name": "all", "target_name": "db", "connection_name": "list_dev_dbt_jyeo_prod"} */
SELECT
table_name,
if(table_type IN ('EXTERNAL', 'MANAGED', 'MANAGED_SHALLOW_CLONE', 'EXTERNAL_SHALLOW_CLONE'), 'table', lower(table_type)) AS table_type,
lower(data_source_format) AS file_format,
table_owner,
if(
table_type IN (
'EXTERNAL',
'MANAGED',
'MANAGED_SHALLOW_CLONE',
'EXTERNAL_SHALLOW_CLONE'
),
lower(table_type),
NULL
) AS databricks_table_type
FROM `system`.`information_schema`.`tables`
WHERE table_catalog = 'dev'
AND table_schema = 'dbt_jyeo_prod'
02:19:11 Opening a new connection, currently in state init
02:19:12 Databricks adapter: Connection(session-id=01f0a7db-01f0-1d0b-9430-1ed1629c3042) - Created
02:19:12 SQL status: OK in 1.510 seconds
02:19:12 Databricks adapter: Cursor(session-id=01f0a7db-01f0-1d0b-9430-1ed1629c3042, command-id=01f0a7db-0214-1f9f-b475-305c8175cc78) - Closing
02:19:12 On list_dev_dbt_jyeo_prod: Close
02:19:12 Databricks adapter: Connection(session-id=01f0a7db-01f0-1d0b-9430-1ed1629c3042) - Closing
02:19:13 Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x11d86a3d0>]}
02:19:13 Began running node model.analytics.foo
02:19:13 1 of 1 START sql table model dbt_jyeo_prod.foo ................................. [RUN]
02:19:13 Databricks adapter: DatabricksDBTConnection(session-id=None, name=model.analytics.foo) - Creating connection
02:19:13 Acquiring new databricks connection 'model.analytics.foo'
02:19:13 Began compiling node model.analytics.foo
02:19:13 Writing injected SQL for node "model.analytics.foo"
02:19:13 Began executing node model.analytics.foo
02:19:13 MATERIALIZING TABLE
02:19:13 [WARNING]: Use revamped materializations based on separating create and insert. This allows more performant column comments, as well as new column features.
You may opt into the new behavior sooner by setting `flags.use_materialization_v2` to `True` in `dbt_project.yml`.
Visit https://docs.getdbt.com/reference/global-configs/behavior-changes for more information.
02:19:13 Sending event: {'category': 'dbt', 'action': 'BehaviorChangeEvent', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x11f9dbc10>]}
02:19:13 Writing runtime sql for node "model.analytics.foo"
02:19:13 Using databricks connection "model.analytics.foo"
02:19:13 On model.analytics.foo: /* {"app": "dbt", "dbt_version": "1.10.13", "dbt_databricks_version": "1.10.12", "databricks_sql_connector_version": "4.0.5", "profile_name": "all", "target_name": "db", "node_id": "model.analytics.foo"} */
create or replace table `dev`.`dbt_jyeo_prod`.`foo`
using delta
as
select 1 as `Account_ID`
02:19:13 Opening a new connection, currently in state init
02:19:13 Databricks adapter: Connection(session-id=01f0a7db-030a-166c-a093-f4756cfb4c12) - Created
02:19:16 SQL status: OK in 3.180 seconds
02:19:16 Databricks adapter: Cursor(session-id=01f0a7db-030a-166c-a093-f4756cfb4c12, command-id=01f0a7db-032f-1968-9adb-bb853f9b97e6) - Closing
02:19:16 Applying tags to relation None
02:19:16 Using databricks connection "model.analytics.foo"
02:19:16 On model.analytics.foo: /* {"app": "dbt", "dbt_version": "1.10.13", "dbt_databricks_version": "1.10.12", "databricks_sql_connector_version": "4.0.5", "profile_name": "all", "target_name": "db", "node_id": "model.analytics.foo"} */
DESCRIBE TABLE EXTENDED `dev`.`dbt_jyeo_prod`.`foo` AS JSON
02:19:16 SQL status: OK in 0.390 seconds
02:19:16 Databricks adapter: Cursor(session-id=01f0a7db-030a-166c-a093-f4756cfb4c12, command-id=01f0a7db-048f-1b0a-a8a9-ef43335fe57b) - Closing
02:19:16 On model.analytics.foo: Close
02:19:16 Databricks adapter: Connection(session-id=01f0a7db-030a-166c-a093-f4756cfb4c12) - Closing
02:19:16 Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '722ca63b-295f-4a97-8b43-3208e464a49d', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x11d723010>]}
02:19:16 1 of 1 OK created sql table model dbt_jyeo_prod.foo ............................ [OK in 3.94s]
02:19:16 Finished running node model.analytics.foo
02:19:16 Databricks adapter: DatabricksDBTConnection(session-id=None, name=master) - Creating connection
02:19:16 Acquiring new databricks connection 'master'
02:19:16
02:19:16 Finished running 1 table model in 0 hours 0 minutes and 8.19 seconds (8.19s).
02:19:16 Command end result
02:19:16 Wrote artifact WritableManifest to /Users/jeremy/git/dbt-basic/target/manifest.json
02:19:16 Wrote artifact SemanticManifest to /Users/jeremy/git/dbt-basic/target/semantic_manifest.json
02:19:17 Wrote artifact RunExecutionResult to /Users/jeremy/git/dbt-basic/target/run_results.json
02:19:17
02:19:17 Completed successfully
02:19:17
02:19:17 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1
02:19:17 Resource report: {"command_name": "run", "command_success": true, "command_wall_clock_time": 9.930657, "process_in_blocks": "0", "process_kernel_time": 0.281232, "process_mem_max_rss": "246988800", "process_out_blocks": "0", "process_user_time": 2.229912}
02:19:17 Command `dbt run` succeeded at 15:19:17.006929 after 9.93 seconds
02:19:17 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10c3e2b90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10c3e3250>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x104c4e810>]}
02:19:17 Flushing usage events
02:19:18 An error was encountered while trying to flush usage eventsExpected behavior
No errors.
Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.
System information
The output of dbt --version:
Core:
- installed: 1.11.0-b3
- latest: 1.10.13 - Ahead of latest version!
Plugins:
- databricks: 1.11.0a2 - Ahead of latest version!
- spark: 1.9.3 - Up to date!
Core:
- installed: 1.10.13
- latest: 1.10.13 - Up to date!
Plugins:
- databricks: 1.10.12 - Up to date!
- spark: 1.9.3 - Up to date!
The operating system you're using:
macOS
The output of python --version:
3.11.9
Additional context
I believe the problem is likely due to the changes made in #1190
Afaict, in 1.10 - when there is a casing mismatch, the persist_docs path is ignored, meaning
- Databricks returns
foohas columnAccount_ID. - Schema yml has column
account_id. - (1) and (2) are not matches - we don't attempt to do persist docs cause they are technically not the same thing.
In 1.11 - with lower casing logic - we reconcile those 2?