Support for COPY INTO

[allisonwang-db]

resolves #41

Description

This PR adds a macro databricks_copy_into to support the COPY INTO command. You can use it with dbt run-operation:

dbt run-operation databricks_copy_into --args ...

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[ueshin]

@allisonwang-db Thanks for working on this!

Could you fix the linter issue by running tox -e black?
Could you also add an item in CHANGELOG.txt for this?

Otherwise, LGTM.

[ueshin]

Does this feature not support UC?

[allisonwang-db]

I tested against a UC cluster and I don't think UC can support this test case:

1. UC does not allow the creation of parquet tables (only Delta is supported)
2. UC does not allow access to the underlying path of a managed Delta table (so we can't get the source in the test)

But UC should support COPY INTO: https://docs.databricks.com/ingestion/copy-into/unity-catalog.html

[ueshin]

Thanks! merging.

[bilalaslamseattle]

@ueshin does this support the inline credentials that newer DBRs support for COPY INTO?

[ueshin]

@bilalaslamseattle Yes, seems so, if you mean this:

dbt-databricks/dbt/include/databricks/macros/copy_into.sql

Lines 27 to 33 in 1079d25

	{% if source_credential -%}
	credential (
	{%- for name in source_credential -%}
	'{{ name }}' = '{{ source_credential[name] }}' {%- if not loop.last %}, {% endif -%}
	{%- endfor -%}
	)
	{%- endif %}

cc @allisonwang-db

[allisonwang-db]

@bilalaslamseattle Yes, it should support the inline credentials (for the source table).