[FEATURE]: Add Disposition parameter in SqlBackend creation #3447
Description
Is there an existing issue for this?
- I have searched the existing issues
Problem statement
When running the databricks labs ucx export-assessment, some customer are facing the following issue:
BadRequest: Inline byte limit exceeded. Statements executed with disposition=INLINE can have a result size of at most 26214400 bytes. Please execute the statement with disposition=EXTERNAL_LINKS if you want to download the full result.
This configuration can also introduce limits in other executions in which SQL statement results are not negligible.
Proposed Solution
Requiring the Disposition value during the UCX installation can allow users to select the proper disposition method to use when running large queries. This parameter can be added then in the config.yml file and used for the SqlBackend definition. Using this approach, large queries will not fail.
Additional Context
I can work on it and introduce this feature if it seems compliant with UCX framework.