Fix: Implement support for fetchone() in the ODBCHook and the Databricks SQL Hook
#36161
Conversation
|
The static check handled in #36162 |
| """Transform the databricks Row objects into a JSON-serializable list of rows.""" | ||
| if result is not None: | ||
| """Transform the databricks Row objects into JSON-serializable lists.""" | ||
| if isinstance(result, list): |
There was a problem hiding this comment.
statement = f"DESCRIBE TABLE {table.name};"
hook.run(statement, parameters=parameters, handler=lambda x: x.fetchall())
with this code hook.run() will return [[val1, val2, val3],[val1, val2, val3], [val1, val2, val3]] instead of [row(), row(), row()] isn't this a breaking change??
There was a problem hiding this comment.
One possible solution is to inherit from row class and introduce deserialize and serialize method that way we can deal with this in a similar way as dataframes.
|
@Joffreybvn -> what's your take ? |
|
Just to clarify, the support for fetchall - with the breaking change - was implemented in #32319 . But yes, it is a breaking change. For ODBC, the Row became a NamedTuple - which is a very 'soft' breaking change. For Databricks, there is no NamedTuple because the 'make_serializable' method from #31780, which was turning Row into tuples already, was simply moved down from the Operator into the Hook. Thus, I can make it a NamedTuple too. Adding serialize and deserialize was also discussed in the first PR, and led to this ADR. If Databricks wants to comply with it, it should return a tuple / namedtuple. |
4 participants
This PR implements support for
fetchone()in the ODBCHook and the Databricks SQL Hook. Fixing the bug described in #32319 (comment). It also refactors and improves a bit the docstring.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.