Add config and context params to KubernetesHook

[dstandish]

Here we add a few params to KubernetesHook

        self.cluster_context = cluster_context
        self.config_file = config_file
        self.in_cluster = in_cluster

We also make it optional to use an airflow connection. In order to use k8s hook with k8s pod op, we need connection to be optional because up to now k8s pod operator does not use airflow connection.

For the new hook params, I parameterized the relevant tests and added a lot of cases in order to verify search path (hook param beats conn extra)

The principal motivator here is enabling the use of KubernetesHook with KubernetesPodOperator while maintaining backward compatibility.

[dstandish]

it seems that merger and loader were backwards in some of these tests

[potiuk]

Do you stll need feedback on it @dstandish ?

[dstandish]

Do you stll need feedback on it @dstandish ?

Yes please, it's ready for review if you have the time. It's a stepping stone towards adding k8s hook to k8s pod op

[potiuk]

Tomorrow morning :)

[jedcunningham]

We also make it optional to use an airflow connection.

Do any other hooks have this behavior? I wonder if KPO should handle backwards compat instead?

[dstandish]

We also make it optional to use an airflow connection.

Do any other hooks have this behavior? I wonder if KPO should handle backwards compat instead?

Well, SubprocessHook doesn't even take a connection id, and postgres and mysql will allow you to not supply a connection id but supply an actual Connection object instead.

To me it seems pretty reasonable because it's a situation where it would be very normal to not need a connection (cus you're in the cluster e.g.) but want to use the hook methods etc. It's not really about backwards compat in my view but making it flexible to use the hook without having defined a connection.

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.