Adding Flink on K8s Operator #9134

[chethanuk]

Adding new Flink on K8s Operator to cncf provider

NOTE: The Flink Kubernetes Operator extends the Kubernetes API with the ability to manage and operate Flink Deployments. The operator features the following amongst others:
Basically allows users to manage Flink applications and their lifecycle through native k8s tooling

• Deploy and monitor Flink Application and Session deployments
• Upgrade, suspend and delete deployments
• Full logging and metrics integration
• Flexible deployments and native integration with Kubernetes tooling

related: #9134

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

I think this shoudl be rather part of a new "apache.flink" provider (depending on cncf.kubernetes) than part of cncf.kubernetes provider. I wonder what others think, but IMHO adding flink to cncf.kubernetes (even if it uses K8S to run) is somewhat counter-intuitive.

@jedcunningham @dstandish @ashb @ephraimbuddy - WDYT?

[chethanuk]

new "apache.flink" provider

That will be better I can make it a new "apache.flink" provider

[potiuk]

Conflicts need resolving.

[potiuk]

still :(

[GezimSejdiu]

Hey @chethanuk-plutoflume ,

are you aware of any blockers that this has to be pushed? Do you need more tests from the community so that this also gets priority to be pushed for the next release? I will try to find some time later this week and test that out.

Just wanted to check it, as I find it quite useful for flink-deployments. I did some tests with argo-workflow and directly using kubectl, btu this is much, much better as it is natively integrated with Airflow. I know that this makes sense indeed when the only batch mode is used, but as a start, it would be great to push it and look for how to handle streaming mode.

Just my two cents.

Once again, thanks a lot for your hard work. b.t.w I also had in mind to push this to Airflow :) and was so happy to see it here.

[xinbinhuang]

fixed in #28512