Fix for LocalKubernetesExecutor scheduler is not serving logs

[snjypl]

*** Fetching from: http://airflow-scheduler-0.airflow-scheduler.airflow.svc.cluster.local:8793/log/dag_id=example_bash_operator/run_id=manual__2022-12-29T08:33:49.310825+00:00/task_id=runme_1/attempt=1.log
*** Failed to fetch log file from worker. [Errno 111] Connection refused

For LocalKubernetesExecutor we are not able to fetch the logs for local tasks.

the scheduler is not starting the serve_logs process.

airflow/airflow/cli/commands/scheduler_command.py

Lines 83 to 87 in 94b3b89

	executor_class, _ = ExecutorLoader.import_default_executor_cls()
	if executor_class.is_local:
	if skip_serve_logs is False:
	sub_proc = Process(target=serve_logs)
	sub_proc.start()

we need to set is_local to true for LocalKubernetesExecutor

airflow/airflow/executors/local_kubernetes_executor.py

Line 46 in 94b3b89

is_local: bool = False

also in the helm chart, when the executor is LocalKubernetesExecutor we need to create the scheduler service so that the tasks logs can be fetched.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[dstandish]

How about we create flag serve_logs and add this to base executor and use this instead of is_local in all executors to control log serving?

CC @potiuk @o-nikolas

[potiuk]

I think if we want to make our Chart fully compatible with AIP-51 and allow "any" executors we should also implement some way to override Helm Chart behaviour for custom executors. But we can do it later. CC: @o-nikolas

[potiuk]

Ah.. I see @dstandish already had some comments on that one too :) Yeah - serve_logs should be fine and it should have default value depending on the "built-in" executors - so that we do not have to configure it manually for built-in executors.

[snjypl]

@potiuk i noticed that airflow-worker service account is not getting created for LocalKubernetesExecutor.

shall i create a new PR with all the helm chart related changes? or it can be part of this PR?

kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6000d700-1b7b-413b-a49f-3eba7114bbf7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'e6bfeec0-ef68-4420-88d8-b7024eed9fea', 'X-Kubernetes-Pf-Prioritylevel-Uid': '0e598fd5-a817-46f7-be85-6ac94aeb7872', 'Date': 'Thu, 29 Dec 2022 18:57:06 GMT', 'Content-Length': '400'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"example-local-kubernetes-executor-task-with-kubernetes-c5u03pq1\" is forbidden: error looking up service account airflow/airflow-worker: serviceaccount \"airflow-worker\" not found","reason":"Forbidden","details":{"name":"example-local-kubernetes-executor-task-with-kubernetes-c5u03pq1","kind":"pods"},"code":403}
[2022-12-29T18:57:06.871+0000] {kubernetes_executor.py:686} WARNING - ApiException when attempting to run task, re-queueing. Reason: 'Forbidden'. Message: pods "example-local-kubernetes-executor-task-with-kubernetes-c5u03pq1" is forbidden: error looking up service account airflow/airflow-worker: serviceaccount "airflow-worker" not found

[o-nikolas]

Hey folks!

I definitely agree with splitting out is_local into more fine grained compatibility flags/properties (see here for more context). Also expect an AIP soon which will deliver native support for hybrid executors (of any combination) rather than these handbuilt classes which combine two explicit executors 🚀

[snjypl]

@dstandish i have added serve_logs flag

[snjypl]

@potiuk i noticed that airflow-worker service account is not getting created for LocalKubernetesExecutor.

shall i create a new PR with all the helm chart related changes? or it can be part of this PR?

kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6000d700-1b7b-413b-a49f-3eba7114bbf7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'e6bfeec0-ef68-4420-88d8-b7024eed9fea', 'X-Kubernetes-Pf-Prioritylevel-Uid': '0e598fd5-a817-46f7-be85-6ac94aeb7872', 'Date': 'Thu, 29 Dec 2022 18:57:06 GMT', 'Content-Length': '400'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"example-local-kubernetes-executor-task-with-kubernetes-c5u03pq1\" is forbidden: error looking up service account airflow/airflow-worker: serviceaccount \"airflow-worker\" not found","reason":"Forbidden","details":{"name":"example-local-kubernetes-executor-task-with-kubernetes-c5u03pq1","kind":"pods"},"code":403}
[2022-12-29T18:57:06.871+0000] {kubernetes_executor.py:686} WARNING - ApiException when attempting to run task, re-queueing. Reason: 'Forbidden'. Message: pods "example-local-kubernetes-executor-task-with-kubernetes-c5u03pq1" is forbidden: error looking up service account airflow/airflow-worker: serviceaccount "airflow-worker" not found

@Chen-Oliver has opened a PR for this particular issue: #28813

[o-nikolas]

Anyone have some time to give this a second review/approval? Would be nice to get this merged for @snjypl
Maybe @potiuk, @dstandish or @pierrejeambrun?

[pierrejeambrun]

LGTM

[pierrejeambrun]

Could use a rebase before merging, it's 20+ commits behind main

[snjypl]

Could use a rebase before merging, it's 20+ commits behind master

Done.