Empty logs from CeleryWorker

[zoid-w]

Our instance is running on Kubernetes.

After conversion from Airflow 2.4.1 to 2.6.0 the logs of our celery worker are completely empty.

I have inspected the network call being made to the worker to get the logs and this request is being handled fine.
However the response is the following :

No changes were made to the configuration compared to our 2.4.1 release. Could this be caused by the integration of the triggerer logs in 2.6.0?

Any advice on how to remediate this issue?
This issue seems to be different compared to old issues referencing issues with logs.

[potiuk]

Yes. It's a change in a behaviour explained by @dstandish here: #31059 (comment)

Celery workers cannot be relied on being run continuously with the same name, and you probably had a similar problem before it (when restarting the celery workers) was just not as prominent as it is now.

From the answer of Daniel - you have two options:

1. make sure your logs are store a location (shared folder) shared by all your components
2. setup remote logging.

[potiuk]

BTW. Seems that it might be a bit more than that and we might need a fix there (hold on until we make deeper analysis and possibly get a better fix in 2.6.1 and it seems to be more common issue) - but switching to one of the above should fix it for now.

[zoid-w]

Thanks for the feedback.

I have tested with a longer running pipeline indeed the logs are available while the task is running. (I was doing tests with a very fast pipeline so missed the fact that the logs were available while the task was being done).

For option 2 : We have been looking at remote logging to our ELK stack, but this will be for a later stage.

A quick question with regards to option 1 (which I think we can use as an intermediate fix):
Could we create a PVC/PV and provide this to the logs section of the helm chart?

[potiuk]

I do not know it by heart, but tou can take a look at our Helm chart. When you specify logs.persistence.enabled (and related parameter) and use --dry-run or possibly even helm-diff plugin you should be able to see how our "reference" helm chart works with log persistence and use it as an example.

(long term - maybe switching to our chart is a good idea if you have your own).

[potiuk]

https://airflow.apache.org/docs/helm-chart/stable/index.html

[potiuk]

Ah - you seem to use our chart :) - then you need to enable logs persistence, yes.

[zoid-w]

Thanks for the comments! Indeed I am using your chart; logs persistence did the trick as an intermediate fix!

[potiuk]

Thanks for confirming it! That will help us to see what options we have.