After upgrading to 2.5.3, Dag Processing time increased dramatically.

[wookiist]

Apache Airflow version

2.5.3

What happened

I upgraded my airflow cluster from 2.5.2 to 2.5.3 , after which strange things started happening.
I'm currently using a standalone dagProcessor, and the parsing time that used to take about 2 seconds has suddenly increased to about 10 seconds.
I'm thinking it's weird because I haven't made any changes other than a version up, but is there something I can look into? Thanks in advance! 🙇🏼

What you think should happen instead

I believe that the time it takes to parse a Dag should be constant, or at least have some variability, but shouldn't take as long as it does now.

How to reproduce

If you cherrypick this commit into 2.5.2 stable code, the issue will recur.

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

• Kubernetes 1.21 Cluster
• 1.7.0 helm chart
• standalone dag processor
• using kubernetes executor
• using mysql database

Anything else

This issue still persists, and restarting the Dag Processor has not resolved the issue.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Thanks. looking at it, seems like a regression., but also could be the "metrics" improvement. Is it possible that you share some excerpts of logs from dag file processor showing parsing the fiels?

[potiuk]

If you cherrypick #30079 into 2.5.2 stable code, the issue will recur.

Ah. Fantastic. That will help us a LOT to see what's going on. Thanks for narrowing it down @wookiist !

[potiuk]

We definitely need the log. I looked at the change and I can think of only one reason why you could have the metrics increased - namely that there is a recurring problem with some database operation related to callbacks, and retries simply increased the time it takes to run the opertions several times until they fail. This might indicate some consistent problem with either bug in Airflow or some deadlock/operational issue with your database, but we need more data to take a closer look and diagnose it fully.

Can you tell us how many dag file processors you have @wookiist ?

So I have two kind requests @wookiist:

1. could you please increase log level for dag file processor to debug (it can be only for the processor to not impact other components): https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging-level (it can be done via adding the AIRFLOW__LOGGING__LOGGING_LEVEL env variable and restarting the processor) and sharing a gist with the logs f dag file processor. Also if you could see if there are any errror messages produced in your database logs, signalling that there are some queries failing from correlated time of when dag file processor runs, that could help a lot as well.

2. Can you please take a look at the setting of scheduler 'max_callbacks_for_loop configuration (default is 20) . And if you could lower it down to say 5 (if you have 20), and see if that helps. I have a theory that this parameter might make your queries either run long or run into deadlocks and that might cause the queries to retry few times.

Longer explanation (and theory I have):

The processing time you see is not only for parsing, it is also for running callbacks related to the DAG being processed. When you have callbacks that cannot be run in the worker related to DAG file, they are executed by the DAG file processor after the file is parsed. How it works is that the main parsing loop will start processor for each file and feed it with the callbacks to execute. The change you mentioned does not directly contribute to the time, but it produces callbacks that the file processor executes. And those callbacks actually contribute to the time you see - so the increase of the time you see might not come from "parsing" but it might come from callback processing, and it could be tha the db_retry mechanism introduced in the PR you mentioned, has a problem that it (accidentally) sends multiple repeated callbacks to the processor - and that increases the processing time.

Having the logs, we will be able to see if that is happening and likely we should be able to see the reason (possibly database logs might give more explanation what's going on).

cc: @mhenc I think that one is closely related to standalone file processor case and might need your insights once we get more information

[wookiist]

Thanks for looking, @potiuk !

0. Can you tell us how many dag file processors you have @wookiist ?

Only one dag processor is being used

1. could you please increase log level for dag file processor to debug

Yes, I tested it on dev airflow cluster, and here is the log file.

log file

[2023-04-13T04:52:47.287+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T04:52:47.287+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T04:52:47.294+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3
[2023-04-13T04:52:47.306+0000] {manager.py:1055} DEBUG - Started a process (PID: 37) to generate tasks for /opt/airflow/dags/data/
[2023-04-13T04:52:47.307+0000] {manager.py:979} DEBUG - Processor for /opt/airflow/dags/data/ finished
[2023-04-13T04:52:47.309+0000] {processor.py:296} DEBUG - Waiting for <ForkProcess name='DagFileProcessor7-Process' pid=36 parent=9 stopped exitcode=0>
[2023-04-13T04:52:47.310+0000] {manager.py:1019} DEBUG - 1/2 DAG parsing processes running
[2023-04-13T04:52:47.310+0000] {manager.py:1021} DEBUG - 35 file paths queued for processing
[2023-04-13T04:52:47.324+0000] {manager.py:864} INFO - 
================================================================================
DAG File Processing Stats

File Path                                                                                PID  Runtime      # DAGs    # Errors  Last Runtime    Last Run
-------------------------------------------------------------------------------------  -----  ---------  --------  ----------  --------------  -------------------
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      37  0.01s             0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      0           0
/opt/airflow/dags/data/                                                                                      1           0          5.01s           2023-04-13T04:52:37
/opt/airflow/dags/data/                                                                                      1           0          5.02s           2023-04-13T04:52:42
/opt/airflow/dags/data/                                                                                      1           0          5.02s           2023-04-13T04:52:22
/opt/airflow/dags/data/                                                                                      1           0          5.01s           2023-04-13T04:52:27
/opt/airflow/dags/data/                                                                                      1           0          5.12s           2023-04-13T04:52:17
/opt/airflow/dags/data/                                                                                      1           0          5.03s           2023-04-13T04:52:32
/opt/airflow/dags/data/                                                                                      1           0          5.04s           2023-04-13T04:52:47
/opt/airflow/dags/data/                                                                                      1           0          5.18s           2023-04-13T04:52:17
================================================================================
[2023-04-13T04:52:47.810+0000] {settings.py:408} DEBUG - Disposing DB connection pool (PID 37)
[2023-04-13T04:52:52.300+0000] {base_job.py:240} DEBUG - [heartbeat]
[2023-04-13T04:52:52.301+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T04:52:52.301+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T04:52:52.308+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3
[2023-04-13T04:52:52.343+0000] {manager.py:1055} DEBUG - Started a process (PID: 38) to generate tasks for /opt/airflow/dags/data/B
[2023-04-13T04:52:52.344+0000] {manager.py:979} DEBUG - Processor for /opt/airflow/dags/data/C finished
[2023-04-13T04:52:52.345+0000] {processor.py:296} DEBUG - Waiting for <ForkProcess name='DagFileProcessor8-Process' pid=37 parent=9 stopped exitcode=0>
[2023-04-13T04:52:52.346+0000] {manager.py:1019} DEBUG - 1/2 DAG parsing processes running
[2023-04-13T04:52:52.346+0000] {manager.py:1021} DEBUG - 34 file paths queued for processing
[2023-04-13T04:52:52.513+0000] {settings.py:408} DEBUG - Disposing DB connection pool (PID 38)
[2023-04-13T04:52:57.299+0000] {base_job.py:240} DEBUG - [heartbeat]
[2023-04-13T04:52:57.300+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T04:52:57.300+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T04:52:57.305+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3
[2023-04-13T04:52:57.316+0000] {manager.py:1055} DEBUG - Started a process (PID: 39) to generate tasks for /opt/airflow/dags/data/A.py
[2023-04-13T04:52:57.319+0000] {manager.py:979} DEBUG - Processor for /opt/airflow/dags/data/B.py finished
[2023-04-13T04:52:57.320+0000] {processor.py:296} DEBUG - Waiting for <ForkProcess name='DagFileProcessor9-Process' pid=38 parent=9 stopped exitcode=0>
[2023-04-13T04:52:57.321+0000] {manager.py:1019} DEBUG - 1/2 DAG parsing processes running
[2023-04-13T04:52:57.321+0000] {manager.py:1021} DEBUG - 33 file paths queued for processing
[2023-04-13T04:52:57.516+0000] {settings.py:408} DEBUG - Disposing DB connection pool (PID 39)
[2023-04-13T04:53:02.314+0000] {base_job.py:240} DEBUG - [heartbeat]
[2023-04-13T04:53:02.314+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T04:53:02.315+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T04:53:02.319+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3
[2023-04-13T04:53:02.328+0000] {manager.py:1055} DEBUG - Started a process (PID: 40) to generate tasks for /opt/airflow/dags/data/D.py
[2023-04-13T04:53:02.329+0000] {manager.py:979} DEBUG - Processor for /opt/airflow/dags/data/A.py finished
[2023-04-13T04:53:02.329+0000] {processor.py:296} DEBUG - Waiting for <ForkProcess name='DagFileProcessor10-Process' pid=39 parent=9 stopped exitcode=0>
[2023-04-13T04:53:02.330+0000] {manager.py:1019} DEBUG - 1/2 DAG parsing processes running
[2023-04-13T04:53:02.330+0000] {manager.py:1021} DEBUG - 32 file paths queued for processing
[2023-04-13T04:53:02.471+0000] {settings.py:408} DEBUG - Disposing DB connection pool (PID 40)
[2023-04-13T04:53:07.321+0000] {base_job.py:240} DEBUG - [heartbeat]
[2023-04-13T04:53:07.322+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T04:53:07.322+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T04:53:07.326+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3

2. Can you please take a look at the setting of scheduler 'max_callbacks_for_loop configuration (default is 20) .

I think you're probably referring to the max_callbacks_per_loop value, and I changed that to 5.
This is the log file after I change the value. And this is the graph change.

log file

[2023-04-13T05:15:43.600+0000] {base_job.py:240} DEBUG - [heartbeat]
[2023-04-13T05:15:43.601+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T05:15:43.601+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T05:15:43.611+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3
[2023-04-13T05:15:43.630+0000] {manager.py:1055} DEBUG - Started a process (PID: 185) to generate tasks for /opt/airflow/dags/data/A.py
[2023-04-13T05:15:43.632+0000] {manager.py:979} DEBUG - Processor for /opt/airflow/dags/data/0.py finished
[2023-04-13T05:15:43.637+0000] {processor.py:296} DEBUG - Waiting for <ForkProcess name='DagFileProcessor91-Process' pid=184 parent=6 stopped exitcode=0>
[2023-04-13T05:15:43.637+0000] {manager.py:1019} DEBUG - 1/2 DAG parsing processes running
[2023-04-13T05:15:43.637+0000] {manager.py:1021} DEBUG - 29 file paths queued for processing
[2023-04-13T05:15:43.684+0000] {manager.py:864} INFO - 
================================================================================
DAG File Processing Stats

File Path                                                                                PID  Runtime      # DAGs    # Errors  Last Runtime    Last Run
-------------------------------------------------------------------------------------  -----  ---------  --------  ----------  --------------  -------------------
/opt/airflow/dags/data/                                                                                         0           1  5.04s           2023-04-13T05:14:43
/opt/airflow/dags/data/                                                                                         0           0  5.21s           2023-04-13T05:10:53
/opt/airflow/dags/data/                                                                                         0           0  4.89s           2023-04-13T05:10:58
/opt/airflow/dags/data/                                                                                         0           0  5.15s           2023-04-13T05:14:53
/opt/airflow/dags/data/                                                                                         0           0  5.01s           2023-04-13T05:11:08
/opt/airflow/dags/data/                                                                                         0           0  5.03s           2023-04-13T05:11:03
/opt/airflow/dags/data/                                                                                         0           0  5.08s           2023-04-13T05:11:13
/opt/airflow/dags/data/                                                                                         0           0  5.02s           2023-04-13T05:11:18
/opt/airflow/dags/data/                                                                                         0           1  5.01s           2023-04-13T05:13:13
/opt/airflow/dags/data/                                                                                         0           0  5.61s           2023-04-13T05:14:38
/opt/airflow/dags/data/                                                                                         0           1  5.07s           2023-04-13T05:13:18
/opt/airflow/dags/data/                                                                                         0           1  5.05s           2023-04-13T05:13:38
/opt/airflow/dags/data/                                                                                         0           1  4.99s           2023-04-13T05:13:43
/opt/airflow/dags/data/                                                                                         0           1  8.10s           2023-04-13T05:13:51
/opt/airflow/dags/data/                                                                                         0           1  4.99s           2023-04-13T05:13:23
/opt/airflow/dags/data/                                                                                         1           0  18.92s          2023-04-13T05:14:43
/opt/airflow/dags/data/                                                                                         1           0  5.09s           2023-04-13T05:14:17
/opt/airflow/dags/data/                                                                                         1           0  5.08s           2023-04-13T05:14:17
/opt/airflow/dags/data/                                                                                         1           0  5.06s           2023-04-13T05:13:33
/opt/airflow/dags/data/                                                                                         1           0  17.68s          2023-04-13T05:14:06
/opt/airflow/dags/data/                                                                                         1           0  5.14s           2023-04-13T05:12:43
/opt/airflow/dags/data/                                                                                         1           0  4.92s           2023-04-13T05:12:48
/opt/airflow/dags/data/                                                                                         1           0  5.02s           2023-04-13T05:13:08
/opt/airflow/dags/data/                                                                                         1           0  5.09s           2023-04-13T05:12:53
/opt/airflow/dags/data/                                                                                         1           0  5.02s           2023-04-13T05:12:38
/opt/airflow/dags/data/                                                                                         1           0  5.01s           2023-04-13T05:12:33
/opt/airflow/dags/data/                                                                                         1           0  5.02s           2023-04-13T05:13:03
/opt/airflow/dags/data/                                                                                         1           0  5.01s           2023-04-13T05:12:58
/opt/airflow/dags/data/                                                                                         1           0  5.05s           2023-04-13T05:15:23
/opt/airflow/dags/data/                                                                                         1           0  4.99s           2023-04-13T05:15:28
/opt/airflow/dags/data/                                                                  185  0.02s             1           0  5.03s           2023-04-13T05:12:13
/opt/airflow/dags/data/                                                                                         1           0  5.09s           2023-04-13T05:15:08
/opt/airflow/dags/data/                                                                                         1           0  5.04s           2023-04-13T05:12:18
/opt/airflow/dags/data/                                                                                         1           0  4.99s           2023-04-13T05:15:38
/opt/airflow/dags/data/                                                                                         1           0  5.06s           2023-04-13T05:15:13
/opt/airflow/dags/data/                                                                                         1           0  5.01s           2023-04-13T05:12:23
/opt/airflow/dags/data/                                                                                         1           0  5.03s           2023-04-13T05:12:28
/opt/airflow/dags/data/                                                                                         1           0  5.12s           2023-04-13T05:15:03
/opt/airflow/dags/data/                                                                                         1           0  5.05s           2023-04-13T05:15:43
/opt/airflow/dags/data/                                                                                         1           0  4.99s           2023-04-13T05:15:18
/opt/airflow/dags/data/                                                                                         1           0  5.05s           2023-04-13T05:15:33
/opt/airflow/dags/data/                                                                                         1           0  4.91s           2023-04-13T05:14:58
/opt/airflow/dags/data/                                                                                         1           0  9.75s           2023-04-13T05:14:06
/opt/airflow/dags/data/                                                                                         1           0  5.02s           2023-04-13T05:13:28
================================================================================
[2023-04-13T05:15:43.853+0000] {settings.py:408} DEBUG - Disposing DB connection pool (PID 185)
[2023-04-13T05:15:48.616+0000] {base_job.py:240} DEBUG - [heartbeat]
[2023-04-13T05:15:48.617+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T05:15:48.618+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T05:15:48.645+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3
[2023-04-13T05:15:48.657+0000] {manager.py:1055} DEBUG - Started a process (PID: 186) to generate tasks for /opt/airflow/dags/data/B.py
[2023-04-13T05:15:48.658+0000] {manager.py:979} DEBUG - Processor for /opt/airflow/dags/data/A.py finished
[2023-04-13T05:15:48.660+0000] {processor.py:296} DEBUG - Waiting for <ForkProcess name='DagFileProcessor92-Process' pid=185 parent=6 stopped exitcode=0>
[2023-04-13T05:15:48.661+0000] {manager.py:1019} DEBUG - 1/2 DAG parsing processes running
[2023-04-13T05:15:48.661+0000] {manager.py:1021} DEBUG - 28 file paths queued for processing
[2023-04-13T05:15:48.974+0000] {settings.py:408} DEBUG - Disposing DB connection pool (PID 186)
[2023-04-13T05:15:53.671+0000] {base_job.py:240} DEBUG - [heartbeat]
[2023-04-13T05:15:53.684+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T05:15:53.686+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T05:15:53.708+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3
[2023-04-13T05:15:53.742+0000] {manager.py:1055} DEBUG - Started a process (PID: 187) to generate tasks for /opt/airflow/dags/data/C.py
[2023-04-13T05:15:53.757+0000] {manager.py:979} DEBUG - Processor for /opt/airflow/dags/data/B.py finished
[2023-04-13T05:15:53.758+0000] {processor.py:296} DEBUG - Waiting for <ForkProcess name='DagFileProcessor93-Process' pid=186 parent=6 stopped exitcode=0>
[2023-04-13T05:15:53.762+0000] {manager.py:1019} DEBUG - 1/2 DAG parsing processes running
[2023-04-13T05:15:53.762+0000] {manager.py:1021} DEBUG - 27 file paths queued for processing
[2023-04-13T05:15:54.682+0000] {settings.py:408} DEBUG - Disposing DB connection pool (PID 187)
[2023-04-13T05:15:58.656+0000] {base_job.py:240} DEBUG - [heartbeat]
[2023-04-13T05:15:58.657+0000] {retries.py:80} DEBUG - Running DagFileProcessorManager._fetch_callbacks_with_retries with retries. Try 1 of 3
[2023-04-13T05:15:58.658+0000] {manager.py:680} DEBUG - Fetching callbacks from the database.
[2023-04-13T05:15:58.664+0000] {retries.py:80} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3

graph

Unfortunately, I don't see any major changes that I can think of. Immediately after deployment, the value changed dramatically, but after an hour or so, it was the same as before.

I'm of the same opinion. It's unfortunate that I can't drill down and see the metrics, but I was suspecting that something in the retry logic was adding multiple behaviors.

Unfortunately I couldn't get the database logs, and based on the dag processor debug logs alone, I'm wondering if there's something wrong with the retry logic as well. Please correct me if I'm misinterpreting the log files.

By any chance, could you explain what callbacks are involved in the DAG?

And here I see that the @provide_session and @retry_db_transaction decorators are used in separate functions, but it's still possible to call a function with the @retry_db_transaction decorator within a function with the @provide_session decorator.
Is there anything going on here that could be causing the problem?

[argibbs]

@wookiist Do you use SLAs?

Edit:
nvm - I was looking at your logs and thinking it reminded me of problems that I've seen in the past with callback overload (in my case SLAs, but not nec yours). But re-reading through the bug some more I realise that's a side-effect of the problem.

As you were.

[wookiist]

Thanks. @argibbs!
If a side effect of the problem is an increase in callback time, what would you define as a problem?
I didn't see any SLAs in the helm template, could you share what they are?. 🙇🏼
Also, if you have solved it, it would be great if you could share how you solved it.

[potiuk]

By any chance, could you explain what callbacks are involved in the DAG?

Sorry for letting this slip a bit -> the callbacks are on_failure/on_success and sla_miss callbacks mainly.

I still cannot see though how the change you mentioned could impact it. Retry logic does not seem to be triggered in your case. Are you absolutely sure that just cherry-picking that one change is causing those changes?

[wookiist]

@potiuk Yes, I can guarantee that we only cherrypicked that commit.
I'll test with 2.5.3 without that commit, and then we can see what goes on!
I'll share the result

[potiuk]

Thanks. You can also test 2.6.0rc1 which has been just released and see if you see the same problem there.