Druid tasks UI becomes very slow when there are hundreds of tasks running at the same time.

[didip]

Affected Version

0.21.1

Description

when there are hundreds of subtasks running, Druid tasks UI became slow and sometimes threw gateway timeout error.

I have 10 routers running each has 10GB of RAM. In comparison, normal Druid queries are running decently fast.

[FrankChen021]

I think it's same as #11042 and #11140. As a workaround, you can set a smaller value on druid.indexer.storage.recentlyFinishedThreshold.

[didip]

Unfortunately, I have set druid.indexer.storage.recentlyFinishedThreshold to 1 day and it's still unresponsive when there are > 700 sub tasks in 1 day.

[didip]

Increasing Overlord heap to 64GB helped a little bit. But sometimes, I still get 504 error.

[didip]

For everyone here with the same problem, I ended up writing a custom cronjob that deletes completed druid_tasks where timestamp is older than current_time - 1 hour.

[vogievetsky]

I believe #12404 greatly improved the situation here. Please re-open if you see this post 24.0