New Unified Console Unusable for Large Clusters (1000+ datasources, 10k+ segments)

[ofirnn]

After upgrading to 0.21.0 we noticed that the legacy UI (old-console) endpoints are not working anymore. our cluster is pretty big (thousands of data sources, a-lot of segments) - making the unified console in the router mode to not work.

Affected Version

0.21.0 (and all versions when legacy UI isn't working anymore)
The Druid version where the problem was encountered.

Description

When having:

• ~3500 data-sources
• ~20,000 segments

Router mode:

• Datasources page isn't working in router mode - the SQL query endpoint hangs
• Segments page isn't working in router mode - SQL hangs

Coordinator mode:

• Datasources page working well in coordinator mode- but without the segment timeline view
• Segments page crashing in coordinator mode - making thousands of requests - one per datasource - crashing chrome with insufficient resources

More:

• no way to filter prior query in both segments and datasources page
• no way to save a preset of filters with the responsive UI
• Ingestion page seems to work well even under high pressure of tasks (20-30k daily tasks)

All of the above worked well in the old-console - which we didn't find a way to enable in 0.21.0 - we saw a commit completely remove the binding so I believe there's no way to use it anymore.

[vogievetsky]

👋 Hi, I am the one you can blame for the new console. I want to understand your issues and make sure they are addressed.

This change #10909 I think will address most of your perf concerns. Sadly it did not make it into Apache Druid 0.21.0. It is available in the latest Imply distribution of Druid you could use that or build your own Druid.

Just to clarify no way to filter prior query in both segments and datasources page and no way to save a preset of filters with the responsive UI these are feature requests right? Not something that worked in the old console. Also can you clarify both point?

The legacy console was removed from the Druid codebase because it had security alerts for its dependancies that no one knew how to fix. It is still available here: https://github.com/druid-io/druid-console you can run it as a standalone server but I personally never could get it to work as it is using many npm dependancies that no longer exist etc.

If I was in your position I would either grab the latest Imply distro Druid. Or check out the master branch on this repo and run the dev mode of the web console (follow the readme here: https://github.com/apache/druid/tree/master/web-console#getting-started) pointing at your druid cluster like so: druid_host=your.router.node:8888 npm start to verify that the fix above works for you.

[ofirnn]

Hi @vogievetsky - Thanks for the quick reply!

re #10909 - sounds like it could help - we'll wait or try to play with our own built druid

Re the filtering/behavior change - the old console wasn't responsive so you could use add the datasource in the URL - to get the specific needed data like segments map - so it's a behavior that helped in large-scale environments -.

in the current behavior - we must wait for all segments to load (even after the optimizations branch) before having the ability to filter. another thing is the request per datasource to the backend in the segments page - which in the old console was only after digging into specific datasource - so not sure if its a feature request/behavior change caused the new approach to hang on large scale environments. I didn't mean for the "saving bookmarks" feature but just letting control the filters before querying the backend endpoints.

I'll take your suggestion to try the master-branch web-console.

[TheRum]

Hi @vogievetsky , I'm also seeing the same issue of very very slow loading of 'Datasources' tab. (Druid - 0.21.1)

Cluster has close to 30,000+ segments and 10+ datasources.

To my experience, not just 'DataSources', actually all the tabs are very slow to load compared to previous versions. And Sometimes it is just slow enough to give up.

[didip]

A hack to get around this is to use "forced overlord/coordinator" mode. But that only works until a certain point.

This query is what's breaking the UI.

SELECT datasource, COUNT(*) FILTER (WHERE (is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1) AS num_segments,
	COUNT(*) FILTER (WHERE is_available = 1 AND ((is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1)) AS num_available_segments,
	COUNT(*) FILTER (WHERE is_published = 1 AND is_overshadowed = 0 AND is_available = 0) AS num_segments_to_load,
	COUNT(*) FILTER (WHERE is_available = 1 AND NOT ((is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1)) AS num_segments_to_drop,
	COUNT(*) FILTER (WHERE ((is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1) AND "start" LIKE '%:00.000Z' AND "end" LIKE '%:00.000Z') AS minute_aligned_segments,
	COUNT(*) FILTER (WHERE ((is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1) AND "start" LIKE '%:00:00.000Z' AND "end" LIKE '%:00:00.000Z') AS hour_aligned_segments,
	COUNT(*) FILTER (WHERE ((is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1) AND "start" LIKE '%T00:00:00.000Z' AND "end" LIKE '%T00:00:00.000Z') AS day_aligned_segments,
	COUNT(*) FILTER (WHERE ((is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1) AND "start" LIKE '%-01T00:00:00.000Z' AND "end" LIKE '%-01T00:00:00.000Z') AS month_aligned_segments,
	COUNT(*) FILTER (WHERE ((is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1) AND "start" LIKE '%-01-01T00:00:00.000Z' AND "end" LIKE '%-01-01T00:00:00.000Z') AS year_aligned_segments,
	SUM("size") FILTER (WHERE is_published = 1 AND is_overshadowed = 0) AS total_data_size,
	SUM("size" * "num_replicas") FILTER (WHERE is_published = 1 AND is_overshadowed = 0) AS replicated_size,
	MIN("num_rows") FILTER (WHERE is_published = 1 AND is_overshadowed = 0) AS min_segment_rows,
	AVG("num_rows") FILTER (WHERE is_published = 1 AND is_overshadowed = 0) AS avg_segment_rows,
	MAX("num_rows") FILTER (WHERE is_published = 1 AND is_overshadowed = 0) AS max_segment_rows,
	SUM("num_rows") FILTER (WHERE (is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1) AS total_rows,
	CASE WHEN SUM("num_rows") FILTER (WHERE is_published = 1 AND is_overshadowed = 0) <> 0 THEN (
		SUM("size") FILTER (WHERE is_published = 1 AND is_overshadowed = 0) / SUM("num_rows") FILTER (WHERE is_published = 1 AND is_overshadowed = 0)
	) ELSE 0 END AS avg_row_size
	FROM sys.segments GROUP BY 1 ORDER BY 1