Web console: improve SQL autocomplete and add JSON autocomplete by vogievetsky · Pull Request #18126 · apache/druid

vogievetsky · 2025-06-11T16:52:24Z

Fixes #17628

Made the SQL autocomplete much smarter

(note how it is not trying to auto-complete functions and other nonsense)

Also added autocomplete to JSON native queries and to all the JSON entry fields

clintropolis

oh man, that is a lot of API to review, i'll be real here and admit that so far i just scanned it and picked on the things that stuck out.

what are we going for here, is the thinking that mostly correct is better than nothing? or should we try to review this more closely to ensure it is accurate? what is the plan for maintenance, should PR authors update this when they update stuff? I don't have many strong opinions either way yet, just some thoughts that come to mind after looking over this.

All that said, this is pretty cool though 😎

clintropolis · 2025-06-18T07:52:31Z

+    isObject: true,
+    condition: obj => obj.type === 'index_parallel' || obj.type === 'index',
+    completions: [
+      { value: 'firehose', documentation: 'Legacy data input (deprecated)' },


this is gone, not just deprecated

clintropolis · 2025-06-18T07:56:21Z

+    completions: [
+      { value: 'uris', documentation: 'List of HTTP/HTTPS URIs' },
+      { value: 'httpAuthenticationUsername', documentation: 'HTTP authentication username' },
+      { value: 'httpAuthenticationPassword', documentation: 'HTTP authentication password' },


there are a couple missing ones here, systemFields and requestHeaders though the latter isn't documented either

I'll add systemFields and will add requestHeaders when it is documented

clintropolis · 2025-06-18T07:58:58Z

+  {
+    path: '$',
+    isObject: true,
+    condition: obj => obj.queryType !== 'dataSourceMetadata',


afaik, everything has a context and intervals, even dataSourceMetadata

Huh, it is not documented here: https://druid.apache.org/docs/latest/querying/datasourcemetadataquery/ I even double cheched this one. It kind of makes sense for dataSourceMetadata not to support intervals since it is reporting back the max ingested time.

clintropolis · 2025-06-18T08:05:52Z

+  {
+    path: '$.filter.type',
+    completions: [
+      { value: 'selector', documentation: 'Matches a specific dimension value' },


this is missing the range filter and typed in filter (which i just realized isn't documented yet, from #16039)

I am sticking only to the documented stuff for now

clintropolis · 2025-06-18T08:10:09Z

+        value: 'populateResultLevelCache',
+        documentation: 'Whether to populate result level cache',
+      },
+      { value: 'bySegment', documentation: 'Return results by segment' },


does this still break the results if you actually set it?

bySegment: true should work :-)

clintropolis · 2025-06-18T08:16:07Z

+ * For automatic compaction, set the type to autocompact."
+ */
+function isSupervisorType(type: string): boolean {
+  return type === 'kafka' || type === 'kinesis' || type === 'rabbit' || type === 'autocompact';


missing the scheduled batch supervisor, #17353

clintropolis · 2025-06-18T08:27:39Z

+      { value: 'timestampSpec', documentation: 'Timestamp specification of data in segments' },
+      { value: 'queryGranularity', documentation: 'Query granularity of data in segments' },
+      { value: 'aggregators', documentation: 'List of aggregators usable for metric columns' },
+      { value: 'rollup', documentation: 'Whether the segments are rolled up' },


#18119 recently added a projection analysis type

clintropolis · 2025-06-18T08:30:08Z

+export const COMPACTION_CONFIG_COMPLETIONS: JsonCompletionRule[] = [
+  // Root level properties
+  {
+    path: '$',


this isn't really correct, after #17803 there is a "type" for a datasource compaction config, with the old spec type being "inline" and the one added in that PR is "catalog"

I'll add that when the docs are up

vogievetsky · 2025-06-20T19:52:04Z

Thank you for the review and feedback @clintropolis in general I have taken the approach of only going after the documented properties. I also think of these JSON completions as best effort. It is ok for them to miss stuff. Even if they are totally wrong it is not a blocker (even though it is best to fix) as the completions are purely cosmetic.

As for updating I expect the completions to be updated like like the properties for rendering all the AutoForms. Ideally done by the dev adding a new property in their PR but more often done after the fact once that property is exposed and documented

clintropolis

approving because i think this does more help than harm, but I still am a bit wary about long term maintainability. In its current state it seems like maybe we just have to like periodically ask claude (or some similar bot) to refresh it to match the docs? It would be really nice to like consolidate the common definitions across the completion files somehow so that human devs could potentially proactively add or adjust stuff here, but not a blocker.

Still, neat overall 🤘

clintropolis · 2025-06-27T00:33:40Z

+  'DOW',
+  'DOW',


do dupes matter? also this doesn't work in druid (and a bunch of others in this list), though I guess calcite recognizes it since i see it in the parser stuff 🤷

clintropolis · 2025-06-27T00:38:36Z

+    path: '$.spec.dataSchema.dimensionsSpec.useSchemaDiscovery',
+    completions: [
+      { value: 'true', documentation: 'Enable automatic type detection and schema discovery' },
+      { value: 'false', documentation: 'Use manual dimension specification (default)' },


this is perhaps misleading since you can still discover dimensions by leaving the list empty OR by specifying includeAllDimension as true.. though maybe that makes it even more confusing..

clintropolis · 2025-06-27T00:39:35Z

+      { value: 'true', documentation: 'Create bitmap index (default, faster filtering)' },
+      { value: 'false', documentation: 'No bitmap index (saves storage)' },


should we indicate that this only works for string type dimensions?

clintropolis · 2025-06-27T00:39:58Z

+      { value: 'sorted_array', documentation: 'Sort multi-values (default)' },
+      { value: 'sorted_set', documentation: 'Sort and deduplicate multi-values' },
+      { value: 'array', documentation: 'Keep multi-values as-is' },


should we mention this only works for string type dimensions?

clintropolis · 2025-06-27T00:42:59Z

+
+  // Aggregation types
+  {
+    path: '$.aggregations.[].type',


i've noticed this repeated quite a lot across the completions (and also some other common components like filters), is there a way to share it perhaps? fine to do as a follow-up, but sharing common elements across query, ingestion, compaction, etc would help make this a ton more maintainable

vogievetsky · 2025-07-01T17:23:29Z

I think the feedback all makes sense. I also think this PR makes things a lot better. In a follow up PR I will change this behaviour to use json-schema which is a standard and allows for recursive schema definition

vogievetsky added 30 commits June 9, 2025 16:04

complete less on '

3afa402

improve autocomplter

0a82a95

tests

5d1d61e

update sql input

4d93917

start hjson

6e6d616

failing test

4f1ee91

hjsonContext

2d490e7

better SQL completions

2d8a37d

completions work

a0442e3

add currentObject

8995fd4

make it work

b5cefe3

rename

8ff15a1

checkpoint

35330a7

query completion

030eddb

query completion

c3b1332

better docs

f665c6b

Refactor

c3dbf9a

Add retention rules

3cf575a

did compaction-dynamic-config-dialog

23da83a

did coordinator-dynamic-config-dialog

0986fbd

add tests

3880bde

add completions to index spec dialog

eb08882

add completions to overlord-dynamic-config-dialog

188f213

add completions to lookup-edit-dialog

f8713ab

add completions to compaction-config-dialog

554b9f5

add ingestion spec

d4cecae

fix bug in hjson parsing

1a02a02

more completions

05c2910

edit context should use JsonInput

9fc8139

fix edit context

3bf9e17

vogievetsky added 2 commits June 10, 2025 11:32

fix bug

8175e72

rewrite getHjsonContext

c20e804

vogievetsky added the Area - Web Console label Jun 11, 2025

github-advanced-security AI found potential problems Jun 11, 2025

View reviewed changes

Comment thread web-console/src/utils/hjson-context.ts Fixed

Comment thread web-console/src/utils/hjson-context.ts Fixed

vogievetsky added 6 commits June 11, 2025 14:35

cleanup

9b4d180

clever feedback fix

f0c2bf7

simpler

1d59819

hash comment support

4099c0b

prettier

edde6af

parse multi-line string also

d3643af

clintropolis reviewed Jun 18, 2025

View reviewed changes

vogievetsky added 2 commits June 20, 2025 12:34

feedback fixes

7d549df

add systemFields

06b48e8

vogievetsky added this to the 34.0.0 milestone Jun 24, 2025

vogievetsky added 2 commits June 25, 2025 09:16

Merge branch 'master' into fix_autocomplete

35af7be

fix typo in readme

4fb4422

clintropolis approved these changes Jun 27, 2025

View reviewed changes

vogievetsky added 2 commits June 27, 2025 10:58

remove duplicates

bd04b13

string only

ec40cc1

vogievetsky merged commit f46124b into apache:master Jul 1, 2025
10 checks passed

vogievetsky deleted the fix_autocomplete branch July 1, 2025 17:23

		{ value: 'true', documentation: 'Create bitmap index (default, faster filtering)' },
		{ value: 'false', documentation: 'No bitmap index (saves storage)' },

Conversation

vogievetsky commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

clintropolis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vogievetsky commented Jun 20, 2025

Uh oh!

clintropolis left a comment

Choose a reason for hiding this comment

Uh oh!

clintropolis Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vogievetsky commented Jul 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clintropolis left a comment •

edited

Loading

clintropolis Jun 27, 2025 •

edited

Loading