Support for Datasource and Column comments

[john-bodley]

Description

Many databases (Hive, MySQL, Postgres, SQLite, etc.) support column and table (datasource) comments which help to provide additional context. I was wondering whether there were any plans to provide similar functionality in Druid?

[vogievetsky]

OMG yes, a million times yes. There is definitely some overhead with column comments as they would be propagated with each segment but it would enable some great abilities things with features like #7572 where a high level comment could be assigned to the auto created columns to later better be able to 're-inflate' the spec for repeated ingestion.

[himanshug]

For consideration of implementors: At the lowest storage level of segment , there is provision of storing "segment level metadata" which currently has "aggregators" and "granularity" used while indexing to generate that segment, these are used to power "queryGranularity" and "aggregators" analysis types in segmentMetadata query. Technically speaking, any information could be stored there. So, if we decide to store comments at segment level then that is the place.

Or, we could think about something drastically different e.g. having concept of "dataSource metadata" stored in metadataStore for these kind of things.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.