Data loader (GUI component)#7572
Conversation
|
Might change some copy soon but felt cute in this gif: https://imgur.com/a/nvzCs4V (too big for GH embed) |
vogievetsky
force-pushed
the
data-loader-gui
branch
from
db6702b to
b2bcd9c
Compare
| type: 'index', | ||
| spec: { | ||
| ioConfig: deepSet(ioConfig, 'type', 'index') | ||
| // dataSchema: { |
There was a problem hiding this comment.
why is this block commented?
There was a problem hiding this comment.
That will be commented in to support #7566
| tuningConfig?: TuningConfig; | ||
| } | ||
|
|
||
| export type IngestionType = 'kafka' | 'kinesis' | 'index_hadoop' | 'index' | 'index_parallel'; |
There was a problem hiding this comment.
It seems like it would be really useful in the future to figure out a way to populate this stuff based on the set of loaded extensions, but I'm not sure the best mechanism to do that at this point.
There was a problem hiding this comment.
Well that is just a type, so in the future it would probably become string. I was planning of proposing a system for extending the data loader to arbitrary extensions. So that every extension could declare a json blob (that I will propose) and then be able to 'register' itself with the data loader
| } | ||
| ]; | ||
|
|
||
| it('works for path, ignore-arrays', () => { |
| type: 'string', | ||
| suggestions: ['uniform', 'arbitrary'], | ||
| info: <> | ||
| This spec is used to generated segments with uniform intervals. |
There was a problem hiding this comment.
nit: this reads strangely, but it's just taken from the docs...
| type: 'number', | ||
| defaultValue: 20000000, | ||
| info: <> | ||
| Total number of rows in segments waiting for being pushed. |
There was a problem hiding this comment.
nit: same about reading a bit strange, but also copied from the docs...
There was a problem hiding this comment.
Yeah I was going to submit an issue about that part being a bit unclean in the druid ingestion spec.
|
|
||
| // copied from http://goo.gl/0ejHHW with small tweak to make dddd not pass on its own | ||
| // tslint:disable-next-line:max-line-length | ||
| export const ISO_MATCHER = new RegExp(/^([\+-]?\d{4}(?!\d{2}\b))((-?)((0[1-9]|1[0-2])(\3([12]\d|0[1-9]|3[01]))?|W([0-4]\d|5[0-2])(-?[1-7])?|(00[1-9]|0[1-9]\d|[12]\d{2}|3([0-5]\d|6[1-6])))(T((([01]\d|2[0-3])((:?)[0-5]\d)?|24:?00)([\.,]\d+(?!:))?)?(\17[0-5]\d([\.,]\d+)?)?([zZ]|([\+-])([01]\d|2[0-3]):?([0-5]\d)?)?)?)$/); |
There was a problem hiding this comment.
oh man, dates and times are just the worst 🙃
| "version": "7.3.4", | ||
| "resolved": "https://registry.npmjs.org/@babel/core/-/core-7.3.4.tgz", | ||
| "integrity": "sha512-jRsuseXBo9pN197KnDwhhaaBzyZr2oIcLHHTt2oDdQrej5Qp57dCCJafWx5ivU8/alEYDpssYqv1MUqcxwQlrA==", | ||
| "version": "7.4.3", |
There was a problem hiding this comment.
just leaving a comment to call out that we'll need to update versions in the license stuff for next release
4 participants
This PR implements the GUI part of #7502 and depends on #7531 being merged.
The data loader / spec editor is fully functional for
indexandindex_paralleltasks and with a few more changes and also #7566 being merged it will be able to handle streaming data also (but there is no need to wait for #7566).Since this PR required so much UI work a lot of other goodies in the console were updated to accommodate it:
AutoFormto be flexible enough for all the new use-cases in the codeSQLtoQueryso it is a verb)npm startscript APIHere is a screenshot of the Schema design page: