Web console: improve make type preservation in ingestion configs in the data loader#10533
Merged
clintropolis merged 27 commits intoapache:masterfrom Oct 31, 2020
Merged
Web console: improve make type preservation in ingestion configs in the data loader#10533clintropolis merged 27 commits intoapache:masterfrom
clintropolis merged 27 commits intoapache:masterfrom
Conversation
Comment on lines
+116
to
+122
| defined: (p: InputFormat) => oneOf(p.type, 'parquet', 'orc', 'avro_ocf'), | ||
| info: ( | ||
| <> | ||
| Specifies if the bytes parquet column which is not logically marked as a string or enum type | ||
| should be treated as a UTF-8 encoded string. | ||
| </> | ||
| ), |
Member
There was a problem hiding this comment.
does this info text always display 'parquet' column even when shown for orc and avro_ocf?
Contributor
Author
There was a problem hiding this comment.
yup, that should be fixed
clintropolis
approved these changes
Oct 30, 2020
JulianJaffePinterest
pushed a commit
to JulianJaffePinterest/druid
that referenced
this pull request
Jan 22, 2021
…he data loader (apache#10533) * improve validation * move to druid modals * adjust specs * oneOf * move transform * segment grans * tidy up webpack * add Partitioning * break out druid models * tidy up * rename to Expression * move druid time * cleanup * update format * better detail * fix e2e test * remove forceGuaranteedRollup from e2e tests * clean up imports * shardSpec -> shard_spec * fix css * adjust snapshot * add granularity to segments view * rename to Time span * use OFFSET in segments view query * update doc links * simplify require * more oneOf
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes an issue that in the data loader if you toggle the rollup it loses any previously entered type information because it will simply rescan the sample. It makes a couple of other fixes also.
This work was done as a result of 2 user studies where people loaded data into Druid via the data loader and then user a couple of other parts of the console.
This PR also refactors all the druid concepts into the
druid-modalsdirectory and breaks them up a bit so they are not just huge files (bad code smell).In addition to that this PR also:
Partitioningcolumn (derived fromshardSpec) to the segment view (see screenshot)Segment granularitycolumn to the datasource view (see screenshot)JsonInputto let it surface higher level errors__timetransform in the parse time step of the data loadertuningConfig.partitionsSpec.typefrom theEdit specstep of the data loaderoneOfutility function to makeAutoFormdefinitions more succinctPartitioning column
Segment granularity column