Skip to content

Web console: fix data loader schema table column ordering bug and other polish#10588

Merged
vogievetsky merged 13 commits intoapache:masterfrom
implydata:polish
Nov 17, 2020
Merged

Web console: fix data loader schema table column ordering bug and other polish#10588
vogievetsky merged 13 commits intoapache:masterfrom
implydata:polish

Conversation

@vogievetsky
Copy link
Copy Markdown
Contributor

@vogievetsky vogievetsky commented Nov 16, 2020 •
edited
Loading

This PR fixes the issue that the columns in the Schema table do not reflect the ordering in the dimensionsSpec (and the ordering in the final segment. This makes it impossible to configure single_dim partitioning effectively.

image

Also this PR contains a bunch of polish:

  • Allow column reordering in the Schema step
  • Default data loader flow to ingest with rollup: false
  • Added ability to AutoForm to fold things into a "Show more" folder
  • Fixed bug where totalNumMergeTasks, and maxNumSegmentsToMerge were not showing up in the form
  • Extracted all the data loader callout messages into their own file for easy editing
  • Fix issue where all the tables were constantly being re-mounted (losing scroll position / column sizing)
  • Misc improvements to data loader flow
  • Allow adding dimensions to dimensionExclusion list via point-and-click
  • Disallow submitting of an unfinished spec
  • Lower cased the segment granularities for aesthetics
  • Extracted dynamic configs to the druid-modals directory
  • Added repartitionTransitionDuration to the form - it was missing
  • Added helpful warnings for non newline delimited JSON

@vogievetsky vogievetsky changed the title Polish Web console: fix data loader schema table column ordering bug and other polish Nov 16, 2020
@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Nov 16, 2020

This pull request introduces 1 alert when merging c066b41 into 3447934 - view on LGTM.com

new alerts:

  • 1 for Syntax error

@vogievetsky vogievetsky merged commit 9964dd4 into apache:master Nov 17, 2020
@vogievetsky vogievetsky deleted the polish branch November 17, 2020 21:25
abhishekagarwal87 pushed a commit to abhishekagarwal87/druid that referenced this pull request Dec 14, 2020
@clintropolis @suneet-s
@capistrant @a2l007 @FrankChen021 @martin-g @vogievetsky @zhangyue19921010
ingest-service (#84)
5fbacec 
* scaffolding

* readme

* adjust

* more better, janky heap metadata store, primitive job queue that can submit to overlord, it works - sort of

* test scaffolding

* move InputFormat into IngestSchema

* imply-5135 Create & list ingest tables

* Addressed PR comments

* Removed bean IngestTable

* job processing + sql metadata job table (#78)

* Add indexed-table-loader (#65)

* Add indexed-table-loader

* Fix checkstyle

* Fix intelliJ inspections

* Fix analyze dependencies

* fix license check job

* Add imply-druid-security (#66)

* Add imply-druid-security

* fix checkstyle

* Fix analyze dependencies

* Fix license check job

* Update license header for all imply extensions

* fix intelliJ inspections

* code review

* modify access to protected SQLMetadataConnector methods to allow extensions to create SQL metadata tables using implementation specific constructs (payload type, serial type, etc) (apache#10573)

* Correct getRandomBalancerSegmentHolderTest (apache#10569)

* Add missing docs for timeout exceptions (apache#10554)

* Add missing docs for timeout exceptions

* Add info on auth failures

* Fix ingestion failure of pretty-formatted JSON message (apache#10383)

* support multi-line text

* add test cases

* split json text into lines case by case

* improve exception handle

* fix CI

* use IntermediateRowParsingReader as base of JsonReader

* update doc

* ignore the non-immutable field in test case

* add more test cases

* mark `lineSplittable` as final

* fix testcases

* fix doc

* add a test case for SqlReader

* return all raw columns when exception occurs

* fix CI

* fix test cases

* resolve review comments

* handle ParseException returned by index.add

* apply Iterables.getOnlyElement

* fix CI

* fix test cases

* improve code in more graceful way

* fix test cases

* fix test cases

* add a test case to check multiple json string in one text block

* fix inspection check

* Add TravisCI job that builds and tests on ARM64 CPU architecture (apache#10562)

* Ensure Krb auth before killing YARN apps in graceful shutdown (apache#9785)

* job processing + sql metadata

* Web console: fix data loader schema table column ordering bug and other polish (apache#10588)

* remove unused fields

* keep tables live

* advanced

* fix schema view

* better indication

* tests pass

* Show more instead of show advanced

* fix tests

* extract dynamic configs

* update snapshots

* fix issues

* update snapshot

* reword without >

* some javadoc

* modify druid.historical.cache.maxEntrySize property in Unified format (apache#10590)

Co-authored-by: yuezhang <yuezhang@freewheel.tv>

* Fix license header for imply extensions (#76)

* Fix license header for imply extensions

* arm64 packaging should use jdk8

* maybe this time

* jobs and states and status and whatever

* use indexing client and coordinator client instead of leader client

* always running

* simplify

* fix readme

* Add zero period support to TIMESTAMPADD (apache#10550)

* Allow zero period for TIMESTAMPADD

* update test cases

* add empty zone test case

* add unit test cases for TimestampShiftMacro

* add -Pimply-saas distribution profile, table exists check

* update readme

Co-authored-by: Suneet Saldanha <suneet.saldanha@imply.io>
Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Atul Mohan <atulmohan.mec@gmail.com>
Co-authored-by: frank chen <frank.chen021@outlook.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>
Co-authored-by: zhangyue19921010 <69956021+zhangyue19921010@users.noreply.github.com>
Co-authored-by: yuezhang <yuezhang@freewheel.tv>

* fix style and headers

* fix fails

* fix auth

Co-authored-by: Agustin Gonzalez <agustin.gonzalez@imply.io>
Co-authored-by: Suneet Saldanha <suneet.saldanha@imply.io>
Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Atul Mohan <atulmohan.mec@gmail.com>
Co-authored-by: frank chen <frank.chen021@outlook.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>
Co-authored-by: zhangyue19921010 <69956021+zhangyue19921010@users.noreply.github.com>
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
@jihoonson jihoonson added this to the 0.21.0 milestone Jan 4, 2021
@jihoonson jihoonson mentioned this pull request Jan 13, 2021
Closed
JulianJaffePinterest pushed a commit to JulianJaffePinterest/druid that referenced this pull request Jan 22, 2021
@vogievetsky @JulianJaffePinterest
Web console: fix data loader schema table column ordering bug and oth…
d781cef 
…er polish (apache#10588)

* remove unused fields

* keep tables live

* advanced

* fix schema view

* better indication

* tests pass

* Show more instead of show advanced

* fix tests

* extract dynamic configs

* update snapshots

* fix issues

* update snapshot

* reword without >
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@clintropolis clintropolis clintropolis approved these changes

Assignees

No one assigned

Labels

Area - Web Console Bug Release Notes

Projects

None yet

Milestone

0.21.0

Development

Successfully merging this pull request may close these issues.

3 participants

@vogievetsky @clintropolis @jihoonson