(Enhancement)[load-json] support simdjson in new json reader #16903

eldenmoon · 2023-02-18T11:53:39Z

Proposed changes

be config:
enable_simdjson_reader=true

related PR #11665
related isue #11663

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

be config: enable_simdjson_reader=true related PR apache#11665

github-actions · 2023-02-18T11:56:55Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T11:58:27Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T12:06:55Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T12:19:28Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T12:38:21Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-02-18T13:22:08Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.44 seconds
stream load tsv: 467 seconds loaded 74807831229 Bytes, about 152 MB/s
stream load json: 36 seconds loaded 2358488459 Bytes, about 62 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230220121041_clickbench_pr_100245.html

github-actions · 2023-02-18T14:41:49Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T14:44:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T15:11:20Z

clang-tidy review says "All clean, LGTM! 👍"

be/src/vec/exec/format/json/new_json_reader.cpp

xiaokang · 2023-02-19T13:59:05Z

be/src/exprs/json_functions.cpp

+                                  fmt::format("failed to access field as array, field: {}", col));
+
+            HANDLE_SIMDJSON_ERROR(
+                    arr.at(index).get(tvalue),


Can simdjson properly process index == -2, which means * ?

no index == -2 is not supported in this function for now

xiaokang · 2023-02-19T14:00:42Z

be/src/exprs/json_functions.cpp

+        if (UNLIKELY(_err)) {                                                                  \
+            if (_err == simdjson::NO_SUCH_FIELD || _err == simdjson::INDEX_OUT_OF_BOUNDS) {    \
+                return Status::NotFound(                                                       \
+                        fmt::format("err: {}, msg: {}", simdjson::error_message(_err), _msg)); \


err: to simdjson err:

what's the meaning of err: to simdjson err:?

be/src/exprs/json_functions.cpp

be/src/vec/exec/format/json/new_json_reader.cpp

be/src/vec/exec/format/json/new_json_reader.h

be/src/vec/exec/format/json/new_json_reader.cpp

eldenmoon · 2023-02-19T15:02:16Z

be/src/exprs/json_functions.cpp

+                                  fmt::format("failed to access field as array, field: {}", col));
+
+            HANDLE_SIMDJSON_ERROR(
+                    arr.at(index).get(tvalue),


no index == -2 is not supported in this function for now

eldenmoon · 2023-02-19T15:03:07Z

be/src/exprs/json_functions.cpp

+        if (UNLIKELY(_err)) {                                                                  \
+            if (_err == simdjson::NO_SUCH_FIELD || _err == simdjson::INDEX_OUT_OF_BOUNDS) {    \
+                return Status::NotFound(                                                       \
+                        fmt::format("err: {}, msg: {}", simdjson::error_message(_err), _msg)); \


what's the meaning of err: to simdjson err:?

be/src/vec/exec/format/json/new_json_reader.cpp

be/src/exprs/json_functions.cpp

github-actions · 2023-02-20T05:14:01Z

clang-tidy review says "All clean, LGTM! 👍"

eldenmoon · 2023-02-20T06:11:56Z

run buildall

eldenmoon · 2023-02-20T07:13:45Z

run p0

xiaokang

LGTM

be/src/exprs/json_functions.cpp

xiaokang · 2023-02-20T08:55:24Z

be/src/exprs/json_functions.cpp

+                                  fmt::format("failed to access field as array, field: {}", col));
+
+            HANDLE_SIMDJSON_ERROR(
+                    arr.at(index).get(tvalue),


github-actions · 2023-02-20T08:56:20Z

PR approved by anyone and no changes requested.

github-actions · 2023-02-20T10:35:38Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-20T10:50:46Z

clang-tidy review says "All clean, LGTM! 👍"

eldenmoon · 2023-02-20T11:02:32Z

run buildall

github-actions · 2023-02-20T11:06:05Z

clang-tidy review says "All clean, LGTM! 👍"

qidaye

LGTM

github-actions · 2023-02-20T12:30:16Z

PR approved by at least one committer and no changes requested.

WinkerDu · 2023-02-24T17:31:09Z

@eldenmoon This patch would cause BE compile error on MacOS with clang below

Apple clang version 13.0.0 (clang-1300.0.27.3)
Target: arm64-apple-darwin21.3.0
Thread model: posix

…16903) be config: enable_simdjson_reader=true related PR apache#11665

* (Enhancement)[load-json] support simdjson in new json reader (apache#16903) be config: enable_simdjson_reader=true related PR apache#11665 * [Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. (apache#17124) * [Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. `_simdjson_set_column_value` could become a hot spot while parsing json in simdjson mode, introduce `_prev_positions` to cache results for previous row (keyed as index in JSON object) due to the json name field order, should be quite the same between each lines * fix case

… (apache#1445) * (Enhancement)[load-json] support simdjson in new json reader (apache#16903) be config: enable_simdjson_reader=true related PR apache#11665 * [Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. (apache#17124) * [Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. `_simdjson_set_column_value` could become a hot spot while parsing json in simdjson mode, introduce `_prev_positions` to cache results for previous row (keyed as index in JSON object) due to the json name field order, should be quite the same between each lines * fix case

eldenmoon changed the title ~~Simdjson~~ (Enhancement)[load-json] support simdjson in new json reader Feb 18, 2023

github-actions bot added area/vectorization kind/test labels Feb 18, 2023

(Enhancement)[load-json] support simdjson in new json reader

9216cc6

be config: enable_simdjson_reader=true related PR apache#11665

eldenmoon force-pushed the simdjson branch from f14a984 to 9216cc6 Compare February 18, 2023 11:55

github-actions bot added the kind/docs Categorizes issue or PR as related to documentation. label Feb 18, 2023

add config doc

0ec87c3

eldenmoon force-pushed the simdjson branch from 0081c6b to 0ec87c3 Compare February 18, 2023 12:15

fix ut compile

998495d

add more malformed json cases

9dd6031

eldenmoon force-pushed the simdjson branch from 18c8e3f to a2bfcbc Compare February 18, 2023 15:07

set enable_simdjson_reader fuzzy

3397b89

xiaokang reviewed Feb 19, 2023

View reviewed changes

eldenmoon commented Feb 19, 2023

View reviewed changes

add more cases and fix bug

ac8fc37

eldenmoon force-pushed the simdjson branch from a2bfcbc to ac8fc37 Compare February 20, 2023 05:10

eldenmoon requested a review from xiaokang February 20, 2023 05:16

xiaokang previously approved these changes Feb 20, 2023

View reviewed changes

github-actions bot added the reviewed label Feb 20, 2023

eldenmoon dismissed xiaokang’s stale review via 9d23118 February 20, 2023 10:31

eldenmoon force-pushed the simdjson branch from 9d23118 to 62104a8 Compare February 20, 2023 10:47

modify

4034e06

eldenmoon force-pushed the simdjson branch from 62104a8 to 4034e06 Compare February 20, 2023 11:00

eldenmoon requested a review from xiaokang February 20, 2023 11:27

qidaye approved these changes Feb 20, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 20, 2023

xiaokang approved these changes Feb 20, 2023

View reviewed changes

qidaye merged commit 113023f into apache:master Feb 21, 2023

yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 9, 2023

(Enhancement)[load-json] support simdjson in new json reader (apache#…

df2bc5d

…16903) be config: enable_simdjson_reader=true related PR apache#11665

(Enhancement)[load-json] support simdjson in new json reader #16903

(Enhancement)[load-json] support simdjson in new json reader #16903

Uh oh!

Conversation

eldenmoon commented Feb 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

github-actions bot commented Feb 18, 2023

Uh oh!

github-actions bot commented Feb 18, 2023

Uh oh!

github-actions bot commented Feb 18, 2023

Uh oh!

github-actions bot commented Feb 18, 2023

Uh oh!

github-actions bot commented Feb 18, 2023

Uh oh!

hello-stephen commented Feb 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 18, 2023

Uh oh!

github-actions bot commented Feb 18, 2023

Uh oh!

github-actions bot commented Feb 18, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 20, 2023

Uh oh!

eldenmoon commented Feb 20, 2023

Uh oh!

eldenmoon commented Feb 20, 2023

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 20, 2023

Uh oh!

github-actions bot commented Feb 20, 2023

Uh oh!

github-actions bot commented Feb 20, 2023

Uh oh!

eldenmoon commented Feb 20, 2023

Uh oh!

eldenmoon commented Feb 18, 2023 •

edited

Loading

hello-stephen commented Feb 18, 2023 •

edited

Loading