Skip to content

Conversation

@weizhengte
Copy link
Contributor

@weizhengte weizhengte commented Apr 19, 2023

Proposed changes

  1. Supports sampling to collect statistics
  2. Improved syntax for collecting statistics
  3. Support histogram specifies the number of buckets
  4. Tweaked some code structure

The syntax supports WITH and PROPERTIES, using the same syntax as before.

Column Statistics Collection Syntax:

ANALYZE [ SYNC ] TABLE table_name
     [ (column_name [, ...]) ]
     [ [WITH SYNC] | [WITH INCREMENTAL] | [WITH SAMPLE PERCENT | ROWS ] ]
     [ PROPERTIES ('key' = 'value', ...) ];

Column histogram collection syntax:

ANALYZE [ SYNC ] TABLE table_name
     [ (column_name [, ...]) ]
     UPDATE HISTOGRAM
     [ [ WITH SYNC ][ WITH INCREMENTAL ][ WITH SAMPLE PERCENT | ROWS ][ WITH BUCKETS ] ]
     [ PROPERTIES ('key' = 'value', ...) ];

Illustrate:

  • sync:Collect statistics synchronously. Return after collecting.
  • incremental:Collect statistics incrementally. Incremental collection of histogram statistics is not supported.
  • sample percent | rows:Collect statistics by sampling. Scale and number of rows can be sampled.
  • buckets:Specifies the maximum number of buckets generated when collecting histogram statistics.
  • table_name: The purpose table for collecting statistics. Can be of the form db_name.table_name.
  • column_name: The specified destination column must be a column that exists in table_name, and multiple column names are separated by commas.
  • properties:Properties used to set statistics tasks. Currently only the following configurations are supported (equivalent to the with statement)
    • 'sync' = 'true'
    • 'incremental' = 'true'
    • 'sample.percent' = '50'
    • 'sample.rows' = '1000'
    • 'num.buckets' = 10

TODO:

  • Supplement the complete p0 test
  • Incremental statistics see pr18653

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Apr 19, 2023
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 19, 2023

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38 seconds
stream load tsv: 432 seconds loaded 74807831229 Bytes, about 165 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230419114607_clickbench_pr_131072.html

@weizhengte
Copy link
Contributor Author

run buildall

Kikyou1997
Kikyou1997 previously approved these changes Apr 20, 2023
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@weizhengte
Copy link
Contributor Author

run buildall

@weizhengte
Copy link
Contributor Author

run clickbench

@weizhengte
Copy link
Contributor Author

run p1

@weizhengte weizhengte closed this Apr 21, 2023
@weizhengte
Copy link
Contributor Author

This pr rebase has been done several times, and the pipeline clickbench has been stuck (Can't connect to MySQL), using the new pr #18880

@weizhengte weizhengte deleted the opt-sample-stats branch April 21, 2023 05:34
@weizhengte weizhengte changed the title [Enchancement](statistics) Support sampling collection of statistics [Enchancement](statistics) Support sampling collection of statistics(discard) Apr 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/nereids area/planner Issues or PRs related to the query planner kind/test reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants