Skip to content

[Optimization](statistics) optimize Incremental statistics collection and statistics cleaning #18653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
weizhengte wants to merge 1 commit into from
Closed

[Optimization](statistics) optimize Incremental statistics collection and statistics cleaning #18653

weizhengte wants to merge 1 commit into from

Conversation

@weizhengte
Copy link
Contributor

@weizhengte weizhengte commented Apr 13, 2023 •
edited
Loading

Proposed changes

This pr mainly optimizes the following items:

  • the collection of statistics: clear up invalid historical statistics before collecting them, so as not to affect the final table statistics.
  • the incremental collection of statistics: in the case of incremental collection, only the corresponding partition statistics need to be collected.

TODO: Supports incremental collection of materialized view statistics.

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@weizhengte
Copy link
Contributor Author

weizhengte commented Apr 13, 2023

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 14, 2023 •
edited
Loading

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.84 seconds
stream load tsv: 440 seconds loaded 74807831229 Bytes, about 162 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 61 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230418004344_clickbench_pr_130377.html

Kikyou1997
Kikyou1997 reviewed Apr 14, 2023
View reviewed changes
Kikyou1997
Kikyou1997 suggested changes Apr 14, 2023
View reviewed changes
Copy link
Contributor

@Kikyou1997 Kikyou1997 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Further modifications and discussions are needed for this PR before it could be merged

@weizhengte weizhengte force-pushed the opt-drop_stats branch 2 times, most recently from d0cf7a1 to a3dc490 Compare April 16, 2023 11:13
@weizhengte
Copy link
Contributor Author

weizhengte commented Apr 16, 2023

run buildall

@weizhengte
Copy link
Contributor Author

weizhengte commented Apr 16, 2023

Further modifications and discussions are needed for this PR before it could be merged

I have removed the code that is not part of this pr~

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Apr 17, 2023
@weizhengte weizhengte changed the title [Optimization](statistics) optimize the problem of slow deletion of statistics . [Optimization](statistics) optimize Incremental statistics collection and statistics cleaning Apr 17, 2023
@weizhengte
Copy link
Contributor Author

weizhengte commented Apr 17, 2023

run buildall

Kikyou1997
Kikyou1997 previously approved these changes Apr 18, 2023
View reviewed changes
@github-actions
Copy link
Contributor

github-actions bot commented Apr 18, 2023

PR approved by anyone and no changes requested.

@weizhengte weizhengte closed this Apr 18, 2023
@weizhengte weizhengte reopened this Apr 18, 2023
@weizhengte weizhengte mentioned this pull request Apr 19, 2023
Closed
5 tasks
@weizhengte
Copy link
Contributor Author

weizhengte commented Apr 19, 2023 •
edited
Loading

Wait until the sampling statistics pr is comerged before modifying

morrySnow pushed a commit that referenced this pull request Apr 21, 2023
@weizhengte
[enchancement](statistics) support sampling collection of statistics (#…
1a6401d 
…18880)

1. Supports sampling to collect statistics
2. Improved syntax for collecting statistics
3. Support histogram specifies the number of buckets
4. Tweaked some code structure

---

The syntax supports WITH and PROPERTIES, using the same syntax as before.

Column Statistics Collection Syntax:
```SQL
ANALYZE [ SYNC ] TABLE table_name
     [ (column_name [, ...]) ]
     [ [WITH SYNC] | [WITH INCREMENTAL] | [WITH SAMPLE PERCENT | ROWS ] ]
     [ PROPERTIES ('key' = 'value', ...) ];
```

Column histogram collection syntax:
```SQL
ANALYZE [ SYNC ] TABLE table_name
     [ (column_name [, ...]) ]
     UPDATE HISTOGRAM
     [ [ WITH SYNC ][ WITH INCREMENTAL ][ WITH SAMPLE PERCENT | ROWS ][ WITH BUCKETS ] ]
     [ PROPERTIES ('key' = 'value', ...) ];
```

Illustrate：
- sync：Collect statistics synchronously. Return after collecting.
- incremental：Collect statistics incrementally. Incremental collection of histogram statistics is not supported.
- sample percent | rows：Collect statistics by sampling. Scale and number of rows can be sampled.
- buckets：Specifies the maximum number of buckets generated when collecting histogram statistics.
- table_name: The purpose table for collecting statistics. Can be of the form `db_name.table_name`.
- column_name: The specified destination column must be a column that exists in `table_name`, and multiple column names are separated by commas.
- properties：Properties used to set statistics tasks. Currently only the following configurations are supported (equivalent to the with statement)
   - 'sync' = 'true'
   - 'incremental' = 'true'
   - 'sample.percent' = '50'
   - 'sample.rows' = '1000'
   - 'num.buckets' = 10

--- 

TODO: 
- Supplement the complete p0 test
- `Incremental` statistics see #18653
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
@weizhengte @gnehil
[enchancement](statistics) support sampling collection of statistics (a…
760b258 
…pache#18880)

1. Supports sampling to collect statistics
2. Improved syntax for collecting statistics
3. Support histogram specifies the number of buckets
4. Tweaked some code structure

---

The syntax supports WITH and PROPERTIES, using the same syntax as before.

Column Statistics Collection Syntax:
```SQL
ANALYZE [ SYNC ] TABLE table_name
     [ (column_name [, ...]) ]
     [ [WITH SYNC] | [WITH INCREMENTAL] | [WITH SAMPLE PERCENT | ROWS ] ]
     [ PROPERTIES ('key' = 'value', ...) ];
```

Column histogram collection syntax:
```SQL
ANALYZE [ SYNC ] TABLE table_name
     [ (column_name [, ...]) ]
     UPDATE HISTOGRAM
     [ [ WITH SYNC ][ WITH INCREMENTAL ][ WITH SAMPLE PERCENT | ROWS ][ WITH BUCKETS ] ]
     [ PROPERTIES ('key' = 'value', ...) ];
```

Illustrate：
- sync：Collect statistics synchronously. Return after collecting.
- incremental：Collect statistics incrementally. Incremental collection of histogram statistics is not supported.
- sample percent | rows：Collect statistics by sampling. Scale and number of rows can be sampled.
- buckets：Specifies the maximum number of buckets generated when collecting histogram statistics.
- table_name: The purpose table for collecting statistics. Can be of the form `db_name.table_name`.
- column_name: The specified destination column must be a column that exists in `table_name`, and multiple column names are separated by commas.
- properties：Properties used to set statistics tasks. Currently only the following configurations are supported (equivalent to the with statement)
   - 'sync' = 'true'
   - 'incremental' = 'true'
   - 'sample.percent' = '50'
   - 'sample.rows' = '1000'
   - 'num.buckets' = 10

--- 

TODO: 
- Supplement the complete p0 test
- `Incremental` statistics see apache#18653
@weizhengte
Copy link
Contributor Author

weizhengte commented Apr 23, 2023

run buildall

@weizhengte
Copy link
Contributor Author

weizhengte commented Apr 23, 2023

run clickbench

@weizhengte weizhengte closed this Apr 23, 2023
Reminiscent pushed a commit to Reminiscent/doris that referenced this pull request May 15, 2023
@weizhengte @Reminiscent
[enchancement](statistics) support sampling collection of statistics (a…
0c20d68 
…pache#18880)

1. Supports sampling to collect statistics
2. Improved syntax for collecting statistics
3. Support histogram specifies the number of buckets
4. Tweaked some code structure

---

The syntax supports WITH and PROPERTIES, using the same syntax as before.

Column Statistics Collection Syntax:
```SQL
ANALYZE [ SYNC ] TABLE table_name
     [ (column_name [, ...]) ]
     [ [WITH SYNC] | [WITH INCREMENTAL] | [WITH SAMPLE PERCENT | ROWS ] ]
     [ PROPERTIES ('key' = 'value', ...) ];
```

Column histogram collection syntax:
```SQL
ANALYZE [ SYNC ] TABLE table_name
     [ (column_name [, ...]) ]
     UPDATE HISTOGRAM
     [ [ WITH SYNC ][ WITH INCREMENTAL ][ WITH SAMPLE PERCENT | ROWS ][ WITH BUCKETS ] ]
     [ PROPERTIES ('key' = 'value', ...) ];
```

Illustrate：
- sync：Collect statistics synchronously. Return after collecting.
- incremental：Collect statistics incrementally. Incremental collection of histogram statistics is not supported.
- sample percent | rows：Collect statistics by sampling. Scale and number of rows can be sampled.
- buckets：Specifies the maximum number of buckets generated when collecting histogram statistics.
- table_name: The purpose table for collecting statistics. Can be of the form `db_name.table_name`.
- column_name: The specified destination column must be a column that exists in `table_name`, and multiple column names are separated by commas.
- properties：Properties used to set statistics tasks. Currently only the following configurations are supported (equivalent to the with statement)
   - 'sync' = 'true'
   - 'incremental' = 'true'
   - 'sample.percent' = '50'
   - 'sample.rows' = '1000'
   - 'num.buckets' = 10

--- 

TODO: 
- Supplement the complete p0 test
- `Incremental` statistics see apache#18653
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@Kikyou1997 Kikyou1997 Kikyou1997 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/planner Issues or PRs related to the query planner kind/test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@weizhengte @hello-stephen @Kikyou1997