-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[opt](cache) enhance cache key computation by removing comments and trimming SQL input #46099
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…rimming SQL input
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 32710 ms |
TPC-DS: Total hot run time: 196492 ms |
ClickBench: Total hot run time: 31.58 s |
924060929
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not cache the normalized sql in SqlCacheContext to skip normalize sql every times
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
…rimming SQL input (#46099) - Currently, the SQL cache system in Doris may miss cache hits due to semantically identical queries being treated as different because of: - Extra whitespace characters in the SQL query - SQL comments that don't affect the query execution - For example, these queries are semantically identical but would generate different cache keys: ```sql SELECT * FROM table; -- Same query with comments and extra spaces /* Comment */ SELECT * FROM table ; ``` - This PR improves the SQL cache hit rate by: - Trimming whitespace from SQL queries - Removing SQL comments before calculating the cache key MD5 - This ensures that queries that are semantically identical but differ only in whitespace or comments will now hit the same cache entry, improving cache efficiency and reducing unnecessary query executions
…rimming SQL input (#46099) - Currently, the SQL cache system in Doris may miss cache hits due to semantically identical queries being treated as different because of: - Extra whitespace characters in the SQL query - SQL comments that don't affect the query execution - For example, these queries are semantically identical but would generate different cache keys: ```sql SELECT * FROM table; -- Same query with comments and extra spaces /* Comment */ SELECT * FROM table ; ``` - This PR improves the SQL cache hit rate by: - Trimming whitespace from SQL queries - Removing SQL comments before calculating the cache key MD5 - This ensures that queries that are semantically identical but differ only in whitespace or comments will now hit the same cache entry, improving cache efficiency and reducing unnecessary query executions
…rimming SQL input (#46099) - Currently, the SQL cache system in Doris may miss cache hits due to semantically identical queries being treated as different because of: - Extra whitespace characters in the SQL query - SQL comments that don't affect the query execution - For example, these queries are semantically identical but would generate different cache keys: ```sql SELECT * FROM table; -- Same query with comments and extra spaces /* Comment */ SELECT * FROM table ; ``` - This PR improves the SQL cache hit rate by: - Trimming whitespace from SQL queries - Removing SQL comments before calculating the cache key MD5 - This ensures that queries that are semantically identical but differ only in whitespace or comments will now hit the same cache entry, improving cache efficiency and reducing unnecessary query executions
add test for #46099
add test for #46099
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)