perf: memory perf, lazy read & stream read#321
Merged
Conversation
c-git
reviewed
May 13, 2026
| } | ||
|
|
||
| #[test] | ||
| #[ignore] |
Contributor
There was a problem hiding this comment.
Why are the test that were added ignored?
Contributor
Author
There was a problem hiding this comment.
I use env to run the special perf. So we don't need to run it every time
Contributor
There was a problem hiding this comment.
I see, thanks for explaining.
Owner
|
@attila-lin |
Owner
|
@attila-lin |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I encountered a fairly serious performance issue when processing large tables. In some cases, memory usage would continue growing and could even reach tens of gigabytes.
I used AI-assisted optimization to refactor the lazy-read path into a streaming-based approach. With this change, memory usage now stays at a much healthier level (around 50 MB in my test case).
That said, I have to admit I did not manually review every part of the generated code in depth.
If you notice any issues or have suggestions, please let me know and I’ll be happy to make further adjustments.