Conversation
…e merge result by modifying the contents.
Return []Changes and a "closer" func to return them to Pool. Update tests for LoadChanges new return values.
…tring entails lots of temporary allocations.
…Mutable treap recycle nodes consistently. Rework Immutable treap node recycling attempting to make it safer in the presence of code that takes snapshots (dbCacheSnapshot) of the treap. Add special mutable PutM and DeleteM methods which DB transaction can use to apply changes more efficiently without creating lots of garbage memory.
Pull Request Test Coverage Report for Build 2430460733
💛 - Coveralls |
|
Good analysis and attempt :-) You've managed to group the changes into logical commits, let's break them down into separate PRs as well.
Thanks! |
|
Syncing from 0. Killed after 1 hour wall clock time. |
|
Syncing from 984287. Killed after 1 hour wall clock time. before.log.txt |
|
node Clone() is a major new source of temporary allocations after 8f95946 It's most noticeable when syncing from 984287. |
The chain got much busier after 800K. before: 4.5GB memory at 1,018,896 I'm running full sync to the tip, which should give us even more noticeable numbers. We'll follow up in #46 |
Are you basing this on: I was not trying to reduce RSS or active heap memory. Just reduce temp allocations that become garbage. I have the before/after.mem.heap files, though I only uploaded before/after.mem.allocs. |
|
Closing this as the changes were unbundled into individual pull requests. |
4 participants
Based on work by @BrannonKing in the mem_pressure_try_2 and reduce_memory_pressure branches...
I think the most impactful thing is reducing time for "Building the entire claim trie in RAM" phase. It's CPU intensive, so time spent in GC reduces performance.
Second thing is allowing the GOGC= environment variable to take effect overriding the hard-coded value (10%). Ten percent is quite low, and leads to constant GC activity. Opening up GOGC= allows tradeoffs to be made.
I don't think reducing GC burden accelerates the overall sync process. (I had hoped it might...) But other things (network/disk?) are limiting as long as there's enough CPU available. I tried running "env GOGC=400 ./lbcd" and although CPU use is low, sync doesn't go faster. :-( Perhaps the way to accelerate sync would be syncing blocks from multiple peers.
Finally, I am not entirely comfortable with the Mutable/Immutable treap changes in database/... It's my best attempt to combine original Immutable treap semantics with the way that ffldb/transaction actually wants to use it. I was very successful in reducing allocations, but I'm still not sure it's safe. It could also be fragile if treap is used in different ways in the future.
Test data forthcoming...