About KhaosDatabase Optimization

[xiaomudan778]

Fork chain selection mechanism is one of the most important feature of blockchain stability. During my code review of java-tron, I discovered that KhaosDatabase is used to store all non-solidified blocks. In local testing analysing KhaosDatabase capacity under non-solidification conditions (running a FullNode on MainNet with solidification logic commented out at updateSolidBlock ), I observed each block consumes ~1.2MB memory on average. Here is one KhaosBlock hump dump result:

At this consumption rate, KhaosDatabase alone would require at least 5.6GB(test observed) memory per maintenance period. Combined with SnapShot and other overhead, this could quickly lead to OOM under our recommended 9GB JVM configuration. It means in abnormal cases, the TRON network only have few hours to found the issue and fix it, or FullNodes have to be restarted to recover because of OOM.

Detailed memory analysis of KhaosDatabase variables reveals TransactionRetCapsule (shown in the image) accounts for at least 1/3 of memory usage. Code review shows this variable is only used to pass data to TransactionRetStore.

Want to ask community whether it is feasible to remove TransactionRetCapsule storage from KhaosDatabase. This optimisation could save ≥33% memory, making nodes more robust during abnormal conditions and improving recovery capability.

[halibobo1205]

@xiaomudan778

running a FullNode on MainNet with solidification logic commented out at updateSolidBlock

What's the rationale behind this design? Is it intended for testing? Theoretically, we shouldn't encounter such an extensive sequence of unsolidified block data.

[waynercheung]

Hi @xiaomudan778 , thanks for your analysis.

Actually, the numKblkMap of the KhaosStore implements the removeEldestEntry method to automatically delete elements with block heights less than the solidified block height (head.num - maxCapacity):

@Override 
protected boolean removeEldestEntry(Map.Entry<Long, ArrayList<KhaosBlock>> entry) { 
    long minNum = Long.max(0L, head.num - maxCapacity); 
    Map<Long, ArrayList<KhaosBlock>> minNumMap = numKblkMap.entrySet().stream() 
        .filter(e -> e.getKey() < minNum) 
        .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)); 
    minNumMap.forEach((k, v) -> { 
        numKblkMap.remove(k); 
        v.forEach(b -> { 
            hashKblkMap.remove(b.id); 
            logger.info("Remove from khaosDatabase: {}.", b.id); 
        }); 
    }); 
    return false; 
}

The maxCapacity is configured as follows:

khaosDb.setMaxSize((int) (chainBaseManager.getDynamicPropertiesStore().getLatestBlockHeaderNumber() 
    - chainBaseManager.getDynamicPropertiesStore().getLatestSolidifiedBlockNum() + 1));

So, in the normal condition, there seems to be no OOM risk.

And about remove TransactionRetCapsule storage from KhaosDatabase, I will do some research to check it.

[Sunny6889]

I think he is conducting the test for abnormal cases: when the network failed with more than 1/3 SR dis-connected. Checking how much time the current FullNode will become OOM when the transaction traffic keeps the same but blocks cannot be solidified. The rationale behind this is whether the current consensus logic leaves enough time buffer to let FullNode or SR recover from disasters.

[halibobo1205]

@xiaomudan778 @Sunny6889

Checking how much time the current FullNode will become OOM when the transaction traffic keeps the same but blocks cannot be solidified.

Please refer to this PR, which includes targeted optimizations to address this particular use case. : #5643

[xiaomudan778]

@halibobo1205 that is really helpful, seems don't need to worry about OOM caused by KhaosDatabase. Since when exceed the node.maxUnsolidifiedBlocks(default is 54), SR will only generate empty blocks, which will cost much less memory（～1KB）

[xiaomudan778]

@halibobo1205 But hope to hear some research result about remove TransactionRetCapsule variable from KhaosDatabase, as from the logic, this variable is not necessary to store there.

[halibobo1205]

Any further developments will be communicated here.

[Sunny6889]

@halibobo1205 I just found the latest code unsolidifiedBlockCheck default is false. Why change it to false? In this case KhaosDatabase will have OOM risk as transactions and block generation continues.

[halibobo1205]

about remove TransactionRetCapsule variable from KhaosDatabase:

The blocks in KhaosDatabase and the blocks being processed are references to the same object instances. If we want to remove TransactionRetCapsule from the block objects in KhaosDatabase, we would need to create new object instances, which would significantly increase memory consumption in scenarios where blocks cannot be solidified/finalized. @xiaomudan778

[halibobo1205]

@Sunny6889
Concerning why unsolidifiedBlockCheck defaults to false: I'm admittedly uncertain about the exact rationale for this configuration. I assume that new features are typically disabled by default initially and are only enabled as the default behavior after comprehensive testing and validation processes have been completed.

[Sunny6889]

@halibobo1205 As I tested on a FullNode, the memory consumption with 4677 unsolidited block is 5.57GB，check the image below.

Then beside KhaosDatabase, there are Chainbase costs total about 4GB. These 2 together is already takes 9.57GB.
To generate 4677 unsolidited block means at most 18 working SRs takes about 5hour 50mins, less than one maintenance cycle.

[xiaomudan778]

about remove TransactionRetCapsule variable from KhaosDatabase:

The blocks in KhaosDatabase and the blocks being processed are references to the same object instances. If we want to remove TransactionRetCapsule from the block objects in KhaosDatabase, we would need to create new object instances, which would significantly increase memory consumption in scenarios where blocks cannot be solidified/finalized. @xiaomudan778

As currently TransactionRetStore already store TransactionRetCapsule, why we need new object.

chainBaseManager.getTransactionRetStore()
          .put(ByteArray.fromLong(block.getNum()), block.getResult()); // here block is also stored in KhaosDatabase

[waynercheung]

Hi @xiaomudan778 @Sunny6889 , do you already have the solution to remove TransactionRetCapsule storage from KhaosDatabase?

As @halibobo1205 mentioned The blocks in KhaosDatabase and the blocks being processed are references to the same object instances. , you need to define a new block object which will also increase memory consumption in scenarios where blocks cannot be solidified/finalized.

Do you have any ideas about it?

[xiaomudan778]

Hi @xiaomudan778 @Sunny6889 , do you already have the solution to remove TransactionRetCapsule storage from KhaosDatabase?

As @halibobo1205 mentioned The blocks in KhaosDatabase and the blocks being processed are references to the same object instances. , you need to define a new block object which will also increase memory consumption in scenarios where blocks cannot be solidified/finalized.

Do you have any ideas about it?

Since The blocks in KhaosDatabase and the blocks being processed are references to the same object instances. , I agree there is no benefit in terms of memory saving for remove TransactionRetCapsule storage from KhaosDatabase. Thank you