Fullnode sync fails or stalls at block heights ending in 99 or 999 from official snapshot

[ZDCELL]

Hi TRON community,

We are consistently encountering a reproducible issue when syncing a Full Node using the official TRON snapshot.

---

🐛 Problem Summary

During sync, if the block height ends with 99 or 999 (e.g., 74450399, 2399999, etc.), the node:

• Either freezes completely (block number never increases), or
• Pauses syncing for 12–24 seconds before resuming

This significantly slows down or halts the sync process.

---

🔁 Confirmed Through Multiple Attempts

• We have tested multiple times by restoring from the same official snapshot
• Every attempt freezes at the exact same block number, particularly at block heights ending in 99 or 999
• Restarting systemctl restart tron does not help
• This confirms that it's not a random runtime issue, but likely something within the snapshot itself or the sync logic

---

🧪 Additional Observations

• Sync speed is normally ~3s/block
• Monitoring shows 6s polling via Prometheus
• At ...99 or ...999, sync often stalls for multiple cycles (12–24s+)
• If sync starts at a ...999 block (e.g. 2399999), sync doesn't start at all
• No errors appear in tron.log, gc.log, or system logs
• Peers are connected (activeConnectCount = 10), RocksDB meta files present and correct

---

🔧 Environment Details

• Java 8/11 with G1GC
• 64GB+ RAM, Ubuntu 20.04/22.04
• RocksDB CURRENT, MANIFEST-*, LOG all present in subdirectories
• Using the latest TRON snapshot
• Containerd, Prometheus, Node Exporter used for monitoring

Config contains:

sync {
  syncVerify = false
}
witness {
  validate-pbft-block = false
}

[barbatos2011]

Hi, could you please share the whole config.conf and the command to start the node?

[lxcmyf]

@ZDCELL
Is this the same issue as #6424? Please provide the startup command parameters and detailed asynchronous logs. Additionally, Java-tron currently does not support any other versions of JDK except for JDK 8.

[ZDCELL]

Thanks for the reply.

Yes, I believe the issue is similar to #6422 but in my case, the sync always halts at blocks ending with 99 or 999, sometimes immediately after startup.

Here are the details:

start..:
ExecStart=/usr/bin/java
-Xms64g
-Xmx64g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-Xloggc:/data/logs/gc.log
-Djava.io.tmpdir=/mnt/ssd4t/java_tmp
-jar /data/FullNode.jar
-c /data/config.conf

config.conf

net {
type = mainnet
activeConnectNumber = 5 //80
passiveConnectNumber = 100 //80
minActiveConnectNumber = 40
maxConnectNumber = 200
}

storage {
db.engine = "ROCKSDB"
db.sync = true
db.directory = "database"
index.directory = "index"
transHistory.switch = "on"

properties = [
]

needToUpdateAsset = true

dbSettings = {
levelNumber = 7
//compactThreads = 32
blocksize = 64 // n * KB
maxBytesForLevelBase = 256 // n * MB
maxBytesForLevelMultiplier = 10
level0FileNumCompactionTrigger = 4
targetFileSizeBase = 256 // n * MB
targetFileSizeMultiplier = 1

}

backup = {
enable = true
propPath = "prop.properties"
bak1path = "/data/bak1/database"
bak2path = "/mnt/ssd4t/bak2/database"
frequency = 600
}

balance.history.lookup = false
txCache.initOptimization = true
}

node.discovery = {
enable = true
persist = true
}

node.backup {
port = 10001
priority = 8
keepAliveInterval = 3000
members = [
]
}

crypto {
engine = "eckey"
}

node.metrics {
prometheus {
enable = true
port = 9527
path = "/metrics"
includeMetrics = ".*"
excludeMetrics = ""
}
}

node {
trustNode = "127.0.0.1:50051"
walletExtensionApi = true

listen.port = 18888
connection.timeout = 5
fetchBlock.timeout = 500

tcpNettyWorkThreadNum = 0
udpNettyWorkThreadNum = 1

maxConnections = 100
minConnections = 20
minActiveConnections = 10
maxConnectionsWithSameIp = 5
maxHttpConnectNumber = 50
minParticipationRate = 15

isOpenFullTcpDisconnect = false
inactiveThreshold = 600

p2p {
version = 11111
}

activeNodes = [
"34.224.96.251:18888",
"34.80.14.200:18888",
"34.80.47.162:18888",
"35.194.220.132:18888",
"52.53.189.99:18888",
"13.124.62.58:18888",
"54.252.224.209:18888",
"52.196.244.176:18888"
]

passive = [
]

fastForward = [
"100.27.171.62:18888",
"15.188.6.125:18888",
"3.225.171.164:18888",
"54.151.226.240:18888",
"35.174.93.198:18888",
"18.210.241.149:18888",
"52.196.244.176:18888",
"54.248.129.19:18888",
"35.197.17.205:18888",
"34.86.86.229:18888",
"35.247.128.170:18888"
]

http {
fullNodeEnable = true
fullNodePort = 8090
solidityEnable = true
solidityPort = 8091
PBFTEnable = true
PBFTPort = 8092
}

rpc {
enable = true
port = 50051
solidityEnable = true
solidityPort = 50061
PBFTEnable = true
PBFTPort = 50071
maxConnectionIdleInMillis = 60000
minEffectiveConnection = 1
}

dns {
treeUrls = [
"tree://AKMQMNAJJBL73LXWPXDI4I5ZWWIZ4AWO34DWQ636QOBBXNFXH3LQS@main.trondisco.net"
]
}

jsonrpc {
maxBlockRange = 5000
maxSubTopics = 1000
}
}

rate.limiter = {
http = [
]
rpc = [
]
}

seed.node = {
ip.list = [
"3.225.171.164:18888",
"52.53.189.99:18888",
"18.196.99.16:18888",
"34.253.187.192:18888",
"18.133.82.227:18888",
"35.180.51.163:18888",
"54.252.224.209:18888",
"18.231.27.82:18888",
"52.15.93.92:18888",
"34.220.77.106:18888",
"13.124.62.58:18888",
"54.151.226.240:18888",
"35.174.93.198:18888",
"18.210.241.149:18888",
"54.177.115.127:18888",
"18.167.171.167:18888",
"35.74.7.196:18888",
"52.196.244.176:18888",
"54.248.129.19:18888",
"43.198.142.160:18888"
]
}

genesis.block = {
assets = [
{
accountName = "Zion"
accountType = "AssetIssue"
address = "TLLM21wteSPs4hKjbxgmH1L6poyMjeTbHm"
balance = "99000000000000000"
},
{
accountName = "Sun"
accountType = "AssetIssue"
address = "TXmVpin5vq5gdZsciyyjdZgKRUju4st1wM"
balance = "0"
},
{
accountName = "Blackhole"
accountType = "AssetIssue"
address = "TLsV52sRDL79HXGGm9yzwKibb6BeruhUzy"
balance = "-9223372036854775808"
}
]

witnesses = [
{ address: THKJYuUmMKKARNf7s2VT51g5uPY6KEqnat, url = "http://GR1.com", voteCount = 100000026 },
{ address: TVDmPWGYxgi5DNeW8hXrzrhY8Y6zgxPNg4, url = "http://GR2.com", voteCount = 100000025 },
{ address: TWKZN1JJPFydd5rMgMCV5aZTSiwmoksSZv, url = "http://GR3.com", voteCount = 100000024 },
{ address: TDarXEG2rAD57oa7JTK785Yb2Et32UzY32, url = "http://GR4.com", voteCount = 100000023 },
{ address: TAmFfS4Tmm8yKeoqZN8x51ASwdQBdnVizt, url = "http://GR5.com", voteCount = 100000022 },
{ address: TK6V5Pw2UWQWpySnZyCDZaAvu1y48oRgXN, url = "http://GR6.com", voteCount = 100000021 },
{ address: TGqFJPFiEqdZx52ZR4QcKHz4Zr3QXA24VL, url = "http://GR7.com", voteCount = 100000020 },
{ address: TC1ZCj9Ne3j5v3TLx5ZCDLD55MU9g3XqQW, url = "http://GR8.com", voteCount = 100000019 },
{ address: TWm3id3mrQ42guf7c4oVpYExyTYnEGy3JL, url = "http://GR9.com", voteCount = 100000018 },
{ address: TCvwc3FV3ssq2rD82rMmjhT4PVXYTsFcKV, url = "http://GR10.com", voteCount = 100000017 },
{ address: TFuC2Qge4GxA2U9abKxk1pw3YZvGM5XRir, url = "http://GR11.com", voteCount = 100000016 },
{ address: TNGoca1VHC6Y5Jd2B1VFpFEhizVk92Rz85, url = "http://GR12.com", voteCount = 100000015 },
{ address: TLCjmH6SqGK8twZ9XrBDWpBbfyvEXihhNS, url = "http://GR13.com", voteCount = 100000014 },
{ address: TEEzguTtCihbRPfjf1CvW8Euxz1kKuvtR9, url = "http://GR14.com", voteCount = 100000013 },
{ address: TZHvwiw9cehbMxrtTbmAexm9oPo4eFFvLS, url = "http://GR15.com", voteCount = 100000012 },
{ address: TGK6iAKgBmHeQyp5hn3imB71EDnFPkXiPR, url = "http://GR16.com", voteCount = 100000011 },
{ address: TLaqfGrxZ3dykAFps7M2B4gETTX1yixPgN, url = "http://GR17.com", voteCount = 100000010 },
{ address: TX3ZceVew6yLC5hWTXnjrUFtiFfUDGKGty, url = "http://GR18.com", voteCount = 100000009 },
{ address: TYednHaV9zXpnPchSywVpnseQxY9Pxw4do, url = "http://GR19.com", voteCount = 100000008 },
{ address: TCf5cqLffPccEY7hcsabiFnMfdipfyryvr, url = "http://GR20.com", voteCount = 100000007 },
{ address: TAa14iLEKPAetX49mzaxZmH6saRxcX7dT5, url = "http://GR21.com", voteCount = 100000006 },
{ address: TBYsHxDmFaRmfCF3jZNmgeJE8sDnTNKHbz, url = "http://GR22.com", voteCount = 100000005 },
{ address: TEVAq8dmSQyTYK7uP1ZnZpa6MBVR83GsV6, url = "http://GR23.com", voteCount = 100000004 },
{ address: TRKJzrZxN34YyB8aBqqPDt7g4fv6sieemz, url = "http://GR24.com", voteCount = 100000003 },
{ address: TRMP6SKeFUt5NtMLzJv8kdpYuHRnEGjGfe, url = "http://GR25.com", voteCount = 100000002 },
{ address: TDbNE1VajxjpgM5p7FyGNDASt3UVoFbiD3, url = "http://GR26.com", voteCount = 100000001 },
{ address: TLTDZBcPoJ8tZ6TTEeEqEvwYFk2wgotSfD, url = "http://GR27.com", voteCount = 100000000 }
]

timestamp = "0"
parentHash = "0xe58f33f9baf9305dc6f82b9f1934ea8f0ade2defb951258d50167028c780351f"
}

localwitness = [
]

block = {
needSyncCheck = true
maintenanceTimeInterval = 21600000
proposalExpireTime = 259200000
}

trx.reference.block = "solid"

vm = {
supportConstant = true
maxEnergyLimitForConstant = 100000000
minTimeRatio = 0.0
maxTimeRatio = 5.0
saveInternalTx = false
#native.vm.enable = true
#native.vm.enable.all.contracts = true
}

committee = {
allowCreationOfContracts = 0
allowAdaptiveEnergy = 0
}

event.subscribe = {
native = {
useNativeQueue = true
bindport = 5555
sendqueuelength = 1000
}
version = 0

path = ""
server = ""
dbconfig = ""
contractParse = true

topics = [
{ triggerName = "block", enable = false, topic = "block", solidified = false },
{ triggerName = "transaction", enable = false, topic = "transaction", solidified = false, ethCompatible = false },
{ triggerName = "contractevent", enable = false, topic = "contractevent" },
{ triggerName = "contractlog", enable = false, topic = "contractlog", redundancy = false },
{ triggerName = "solidity", enable = true, topic = "solidity" },
{ triggerName = "solidityevent", enable = false, topic = "solidityevent" },
{ triggerName = "soliditylog", enable = false, topic = "soliditylog", redundancy = false }
]

filter = {
fromblock = ""
toblock = ""
contractAddress = [ "" ]
contractTopic = [ "" ]
}
}

[ZDCELL]

Phenomenon:
After restoring from the foundation’s backup file (FullNode_output-directory.tgz), the TRON Full Node consistently experiences sync delays when it reaches blocks ending in 99, 999, or 9999.
• Reproduction conditions:
• During sync, when the block number ends with ...99, ...999, or ...9999, a noticeable delay occurs.
• Block height remains frozen for about 90–102 seconds before proceeding to the next block.
• Monitoring evidence:
• We monitor the block height every 6 seconds.
• When the issue occurs, the same block height is observed 15–17 times consecutively, indicating a delay of 1 minute 30 seconds to 1 minute 42 seconds.
• After this delay, the sync resumes normally.
• Additional observations:
• This delay seems to get longer over time.
• In many cases, there are no explicit error logs during the delay.
• Occasionally, logs such as Tapos failed, different block hash, verifyBlockHeader failed, or fail to switch fork appear.

[halibobo1205]

@ZDCELL, can you provide the backup of all store usage logs?

long timeUsed = System.currentTimeMillis() - t1;
    logger
        .info("Current block number is {}, backup all store use {} ms!", block.getNum(), timeUsed);
    if (timeUsed >= 3000) {
      logger.warn("Backing up db uses too much time. {} ms.", timeUsed);
    }

[halibobo1205]

From what I understand, there are currently two problems with RocksDB's backup feature:

1. The backup process becomes progressively slower as backup data accumulates, affecting block synchronization.

2. The current backup is incomplete due to missing database portions, rendering the backup data temporarily unavailable.

Please have a look at #5698 for details.

[ZDCELL]

@ZDCELL, can you provide the backup of all store usage logs?

long timeUsed = System.currentTimeMillis() - t1;
logger
.info("Current block number is {}, backup all store use {} ms!", block.getNum(), timeUsed);
if (timeUsed >= 3000) {
logger.warn("Backing up db uses too much time. {} ms.", timeUsed);
}

store_backup_times.log

[halibobo1205]

The logs demonstrate that backup operations are excessively time-consuming, severely affecting block sync performance, and the backup data is incomplete. I suggest disabling the backup functionality:

backup = {
    enable = false
    propPath = "prop.properties"
    bak1path = "/data/bak1/database"
    bak2path = "/mnt/ssd4t/bak2/database"
    frequency = 600
}