Warp sync stops >1m blocks below latest block before switching to full sync #1688
Description
Describe the bug
When starting a fresh light node, warp sync stops a seemingly random count of blocks before the latest block on the chain before switching to full sync.
Curtailed logs:
2025-05-27 00:00:00 16:43:38 INFO: [core] original dial target is: "unix:///Users/xxx/.docker/run/user-analytics.otlp.grpc.sock"
2025-05-27 00:00:00 16:43:38 INFO: [core] [Channel #1]Channel created
2025-05-27 00:00:00 16:43:38 INFO: [core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"unix", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/Users/jeff/.docker/run/user-analytics.otlp.grpc.sock", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025-05-27 00:00:00 16:43:38 INFO: [core] [Channel #1]Channel authority set to "localhost"
2025-05-27 00:00:00 16:43:38 INFO: [core] original dial target is: "unix:///Users/aaa/.docker/run/user-analytics.otlp.grpc.sock"
2025-05-27 00:00:00 16:43:38 INFO: [core] [Channel #2]Channel created
2025-05-27 00:00:00 16:43:38 INFO: [core] [Channel #2]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"unix", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/Users/jeff/.docker/run/user-analytics.otlp.grpc.sock", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025-05-27 00:00:00 16:43:38 INFO: [core] [Channel #2]Channel authority set to "localhost"
2025-05-27 16:38:37 2025-05-27 15:38:37 Subtensor Node
2025-05-27 16:38:37 2025-05-27 15:38:37 ✌️ version 4.0.0-dev-719a6f5e9ec
2025-05-27 16:38:37 2025-05-27 15:38:37 ❤️ by Substrate DevHub <https://github.com/substrate-developer-hub>, 2017-2025
2025-05-27 16:38:37 2025-05-27 15:38:37 📋 Chain specification: Bittensor
2025-05-27 16:38:37 2025-05-27 15:38:37 🏷 Node name: greasy-bait-2607
2025-05-27 16:38:37 2025-05-27 15:38:37 👤 Role: FULL
2025-05-27 16:38:37 2025-05-27 15:38:37 💾 Database: RocksDb at /tmp/blockchain/chains/bittensor/db/full
2025-05-27 16:38:38 2025-05-27 15:38:38 🔨 Initializing Genesis block/state (state: 0x4015…9643, header-hash: 0x2f05…6c03)
2025-05-27 16:38:38 2025-05-27 15:38:38 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2025-05-27 16:38:38 2025-05-27 15:38:38 🏷 Local node identity is: 12D3KooWP2UNVmH3AiKMb7eenqCvimNiUMLNeB2AFMNeVzWe8U9h
2025-05-27 16:38:38 2025-05-27 15:38:38 Running libp2p network backend
2025-05-27 16:38:38 2025-05-27 15:38:38 💻 Operating system: linux
2025-05-27 16:38:38 2025-05-27 15:38:38 💻 CPU architecture: aarch64
2025-05-27 16:38:38 2025-05-27 15:38:38 💻 Target environment: gnu
2025-05-27 16:38:38 2025-05-27 15:38:38 💻 Memory: 32045MB
2025-05-27 16:38:38 2025-05-27 15:38:38 💻 Kernel: 6.10.14-linuxkit
2025-05-27 16:38:38 2025-05-27 15:38:38 💻 Linux distribution: Ubuntu 24.04.1 LTS
2025-05-27 16:38:38 2025-05-27 15:38:38 💻 Virtual machine: no
2025-05-27 16:38:38 2025-05-27 15:38:38 📦 Highest known block at #0
2025-05-27 16:38:38 2025-05-27 15:38:38 〽️ Prometheus exporter started at 127.0.0.1:9615
2025-05-27 16:38:38 2025-05-27 15:38:38 Running JSON-RPC server: addr=0.0.0.0:9944,[::]:40479
2025-05-27 16:38:38 2025-05-27 15:38:38 🔍 Discovered new external address for our node: /ip4/1.2.3.4/tcp/30333/ws/p2p/12D3KooWP2UNVmH3AiKMb7eenqCvimNiUMLNeB2AFMNeVzWe8U9h
2025-05-27 16:38:43 2025-05-27 15:38:43 ⏩ Warping, Downloading finality proofs, 0.00 Mib (7 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 34.3kiB/s ⬆ 12.1kiB/s
2025-05-27 16:38:48 2025-05-27 15:38:48 ⏩ Warping, Downloading finality proofs, 0.00 Mib (11 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 12.3kiB/s ⬆ 1.7kiB/s
2025-05-27 16:38:53 2025-05-27 15:38:53 ⏩ Warping, Downloading finality proofs, 0.00 Mib (20 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 100.6kiB/s ⬆ 17.2kiB/s
2025-05-27 16:38:58 2025-05-27 15:38:58 ⏩ Warping, Downloading finality proofs, 0.00 Mib (29 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 106.1kiB/s ⬆ 29.3kiB/s
2025-05-27 16:39:01 2025-05-27 15:39:01 received notification from 12D3KooWHXvcms5q5TfVqU2sAZKZgasEbCgqhrH9HACZyuKf6ghd who had been earlier refused by `SyncingEngine`
2025-05-27 16:39:03 2025-05-27 15:39:03 ⏩ Warping, Downloading finality proofs, 0.00 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 123.3kiB/s ⬆ 26.0kiB/s
2025-05-27 16:39:08 2025-05-27 15:39:08 ⏩ Warping, Downloading finality proofs, 0.00 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 31.0kiB/s ⬆ 5.0kiB/s
2025-05-27 16:39:13 2025-05-27 15:39:13 ⏩ Warping, Downloading finality proofs, 0.00 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 117.2kiB/s ⬆ 3.6kiB/s
2025-05-27 16:39:18 2025-05-27 15:39:18 ⏩ Warping, Downloading finality proofs, 0.00 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 28.3kiB/s ⬆ 3.2kiB/s
2025-05-27 16:39:23 2025-05-27 15:39:23 ⏩ Warping, Downloading finality proofs, 0.00 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 4.4kiB/s ⬆ 2.9kiB/s
2025-05-27 16:39:28 2025-05-27 15:39:28 ⏩ Warping, Downloading finality proofs, 0.00 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 49.0kiB/s ⬆ 3.6kiB/s
2025-05-27 16:39:33 2025-05-27 15:39:33 ⏩ Warping, Downloading finality proofs, 0.00 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 15.8kiB/s ⬆ 3.3kiB/s
2025-05-27 16:39:38 2025-05-27 15:39:38 ⏩ Warping, Downloading finality proofs, 0.00 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 126.4kiB/s ⬆ 7.5kiB/s
2025-05-27 16:39:43 2025-05-27 15:39:43 ⏩ Warping, Downloading finality proofs, 0.00 Mib (31 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 31.9kiB/s ⬆ 6.4kiB/s
2025-05-27 16:39:43 2025-05-27 15:39:43 Report 12D3KooWEhbi2zHiqzJMNPWBEUhbNSM7HsCCvXtPCwA8ZF6QN6vb: -536870912 to -1984132567. Reason: Bad warp proof. Banned, disconnecting.
...
2025-05-27 16:43:43 2025-05-27 15:43:43 Report 12D3KooWEhbi2zHiqzJMNPWBEUhbNSM7HsCCvXtPCwA8ZF6QN6vb: -536870912 to -1783245858. Reason: Bad warp proof. Banned, disconnecting.
2025-05-27 16:43:43 2025-05-27 15:43:43 ⏩ Warping, Downloading finality proofs, 0.00 Mib (31 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 21.0kiB/s ⬆ 4.0kiB/s
2025-05-27 00:00:00 16:43:43 INFO: [core] [Channel #2]Resolver state updated: {
"Addresses": [
{
"Addr": "/Users/jeff/.docker/run/user-analytics.otlp.grpc.sock",
"ServerName": "",
"Attributes": {
"\u003c%!p(networktype.keyType=grpc.internal.transport.networktype)\u003e": "unix"
},
"BalancerAttributes": null,
"Metadata": null
}
],
"Endpoints": [
{
"Addresses": [
{
"Addr": "/Users/jeff/.docker/run/user-analytics.otlp.grpc.sock",
"ServerName": "",
"Attributes": {
"\u003c%!p(networktype.keyType=grpc.internal.transport.networktype)\u003e": "unix"
},
"BalancerAttributes": null,
"Metadata": null
}
],
"Attributes": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2025-05-27 00:00:00 16:43:43 INFO: [core] [Channel #2]Channel switches to new LB policy "pick_first"
2025-05-27 00:00:00 16:43:43 INFO: [pick-first-lb] [pick-first-lb 0x14000354660] Received new config {
"shuffleAddressList": false
}, resolver state {
"Addresses": [
{
"Addr": "/Users/jeff/.docker/run/user-analytics.otlp.grpc.sock",
"ServerName": "",
"Attributes": {
"\u003c%!p(networktype.keyType=grpc.internal.transport.networktype)\u003e": "unix"
},
"BalancerAttributes": null,
"Metadata": null
}
],
"Endpoints": [
{
"Addresses": [
{
"Addr": "/Users/jeff/.docker/run/user-analytics.otlp.grpc.sock",
"ServerName": "",
"Attributes": {
"\u003c%!p(networktype.keyType=grpc.internal.transport.networktype)\u003e": "unix"
},
"BalancerAttributes": null,
"Metadata": null
}
],
"Attributes": null
}
],
"ServiceConfig": null,
"Attributes": null
}
...
2025-05-27 16:46:03 2025-05-27 15:46:03 Report 12D3KooWLUW5hGkmLhfVdns4aiovR9RBYM5rYw7tNs5PQ1LWSxqS: -536870912 to -1570210921. Reason: Bad warp proof. Banned, disconnecting.
2025-05-27 16:46:03 2025-05-27 15:46:03 ⏩ Warping, Downloading finality proofs, 0.00 Mib (14 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 42.9kiB/s ⬆ 1.8kiB/s
2025-05-27 16:46:03 2025-05-27 15:46:03 Report 12D3KooWPNK2Mfj1pxHvBQAaRntJUzi39MBnSyJysD865PZZd9tT: -536870912 to -2034637429. Reason: Bad warp proof. Banned, disconnecting.
2025-05-27 16:46:05 2025-05-27 15:46:05 Report 12D3KooWRjp2xBz67uje6y7CbRDHfvVoG4rYcPpeQCcLrKxqUGeG: -536870912 to -1719225551. Reason: Bad warp proof. Banned, disconnecting.
2025-05-27 16:46:06 2025-05-27 15:46:06 Report 12D3KooWPciDbUfp6bFu4bUMSsy8dc1wQk5iWb9JyunsZtZvudKg: -536870912 to -1750256648. Reason: Bad warp proof. Banned, disconnecting.
2025-05-27 16:46:07 2025-05-27 15:46:07 Report 12D3KooWDC2CLxsC6F5aUc4meVYVCFotrcE92c5eL6xgLr26FqUE: -536870912 to -1946332878. Reason: Bad warp proof. Banned, disconnecting.
2025-05-27 16:46:08 2025-05-27 15:46:08 Report 12D3KooWGaKvqbYxJy9ihjgmYRixhDYugiYBPvMZxAd2Eh76HABo: -536870912 to -1830156136. Reason: Bad warp proof. Banned, disconnecting.
2025-05-27 16:46:08 2025-05-27 15:46:08 ⏩ Warping, Downloading target block, 0.00 Mib (9 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 10.6kiB/s ⬆ 1.5kiB/s
2025-05-27 16:46:12 2025-05-27 15:46:12 Warp sync is complete, continuing with state sync.
2025-05-27 16:46:13 2025-05-27 15:46:13 ⚙️ State sync, Downloading state, 0%, 0.00 Mib (12 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 17.1kiB/s ⬆ 2.1kiB/s
2025-05-27 16:46:18 2025-05-27 15:46:18 ⚙️ State sync, Downloading state, 0%, 0.00 Mib (13 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 6.4kiB/s ⬆ 1.2kiB/s
2025-05-27 16:46:23 2025-05-27 15:46:23 ⚙️ State sync, Downloading state, 14%, 2.41 Mib (16 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 284.7kiB/s ⬆ 1.5kiB/s
2025-05-27 16:46:28 2025-05-27 15:46:28 ⚙️ State sync, Downloading state, 14%, 2.41 Mib (20 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 176.3kiB/s ⬆ 6.0kiB/s
2025-05-27 16:46:33 2025-05-27 15:46:33 ⚙️ State sync, Downloading state, 14%, 4.85 Mib (23 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 178.1kiB/s ⬆ 1.6kiB/s
2025-05-27 16:46:38 2025-05-27 15:46:38 ⚙️ State sync, Downloading state, 14%, 7.29 Mib (30 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 319.6kiB/s ⬆ 1.9kiB/s
2025-05-27 16:46:43 2025-05-27 15:46:43 ⚙️ State sync, Downloading state, 14%, 9.74 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 306.7kiB/s ⬆ 2.9kiB/s
2025-05-27 16:46:48 2025-05-27 15:46:48 ⚙️ State sync, Downloading state, 14%, 12.18 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 381.9kiB/s ⬆ 1.5kiB/s
2025-05-27 16:46:53 2025-05-27 15:46:53 ⚙️ State sync, Downloading state, 14%, 14.64 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 347.7kiB/s ⬆ 1.2kiB/s
2025-05-27 16:46:58 2025-05-27 15:46:58 ⚙️ State sync, Downloading state, 14%, 17.08 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 298.1kiB/s ⬆ 1.2kiB/s
2025-05-27 16:47:03 2025-05-27 15:47:03 ⚙️ State sync, Downloading state, 14%, 19.54 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 331.5kiB/s ⬆ 0.7kiB/s
2025-05-27 16:47:08 2025-05-27 15:47:08 ⚙️ State sync, Downloading state, 14%, 21.98 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 295.8kiB/s ⬆ 0.9kiB/s
2025-05-27 16:47:13 2025-05-27 15:47:13 ⚙️ State sync, Downloading state, 14%, 24.43 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 399.9kiB/s ⬆ 1.2kiB/s
2025-05-27 16:47:18 2025-05-27 15:47:18 ⚙️ State sync, Downloading state, 14%, 26.87 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 299.8kiB/s ⬆ 0.6kiB/s
2025-05-27 16:47:23 2025-05-27 15:47:23 ⚙️ State sync, Downloading state, 14%, 29.32 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 568.8kiB/s ⬆ 3.5kiB/s
2025-05-27 16:47:28 2025-05-27 15:47:28 ⚙️ State sync, Downloading state, 39%, 31.76 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 116.6kiB/s ⬆ 2.8kiB/s
2025-05-27 16:47:33 2025-05-27 15:47:33 ⚙️ State sync, Downloading state, 39%, 31.76 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 10.6kiB/s ⬆ 0.6kiB/s
2025-05-27 16:47:38 2025-05-27 15:47:38 ⚙️ State sync, Downloading state, 39%, 31.76 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 107.1kiB/s ⬆ 2.0kiB/s
2025-05-27 16:47:43 2025-05-27 15:47:43 ⚙️ State sync, Downloading state, 39%, 34.99 Mib (31 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 264.7kiB/s ⬆ 2.2kiB/s
2025-05-27 16:47:48 2025-05-27 15:47:48 ⚙️ State sync, Downloading state, 39%, 37.98 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 294.5kiB/s ⬆ 1.4kiB/s
2025-05-27 16:47:53 2025-05-27 15:47:53 ⚙️ State sync, Downloading state, 39%, 44.03 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 583.8kiB/s ⬆ 2.0kiB/s
2025-05-27 16:47:58 2025-05-27 15:47:58 ⚙️ State sync, Downloading state, 39%, 47.10 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 281.0kiB/s ⬆ 0.9kiB/s
2025-05-27 16:48:03 2025-05-27 15:48:03 ⚙️ State sync, Downloading state, 39%, 50.13 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 382.6kiB/s ⬆ 0.6kiB/s
2025-05-27 16:48:08 2025-05-27 15:48:08 ⚙️ State sync, Downloading state, 39%, 56.25 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 558.2kiB/s ⬆ 1.2kiB/s
2025-05-27 16:48:13 2025-05-27 15:48:13 ⚙️ State sync, Downloading state, 39%, 61.92 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 539.1kiB/s ⬆ 1.0kiB/s
2025-05-27 16:48:18 2025-05-27 15:48:18 ⚙️ State sync, Downloading state, 39%, 68.17 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 579.8kiB/s ⬆ 0.7kiB/s
2025-05-27 16:48:23 2025-05-27 15:48:23 ⚙️ State sync, Downloading state, 39%, 70.64 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 220.9kiB/s ⬆ 5.4kiB/s
2025-05-27 16:48:28 2025-05-27 15:48:28 ⚙️ State sync, Downloading state, 39%, 75.96 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 595.8kiB/s ⬆ 0.8kiB/s
2025-05-27 16:48:33 2025-05-27 15:48:33 ⚙️ State sync, Downloading state, 39%, 78.42 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 216.8kiB/s ⬆ 0.7kiB/s
2025-05-27 16:48:38 2025-05-27 15:48:38 ⚙️ State sync, Downloading state, 39%, 80.88 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 237.1kiB/s ⬆ 3.0kiB/s
2025-05-27 16:48:43 2025-05-27 15:48:43 ⚙️ State sync, Downloading state, 39%, 83.33 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 232.9kiB/s ⬆ 3.4kiB/s
2025-05-27 16:48:48 2025-05-27 15:48:48 ⚙️ State sync, Downloading state, 39%, 87.93 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 564.6kiB/s ⬆ 1.1kiB/s
2025-05-27 16:48:53 2025-05-27 15:48:53 ⚙️ State sync, Downloading state, 39%, 96.50 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 1.3MiB/s ⬆ 5.0kiB/s
2025-05-27 16:48:58 2025-05-27 15:48:58 ⚙️ State sync, Downloading state, 39%, 101.93 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 474.9kiB/s ⬆ 4.4kiB/s
2025-05-27 16:49:03 2025-05-27 15:49:03 ⚙️ State sync, Downloading state, 39%, 110.69 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 923.9kiB/s ⬆ 5.3kiB/s
2025-05-27 16:49:08 2025-05-27 15:49:08 ⚙️ State sync, Downloading state, 39%, 123.30 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 1.3MiB/s ⬆ 1.6kiB/s
2025-05-27 16:49:13 2025-05-27 15:49:13 ⚙️ State sync, Downloading state, 39%, 130.39 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 991.6kiB/s ⬆ 0.9kiB/s
2025-05-27 16:49:18 2025-05-27 15:49:18 ⚙️ State sync, Downloading state, 39%, 130.39 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 21.6kiB/s ⬆ 0.6kiB/s
2025-05-27 16:49:23 2025-05-27 15:49:23 ⚙️ State sync, Downloading state, 39%, 130.39 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 14.5kiB/s ⬆ 5.9kiB/s
2025-05-27 16:49:28 2025-05-27 15:49:28 ⚙️ State sync, Downloading state, 39%, 132.83 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 256.2kiB/s ⬆ 0.9kiB/s
2025-05-27 16:49:33 2025-05-27 15:49:33 ⚙️ State sync, Downloading state, 39%, 132.83 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 5.8kiB/s ⬆ 0.7kiB/s
2025-05-27 16:49:38 2025-05-27 15:49:38 ⚙️ State sync, Downloading state, 39%, 144.10 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 1.0MiB/s ⬆ 2.4kiB/s
2025-05-27 16:49:43 2025-05-27 15:49:43 ⚙️ State sync, Downloading state, 39%, 153.54 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 759.1kiB/s ⬆ 1.6kiB/s
2025-05-27 16:49:49 2025-05-27 15:49:48 ⚙️ State sync, Downloading state, 63%, 171.23 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 1.9MiB/s ⬆ 1.4kiB/s
2025-05-27 16:49:53 2025-05-27 15:49:53 ⚙️ State sync, Downloading state, 63%, 174.87 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 531.7kiB/s ⬆ 1.8kiB/s
2025-05-27 16:49:58 2025-05-27 15:49:58 ⚙️ State sync, Importing state, 78%, 179.45 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 590.0kiB/s ⬆ 0.8kiB/s
2025-05-27 16:50:04 2025-05-27 15:50:03 ⚙️ State sync, Importing state, 78%, 179.45 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 139.0kiB/s ⬆ 0.7kiB/s
2025-05-27 16:50:08 2025-05-27 15:50:08 ⚙️ State sync, Importing state, 78%, 179.45 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 69.9kiB/s ⬆ 1.2kiB/s
2025-05-27 16:50:13 2025-05-27 15:50:13 ⚙️ State sync, Importing state, 78%, 179.45 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 19.5kiB/s ⬆ 0.6kiB/s
2025-05-27 16:50:18 2025-05-27 15:50:18 ⚙️ State sync, Importing state, 78%, 179.45 Mib (32 peers), best: #0 (0x2f05…6c03), finalized #0 (0x2f05…6c03), ⬇ 14.3kiB/s ⬆ 2.8kiB/s
2025-05-27 16:50:23 2025-05-27 15:50:23 ⚙️ State sync, Importing state, 78%, 179.45 Mib (32 peers), best: #4519424 (0xf3a0…eaca), finalized #4519424 (0xf3a0…eaca), ⬇ 5.1kiB/s ⬆ 3.2kiB/s
2025-05-27 16:50:26 2025-05-27 15:50:26 State sync is complete, continuing with block sync.
2025-05-27 16:50:29 2025-05-27 15:50:28 ⚙️ Syncing 0.7 bps, target=#5650634 (32 peers), best: #4519428 (0xfc92…1d62), finalized #4519424 (0xf3a0…eaca), ⬇ 5.6MiB/s ⬆ 4.2kiB/s
2025-05-27 16:50:34 2025-05-27 15:50:33 ⚙️ Syncing 2.5 bps, target=#5650634 (32 peers), best: #4519441 (0xabff…2392), finalized #4519424 (0xf3a0…eaca), ⬇ 406.8kiB/s ⬆ 2.2kiB/s
...
To Reproduce
- Clone the repo
- Ensure branch set to
main - Ensure a fresh docker environment (images and volumes cleaned)
- Run
./scripts/run/subtensor.sh -e docker --network mainnet --node-type lite - Observe the issue using
docker logs
Expected behavior
Warp sync should swap over to full syncing significantly closer to, or at, the current chain block.
Screenshots
No response
Environment
macOS Sonoma (arm64), multiple versions main, v2.0.10 (latest), v3.0.0
Additional context
We run a pool of ghcr.io/opentensor/subtensor:v2.0.7 images (the last good amd64 build AFAICT) which started exhibiting this issue when we tried to scale up the pool today. These were working correctly when they were restarted 7 days ago.