[BUG] Mainnet halts at block 100

[shekohex]

Bug

While #369 using Alice and Bob. The blockchain halts and no more blocks are produced at block 100.

Errors

Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0xc2330e0d164ed9981d40ae2d54d29852df191315800ab19e243ef2c6f9f20243, s283931986

Logs

2023-12-26 13:52:12 ✨ Imported #99 (0x3051…66ee)
2023-12-26 13:52:16 💤 Idle (1 peers), best: #99 (0x3051…66ee), finalized #97 (0x56c6…4067), ⬇ 0.7kiB/s ⬆ 0.5kiB/s
2023-12-26 13:52:18 🙌 Starting consensus session on top of parent 0x30517c5fe558e96dcaf7bd49cc0f2e0e05f3800fce455fe30d0298b2412266ee
2023-12-26 13:52:18 🎁 Prepared block for proposing at 100 (0 ms) [hash: 0xf3a1d422f8c87d74b9bd76c0d72ede19ea1bcf0fa035b8da9a4960cd6df044cf; parent_hash: 0x3051…66ee; extrinsics (1): [0x5465…2920]
2023-12-26 13:52:18 🔖 Pre-sealed block for proposal at 100. Hash now 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69, previously 0xf3a1d422f8c87d74b9bd76c0d72ede19ea1bcf0fa035b8da9a4960cd6df044cf.
2023-12-26 13:52:18 ✨ Imported #100 (0x0792…ef69)
2023-12-26 13:52:21 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.5kiB/s ⬆ 0.6kiB/s
2023-12-26 13:52:24 🙌 Starting consensus session on top of parent 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69
2023-12-26 13:52:24 🎁 Prepared block for proposing at 101 (0 ms) [hash: 0x842b715357c1cc5eae7c594c0ec08477d152841bb4bda10a29b8ece6439b3a2d; parent_hash: 0x0792…ef69; extrinsics (1): [0xd32d…9c37]
2023-12-26 13:52:24 🔖 Pre-sealed block for proposal at 101. Hash now 0xf897d7740221351df269b7702bd823949ca6ec0c1e9a9fcff7d830acc763788a, previously 0x842b715357c1cc5eae7c594c0ec08477d152841bb4bda10a29b8ece6439b3a2d.
2023-12-26 13:52:24 Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0xf897d7740221351df269b7702bd823949ca6ec0c1e9a9fcff7d830acc763788a, s283931924
2023-12-26 13:52:26 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.8kiB/s ⬆ 0.8kiB/s
2023-12-26 13:52:30 🙌 Starting consensus session on top of parent 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69
2023-12-26 13:52:30 🎁 Prepared block for proposing at 101 (0 ms) [hash: 0xb040741981dd2b40548c83776f2b26fd610f34903f9edbe0f408e88ea662d438; parent_hash: 0x0792…ef69; extrinsics (1): [0x1607…b16e]
2023-12-26 13:52:30 🔖 Pre-sealed block for proposal at 101. Hash now 0x7ea2dfc6d2729d581773ce1981c0d240c2d245056092644d768ac5aaed73753a, previously 0xb040741981dd2b40548c83776f2b26fd610f34903f9edbe0f408e88ea662d438.
2023-12-26 13:52:30 Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0x7ea2dfc6d2729d581773ce1981c0d240c2d245056092644d768ac5aaed73753a, s283931925
2023-12-26 13:52:31 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.4kiB/s ⬆ 0.4kiB/s
2023-12-26 13:52:36 🙌 Starting consensus session on top of parent 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69
2023-12-26 13:52:36 🎁 Prepared block for proposing at 101 (0 ms) [hash: 0xf926a385c49024a87df1993a32f5a67a991529a8c3a3634ab2c33e1e1e6eb586; parent_hash: 0x0792…ef69; extrinsics (1): [0x7767…0a98]
2023-12-26 13:52:36 🔖 Pre-sealed block for proposal at 101. Hash now 0x932c363ae23944a04fdb5e7fe7531fdef86ccfb3d4266fb9579cd874bf78bcb1, previously 0xf926a385c49024a87df1993a32f5a67a991529a8c3a3634ab2c33e1e1e6eb586.
2023-12-26 13:52:36 Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0x932c363ae23944a04fdb5e7fe7531fdef86ccfb3d4266fb9579cd874bf78bcb1, s283931926
2023-12-26 13:52:36 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.5kiB/s ⬆ 0.5kiB/s
2023-12-26 13:52:41 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.5kiB/s ⬆ 0.5kiB/s
2023-12-26 13:52:42 🙌 Starting consensus session on top of parent 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69
2023-12-26 13:52:42 🎁 Prepared block for proposing at 101 (0 ms) [hash: 0xc34906cd5bf1813bcf896ca69a4785515312608e55f247eaaa3ccc5fd6e74184; parent_hash: 0x0792…ef69; extrinsics (1): [0x503a…10d7]
2023-12-26 13:52:42 🔖 Pre-sealed block for proposal at 101. Hash now 0x496b499a83d8bca61902f564d9d284c2b9a814783b6ede770308e34832a4acd3, previously 0xc34906cd5bf1813bcf896ca69a4785515312608e55f247eaaa3ccc5fd6e74184.
2023-12-26 13:52:42 Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0x496b499a83d8bca61902f564d9d284c2b9a814783b6ede770308e34832a4acd3, s283931927
2023-12-26 13:52:46 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.5kiB/s ⬆ 0.5kiB/s
2023-12-26 13:52:48 🙌 Starting consensus session on top of parent 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69
2023-12-26 13:52:48 🎁 Prepared block for proposing at 101 (0 ms) [hash: 0x3c8017b17cccb215c2169a940313a82ce74a8cc205df514275fc679615b56841; parent_hash: 0x0792…ef69; extrinsics (1): [0x60b5…5f5f]
2023-12-26 13:52:48 🔖 Pre-sealed block for proposal at 101. Hash now 0x0e19fa9a695a6b1a1fdb41f68a8b046fcd052a80a3e1fa21304afaea23560cc5, previously 0x3c8017b17cccb215c2169a940313a82ce74a8cc205df514275fc679615b56841.
2023-12-26 13:52:48 Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0x0e19fa9a695a6b1a1fdb41f68a8b046fcd052a80a3e1fa21304afaea23560cc5, s283931928
2023-12-26 13:52:51 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.7kiB/s ⬆ 0.7kiB/s
2023-12-26 13:52:54 🙌 Starting consensus session on top of parent 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69
2023-12-26 13:52:54 🎁 Prepared block for proposing at 101 (0 ms) [hash: 0xf9e9dd0b18d8e6e836c13f0cb4e04372d42b74f85bac0fbc8a851c380b382c79; parent_hash: 0x0792…ef69; extrinsics (1): [0x1405…b6f9]
2023-12-26 13:52:54 🔖 Pre-sealed block for proposal at 101. Hash now 0x715c76d84191a5d6093d9a6899a2122431920cc1d68ca1d3a2853e2da236bfd4, previously 0xf9e9dd0b18d8e6e836c13f0cb4e04372d42b74f85bac0fbc8a851c380b382c79.
2023-12-26 13:52:54 Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0x715c76d84191a5d6093d9a6899a2122431920cc1d68ca1d3a2853e2da236bfd4, s283931929
2023-12-26 13:52:56 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.6kiB/s ⬆ 0.6kiB/s
2023-12-26 13:53:00 🙌 Starting consensus session on top of parent 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69
2023-12-26 13:53:00 🎁 Prepared block for proposing at 101 (0 ms) [hash: 0x04ff50a3e43c34d61ca5ec7ec3d755f86112599d04535b929021ee170ef0a9be; parent_hash: 0x0792…ef69; extrinsics (1): [0xa0ce…31ea]
2023-12-26 13:53:00 🔖 Pre-sealed block for proposal at 101. Hash now 0x0fef66642f67ea442511fd71e3f43690b7c327d1089cd63eed17afd1f0198083, previously 0x04ff50a3e43c34d61ca5ec7ec3d755f86112599d04535b929021ee170ef0a9be.
2023-12-26 13:53:00 Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0x0fef66642f67ea442511fd71e3f43690b7c327d1089cd63eed17afd1f0198083, s283931930
2023-12-26 13:53:01 💤 Idle (1 peers), best: #100 (0x0792…ef69), finalized #98 (0x78b8…2f4c), ⬇ 0.4kiB/s ⬆ 0.4kiB/s
2023-12-26 13:53:06 🙌 Starting consensus session on top of parent 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69
2023-12-26 13:53:06 🎁 Prepared block for proposing at 101 (0 ms) [hash: 0x9bdd8e40b94f5b69456913a190b7f8481d885b933422c2751413b8cdcc29572a; parent_hash: 0x0792…ef69; extrinsics (1): [0x5bf0…9227]
2023-12-26 13:53:06 🔖 Pre-sealed block for proposal at 101. Hash now 0x6431213c9b994f9d4e612cf96552e314a94f429c1ba37f815b2e1dd527828cdc, previously 0x9bdd8e40b94f5b69456913a190b7f8481d885b933422c2751413b8cdcc29572a.
2023-12-26 13:53:06 Error with block built on 0x07924a3bde246b14361a8b42ee6966cc93e9469c713f88a0a6530092daddef69: Import failed: Expected epoch change to happen at 0x6431213c9b994f9d4e612cf96552e314a94f429c1ba37f815b2e1dd527828cdc, s283931931

[drewstone]

I am able to replicate things using the ./scripts/run-standalone-local.sh. CC @1xstj

[drewstone]

Possibly dev issue? Unsure lack of explanation here:

• Babe block producing hacks paritytech/substrate#5862
• How to prevent and deal with unexpected epoch changes paritytech/substrate#4464

I posted on Substrate stackexchange