bitmarkd compiled from source doesn't sync after block 450627

[jarr0s]

I've compiled bitmarkd from source on ubuntu 16.04 and tried to sync blockchain, but sync got stuck on block 450627. Part of the log:

UpdateTip: new best=9df459786d64f926e91ac357146e0a11b78fb8d4e70bf2cc10943434a936f528 height=450627 log2_work=72.835175 tx=1089800 date=1526090544 progress=0.712616 nbits=456215360 algo=0
ProcessBlock: ACCEPTED
Requesting block 1ee33a0c2faa5795f82c33d79e1f5809608f11a399b0d23447e9de82632da27b from 139.162.128.92:9265
sending: getdata (37 bytes)
sending: inv (37 bytes)
received: block (1700 bytes)
received block c08f9fa005400ffbd0f750b5e3c67d0bf79674e9b9343ba88cbd62f2a3442481
ERROR: CheckTransaction() : vout empty
ERROR: CheckBlock() : CheckTransaction failed
ERROR: ProcessBlock() : CheckBlock FAILED
Requesting block f6c734f3bd176651f7e9e3d22a7577dafb575dce8c365ca38af488668de1d752 from 139.162.128.92:9265
sending: getdata (37 bytes)
received: block (7347 bytes)
received block c60a4a65d0dd66cabbd56722d6d4ffb0126674e878641f30638e4f06e061d1e3
check auxpow err 1
ERROR: AuxPow is not a generate
auxpow err 5
ERROR: CheckAuxPowProofOfWork : AUX POW is not valid
ERROR: CheckBlock() : auxpow proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (9376 bytes)
received block 44450bff453b2f67782fa2ffd0a53aca3e7ab5d65a8c04f570209be25ee4f9fa
ERROR: CheckBlock() : size limits failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (7684 bytes)
received block 18fae94924efeb2b28e08384f4f6b4fc4673f5b72ad0c48bca812f46d3988907
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (213 bytes)
received block f973da5bfd02e44d99236aa255d24f37a7c3ad905c2d4ffad296e31ae24ded06
ERROR: CheckBlock() : size limits failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (91 bytes)
received block 478bac48864a27b6ff87f9a91faf84039388da0cf8b6afe0a9c653172ff13cdc
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (7221 bytes)
received block 880f5dd6b2010f17bd457649326df77fefc86353073cd6666a5209c8ea920b78
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (7308 bytes)
received block bda77e701135bad77b87e4dfa8df84a0d1c220c4d898815679d4df0a070a981d
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (14037 bytes)
received block 5578fecf8f6511bab1216da42694deec24cc7b53e788bee9510b87a36c0f134b
ERROR: CheckBlock() : size limits failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (19691 bytes)
received block 8dade054a23459eb1688a6d296a20fd1f5eab40a5662e764cb6024879affc3f1
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (172 bytes)
received block 4c70ebda2df275c9dd2d152ab2b9968e3a2cc32af0c3114a9ebb5eaa2710716c
ERROR: CheckBlock() : size limits failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (8417 bytes)
received block 99f4d77fefff4fa76c404b3ba6f5f99b5a5cc40df16669ff2c609a7e2031db5a
check auxpow err 1
ERROR: AuxPow is not a generate
auxpow err 5
ERROR: CheckAuxPowProofOfWork : AUX POW is not valid
ERROR: CheckBlock() : auxpow proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (11123 bytes)
received block 47986fd13d63ce2aa0e87bf3826739cae3cd9a45bb88f9fce7d0efedeca16e50
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (208 bytes)
received block b6a70fc6845f878546cb916e201e8df644d0ac4b2597d686a6e6400c64644773
ERROR: CheckBlock() : size limits failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (39299 bytes)
received block a87cf133d4c6b74e1a03f7718f33a810a17ec2e1f7de7cf4fc4decfcce2c60aa
check auxpow err 1
ERROR: AuxPow is not a generate
auxpow err 5
ERROR: CheckAuxPowProofOfWork : AUX POW is not valid
ERROR: CheckBlock() : auxpow proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (11337 bytes)
received block 5fa092b1844aa839ba894f8cb81d0dc29ed4da4c7aaade6319ff9f1f465ff71a
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (12681 bytes)
received block da1d459a890702636cabab7aae29119ba9a8a9aa275342ecf3ea0f6928278bda
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (1402 bytes)
received block 32df0aaaec4a2ef37b4a8362f25b175e339db332f13b70e752eecbf5affa3dd1
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (4438 bytes)
received block 824f753966cab50ed248c2e3ce210998bdb00a67501d1ed5cae73c9b9e9ba6f5
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (16460 bytes)
received block 803879d05d90b1eee6c041216febd789f1415ee2181d9abf55f9875d6aad0b8f
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (11741 bytes)
received block 1ee431f32e74a0d2ff61f2556a50fdc44de7947a7585642be87ece528d50b5d2
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (81 bytes)
received block dd8cf762dbd490d0a234796ff5f5fd7cb7f13344ae31b1964b17aa97562879ac
ERROR: CheckBlock() : size limits failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (5718 bytes)
received block 507aa2e8f8bcc8c4792f50686f8115396346ede5a021ef9b2cf0699d8547e5b3
ERROR: CheckProofOfWork() : nBits below minimum work
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (15832 bytes)
received block f997af18bba58e6f732416e2626aa04d0b06c432e10a1473a12214b2ef5e5bd6
auxpow err 1
ERROR: CheckAuxPowProofOfWork : block does not have our chain ID (got 22888, expected 91, full nVersion 1500000000)
ERROR: CheckBlock() : auxpow proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED
received: block (8077 bytes)
received block f29045e04deaaebe1047fee54649809769b6f270f88148e2ebc43551b3b551cd
auxpow err 1

[jarr0s]

Release binaries provided don't have this problem, only when building from source.

[dbkeys]

Tried to reproduce issue, compiled from latest 'master' branch source (v0.9.7.1), starting from scratch, worked fine.

partial debug log:
.......
2018-06-17 09:27:05 UpdateTip: new best=a5c07bcd1ee95258ac2edeb7d93a87d3205272d6c712a9fac94d95ae38c96486 height=450626 log2_work=72.835167 tx=1089797 date=1526090543 progress=0.709178 nbits=456215360 algo=0
2018-06-17 09:27:05 ProcessBlock: ACCEPTED
2018-06-17 09:27:05 fScriptChecks true
2018-06-17 09:27:05 fScriptChecks true
2018-06-17 09:27:05 UpdateTip: new best=9df459786d64f926e91ac357146e0a11b78fb8d4e70bf2cc10943434a936f528 height=450627 log2_work=72.835175 tx=1089800 date=1526090544 progress=0.709181 nbits=456215360 algo=0
2018-06-17 09:27:05 ProcessBlock: ACCEPTED
2018-06-17 09:27:05 fScriptChecks true
2018-06-17 09:27:05 fScriptChecks true
2018-06-17 09:27:05 UpdateTip: new best=c08f9fa005400ffbd0f750b5e3c67d0bf79674e9b9343ba88cbd62f2a3442481 height=450628 log2_work=72.835183 tx=1089803 date=1526090545 progress=0.709184 nbits=456215360 algo=0
2018-06-17 09:27:05 ProcessBlock: ACCEPTED
2018-06-17 09:27:05 fScriptChecks true
2018-06-17 09:27:05 fScriptChecks true
2018-06-17 09:27:05 UpdateTip: new best=a75a5eae1f633fc1890ad977b76322fec2f51db5006aef40c5a8d1988b289c19 height=450629 log2_work=72.835191 tx=1089806 date=1526090546 progress=0.709187 nbits=456215360 algo=0
2018-06-17 09:27:05 ProcessBlock: ACCEPTED
2018-06-17 09:27:05 fScriptChecks true
2018-06-17 09:27:05 fScriptChecks true
.......

Try reindexing, 'bitmarkd --reindex', or downloading the blockchain from scratch.

[jarr0s]

It seems it happens randomly. I've tired downloading blockchain from scratch and got stuck again, but this time I continued restarting bitmarkd until it pull through.

[piratelinux]

We should add a new checkpoint soon... Also when we sync with the latest Bitcoin Core, network issues should improve.

[dbkeys]

Also, try starting daemon with -reindex option.

[jarr0s]

Guys, why did you closed the issue without fixing it.
I tried again to download blockchain on a new machine, and I couldn't sync up the blockchain. After a multiple attempts, I had to copy blockchain data manually from one machine to another.

@dbkeys you wrote on bitcointalk that cryptopia has problems syncing up the chain, I believe this is the same issue.
Problem is more than reproducible, and any exchange should have problem with blockchain sync, not just cryptopia.
This is critical bug at this moment.
@piratelinux

[dbkeys]

Ok, have re-opened for further investigation. If you can post logs, particularly at point where it gets stuck it would be useful.

[piratelinux]

Strange because I've been consistently compiling and syncing nodes from scratch. Maybe because I'm using the dev branch? Let's see today, we should have a new release merged from dev.

[jarr0s]

@dbkeys Log more or less looks like as the one I already posted. But here is the one with little more data, before the stuck.
btm_debug.log

It is interesting that all this block hashes received are non existent block hashes.

Managed to reproduce on release binaries also.

[melvincarvalho]

FYI : I compiled this from source this week and it synced correctly