panic: unable to sanity check wallet birthday block

[stridentbean]

Background

I upgraded an lnd node from commit d8863be to b07499f (0.5.1-beta)

Then started seeing the error "panic: unable to sanity check wallet birthday block: failed to store sync information 000000000000000000030c1ff057d297fd8ad5b29a882e7f0a0e16f9da9f9ee5: failed to fetch block hash for height 554507: block not found"

I've done this on several raspberry pi's and most worked fine, but a couple have seen the error above. One of the devices that had an error corresponding to an out of memory error. My working theory is that during the db migration from the first commit to the second, the pi sometimes runs out of memory.

Potential fix

I'm curious if deleting the wallet.db file and recreating it with the same seed words would work. I would leave the channel.db and sphinxreplay.db to preserve the channels states. Any thoughts on that?

Your environment

• version of lnd
  0.5.1-beta commit=v0.5.1-beta
• which operating system (uname -a on *Nix)
  Linux lnd 4.14.70-v7+ Fixing some README typos #2 SMP Wed Sep 19 07:49:26 UTC 2018 armv7l Linux
• version of btcd, bitcoind, or other backend
  bitcoind: 0.17.0
• any other relevant environment details
  bitcoind and lnd are running in docker containers
  raspberry pi 3b+

Steps to reproduce/ stack traces

Out of memory error at time of lnd upgrade

2018-12-21T12:23:08Z lnd UNKNOWN[4088] runtime: out of memory: cannot allocate 2703360-byte block (1793949696 in use)
2018-12-21T12:23:08Z lnd UNKNOWN[4088] fatal error: out of memory
2018-12-21T12:23:08Z lnd UNKNOWN[4088] 
2018-12-21T12:23:08Z lnd UNKNOWN[4088] runtime stack:
2018-12-21T12:23:08Z lnd UNKNOWN[4088] runtime.throw(0x9962e8, 0xd)
2018-12-21T12:23:08Z lnd UNKNOWN[4088] #011/usr/local/go/src/runtime/panic.go:608 +0x5c
2018-12-21T12:23:08Z lnd UNKNOWN[4088] runtime.largeAlloc(0x292dd4, 0x1c30100, 0x6a418)
2018-12-21T12:23:08Z lnd UNKNOWN[4088] #011/usr/local/go/src/runtime/malloc.go:1021 +0x120
2018-12-21T12:23:08Z lnd UNKNOWN[4088] runtime.mallocgc.func1()
2018-12-21T12:23:08Z lnd UNKNOWN[4088] #011/usr/local/go/src/runtime/malloc.go:914 +0x38
2018-12-21T12:23:08Z lnd UNKNOWN[4088] runtime.systemstack(0x76f6c6c8)
2018-12-21T12:23:08Z lnd UNKNOWN[4088] #011/usr/local/go/src/runtime/asm_arm.s:354 +0x84
2018-12-21T12:23:08Z lnd UNKNOWN[4088] runtime.mstart()
2018-12-21T12:23:08Z lnd UNKNOWN[4088] #011/usr/local/go/src/runtime/proc.go:1229

sanity check error

2019-01-01T23:01:54Z lnd UNKNOWN[29439] 2019-01-01 23:01:54.092 [ERR] LNWL: unable to sanity check wallet birthday block: failed to store sync information 00000000000000000010091f141b0a06337eb99d16dbc09643abec28739ccfd3: failed to fetch block hash for height 554372: block not found
2019-01-01T23:01:54Z lnd UNKNOWN[29439] panic: unable to sanity check wallet birthday block: failed to store sync information 00000000000000000010091f141b0a06337eb99d16dbc09643abec28739ccfd3: failed to fetch block hash for height 554372: block not found
2019-01-01T23:01:54Z lnd UNKNOWN[29439] 
2019-01-01T23:01:54Z lnd UNKNOWN[29439] goroutine 86 [running]:
2019-01-01T23:01:54Z lnd UNKNOWN[29439] github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet.(*Wallet).handleChainNotifications(0x2428000)
2019-01-01T23:01:54Z lnd UNKNOWN[29439] #011/go/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet/chainntfns.go:126 +0xdd0
2019-01-01T23:01:54Z lnd UNKNOWN[29439] created by github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet.(*Wallet).SynchronizeRPC
2019-01-01T23:01:54Z lnd UNKNOWN[29439] #011/go/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet/wallet.go:186 +0x124

Expected behaviour

Upgrading lnd failed and now yeilds a sanity check error.

Actual behaviour

Upgrade should work fine

[Roasbeef]

You ran out of memory. Add swap. Also looks like there was a re-org at that height?

[jsindy]

My Casanode hit the same issue, any recommendations to reset LND?

[wpaulino]

@stridentbean do you have the logs that led to the crash?

[stridentbean]

Here is one set of logs where we did encounter a mem issue.
logs.txt

[Roasbeef]

From that log message, it looks like bitcoind was restarted, or somehow the zmq connection was borked. I'd ensure you're on the latest version of lnd as we've fixed a number of issues with zmq, and also add swap if you must run both bitcoind and lnd on the same rasp pi as bitcoind can be very memory heavy.