HBRT infinite loop on ECC error during startup

Start opal-prd and observe this log before opal-prd gets stuck at 100% CPU.

HBRT: PRDF:>>PRDF::main() Global attnType=0004
HBRT: PRDF:>>PRDF::noLock_initialize() 
HBRT: PRDF:>>PegasusConfigurator::build()
HBRT: PRDF:<<PegasusConfigurator::build()
HBRT: PRDF:<<PRDF::noLock_initialize() 
HBRT: ERRL:>>ErrlManager::ErrlManager constructor.
HBRT: ERRL:iv_hiddenErrorLogsEnable = 0x0
HBRT: ERRL:>>setupPnorInfo
HBRT: PNOR:>>RtPnor::getSectionInfo
HBRT: PNOR:>>RtPnor::readFromDevice: i_offset=0x0, i_procId=0 sec=11 size=0x20000 ecc=1
HBRT: PNOR:RtPnor::readFromDevice: removing ECC...
HBRT: PNOR:RtPnor::readFromDevice> Uncorrectable ECC error : chip=0,offset=0x0

(at which point everything stops with opal-prd chewing 100% CPU)

Which ends up being a fairly classic race in trying to log an error before everything has been initialized.

Consequently, opal-prd spins a core and is right off into the weeds.

[dcrowell77]

Ironically enough we just encountered this in a different environment. I'm surprised it took this long. I've got some hacks that need to be ironed out. Since this is a secondary fail it will go on the backburner for a little while.

FWIW, the workaround is to erase the HBEL partition:

pflash -P HBEL -e

and then you'll be okay.

Or, rather, okay until you reboot, then you fail to IPL:

  0.61819|ECC error in PNOR flash in section offset 0x00008000

  0.62322|System shutting down with error status 0x60F
  3.22583|Ignoring boot flags, incorrect version 0x0
  3.30757|ISTEP  6. 3
  1.09243|ECC error in PNOR flash in section offset 0x00008000

  1.09246|System shutting down with error status 0x60F
  3.68140|Ignoring boot flags, incorrect version 0x0
  3.76426|ISTEP  6. 3

and then you get into this stupid inescapable mess that is the golden side and your life is horrible.

But it seems that by flashing zeros over rather than just erasing will do the trick.

[dcrowell77]

Resolved with 1e784c0. HBRT will now assert and crash if we get early life PNOR failures instead of doing an infinite loop that pegs the cpu.