borg check --repair --verify-data: inconsistent/sporadic chunk id verification failure

[jurgenhaas]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

ISSUE

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.1.16

Operating system (distribution) and version.

Ubuntu 20.04

Hardware / network configuration, and filesystems used.

Intel NUC with ect4

How much data is handled by borg?

400GB on that host, 274GB in the problematic repository

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg check --repair --verify-data --verbose /var/backups/borg/nextcloud_privat

Describe the problem you're observing.

My monthly check reported a problem with the integrity of some chunks, so I started to analyse the repository. The command above logs the following statement:

Starting repository check
Starting repository index check
Completed repository check, no problems found.
Starting archive consistency check...
Enter passphrase for key /var/backups/borg/nextcloud_privat:
Starting cryptographic data integrity verification...
chunk f856bf715978bb1a4acc9d2f4b23bfc319a2452058541ae5c8ee086b75279ff9, integrity error: Data integrity error: Chunk f856bf715978bb1a4acc9d2f4b23bfc319a2452058541ae5c8ee086b75279ff9: id verification failed
Found defect chunks. They will be deleted now, so affected files can get repaired now and maybe healed later.
chunk %s not deleted, did not consistently fail.
Finished cryptographic data integrity verification, verified 298463 chunks with 1 integrity errors.
Analyzing archive ps1-2020-05-31T04:07:42.711754 (1/28)
Analyzing archive ps1-2020-06-30T04:12:33.529670 (2/28)
Analyzing archive ps1-2020-07-31T04:13:08.699011 (3/28)


Analyzing archive ps1-2021-06-03T04:33:13.815852 (27/28)
Analyzing archive ps1-2021-06-04T05:15:33.195727 (28/28)
Writing Manifest.
Committing repo (may take a while, due to compact_segments)...
Finished committing repo.
Archive consistency check complete, problems found.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, no matter how often I run this. It continues reporting that integrity error but still doesn't delete the reported chunk.

[ThomasWaldmann]

chunk %s not deleted, did not consistently fail.

this is obviously a bug (you should not see %s but a chunkid), there was an argument missing there.

Fixed by #5823.

[ThomasWaldmann]

About your "bug" report: AFAICS now, this is not a bug, but expected borg behaviour:

chunk f856bf715978bb1a4acc9d2f4b23bfc319a2452058541ae5c8ee086b75279ff9, integrity error: Data integrity error: Chunk f856bf715978bb1a4acc9d2f4b23bfc319a2452058541ae5c8ee086b75279ff9: id verification failed
Found defect chunks. They will be deleted now, so affected files can get repaired now and maybe healed later.
chunk (like same id as above) not deleted, did not consistently fail.

Before actually deleting stuff with integrity problems, borg is careful:

It just tries a second time to get the chunk from the repo and verifies the mac, decrypts, decompresses the data and then checks if the wanted-chunkid corresponds to the computed-chunkid:

                        encrypted_data = self.repository.get(defect_chunk)
                        _chunk_id = None if defect_chunk == Manifest.MANIFEST_ID else defect_chunk
                        self.key.decrypt(_chunk_id, encrypted_data)

The main idea with that code is not to delete chunks just because of sporadic failures.

If it doesn't cause issues again, you get that "did not consistently fail" message and it does not delete the chunk.
If it would cause issues again, it would delete the corrupt chunk.

The bad news is of course:

• that there must be some root cause for sporadic failures and "id verification" one time failing and one time succeeding
• that can't be explained by e.g. some I/O problem, but IMHO points to unreliable CPU, RAM or mainboard.

So I suggest you run some passes of memtest86+ on that system.

[ThomasWaldmann]

After you have verified that CPU/RAM is working ok, you could also run the borg repair command again.

It would be interesting to see if it still finds problematic chunks and if so, whether they have same of different ids.

[ThomasWaldmann]

Theoretically, there could be also a bug in borg's code (incorrect detection of inconsistent behaviour), but let's first assume that there actually is inconsistent behaviour.

[jurgenhaas]

Sorry for the delay, I'm still trying to get to the GRUB menu during startup to run the memtest from there. Still trying, getting back.

[ThomasWaldmann]

@jurgenhaas for memtest86+ you need to boot with legacy BIOS and you usually can get to the grub menu with hammering shift or esc(?) after the bios screen (or reconfigure grub to always show the menu for some seconds).

you could also make a USB stick with memtest86+ and boot from that via BIOS boot order (F12 or so).

[jurgenhaas]

OK, I get the Grub menu displayed during boot but it doesn't contain memtest86+. I've looked into /etc/grub.d/20-memtest86+ and there is an if clause that exits because /sys/firmware/efi is available. When I switch to legacy boot mode in BIOS, no bootable device is being found. Then I flushed memtest86+ to a USB stick, but that's not recognized as bootable either.
I know this is off-topic here but I still appreciate any help on getting this somehow tested.

[ThomasWaldmann]

Some devices might be only bootable via UEFI, e.g. NVME PCIe SSDs.

But if your system still supports legacy boot, you should be able to boot from a memtest86+ usb stick.
Some BIOSes might need the USB device to be added to the boot order or enable USB init at boot time or so.

If you still have a DVD drive (internal SATA or external via USB), you could also use a ubuntu 18.04 DVD, it has a working version of memtest86+.