Strange behaviour

[ddundovic]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Bug/issue

System information. For client/server mode post info for both machines.

Client only setup.

Your borg version (borg -V).

borg 1.1.10

Operating system (distribution) and version.

Fedora 30 x64, all the latest updates.

uname -a:
Linux xxx 5.3.8-200.fc30.x86_64 #1 SMP Tue Oct 29 14:46:22 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Hardware / network configuration, and filesystems used.

Intel C2Q CPU Q9300, 8GB DDR2 RAM, 5 SATA HDDs
On two disks involved in the backup process EXT4 filesystem is used.
Wired network.

How much data is handled by borg?

24GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

I've created first backup by these two lines:

borg init --encryption='repokey-blake2' /mnt/sde1/xxx/blake2 --make-parent-dirs --append-only
borg create --stats --compression lzma,9 /mnt/sde1/xxx/blake2::pics-{now:'%Y-%m-%d'} ~/Pictures

Backup was created and no errors reported.
Stats output was:

------------------------------------------------------------------------------
Archive name: pics-2019-11-08
Archive fingerprint: 71440fc831651d5708e2d6ac81cd4c0cb6cfa2538fcff1c6287b5eebd8afbb4a
Time (start): Fri, 2019-11-08 03:33:35
Time (end):   Fri, 2019-11-08 06:28:16
Duration: 2 hours 54 minutes 40.85 seconds
Number of files: 6548
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               24.95 GB             24.85 GB             24.74 GB
All archives:               25.04 GB             24.94 GB             24.74 GB

                       Unique chunks         Total chunks
Chunk index:                   14927                15105
------------------------------------------------------------------------------

Side note: It appears that when using archive placeholder "{now}" and want to apply format to it, the format string has to be in quotes i.e. '%Y-%m-%d'
That's not how it's written in the docs and examples.

Describe the problem you're observing.

Borg behaves erratically. Same command sometimes fails, sometime succedes, sometimes fails with different error message.

For instance:

$ borg list /mnt/sde1/xxx/blake2
pics-2019-11-08                      Fri, 2019-11-08 03:33:35 [71440fc831651d5708e2d6ac81cd4c0cb6cfa2538fcff1c6287b5eebd8afbb4a]

$ borg list /mnt/sde1/xxx/blake2
self test test_chunkify (borg.testsuite.chunker.ChunkerTestCase) FAILED:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/site-packages/borg/testsuite/chunker.py", line 17, in test_chunkify
    self.assert_equal(b''.join(parts), data)
AssertionError: b'000[1396729 chars]0000080000000000000000000000000000000000000000[11186131 chars]000Y' != b'000[1396729 chars]0000000000000000000000000000000000000000000000[11186131 chars]000Y'

self test failed
This is a bug either in Borg or in the package / distribution you use.

These two commands were issued one after the other, few seconds between them.

When I try to check the backup, I get lots of errors.

$ borg check /mnt/sde1/xxx/blake2
Data integrity error: Segment entry checksum mismatch [segment 2, offset 260756192]
Data integrity error: Segment entry checksum mismatch [segment 6, offset 521544632]

...

Index object count mismatch.
committed index: 14928 objects
rebuilt index:   10607 objects
ID                                                               rebuilt index    committed index 
f0ec44d44323adc19b6549656cd5b8d4e29d96ba592090c69130a65f68ce067d (57, 33858225)   <not found>     
7a852f986de65ec6dad95862ed7525dec673a482217861cdd41dff821952ad19 (33, 2311117)    <not found>     
aae473bb0d07b557af16a77d9629c6545ff952816d3bae5ba12b3b81fc5d10a5 <not found>      (25, 327251456) 
46dab8f998eadd92dcdf8c1c8d1e03a2c95c009d490b756dc983fc7274d3d3ea <not found>    

...

Completed repository check, errors found.

4344 lines in total.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes. Every time I create new repository and archive using the two lines from above and later check the repo, I get errors.
When I try to list repo, sometimes I get "self test failed", other times it lists the archives in the repo.

I saw in the docs and here, in the Issues section, that this might be a hw problem.
So I checked the backup disk, smartctl -t long and wd diag long test, no errors were found.
I also ran Memtest86+, it ran for 24 hours/10 passes, no errors found.

Include any warning/errors/backtraces from the system logs

[ThomasWaldmann]

AssertionError:
# -----------------------v-------------------------
b'000[1396729 chars]0000080000000000000000000000000000000000[11186131 chars]000Y' !=
b'000[1396729 chars]0000000000000000000000000000000000000000[11186131 chars]000Y'
# -----------------------^-------------------------

A single bit corruption does not get much more obvious than that.

Might be your RAM, your CPU or some other component.

[ThomasWaldmann]

That system also seems to be pretty old (CPU came out 1/2008), so age related issues are somehow expected.

[ddundovic]

Just a quick follow-up:

Originally I've created this backup using two different physical disks, one as source and one for destination.
Yesterday, on the same computer, I've tried again using only one disk both as source and destination (first one disk from the original post and then the other).
Results are exactly the same, erratic behaviour, "self test failed" in "list" command, "data integrity error" in "check" command.

[ThomasWaldmann]

Can you reproduce the issues after upgrading to 1.1.11 (see changelog and advisory at the top of it, there are some actions required to deal with potential corruption).

[ThomasWaldmann]

Pretty sure this is a hw issue, closing.