Files cache: use ctime instead of mtime

[enkore]

ctime can't be set via userspace, while mtime is easily manipulated.

For most files (e.g. not extracted from tarballs, or installed by a package manager) these two are the same, which limits the impact of the change on existing file caches.

• Works with sshfs? -> (YES) (see below)

@verygreen noted that in the Windows world ctime usually means "creation" not "change" time

• Works with windows+cygwin? -> YES
• Works with native windows? -> NO

[ThomasWaldmann]

as we always backup file metadata, we are only interested in file contents changes (not also in file metadata changes). thus mtime fits better.

[verygreen]

Well, here lies the risk.
Imagine I have a file, run borg backup, it stores the fiel data, remembers the mtime, size and stuff.

Now I modify the file (without changing the size), and then use utimes to return the mtime back to what it was - now next run of borg would miss the changes.

[ThomasWaldmann]

then you have shot yourself in the foot. :-P

[verygreen]

What if it was not me but:

1. package manager updated a file (pretty common, though changed content is probably a bit less common).
   1.1 untar or cp -a or something similar updated a file
2. evil hackers updated a file (in conjunction with people that use borg diff to detect intrusions potentially, there was a ticket about this).

[ThomasWaldmann]

well, borg is not meant to be a intrusion detection system.

as far as a backup program is concerned, it would just not backup the hacked binary.

even if we used ctime, an attacker could still modify the binary, then modify the borg cache and update the ctime there, so there is no safety against this.

[ThomasWaldmann]

Besides "hackability" (detecting which is not our goal), mtime seems a better fit as we need the files cache to detect content changes (to re-chunk, if needed, but only if needed).

So, isn't ctime the wrong timestamp anyway for this?

---

ctime

ctime is the inode or file change time. The ctime gets updated when the file attributes are changed, like changing the owner, changing the permission or moving the file to an other filesystem but will also be updated when you modify a file.

mtime

mtime is the file modify time. The mtime gets updated when you modify a file. Whenever you update content of a file or save a file the mtime gets updated.

Most of the times ctime and mtime will be the same, unless only the file attributes are updated. In that case only the ctime gets updated.

---

[ThomasWaldmann]

we do not want to detect file attribute changes like owner - borg stores them always anyway.
we ONLY want to detect if file content changes (and then we need to re-chunk the file).

so, close?

[verygreen]

As long as you ignore deliberate abusers that update a few bytes in the file (leaving hte size intact) and then reset the mtime to what it was (mtime is changeable by user, ctime is not), it's ok to close.

[d3zd3z]

I have had the debian package manager, on Ubuntu, replace files, setting the mtime back to what it was before. Since the file was the same size, rsync did not back it up (neither would have Borg). The other backup software I used that correctly used the ctime backed it up just fine.

I'm not sure I see why this is a debate. Using ctime will always back up a file when it changes. Using mtime will usually back up a file that has changed. Why would we want to only do the correct thing most of the time. The cost, as mentioned above is possibly having to reread a file that had its metadata changed.

Could we not somehow make this an option? For those that want the mostly-correct behavior, it could use mtime, and those of us that would actually like everything backed up, to use the ctime?

[ThomasWaldmann]

@d3zd3z so, that sounds like a bit stupid behaviour of the package manager in the first place. did it also not change the inode number? because borg considers (mtime, size, inode) by default.

[d3zd3z]

At least dpkg will overwrite files if already present, so the inode number doesn't change. The only thing that was different on the file was the ctime. There are also revision control systems that set the mtime on files. At least those would be expected to put a time on the file that matched its contents, so if it ended up with the same mtime, it would also have the same contents.

But there are lots of potential pieces of software out there, and it is hard to know what kinds of crazy things people could do. Even the 'touch' command can easily put an mtime back.

[d3zd3z]

Someone asked for when ctime/mtime are updated:

• mtime is updated when there is a change to the content of the file. It can also be set by the utime() .
• ctime is updated when any change is made to the file, whether it be metadata, or the contents of the file.

[d3zd3z]

Additional information:

• Per POSIX, ctime will be updated when a change of any kind is made to a file. This will apply to most native filesystems on POSIX or POSIX-like systems.
• The ctime cannot be set by the user. The only way it can be faked is to change the system clock before modifying a file.
• The mtime is updated when the data of the file changes, it can also be set to an arbitrary value with the utime() or utimes() call.
• The only time ctime wouldn't be updated is when a filesystem that doesn't support it is being backed up. For example, backing up a FAT filesystem.
• Reading a file will modify the atime (but not the ctime).
• Setting the atime with utime() or utimes() will cause the ctime to be set to the current time.

[d3zd3z]

Just verified, OSX also returns mtime in the ctime field when mounting a filesystem that doesn't support a ctime. This means if we use 'ctime' on one of these filesystems, we would get the same behavior we have now (barring consistent inodes).