DVC should be able to throw an error if there are multiple versions of the same file in S3

[lsb]

(I am happy to write+contribute this feature, and am mostly interested in community interest in this feature.)

There is an S3 feature ("object lock") for compliance purposes that will keep arbitrarily many version of any object in a bucket.
When overwriting an object at a particular key, this will keep the prior version for the specified duration.

There are one in a trillion odds of two objects hashing to the same 128 bits, after storing 26 trillion objects, more or less.
If there is more than one version of an object, someone is doing something funny!

I think it might be useful to have a flag that one can set, to allow the S3 backend to check how many versions of each file exist before fetching, and then abort when the number is >1 for any file.

[efiop]

Hi @lsb !

Do you mean in dvc import-url scenario? Or push/pull?

[lsb]

I think I mean in the push/pull scenario, when fetching objects from the remote, ensuring that there's only one version of an object with a particular hash on the remote.

[efiop]

@lsb We try to push in an atomic way, so if there was a previous version - new version won't be uploaded. The scenario you describe might happen if someone does this by-hand. Or am I missing something?

[lsb]

The workflow I am foreseeing is this:

One. Alice uses DVC to add file F1, with hash H1, to key H1. Version 1.
Two𝛼. Mallory observes F1@H1, adds F2@H2 to key H1. Version 2, obviously broken.
or
Twoβ. Mallory observes F1@H1, constructs F2@H1 because MD5 is less than secure, adds F2@H1 to key H1. Version 2, less obviously broken.
Three. Bob uses DVC to retrieve key H1, checks all the versions of objects at key H1, sees two versions, aborts.

S3 Object Lock preserves prior versions of objects at keys for specified durations, so it would need to be enabled on any bucket that is a remote for dvc. Other backends might support this but I haven't checked.

[efiop]

Two𝛼. Mallory observes F1@H1, adds F2@H2 to key H1. Version 2, obviously broken.

@lsb Are you talking about a hack? This is not meant to happen during normal dvc operation.

[lsb]

That's exactly right: Mallory is maliciously accessing the blob storage through a side channel outside of dvc shell commands.

Specifically, if I'm using dvc in dev and staging and prod, I would expect no one steps on each others' toes, because of the length of the hashes for the content-addressable storage, and I might want to confirm that there isn't any funny business happening from any machines overwriting prod files.

[efiop]

@lsb Sure, that would be a nice check to have, if it doesn't cost us performance (if it does - maybe as an opt-in config option). Still, if someone malicious has access to your s3, you probably have bigger problems than this. We also have verify config option for remotes, that makes dvc manually verify hashes that it receives from remote, instead of trusting them - that is probably a more general guard against such funny business.

[efiop]

Closing as stale for now. @lsb if you are still interested, feel free to give it a shot, but be aware of #5683