Multibucket objectstore

[icewind1991]

The current primary objectstore runs in scaleability issues with very large instances since it stores all objects into a single bucket and not all s3/swift implementations seem to like having multiple million objects in a bucket.

To work around this the plan is to add the ability balance objects between multiple buckets

Balance methods

There are 2 main distinct options to handle the balancing

1. Per user balancing

   Either each user gets their own bucket (need to ensure that the used objectstore can handle very large amounts of buckets) or spread users evenly over N buckets (larger but less buckets).
   Either way, all user files stay on a single bucket which has the disadvantage that, since users probably don't have the same usage patterns, files are distributed over buckets unevenly

2. Per file balancing

   Unlike 1. this doesn't keep user files together, instead this spreads all files evenly over N buckets (something like $bucketId = $fileId % $numberOfBuckets), this makes sure all files are spread evenly no matter what the usage pattern is, even if a single user has tens of millions of files it will still balance.

Personally I favor 2. since it's a simpler solution and I feel that it solves the problem (the storage not handling very large buckets well) better, although it's not without it's downsides

Changing balance methods

A thing that should be taken into account is whether we want to support changing balance methods (like increasing the number of buckets used) on an existing system.

Since moving all objects around according to the updated balancing scheme is not practical we would need to add some way where existing files/users still use the old scheme while new ones are on the updated scheme.

For per-user balancing this can simply be done by storing the bucket id per user. For per-file balancing storing the bucket for each file is probably not practical.
One way to handle per-file balancing is instead storing the balancing scheme user per range of file ids. Since ids are incremental we only need to store that files 1 to 100000 use 10 buckets, and all newer files use 20

[icewind1991]

One of the main reasons why I feel per-file balancing is "easier" is that it can be done in the objectstore implementation level without any additional information (such as the user id for the owner of a file)

[rullzer]

As discussed the big problem with 2 is that you have to be very very careful when every chaning the number of buckets.

[despens]

I believe solution 2 will make it more difficult to migrate to another object store at a later point, or to migrate away from nextcloud.

[icewind1991]

migrate away from nextcloud

migration is not a supported usecase in the first place when using a primary object store, since we use file ids as object names in the first place

[despens]

If that means I can store .htaccess files on object store, count me in :)

[benmichael]

Either way, we need to keep in mind that AWS S3 (which will be the majority of the users of this) has a default limit of 100 buckets per account. This can be lifted by contacting customer support, but that's a poor experience for the users of NextCloud.

Also, obviously, we'll need to have some bucket name resolution when we try create a bucket, and find that the name is taken.

[rullzer]

@benmichael ah that is a good remark. That would probabaly mean we need to make it configureable. But that should be doable.

[icewind1991]

per-user multibucket is currently merged for Nextcloud 11

[jospoortvliet]

@icewind1991 what about the issue @benmichael mentioned, of a potential limit on the nr of separate buckets? Does this mean Nextcloud 11 can't handle more than 100 users per account on S3 right now?

[icewind1991]

you can limit the number of buckets used

[petejodo]

This is an old issue but I'm curious what occurs when I have a single objectstore as primary storage (currently configured using a Digitalocean space) and want to change it to multi-bucket and add an additional Digitalocean space, and then down the line I again want to add another one? Specifically in the case of group folders.