Skip to content

Mark object store bucket as full and move new files into new bucket #22039

@MorrisJobke

Description

@MorrisJobke

Problem

  • the vendors of object stores give out a recommendation the maximum number of objects per bucket
  • Nextcloud should respect those limits and if a bucket is close to this value it should not put more files into it
  • new files should be put into a newly created bucket
  • multibucket setups should be able to be marked as "extensible" - so there is a default bucket count and as it's capacity of a bucket is reached then a new bucket is created

Things to keep in mind

  • we should look into storing the bucket a file is located in instead of calculate it on the fly:
    • instead of having this implicitly (calculating the hash and have a formula in which bucket this hash goes) we should store the result in the database so we can move different files that would be in the same bucket can also be moved based on the amount of files in the bucket
  • what about a migration? this would allow to rebalance buckets that are already over this limit and misbehave for that reason

@rullzer @kesselb @icewind1991 Feedback and ideas are welcome.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions