Skip to content

Conversation

@icewind1991
Copy link
Member

@icewind1991 icewind1991 commented Oct 18, 2024

Support migrating existing instances to sharding.

Enabled by setting the sharding configuration and setting the from_shard_key and from_primary to the sharding configuration with the next storage/file id (so the first storage/file that will be created after sharding is configured).

The schema from the main db will need to be copied to the shard db's before enabling the sharding (having a command to help with this would be a nice stretch-goal)

Any files on the existing storages will still go to the main db, only newly created storages will be sharded.

@tcitworld
Copy link
Member

Things seem to work, but in some cases, the "File entry could not be inserted but could also not be selected with getId() in order to perform an update. Please try again." error gets thrown. (creating a new user is the most reliable way I found).
While things seem to work otherwise, I suspect that this issue is a sign of a bigger connection handling issue.

I had tried #36313 a while ago

@icewind1991 icewind1991 force-pushed the sharding-existing branch 6 times, most recently from ec107dc to 756e35a Compare November 11, 2024 14:35
@icewind1991 icewind1991 changed the title [wip] Support sharding migration feat: support migrating an instance to sharding Dec 16, 2024
@icewind1991 icewind1991 marked this pull request as ready for review December 16, 2024 19:23
@icewind1991 icewind1991 requested a review from come-nc December 20, 2024 13:09
@icewind1991 icewind1991 added the 3. to review Waiting for reviews label Dec 20, 2024
@icewind1991 icewind1991 added this to the Nextcloud 31 milestone Dec 20, 2024
@joshtrichards joshtrichards added the feature: database Database related DB label Jan 2, 2025
Copy link
Contributor

@come-nc come-nc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks sane, did not test.

@blizzz blizzz mentioned this pull request Jan 8, 2025
@skjnldsv skjnldsv mentioned this pull request Jan 14, 2025
@icewind1991 icewind1991 requested review from a team, Altahrim, provokateurin and yemkareems and removed request for a team January 16, 2025 16:27
@skjnldsv skjnldsv mentioned this pull request Jan 16, 2025
This was referenced Jan 21, 2025
@blizzz blizzz mentioned this pull request Jan 29, 2025
1 task
@blizzz blizzz modified the milestones: Nextcloud 31, Nextcloud 32 Jan 29, 2025
Signed-off-by: Robin Appelman <robin@icewind.nl>
@icewind1991 icewind1991 merged commit b9aa397 into master Feb 25, 2025
190 checks passed
@icewind1991 icewind1991 deleted the sharding-existing branch February 25, 2025 18:42
@AndyScherzinger
Copy link
Member

/backport to stable31

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review Waiting for reviews feature: database Database related DB

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants