fix: sqlite may dead lock in ci

[yihong0618]

Which issue does this PR close?

Closes #5737

This patch fix 5737 by using max_connections=1

root:
for PR #3233 bring sqlite
and metion that:

Hi, @Xuanwo The test_delete_stream test also causes a database is locked error on concurrent deletions in the local file database. Do you have any suggestions?

• the root cause is in CI the sqlite database is holding the lock but waiting connections
• the issue for sqlx Add a specialized pool for SQLite that has a reader / writer split launchbadge/sqlx#459
• they also introduce a great article which is deserve to read: https://beets.io/blog/sqlite-nightmare.html for this
• the fix idea for this PR is from max_connections(1) to avoid using multiple writers w sqlite. boltlabs-inc/zeekoe#305 I think which is also right for OpenDAL

I tested local seems work fine, and waiting for CI

[Xuanwo]

I don't like this idea because it turns SqlitePool into a simple Mutex, preventing our users from reading concurrently. Maybe we could allow users to retry the request when they encounter a locked error? We can implement this by Error::set_temporary.

[yihong0618]

I don't like this idea because it turns SqlitePool into a simple Mutex, preventing our users from reading concurrently. Maybe we could allow users to retry the request when they encounter a locked error? We can implement this by Error::set_temporary.

preventing our users from reading concurrently.

for the one connection it will not, and its sqlite most of the case is one user

[Xuanwo]

for the one connection it will not, and its sqlite most of the case is one user

Hi, opendal has native support for op.read_with(path).concurrent(8).

[yihong0618]

I don't like this idea because it turns SqlitePool into a simple Mutex, preventing our users from reading concurrently. Maybe we could allow users to retry the request when they encounter a locked error? We can implement this by Error::set_temporary.

preventing our users from reading concurrently.

for the one connection it will not, and its sqlite most of the case is one user

seems you are right for opendal ci tests case, will try to a better way
P.S. I do not think error handle is better

[yihong0618]

I don't like this idea because it turns SqlitePool into a simple Mutex, preventing our users from reading concurrently. Maybe we could allow users to retry the request when they encounter a locked error? We can implement this by Error::set_temporary.

preventing our users from reading concurrently.

for the one connection it will not, and its sqlite most of the case is one user

seems you are right for opendal ci tests case, will try to a better way P.S. I do not think error handle is better

AND FYI: for most case, like lldap also force the connection to 1, I will try other way

lldap/lldap@884031d

[Xuanwo]

It makes sense to me to add a new configuration option called max_connections, allowing users to set this value, similar to how lldap does. Hardcoding this value into max_connections=1 is not acceptable to me.

Even after adding max_connections, we still need to ensure that OpenDAL functions properly when max_connections is greater than 1. In OpenDAL's context, every operation is safe to retry since we don't have large transactions here. Additionally, db locked is indeed a retryable error in opendal context.

[yihong0618]

It makes sense to me to add a new configuration option called max_connections, allowing users to set this value, similar to how lldap does. Hardcoding this value into max_connections=1 is not acceptable to me.

Even after adding max_connections, we still need to ensure that OpenDAL functions properly when max_connections is greater than 1. In OpenDAL's context, every operation is safe to retry since we don't have large transactions here. Additionally, db locked is indeed a retryable error in opendal context.

yes agree with you, will learn to find some better solution and force to 1 is a bad solution for this after check
the failed tests, if not a better way I'd like to use error and retry way, thanks

[Xuanwo]

Thank you @yihong0618 for working on this.