upload of large files fails with connection error

[alecbl]

Steps to reproduce

1. Upload a large file
2. Wait for upload of all chunks
3. Profit

Expected behaviour

• App waits for assmembly of chunks up to 5 or 10 minutes.

Actual behaviour

• After about 30s at 100% progress the upload fails with connection error on the client. A minute later the file shows up in place at the server.

Environment data

Android version:
LineageOS 17.1
(I observe this behaviour on several devices, all running LineageOS versions 16 and 17.1)

Device model:
Xiaomi MiA2, Xiaomi Mi5, Xiaomi Redmi Note 7
Stock or customized system:
customized

Nextcloud app version:
3.10.1

Nextcloud server version:
18.0.1

Logs

Web server error log

No relevant errors that I can see on server logs.

Nextcloud log (data/nextcloud.log)

Insert your Nextcloud log here

NOTE: Be super sure to remove sensitive data like passwords, note that everybody can look here! You can use the Issue Template application to prefill some of the required information: https://apps.nextcloud.com/apps/issuetemplate

This seems similar to Bug #2637.
The server in question has decent cpu and ram specs. The I/O performance for the data disk isn't great, and the 30s window only allows assembly of files up to around 600mb, more than that the upload fails.

[Unclezz]

Hi,
very similar issue for me.
Actually I cannot upload files larger than 1MB from mobile, while no issue from my computers via web interface.

It's happening with same version of Nexctloud (application and Docker container official image) mentioned above. Tried on different smartphones, same problem.

Logs from NGINX Proxy:

nginx--proxy | MY_IP - - [09/Mar/2020:18:30:29 +0000] "GET /index.php/204 HTTP/1.1" 204 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:29 +0000] "HEAD /remote.php/webdav/Documents/52.pdf HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "HEAD /remote.php/webdav/Documents/52.pdf HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "MKCOL /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "MKCOL /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.1" 405 247 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "PROPFIND /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:30 +0000] "PROPFIND /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.1" 207 2850 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"
nginx--proxy | MY_IP - MY_USER [09/Mar/2020:18:30:31 +0000] "MOVE /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e/.file HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1" "-"

Other attempt of same file from NextCloud Webserver:

cloud-app | 172.20.0.4 - MY_USER [09/Mar/2020:18:35:51 +0000] "HEAD /remote.php/webdav/Documents/52.pdf HTTP/1.0" 404 556 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1"
cloud-app | 172.20.0.4 - MY_USER [09/Mar/2020:18:35:51 +0000] "MKCOL /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.0" 405 914 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1"
cloud-app | 172.20.0.4 - MY_USER [09/Mar/2020:18:35:52 +0000] "PROPFIND /remote.php/dav/uploads/MY_USER/10680ce6d25577b01226fbd15ef3aa4e HTTP/1.0" 207 3585 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.10.1"
cloud-app | 172.20.0.4 - - [09/Mar/2020:18:35:53 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.0" 200 836 "-" "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"
cloud-app | 172.20.0.4 - - [09/Mar/2020:18:35:53 +0000] "POST /apps/text/session/sync HTTP/1.0" 200 1042 "-" "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"
cloud-app | 172.20.0.4 - - [09/Mar/2020:18:35:59 +0000] "POST /apps/text/session/sync HTTP/1.0" 200 1042 "-" "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

Server configuration:

Operating system: Linux 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64

Webserver: Apache/2.4.38 (Debian) (apache2handler)

Database: mysql 10.4.12

PHP version: 7.3.15

Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, ftp, hash, iconv, json, mbstring, SPL, PDO, session, posix, Reflection, standard, SimpleXML, pdo_sqlite, Phar, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, apache2handler, apcu, exif, gd, gmp, imagick, intl, ldap, memcached, pcntl, pdo_mysql, pdo_pgsql, redis, sodium, zip, Zend OPcache

Nextcloud version: 18.0.1 - 18.0.1.3

[AndyScherzinger]

@tobiasKaminsky the original description by @alecbl points to a timeout issue for assembly of really large files.

@Unclezz your issue rather seems to be a server missconfiguration issue since 1mb is the chunk size so chunking seems to be broken on the server side. Cc @rullzer

[tobiasKaminsky]

How big is the file?
How long are the php timeouts?
What kind of server do you have? If you have one with slow I/O then it might just take too long to write the file…

[alecbl]

@tobiasKaminsky
if I do a 2gb file it always fails because it always misses the window, as it goes smaller there is a chance it succeeds. Around the 600mb size it almost always succeeds.

I've virtually disabled the php timeouts, either infinite or pretty big numbers (max_execution_time = 21600, max_input_time = -1).
The operation completes server side and the file does show up in place.

Nextcloud is running in a VM with 8 virtual cpu threads on a amd threadripper gen. 1. The VM has 3GB ram. Neither load nor RAM usage gets anywhere high enough for it to be a bottleneck.
The I/O is slow yes, but the file assembles and shows up in place in a sane ammount of time (about 2 minutes for a 1.9gb file I was trying to upload the other day). Like I said, the "takes too long" amounts to less than a minute. I'm not sure it's intended for the window of waiting for the file assembly to run out after such a short while. Looking at android-library I get the impression client does a remoteMoveOperation after all chunks are uploaded and that's when assembly occurs, and these lines suggest to me it was intended to wait up to 10 minutes?:
"
private static final int MOVE_READ_TIMEOUT = 600000;
private static final int MOVE_CONNECTION_TIMEOUT = 5000;
"

It errors with connection error in less than a minute. Also, faling in that particular place leads to a retry starting over at 0 and with big files I often end up with the same file uploaded multiple times with (2) (3) etc. appended and the client still listing a failed upload job.

Why does it timeout so soon? Is that intended?

[q-wertz]

Same problem here.
I have autoupload enabled for Signal backups (~1GB file size) and the Upload always fails and gets repeated which leads to a battery usage on my phone by nextcloud of about 80%

[tobiasKaminsky]

private static final int MOVE_READ_TIMEOUT = 600000;

Assembling the chunks uses MoveMethod, but not MoveFileRemoteOperation, so it has its default timeout.

@nextcloud/server-triage do you have an idea how long we should wait?

@alecbl if I provide you a test APK with a much longer timeout, can you test this?

[alecbl]

@tobiasKaminsky
Yes, I will test it. Can it coexist with other versions (like the dev and regular versions)?

[rullzer]

@nextcloud/server-triage do you have an idea how long we should wait?

Until it is done basically.
If you upload a serveral GB file we can't guarantee it is done within X seconds. Esp not if you do it on underpowered hardware

[alecbl]

I'm guessing the timing out is supposed to provide a way for the client to error out eventually in case the assembly fails, so that it can retry or notify the user and not leave the waiting state indefinetly.
I think ideally one would pick a reasonable timeout that accomplishes that UX for a certain size (say 10mb file), and then scale the timeout linearly with file size / number of chunks.

Alternatively maybe the server could inform the client as to the state of the ongoing operation, or notify the client when the state changes (pending, done, failed).

[tobiasKaminsky]

Until it is done basically.

So, if I increase the timeout on final move operation to e.g. 20mins?
I always assumed that php has a time out?

[stale]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!