cannot download large folder with lots of small files

[andreicociuba]

Steps to reproduce

0. install nextcloud on a 10 year old server running ubuntu 32 bit.
   1.mount external smb share (happens to be on the very same computer. its the only way i could figure to add external storage to the server without actually uploading all the files again. plus, when i am at home, i have direct access without the web interface, which is important, since the following bug happens:)

2.in said share, i have a folder. about 9GB, but none of the files inside are really big, they shouldnt trip any 2GB limit of integers in php. i hope. anyway... moving on

3.share that folder with a public link. this public link: https://cociubaandrei.ddns.net/nextcloud/index.php/s/ke7YsxkLrDjZRZx
its some ham radio stuff.

4. visit that link, and try to download everything in that folder. in my particular case, i was trying to download on chrome on a 64 bit laptop running windows 10

Expected behaviour

I expect to have an archive that, unpacked, would have around 9GB,

Actual behaviour

i get an archive about 4.11GB in size, that cannot be opened with windows explorer. 7zip can open it, but theres a single folder with a single file it can see inside

Server configuration

Server configuration detail

Operating system: Linux 4.13.0-19-lowlatency #22-Ubuntu SMP PREEMPT Mon Dec 4 12:49:12 UTC 2017 i686

Webserver: Apache/2.4.27 (Ubuntu) (apache2handler)

Database: mysql 5.7.21

PHP version: 7.1.11-0ubuntu0.17.10.1
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, apache2handler, smbclient, mysqlnd, PDO, xml, apcu, apc, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, intl, json, exif, mcrypt, mysqli, pdo_mysql, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, libsmbclient, Zend OPcache, xdebug

Nextcloud version: 13.0.0 Build:2018-02-16T23:01:27+00:00 964b511 - 13.0.0.14

Updated from an older Nextcloud/ownCloud or fresh install:
updated from 12.something....

Where did you install Nextcloud from:
github

List of activated apps

Enabled:
 - activity: 2.6.1
 - admin_notifications: 1.0.1
 - apporder: 0.4.1
 - audioplayer: 2.2.5
 - bookmarks: 0.10.1
 - bruteforcesettings: 1.0.3
 - calendar: 1.6.0
 - caniupdate: 0.1.2
 - circles: 0.13.6
 - cms_pico: 0.9.7
 - comments: 1.3.0
 - contacts: 2.1.0
 - dav: 1.4.6
 - external: 3.0.2
 - federatedfilesharing: 1.3.1
 - federation: 1.3.0
 - files: 1.8.0
 - files_downloadactivity: 1.2.0
 - files_external: 1.4.1
 - files_pdfviewer: 1.2.0
 - files_reader: 1.2.2
 - files_sharing: 1.5.0
 - files_texteditor: 2.5.1
 - files_trashbin: 1.3.0
 - files_versions: 1.6.0
 - files_videoplayer: 1.2.0
 - firstrunwizard: 2.2.1
 - flowupload: 0.0.7
 - gallery: 18.0.0
 - gpxpod: 2.2.2
 - impersonate: 1.0.2
 - issuetemplate: 0.3.0
 - logreader: 2.0.0
 - lookup_server_connector: 1.1.0
 - metadata: 0.6.0
 - music: 0.5.6
 - nextcloud_announcements: 1.2.0
 - notes: 2.3.2
 - notifications: 2.1.2
 - oauth2: 1.1.0
 - ocsms: 1.12.2
 - passman: 2.1.4
 - password_policy: 1.3.0
 - phonetrack: 0.2.0
 - polls: 0.8.1
 - previewgenerator: 1.0.9
 - provisioning_api: 1.3.0
 - registration: 0.3.1
 - serverinfo: 1.3.0
 - sharebymail: 1.3.0
 - socialsharing_facebook: 1.0.2
 - spreed: 3.1.0
 - survey_client: 1.1.0
 - systemtags: 1.3.0
 - tasks: 0.9.6
 - theming: 1.4.1
 - twofactor_backupcodes: 1.2.3
 - unsplash: 1.0.5
 - updatenotification: 1.3.0
 - weather: 1.5.1
 - workflowengine: 1.3.0
Disabled:
 - admin_audit
 - deck
 - encryption
 - files_clipboard
 - folderplayer
 - gpxmotion
 - ojsxc
 - onlyoffice
 - sensorlogger
 - user_external
 - user_ldap

Configuration (config/config.php)

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "cociubaandrei.ddns.net:8082",
        "cociubaandrei.ddns.net",
        "192.168.0.230"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "http:\/\/cociubaandrei.ddns.net:8082\/nextcloud",
    "dbtype": "mysql",
    "version": "13.0.0.14",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 6379
    },
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "PLAIN",
    "mail_smtpauth": 1,
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "465",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpsecure": "ssl",
    "maintenance": false,
    "updater.release.channel": "daily",
    "loglevel": 0,
    "theme": ""
}

Are you using external storage, if yes which one: smb

Are you using encryption: no

**Are you using an external user-backend, if yes which one:**no

Client configuration

Browser: CHrome

Operating system: Windows 10

Logs

Web server error log

Details

none that i deem relevant, but if requested i can provide.

Nextcloud log (data/nextcloud.log)

Nextcloud log

none that i deem relevant, but if requested i can provide.

Browser log

Browser log

none that i deem relevant, but if requested i can provide.

[Schmuuu]

Hi,

My first guess on this issue: you reach the maximum file size on a 32 bit system. At least 4 GB pretty much sound like that. 2^32 = 4096 MB

What filesystem are you using? Is there a limit to your temp directory (defined in php.ini)?

[Nils160988]

Not related to your issue, but you should remove your public shares/links as soon as possible!
They contain a lot of sensitive information about your server and even your email password. Better change them, because you do not know who had access...

It is better to use the "issue template" app, which can generate all the information and removes the sensitive information.

[andreicociuba]

@Nils160988 there shouldnt be any email password anywhere. just an app password for gmail that i already changed

[andreicociuba]

@Schmuuu i have a 32 bit processor with 64 bit instruction set extension. i deffinitely have bigger files than 496MB on the server just fine, besides, the download is 4.11GB.

There is no activated option containing the word "temp" in my php.ini they are all commented out.

the files are in a smb share that lives on a mdadm raid5 array with ext4.

the root of the server is on a partition on a hardware raid array, 250GB in size, of wich 200GB are free....

updated description with issue template app information

[andreicociuba]

.... no other ideas?

[tflidd]

Just to exclude external storage and SMB, I'd try the same with data on a local folder (there is a reported bug on larger files with external storage: #1655).

NC13 was supposed to come with some improvements regarding the large file handling in 32-bit systems. Not sure if anything was applied at all.

[andreicociuba]

i tried uploading the entire folder structure through the web client in my account root folder, by dragging and dropping.

didnt work, the folder structure that was uploaded was only 1 level deep. any files in this folder, and any folders were succesfully updated, but the subfolders were empty.

so i manually transferred the folder i wanted to test to the appropriate /data folder in the nextcloud installation, and then ran occ files:scan on that particular folder.

when it was finished, downloading it behaved exactly like from the SMB share.

the resulting zip file is 4.15GB big, does not open with windows explorer, and opening it with 7zip yields an incomplete folder structure, with one single file in the last sub-sub-sub-sub.....sub-folder

[tflidd]

Is this related to McNetic/PHPZipStreamer#38 ?
@McNetic @nickvergessen

[nextcloud-bot]

Hey, this issue has been closed because the label stale is set and there were no updates for 14 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)