Optimize Openstack Swift large file download #14116
Description
Steps to reproduce
- Set Openstack Swift as your primary storage (I'm using one from OVH on S1-8 server).
- Upload large file (like 512 MB or so).
- Download it (either desktop client or from web interface).
Expected behaviour
NextCloud server downloads file segments from object storage and simultaneously transfers them to client.
Actual behaviour
First entire file is downloaded to NextCloud server (OVH provides ~100Mb download link from their Object Storage from what I can see). Only after it's downloaded it's transferred to client. In the end it takes twice as much time.
I needed to change max_execution_time on server to prevent timeout errors.
Server configuration
Using Docker 15.0.4-php-alpine image.
Updated from an older Nextcloud/ownCloud or fresh install: Started at version 14.0
Signing status:
Signing status
No errors have been found.
List of activated apps:
App list
Enabled:
- accessibility: 1.1.0
- activity: 2.8.2
- admin_audit: 1.5.0
- calendar: 1.6.4
- camerarawpreviews: 0.6.3
- checksum: 0.4.2
- cloud_federation_api: 0.1.0
- comments: 1.5.0
- contacts: 3.0.3
- data_request: 1.2.0
- dav: 1.8.1
- deck: 0.5.2
- drawio: 0.9.2
- encryption: 2.3.0
- event_update_notification: 0.3.2
- federatedfilesharing: 1.5.0
- federation: 1.5.0
- files: 1.10.0
- files_downloadactivity: 1.4.0
- files_external: 1.6.0
- files_markdown: 2.0.5
- files_mindmap: 0.0.11
- files_pdfviewer: 1.4.0
- files_rightclick: 0.11.0
- files_sharing: 1.7.0
- files_texteditor: 2.7.0
- files_trashbin: 1.5.0
- files_versions: 1.8.0
- files_videoplayer: 1.4.0
- firstrunwizard: 2.4.0
- flowupload: 0.0.9
- gallery: 18.2.0
- impersonate: 1.2.0
- logreader: 2.0.0
- lookup_server_connector: 1.3.0
- music: 0.9.3
- news: 13.0.3
- nextcloud_announcements: 1.4.0
- notes: 2.5.1
- notifications: 2.3.0
- oauth2: 1.3.0
- password_policy: 1.5.0
- previewgenerator: 2.0.0
- provisioning_api: 1.5.0
- quota_warning: 1.4.0
- ransomware_protection: 1.3.0
- richdocuments: 3.2.0
- scanner: 0.1.1
- serverinfo: 1.5.0
- sharebymail: 1.5.0
- spreed: 5.0.2
- support: 1.0.0
- survey_client: 1.3.0
- systemtags: 1.5.0
- tasks: 0.9.8
- theming: 1.6.0
- twofactor_backupcodes: 1.4.1
- twofactor_gateway: 0.11.0
- twofactor_nextcloud_notification: 1.0.3
- twofactor_totp: 2.1.0
- twofactor_u2f: 2.1.0
- twofactor_yubikey: 0.4.1
- unsplash: 1.1.3
- updatenotification: 1.5.0
- weather: 1.5.5
- workflowengine: 1.5.0
Disabled: - admin_notifications
- end_to_end_encryption
- files_accesscontrol
- files_fulltextsearch
- fulltextsearch
- twofactor_rcdevsopenotp
- user_ldap
Nextcloud configuration:
Config report
{
"system": {
"memcache.local": "\\OC\\Memcache\\APCu",
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"port": 6379
},
"memcache.locking": "\\OC\\Memcache\\Redis",
"memcache.distributed": "\\OC\\Memcache\\Redis",
"apps_paths": [
{
"path": "\/var\/www\/html\/apps",
"url": "\/apps",
"writable": false
},
{
"path": "\/var\/www\/html\/custom_apps",
"url": "\/custom_apps",
"writable": true
}
],
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"archive.cloud.pikula.co"
],
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"dbtype": "mysql",
"version": "15.0.4.0",
"overwrite.cli.url": "https:\/\/new.cloud.pikula.co",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"objectstore": {
"class": "OC\\Files\\ObjectStore\\Swift",
"arguments": {
"username": "dEcCjjVA5ghy",
"password": "***REMOVED SENSITIVE VALUE***",
"bucket": "cloud.pikula.co-data",
"autocreate": false,
"region": "WAW1",
"url": "https:\/\/auth.cloud.ovh.net\/v2.0",
"tenantName": "7435064151262691",
"serviceName": "swift"
}
},
"maintenance": false,
"loglevel": 2,
"theme": ""
}
}
Are you using external storage, if yes which one: OpenStack Swift as primary storage
Are you using encryption: no
Are you using an external user-backend, if yes which one: No
Client configuration
Browser: Firefox 65.0 and NextCloud client 2.5.1final (build 20181204) AppImage
Operating system:
Details
``` NAME=Fedora VERSION="29 (Server Edition)" ID=fedora VERSION_ID=29 VERSION_CODENAME="" PLATFORM_ID="platform:f29" PRETTY_NAME="Fedora 29 (Server Edition)" ANSI_COLOR="0;34" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:29" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/" SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=29 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=29 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="Server Edition" VARIANT_ID=server ```Logs
Well, in the end it works, but not in the optimal way. No errors are present (besides timeout, which was mitigated by changing timeout in php.ini).