S3 storage has folder size "pending"

[In-eX]

Steps to reproduce

1. Mount an S3 bucket
2. Upload Data via Windows client

Expected behaviour

The correct folder size should be displayed.

Actual behaviour

Folder size says "pending" on all levels in the file structure.

Server configuration

Operating system:
Ubuntu 16.04.3 64bit

Web server:
Apache/2.4.18
nginx/1.11.10

Database:
mysql Ver 14.14 Distrib 5.7.20

PHP version:
PHP-FPM 5.6.32

Nextcloud version: (see Nextcloud admin page)
12.0.4

Updated from an older Nextcloud/ownCloud or fresh install:
Updated to the newest release since ownCloud 9

Where did you install Nextcloud from:
php installation script

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - activity: 2.5.2
  - bruteforcesettings: 1.0.2
  - calendar: 1.5.6
  - comments: 1.2.0
  - contacts: 2.0.1
  - dav: 1.3.0
  - federatedfilesharing: 1.2.0
  - federation: 1.2.0
  - files: 1.7.2
  - files_external: 1.3.0
  - files_pdfviewer: 1.1.1
  - files_sharing: 1.4.0
  - files_texteditor: 2.4.1
  - files_trashbin: 1.2.0
  - files_versions: 1.5.0
  - files_videoplayer: 1.1.0
  - firstrunwizard: 2.1
  - gallery: 17.0.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - mail: 0.7.5
  - nextcloud_announcements: 1.1
  - notifications: 2.0.0
  - oauth2: 1.0.5
  - password_policy: 1.2.2
  - provisioning_api: 1.2.0
  - serverinfo: 1.2.0
  - sharebymail: 1.2.0
  - survey_client: 1.0.0
  - systemtags: 1.2.0
  - theming: 1.3.0
  - twofactor_backupcodes: 1.1.1
  - updatenotification: 1.2.0
  - workflowengine: 1.2.0
Disabled:
  - admin_audit
  - encryption
  - user_external
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "oc2fyztrch2c",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "domain.com"
        ],
        "datadirectory": "\/home\/andy\/oc-data",
        "overwrite.cli.url": "https:\/\/domain.com",
        "dbtype": "mysql",
        "version": "12.0.4.3",
        "dbname": "cloud_domain_com_1",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "Europe\/Berlin",
        "installed": true,
        "knowledgebaseenabled": false,
        "enable_avatars": true,
        "mail_from_address": "cloud",
        "mail_smtpmode": "php",
        "mail_domain": "domain.com",
        "allow_user_to_change_display_name": true,
        "loglevel": 3,
        "maintenance": false,
        "theme": "",
        "singleuser": false,
        "appstore.experimental.enabled": true,
        "htaccess.RewriteBase": "\/",
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "filelocking.enabled": false,
        "redis": {
            "host": "localhost",
            "port": 6379
        },
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "auth.bruteforce.protection.enabled": true
    },
    "apps": {
        "activity": {
            "enabled": "yes",
            "installed_version": "2.5.2",
            "types": "filesystem"
        },
        "announcementcenter": {
            "enabled": "no",
            "installed_version": "1.1.1",
            "ocsid": "173921",
            "signed": "true",
            "types": ""
        },
        "backgroundjob": {
            "lastjob": "39"
        },
        "bruteforcesettings": {
            "enabled": "yes",
            "installed_version": "1.0.2",
            "types": ""
        },
        "calendar": {
            "enabled": "yes",
            "installed_version": "1.5.6",
            "signed": "true",
            "types": ""
        },
        "comments": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "logging"
        },
        "contacts": {
            "enabled": "yes",
            "installed_version": "2.0.1",
            "ocsid": "168708",
            "types": ""
        },
        "core": {
            "OC_Channel": "production",
            "backgroundjobs_mode": "cron",
            "incoming_server2server_share_enabled": "no",
            "installed.bundles": "[\"CoreBundle\"]",
            "installedat": "1459469183.8086",
            "lastcron": "1512672301",
            "lastupdateResult": "[]",
            "lastupdatedat": "1512672131",
            "moveavatarsdone": "yes",
            "oc.integritycheck.checker": "[]",
            "outgoing_server2server_share_enabled": "no",
            "previewsCleanedUp": "1",
            "public_files": "files_sharing\/public.php",
            "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
            "remote_caldav": "dav\/appinfo\/v1\/caldav.php",
            "remote_calendar": "dav\/appinfo\/v1\/caldav.php",
            "remote_carddav": "dav\/appinfo\/v1\/carddav.php",
            "remote_contacts": "dav\/appinfo\/v1\/carddav.php",
            "remote_dav": "dav\/appinfo\/v2\/remote.php",
            "remote_files": "dav\/appinfo\/v1\/webdav.php",
            "remote_webdav": "dav\/appinfo\/v1\/webdav.php",
            "repairlegacystoragesdone": "yes",
            "scss.variables": "28b56eb75bff149ec8df6d6bdbcaacfa",
            "shareapi_allow_mail_notification": "yes",
            "shareapi_allow_public_notification": "yes",
            "shareapi_default_expire_date": "yes",
            "shareapi_expire_after_n_days": "30",
            "umgmt_send_email": "false",
            "umgmt_show_backend": "false",
            "umgmt_show_email": "false",
            "umgmt_show_last_login": "true",
            "umgmt_show_storage_location": "true",
            "updater.secret.created": "1506461277",
            "vendor": "nextcloud"
        },
        "dav": {
            "OCA\\DAV\\Migration\\ValueFixInsert_ran": "true",
            "buildCalendarSearchIndex": "yes",
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": "filesystem"
        },
        "federatedfilesharing": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": ""
        },
        "federation": {
            "autoAddServers": "0",
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "authentication"
        },
        "files": {
            "cronjob_scan_files": "500",
            "enabled": "yes",
            "installed_version": "1.7.2",
            "types": "filesystem"
        },
        "files_antivirus": {
            "av_chunk_size": "1024",
            "av_cmd_options": "",
            "av_host": "",
            "av_infected_action": "only_log",
            "av_mode": "socket",
            "av_path": "\/usr\/bin\/clamscan",
            "av_port": "0",
            "av_socket": "\/var\/run\/clamav\/clamd.ctl",
            "enabled": "no",
            "installed_version": "0.8.0.2",
            "ocsid": "157439",
            "signed": "true",
            "types": "filesystem"
        },
        "files_external": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "ocsid": "166048",
            "types": "filesystem"
        },
        "files_pdfviewer": {
            "enabled": "yes",
            "installed_version": "1.1.1",
            "ocsid": "166049",
            "types": ""
        },
        "files_sharing": {
            "enabled": "yes",
            "incoming_server2server_share_enabled": "no",
            "installed_version": "1.4.0",
            "outgoing_server2server_share_enabled": "no",
            "types": "filesystem"
        },
        "files_texteditor": {
            "enabled": "yes",
            "installed_version": "2.4.1",
            "ocsid": "166051",
            "types": ""
        },
        "files_trashbin": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        },
        "files_versions": {
            "enabled": "yes",
            "installed_version": "1.5.0",
            "types": "filesystem"
        },
        "files_videoplayer": {
            "enabled": "yes",
            "installed_version": "1.1.0",
            "types": ""
        },
        "firstrunwizard": {
            "enabled": "yes",
            "installed_version": "2.1",
            "types": "logging"
        },
        "gallery": {
            "enabled": "yes",
            "installed_version": "17.0.0",
            "types": ""
        },
        "galleryplus": {
            "enabled": "no",
            "installed_version": "15.1.1",
            "ocsid": "169116",
            "signed": "true",
            "types": ""
        },
        "logreader": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "ocsid": "170871",
            "types": ""
        },
        "lookup_server_connector": {
            "enabled": "yes",
            "installed_version": "1.0.0",
            "types": "authentication"
        },
        "mail": {
            "enabled": "yes",
            "installed_version": "0.7.5",
            "signed": "true",
            "types": ""
        },
        "nextcloud_announcements": {
            "enabled": "yes",
            "installed_version": "1.1",
            "types": "logging"
        },
        "notifications": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "types": "logging"
        },
        "oauth2": {
            "enabled": "yes",
            "installed_version": "1.0.5",
            "types": "authentication"
        },
        "password_policy": {
            "enabled": "yes",
            "installed_version": "1.2.2",
            "types": ""
        },
        "provisioning_api": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "prevent_group_restriction"
        },
        "serverinfo": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": ""
        },
        "sharebymail": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        },
        "spreedme": {
            "OWNCLOUD_TEMPORARY_PASSWORD_LOGIN_ENABLED": "true",
            "OWNCLOUD_TEMPORARY_PASSWORD_SIGNING_KEY": "",
            "SPREED_WEBRTC_BASEPATH": "\/",
            "SPREED_WEBRTC_IS_SHARED_INSTANCE": "false",
            "SPREED_WEBRTC_ORIGIN": "https:\/\/lm.domain.com:8082\/",
            "SPREED_WEBRTC_SHAREDSECRET": "",
            "enabled": "no",
            "installed_version": "0.3.5",
            "is_set_up": "true",
            "ocsid": "174436",
            "types": ""
        },
        "survey_client": {
            "apps": "no",
            "database": "no",
            "enabled": "yes",
            "encryption": "no",
            "files_sharing": "no",
            "installed_version": "1.0.0",
            "php": "no",
            "server": "no",
            "stats": "no",
            "types": ""
        },
        "systemtags": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "logging"
        },
        "templateeditor": {
            "enabled": "no",
            "installed_version": "0.2",
            "types": ""
        },
        "theming": {
            "cachebuster": "4",
            "color": "#1D2D44",
            "enabled": "yes",
            "installed_version": "1.3.0",
            "name": "1n3xCloud",
            "slogan": "nur wir sind cool!",
            "types": "logging",
            "url": "https:\/\/domain.com"
        },
        "twofactor_backupcodes": {
            "enabled": "yes",
            "installed_version": "1.1.1",
            "types": ""
        },
        "updatenotification": {
            "calendar": "1.4.1",
            "contacts": "1.5.1",
            "core": "9.1.2.2",
            "enabled": "[\"admin\"]",
            "installed_version": "1.2.0",
            "spreedme": "0.3.5",
            "types": ""
        },
        "workflowengine": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        }
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
S3

Are you using encryption: yes/no
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser:
any

Operating system:
any

Logs

Web server error log

Web server error log

No errors

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"XA4tA1FzLYXAm8STtEoS","level":3,"time":"2017-12-07T19:53:05+01:00","remoteAddr":"88.130.52.26","user":"a.martin","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/Dokumente%20-%20S3","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/domain.com\/httpdocs\/apps\/files_external\/lib\/Lib\/Storage\/AmazonS3.php#291","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/62.0.3202.94 Safari\/537.36","version":"12.0.4.3"}
{"reqId":"XA4tA1FzLYXAm8STtEoS","level":3,"time":"2017-12-07T19:53:05+01:00","remoteAddr":"88.130.52.26","user":"a.martin","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/Dokumente%20-%20S3","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/domain.com\/httpdocs\/apps\/files_external\/lib\/Lib\/Storage\/AmazonS3.php#291","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/62.0.3202.94 Safari\/537.36","version":"12.0.4.3"}
{"reqId":"XA4tA1FzLYXAm8STtEoS","level":3,"time":"2017-12-07T19:53:05+01:00","remoteAddr":"88.130.52.26","user":"a.martin","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/Dokumente%20-%20S3","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/domain.com\/httpdocs\/apps\/files_external\/lib\/Lib\/Storage\/AmazonS3.php#291","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/62.0.3202.94 Safari\/537.36","version":"12.0.4.3"}
{"reqId":"ivLnfMFaRFsTZlIbQMSJ","level":3,"time":"2017-12-07T19:53:06+01:00","remoteAddr":"88.130.52.26","user":"a.martin","app":"PHP","method":"GET","url":"\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FDokumente%20-%20S3","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/domain.com\/httpdocs\/apps\/files_external\/lib\/Lib\/Storage\/AmazonS3.php#291","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/62.0.3202.94 Safari\/537.36","version":"12.0.4.3"}

[MorrisJobke]

cc @icewind1991

[MorrisJobke]

S3 should be a lot smoother with 13. Could you test it again?

[In-eX]

After upgrading from 12.0.4 to 13beta3 the problem persisted. A installation with the web installer (12.0.3) fixes the problem, but when I upgrade to 12.0.4 I'm back to my old problem. Note, that I'm using the same database for all versions (not at the same time ofc). I don't have any issues with 13beta3 and a new database.

Is there something wrong with the updater? Or did something get lost after all those small updates I did in the past?

[lesaff]

I am running into the same problem, running 13 b4

Error

Invalid argument supplied for foreach() at /###REDACTED###/apps/files_external/lib/Lib/Storage/AmazonS3.php#290

[MickVanDuijn]

After trying some stuff out, I noticed that when the root of the S3 bucket only contains folders, then the "Contents" attribute in the result paginator is not present. It should therefore be checked whether the "Contents" attribute is an array before iterating it with foreach similar to the CommonPrefixes.

@TecJon @lesaff For a quick workaround, you can add a dummy file .dummy with conents placeholder to the root of the S3 bucket.

[MickVanDuijn]

Sorry, I just noticed it still does not always show the correct size (it did at least once), but at least the logs don't overflow with Invalid argument supplied for foreach()

Edit: it sometimes shows the correct size on the shared folder, but for folders inside it is still pending.

[MorrisJobke]

cc @icewind1991

[TecJon]

@MickVanDuijn thank you, unfortunately it didn't change for me.

This issue might only be a side issue to a bigger problem with s3 that I am facing. #8299

[k0ste]

Issue is still persist with nextCloud 13.0.1.

[jaakj]

Same problem here: NC 13.0.2, external storage is DigitalOcean's Spaces. All external folders showing "Pending" and log is flooded with error:

Invalid argument supplied for foreach() at /......./apps/files_external/lib/Lib/Storage/AmazonS3.php#295

[MorrisJobke]

Same problem here: NC 13.0.2, external storage is DigitalOcean's Spaces. All external folders showing "Pending" and log is flooded with error:

Invalid argument supplied for foreach() at /......./apps/files_external/lib/Lib/Storage/AmazonS3.php#295

See #8842 and #9403

[k0ste]

I tested 13.0.4. Still in pending state.