lookup_server_connector creates a lot of oc_jobs entries, high DB load during cron, and tries to sync even if disabled in options

[alx-tuilmenau]

Steps to reproduce

1. Upgraded to NC 14 some days ago
2. All Federation options are disabled
3. Users synced from LDAP (ca. 5k)

Expected behaviour

There should be no connections to external servers. The system should run stable.

Actual behaviour

A lot of traffic to lookup.nextcloud.com. Increasing, high DB load. Leak of personal information (usernames) to nextcloud.com without permission.

It looks like lookup_server_connector app is trying to update lookup.nextcloud.com on ervery user change, also if users are synced via LDAP (and nothing is changed), even if the global adressbook and all other federation options are disabled.
This often fails (maybe server fail / network / proxy timeout) and creates a entry in oc_jobs for retrying during cronjob.
I got about 300k of lookup retries in the oc_jobs table, The cron job has no chance to get this done, so all other jobs (mails) are not processed.

My temporary solution was to delete the lookup_server_connector app completely (occ app:disable didn't work) and deleting all entries for this class in oc_jobs.

Server configuration

Operating system:
Debian 9
Web server:
Apache 2.4
Database:
MariaDB (10.1.37)
PHP version:
7.2
Nextcloud version: (see Nextcloud admin page)
14.0.4.2
Updated from an older Nextcloud/ownCloud or fresh install:
Updated

Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

No errors have been found.

Nextcloud configuration:

Config report

{ "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***", "c***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "14.0.4.2", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "dbdriveroptions": { "1002": "SET wait_timeout = 28800" }, "installed": true, "default_language": "de", "updatechecker": true, "has_internet_connection": true, "memcached_servers": [ [ "localhost", 11211 ] ], "appstoreenabled": true, "ldapIgnoreNamingRules": false, "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory", "loglevel": 1, "trusted_proxies": "***REMOVED SENSITIVE VALUE***", "forwarded_for_headers": [ "HTTP_X_FORWARDED_FOR" ], "overwritehost": ***", "overwriteprotocol": "https", "overwrite.cli.url": "***", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_domain": "***REMOVED SENSITIVE VALUE***", "skeletondirectory": "\/data\/nextcloud_skel", "theme": "tui", "activity_expire_days": 14, "integrity.check.disabled": true, "memcache.local": "\\OC\\Memcache\\APCu", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 6379, "password": "***REMOVED SENSITIVE VALUE***" }, "knowledgebaseenabled": false, "defaultapp": "apporder", "sharing.minSearchStringLength": 4, "sharing.maxAutocompleteResults": 5, "maintenance": false, "debug": false, "htaccess.RewriteBase": "\/", "proxy": "****", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "25", "updater.release.channel": "production" }, "apps": { "QuotaAdmin": { "enabled": "no", "installed_version": "0.0.1", "types": "" }, "accessibility": { "enabled": "yes", "installed_version": "1.0.1", "types": "" }, "activity": { "enabled": "yes", "installed_version": "2.7.0", "notify_email_announcementcenter": "1", "notify_email_calendar": "0", "notify_email_calendar_event": "0", "notify_email_calendar_todo": "0", "notify_email_circles_as_member": "0", "notify_email_circles_as_moderator": "1", "notify_email_comments": "0", "notify_email_favorite": "0", "notify_email_file_changed": "0", "notify_email_file_created": "0", "notify_email_file_deleted": "0", "notify_email_file_downloaded": "0", "notify_email_file_restored": "0", "notify_email_public_links": "0", "notify_email_remote_share": "0", "notify_email_shared": "0", "notify_email_systemtags": "0", "notify_setting_batchtime": "86400", "notify_setting_self": "1", "notify_setting_selfemail": "0", "notify_stream_calendar": "1", "notify_stream_calendar_event": "1", "notify_stream_calendar_todo": "1", "notify_stream_circles_as_member": "1", "notify_stream_circles_as_moderator": "1", "notify_stream_comments": "1", "notify_stream_favorite": "1", "notify_stream_file_changed": "1", "notify_stream_file_created": "1", "notify_stream_file_deleted": "1", "notify_stream_file_downloaded": "1", "notify_stream_file_favorite": "0", "notify_stream_file_restored": "1", "notify_stream_public_links": "1", "notify_stream_remote_share": "1", "notify_stream_shared": "1", "notify_stream_systemtags": "1", "types": "filesystem" }, "admin_audit": { "enabled": "no", "installed_version": "1.2.0", "types": "logging" }, "announcementcenter": { "enabled": "yes", "installed_version": "3.3.1", "types": "logging" }, "apporder": { "enabled": "yes", "hidden": "[]", "installed_version": "0.6.0", "order": "[\"\/apps\/files\/\",\"\/apps\/activity\/\",\"\/apps\/calendar\/\",\"\/apps\/tasks\/\",\"\/apps\/circles\/\",\"\/apps\/spreed\/\",\"\/apps\/notes\/\",\"\/apps\/contacts\/\",\"\/apps\/gallery\/\",\"\/apps\/announcementcenter\/\",\"\/apps\/gpxedit\/\",\"\/apps\/quotaadmin\/\"]", "types": "" }, "audioplayer": { "enabled": "no", "installed_version": "2.1.0", "types": "filesystem" }, "backgroundjob": { "lastjob": "515531" }, "bookmarks": { "enabled": "no", "installed_version": "0.10.1", "types": "" }, "bruteForce": { "whitelist_1": "***" }, "bruteforcesettings": { "enabled": "yes", "installed_version": "1.3.0", "types": "" }, "calendar": { "enabled": "yes", "installed_version": "1.6.4", "types": "" }, "circles": { "allow_federated": "0", "allow_linked_groups": "0", "enabled": "yes", "installed_version": "0.15.2", "members_limit": "1000", "types": "" }, "cloud_federation_api": { "enabled": "yes", "installed_version": "0.0.1", "types": "filesystem" }, "comments": { "enabled": "yes", "installed_version": "1.4.0", "types": "logging" }, "contacts": { "enabled": "yes", "installed_version": "2.1.8", "types": "" }, "core": { "backgroundjobs_mode": "cron", "installed.bundles": "[\"CoreBundle\"]", "installedat": "1506428024.7302", "lastcron": "1546800408", "lastupdateResult": "[]", "lastupdatedat": "1546799081", "moveavatarsdone": "yes", "previewsCleanedUp": "1", "public_files": "files_sharing\/public.php", "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php", "scss.variables": "385ed5e08336aefb25c8b98f0028dbe6", "shareapi_allow_group_sharing": "yes", "shareapi_default_permission_canshare": "no", "shareapi_default_permissions": "15", "umgmt_show_backend": "false", "umgmt_show_email": "false", "umgmt_show_last_login": "false", "updater.secret.created": "1545907978", "vendor": "nextcloud" }, "dashboard": { "enabled": "no", "installed_version": "4.0.5", "types": "" }, "dav": { "buildCalendarSearchIndex": "yes", "enabled": "yes", "installed_version": "1.6.0", "types": "filesystem" }, "drop_account": { "enabled": "yes", "installed_version": "0.0.13", "types": "" }, "federatedfilesharing": { "enabled": "yes", "installed_version": "1.4.0", "types": "" }, "federation": { "enabled": "yes", "installed_version": "1.4.0", "types": "authentication" }, "files": { "cronjob_scan_files": "500", "default_quota": "10 GB", "enabled": "yes", "installed_version": "1.9.0", "types": "filesystem" }, "files_clipboard": { "enabled": "no", "installed_version": "0.7.1", "types": "" }, "files_downloadactivity": { "enabled": "yes", "installed_version": "1.3.0", "types": "filesystem" }, "files_external": { "allow_user_mounting": "no", "enabled": "no", "installed_version": "1.3.0", "types": "filesystem", "user_mounting_backends": "ftp,dav,owncloud,sftp,amazons3,dropbox,googledrive,swift,\\OC\\Files\\Storage\\SFTP_Key" }, "files_fulltextsearch": { "enabled": "yes", "files_audio": "0", "files_encrypted": "0", "files_external": "0", "files_federated": "0", "files_group_folders": "0", "files_image": "0", "files_local": "1", "files_office": "1", "files_pdf": "1", "files_size": "5", "installed_version": "1.1.1", "types": "filesystem" }, "files_markdown": { "enabled": "yes", "installed_version": "2.0.5", "types": "" }, "files_pdfviewer": { "enabled": "yes", "installed_version": "1.3.2", "types": "" }, "files_retention": { "enabled": "yes", "installed_version": "1.3.1", "types": "filesystem" }, "files_rightclick": { "enabled": "yes", "installed_version": "0.8.4", "types": "" }, "files_sharing": { "enabled": "yes", "incoming_server2server_share_enabled": "no", "installed_version": "1.6.2", "lookupServerUploadEnabled": "no", "outgoing_server2server_share_enabled": "no", "types": "filesystem" }, "files_texteditor": { "enabled": "yes", "installed_version": "2.6.0", "types": "" }, "files_trashbin": { "enabled": "yes", "installed_version": "1.4.1", "types": "filesystem,dav" }, "files_versions": { "enabled": "yes", "installed_version": "1.7.1", "types": "filesystem,dav" }, "files_videoplayer": { "enabled": "yes", "installed_version": "1.3.0", "types": "" }, "firstrunwizard": { "enabled": "no", "installed_version": "2.1", "types": "logging" }, "fulltextsearch": { "app_navigation": "0", "cron_err_reset": "1546796703", "enabled": "yes", "installed_version": "1.1.0", "provider_indexed": "{\"files\":\"1\",\"test_provider\":\"0\"}", "search_platform": "OCA\\FullTextSearch_ElasticSearch\\Platform\\ElasticSearchPlatform", "types": "" }, "fulltextsearch_elasticsearch": { "analyzer_tokenizer": "standard", "elastic_host": "***", "elastic_index": "my_index", "enabled": "yes", "installed_version": "1.0.2", "types": "" }, "gallery": { "enabled": "yes", "installed_version": "18.1.0", "types": "" }, "gpxedit": { "enabled": "yes", "installed_version": "0.0.10", "types": "" }, "groupfolders": { "enabled": "no", "installed_version": "1.1.0", "types": "filesystem" }, "guests": { "allow_external_storage": "false", "enabled": "yes", "group": "guests", "installed_version": "1.0.0", "types": "authentication", "usewhitelist": "true", "whitelist": "activity,apporder,circles,files_markdown,files_sharing,files_texteditor,files_trashbin,files_versions,gallery,notifications,spreed" }, "logreader": { "enabled": "yes", "installed_version": "2.0.0", "levels": "00111", "ocsid": "170871", "relativedates": "1", "types": "" }, "lookup_server_connector": { "enabled": "yes", "installed_version": "1.2.0", "types": "authentication" }, "metadata": { "enabled": "yes", "installed_version": "0.8.0", "types": "" }, "nextcloud_announcements": { "enabled": "no", "installed_version": "1.1", "types": "logging" }, "notes": { "enabled": "yes", "installed_version": "2.5.1", "types": "" }, "notifications": { "enabled": "yes", "installed_version": "2.2.1", "types": "logging" }, "oauth2": { "enabled": "yes", "installed_version": "1.2.1", "types": "authentication" }, "password_policy": { "enabled": "no", "installed_version": "1.2.2", "types": "" }, "provisioning_api": { "enabled": "yes", "installed_version": "1.4.0", "types": "prevent_group_restriction" }, "quota_warning": { "alert_email": "yes", "enabled": "yes", "installed_version": "1.3.0", "repeat_warning": "14", "types": "filesystem", "warning_email": "yes" }, "quotaadmin": { "enabled": "yes", "installed_version": "0.0.1", "types": "" }, "serverinfo": { "enabled": "yes", "installed_version": "1.4.0", "types": "" }, "sharebymail": { "enabled": "no", "installed_version": "1.2.0", "types": "filesystem" }, "spreed": { "enabled": "yes", "installed_version": "4.0.2", "signaling_ticket_secret": *** "stun_servers": "[\"stun.nextcloud.com:443\"]", "types": "prevent_group_restriction" }, "support": { "enabled": "no", "installed_version": "1.0.0", "types": "" }, "survey_client": { "enabled": "no", "installed_version": "1.0.0", "types": "" }, "systemtags": { "enabled": "yes", "installed_version": "1.4.0", "types": "logging" }, "tasks": { "enabled": "yes", "installed_version": "0.9.8", "types": "" }, "theming": { "enabled": "no", "installed_version": "1.3.0", "types": "logging" }, "twofactor_backupcodes": { "enabled": "yes", "installed_version": "1.3.1", "types": "" }, "twofactor_totp": { "enabled": "yes", "installed_version": "1.5.0", "types": "" }, "updatenotification": { "enabled": "yes", "installed_version": "1.4.1", "types": "", "update_check_errors": "0" }, "user_ldap": { "_lastChange": "1545915889", "background_sync_interval": "1800", "background_sync_offset": "0", "background_sync_prefix": "", "cleanUpJobOffset": "4600", "enabled": "yes", "has_memberof_filter_support": "0", "home_folder_naming_rule": "attr:uid", "installed_version": "1.4.0", "last_jpegPhoto_lookup": "0", "ldap_agent_password": "***REMOVED SENSITIVE VALUE***", "ldap_attributes_for_group_search": "", "ldap_attributes_for_user_search": "sn\nuid\nmail", "ldap_backup_host": "", "ldap_backup_port": "", "ldap_base": "ou=user,o=uni", "ldap_base_groups": "ou=user,o=uni", "ldap_base_users": "ou=user,o=uni", "ldap_cache_ttl": "3600", "ldap_configuration_active": "1", "ldap_default_ppolicy_dn": "", "ldap_display_name": "sn", "ldap_dn": "cn=proxy-nextcloud,***i", "ldap_dynamic_group_member_url": "", "ldap_email_attr": "mail", "ldap_experienced_admin": "1", "ldap_expert_username_attr": "uid", "ldap_expert_uuid_group_attr": "", "ldap_expert_uuid_user_attr": "uid", "ldap_gid_number": "gidNumber", "ldap_group_display_name": "cn", "ldap_group_filter": "", "ldap_group_filter_mode": "0", "ldap_group_member_assoc_attribute": "uniqueMember", "ldap_groupfilter_groups": "", "ldap_groupfilter_objectclass": "", "ldap_host": "ldap:\/\/localhost", "ldap_login_filter": "***", "ldap_login_filter_mode": "0", "ldap_loginfilter_attributes": "mail", "ldap_loginfilter_email": "0", "ldap_loginfilter_username": "1", "ldap_nested_groups": "0", "ldap_override_main_server": "", "ldap_paging_size": "0", "ldap_port": "389", "ldap_quota_attr": "", "ldap_quota_def": "", "ldap_tls": "0", "ldap_turn_off_cert_check": "0", "ldap_turn_on_pwd_change": "0", "ldap_user_display_name_2": "mail", "ldap_user_filter_mode": "0", "ldap_userfilter_groups": "", "ldap_userfilter_objectclass": "inetOrgPerson", "ldap_userlist_filter": "***", "types": "authentication", "use_memberof_to_detect_membership": "1" }, "workflowengine": { "enabled": "yes", "installed_version": "1.4.0", "types": "filesystem" } } }

Are you using external storage, if yes which one: local/smb/sftp/...
no
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
LDAP

LDAP configuration (delete this part if not used)

LDAP config

hasMemberOfFilterSupport | 0 | | hasPagedResultSupport | | | homeFolderNamingRule | attr:uid | | lastJpegPhotoLookup | 0 | | ldapAgentName | cn=proxy-nextcloud | | ldapAgentPassword | *** | | ldapAttributesForGroupSearch | | | ldapAttributesForUserSearch | sn;uid;mail | | ldapBackupHost | | | ldapBackupPort | | | ldapBase | ou=user,o=uni | | ldapBaseGroups | ou=user,o=uni | | ldapBaseUsers | ou=user,o=uni | | ldapCacheTTL | 3600 | | ldapConfigurationActive | 1 | | ldapDefaultPPolicyDN | | | ldapDynamicGroupMemberURL | | | ldapEmailAttribute | mail | | ldapExperiencedAdmin | 1 | | ldapExpertUUIDGroupAttr | | | ldapExpertUUIDUserAttr | uid | | ldapExpertUsernameAttr | uid | | ldapGidNumber | gidNumber | | ldapGroupDisplayName | cn | | ldapGroupFilter | | | ldapGroupFilterGroups | | | ldapGroupFilterMode | 0 | | ldapGroupFilterObjectclass | | | ldapGroupMemberAssocAttr | uniqueMember | | ldapHost | ldap://localhost | | ldapIgnoreNamingRules | | | ldapLoginFilter |(objectclass=inetOrgPerson) [modified to hide internal attributes] | ldapLoginFilterAttributes | mail | | ldapLoginFilterEmail | 0 | | ldapLoginFilterMode | 0 | | ldapLoginFilterUsername | 1 | | ldapNestedGroups | 0 | | ldapOverrideMainServer | | | ldapPagingSize | 0 | | ldapPort | 389 | | ldapQuotaAttribute | | | ldapQuotaDefault | | | ldapTLS | 0 | | ldapUserAvatarRule | default | | ldapUserDisplayName | sn | | ldapUserDisplayName2 | mail | | ldapUserFilter (objectclass=inetOrgPerson) [modified to hide internal attributes] | ldapUserFilterGroups | | | ldapUserFilterMode | 0 | | ldapUserFilterObjectclass | inetOrgPerson | | ldapUuidGroupAttribute | auto | | ldapUuidUserAttribute | auto | | turnOffCertCheck | 0 | | turnOnPasswordChange | 0 | | useMemberOfToDetectMembership | 1 |

</details>

### Client configuration
**Browser:**
not relevant
**Operating system:**

### Logs
#### Web server error log
<details>
<summary>Web server error log</summary>
no errors
</details>

#### Nextcloud log (data/nextcloud.log)
<details>
<summary>Nextcloud log</summary>
no errors
</details>

#### Browser log
<details>
<summary>Browser log</summary>

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

</details>

[rullzer]

Right. I see what is happening. The lookup server is indeed disabled. But since we do not have the state we try to tell it to remove all the data.

@schiessle any idea how to solve this?

[rullzer]

Ok I think I have a solution in mind.

The moment that the lookupserver uploads get disabled. We should insert a background job. This job should

1. Iterate over all users that have logged in
2. Check if any of their properties are marked as public

• if so insert a job to remove this info from the server
• if not skip the user

[juliusknorr]

@alx-tuilmenau @rullzer One workaround is to set an empty value for the lookup_server in config.php. The background jobs will not be rescheduled then, if the lookup server isn't reachable.

[schiessle]

One workaround is to set an empty value for the lookup_server in config.php. The background jobs will not be rescheduled then, if the lookup server isn't reachable.

yes, this would be a quick workaround to solve the problem.

As a more general solution, I wounder if we should introduce a "dsiable_lookup_server" config.php switch which would be:

1. more obvious then setting 'lookup_server' to an empty string
2. could skip the process of updating the lookupserver right at the beginning
3. would allow us to hide the additional visibility settings on the personal page if the lookup server is disabled.

[alx-tuilmenau]

In my opinion, if the "global adressbook" is not activated in the admin options, there should be no requests at all. Noone expects the leak of the usernames to an external server if the option is turned off, it's also a GDPR problem.
Of course, on turning off the option it's ok to remove users, but this should happen once on turning off and not on every user change and not if the option was never turned on.

If I set lookup_server to an empty value and re-activate the app, a lot of jobs are created very fast again, it's the same problem (it retries 5 times via cron). Maybe it depends on the configuration here, because outgoing http(s) is done via proxy. It looks like every LDAP sync, maybe on every login, will trigger the lookup_server sync. Setting lookup_server to an empty string is not a solution.

[schiessle]

@alx-tuilmenau ah, now I saw that a setting already exists "Allow users to publish their data to a global and public address book", I missed it when I looked at the settings matrix the first time.

Yes, we could use this to disable the connection to the lookup server completely which would come with the draw back that already published data would stay at the lookup server. So we would probably need something like suggested from @rullzer #13396 (comment)

If I set lookup_server to an empty value and re-activate the app, a lot of jobs are created very fast again, it's the same problem (it retries 5 times via cron).

With a quick look at the code I confirm this behavior. The connector will try to update the lookup server, will fail and create the background job, the background job will disappear after first try. But of course on every sync with the LDAP we will create them again and again. So setting the lookup_server to '' will at least help in that regard that the background jobs are cleaned up again.

I will work on a fix for this part, this should be straight forward.

[jngeiger]

We can see the same behaviour after upgrading from NC13 to NC14. Unfortunately the enormous job queue kills the whole server after a while.

Is there something we can do right now to resolve the issue? (empty vaule for lookup_server parameter didn't help)

[alx-tuilmenau]

@jngeiger See my first post, because I don't use federation, I moved apps/lookup_server_connector folder to a different location (outside nextcloud/docroot) and deleted all entries with class like %LookupServer% from oc_jobs table.

[bremensaki]

I'm also seeing this on my company server with AD-backed authentication. I noticed CPU load slowly creeping up after upgrading to 14, and finally tracked it back to this issue. Removing lookup_server_connector to mitigate the issue has it under control for now, but I'd like to not have to do that after every upgrade.

[bremensaki]

Is this fix likely to get a backport to 15.x?