Too specific locale fi_FI

[kohtala]

Steps to reproduce

1. Open Firefox or Chrome, configure Finnish as preferred language
2. Remove config/config.php default_language.
3. Share a link, open the link in Firefox or Chrome

Expected behaviour

Should see the link page in Finnish language.

Actual behaviour

Shows the link page in English.

Browser sends "Accept-Language: fi,en-US;q=0.7,en;q=0.3". The undefined default_language lets Nextcloud determine language from this request header. Nextcloud has language fi_FI, but no fi. Therefore Nextcloud defaults to en. Internet Explorer 11 has option to select both fi and fi-FI language, where fi-FI correctly selects the Nextcloud fi_FI language. Firefox and Chrome allow only fi language. Also, if Nextcould had language fi, but no fi_FI, I think Accept-Language of fi_FI would fall back to fi (see OC\L10N\Factory::findAvailableLanguages in lib/lib/private/l10n/factory.php). Therefore I think it is wrong that Nextcloud has specific fi_FI instead of the less specific fi. The same issue seems to be with other languages as well (af, am, bg, cs, cy, fy, hu, ka, ku, ms, mt, my, nb, nn, sw, tg, th, tl, ur, by looking at core/l10n), and I'm not sure if every language should have a generic fallback.

Server configuration

Operating system: Linux, openSUSE 13.2

Web server: Apache

Database: MySQL

PHP version: 5.6.1

Nextcloud version: Nextcloud 9.0.53 (stable)

Updated from an older Nextcloud/ownCloud or fresh install: fresh

Where did you install Nextcloud from: nextcloud.com nextcloud-9.0.53.tar.bz2

Signing status:
No errors have been found.

List of activated apps:
Enabled:

• activity: 2.2.1
• admin_audit: 1.0.0
• comments: 0.2
• dav: 0.1.6
• federatedfilesharing: 0.1.0
• federation: 0.0.4
• files: 1.4.4
• files_pdfviewer: 0.8.1
• files_sharing: 0.9.1
• files_texteditor: 2.1
• files_trashbin: 0.8.0
• files_versions: 1.2.0
• files_videoplayer: 0.9.8
• firstrunwizard: 1.1
• gallery: 14.5.0
• notifications: 0.2.3
• password_policy: 1.0.0
• provisioning_api: 0.4.1
• systemtags: 0.2
• templateeditor: 0.1
• theming: 0.1.0
• updatenotification: 0.1.0
  Disabled:
• encryption
• external
• files_external
• user_external
• user_ldap
• user_saml

The content of config/config.php:
{
"system": {
"passwordsalt": "_REMOVED SENSITIVE VALUE",
"secret": "_REMOVED SENSITIVE VALUE",
"trusted_domains": [
"_REMOVED SENSITIVE VALUE"
],
"datadirectory": "_REMOVED SENSITIVE VALUE",
"overwrite.cli.url": "_REMOVED SENSITIVE VALUE",
"htaccess.RewriteBase": "/",
"dbtype": "mysql",
"version": "9.0.53.0",
"dbname": "_REMOVED SENSITIVE VALUE",
"dbhost": "localhost",
"dbtableprefix": "oc_",
"dbuser": "_REMOVED SENSITIVE VALUE",
"dbpassword": "_REMOVED SENSITIVE VALUE",
"logtimezone": "UTC",
"installed": true,
"instanceid": "_REMOVED SENSITIVE VALUE",
"singleuser": false,
"skeletondirectory": "_REMOVED SENSITIVE VALUE",
"mail_from_address": "noreply",
"mail_smtpmode": "php",
"mail_domain": "_REMOVED SENSITIVE VALUE_"
}
}

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Firefox 48.0 or Chrome 52.0.2743.116 m

Operating system: Windows 7 Pro

Logs

Web server error log

Empty

Nextcloud log (data/nextcloud.log)

Unrelated errors
Error PHP Undefined variable: share at .../lib/private/share20/manager.php#996
Error PHP Undefined index: id at .../core/ajax/share.php#229
Error PHP Cannot modify header information - headers already sent by (output started at .../3rdparty/sabre/http/lib/Sapi.php:83) at .../3rdparty/sabre/http/lib/Sapi.php#63

Browser log

[LukasReschke]

cc @MorrisJobke Thoughts?

[rullzer]

Mmmm this is a trick one. In this specific case we could just check if there is no match for xx and there is only one match for xx_YY then take that one.

It becomes more troublesome if there are multiple. So there is xx_YY and xx_ZZ and wich one do we pick?

But we can't be the only one with this issue. So lets see how others do it.

[rullzer]

It seems django just pickes one if from xx_YY if xx is not available:

1. https://github.com/django/django/blob/master/django/utils/translation/trans_real.py#L498
2. https://github.com/django/django/blob/master/django/utils/translation/trans_real.py#L536
3. https://github.com/django/django/blob/master/django/utils/translation/trans_real.py#L474

I vote to do this as well

[kohtala]

Hi.

Seems to me Django foremost has only the generic language "xx"

1. https://github.com/django/django/blob/master/django/conf/global_settings.py#L55
2. https://github.com/django/django/blob/master/django/conf/locale/__init__.py#L13

Therefore, in Django, the get_supported_language_variant falls back from fi-FI to fi. Only for Chinese there is no generic language and for it, if get_supported_language_variant is called with strict parameter False, it'll take first zh-hans (Simplified Chinese) if zh-YY is not found.

It seems to me a proper fix would be to change the fi_FI and other such languages where there is only one variant without generic language to fi or the corresponding generic language. I do see that may be complicated with Transifex, with any merges from ownCloud, and it affecting so many languages and translators. I'm a newcomer in this community, I do not fully understand the situation, and feel this is too big change for me to suggest as first thing I do here.

The option to select some arbitrary variant seems like a practical first step to help transition from variants to generic languages. I do think it needs a comment explaining what problem it is solving and which direction to take to avoid the problem properly.

[rullzer]

@MorrisJobke can you check if there is some setting in transifex that allow this?

Else I'll add the fallback code.

[MorrisJobke]

Yes we already do exactly this for Japanese: https://github.com/nextcloud/server/blob/master/l10n/.tx/config#L3

So the way to address this is to add fi_FI there too, right? We want to map fi_FI to the generic one fi.

[MorrisJobke]

I will add this tomorrow, because all repos need to be adjusted.

[rullzer]

Ah yes that would solve it as well! 👍

[kohtala]

Thanks, that looks like a clean solution.

Comparing core/l10n and Firefox 48.0.1 language list in Option / Content / Languages / Add, in addition to fi_FI:fi there should be also af_zA:af, am_ET:am, bg_BG:bg, cs_CZ:cs, cy_GB:cy, fy_NL:fy, hu_HU:hu, ka_GE:ka, ku_IQ:ku, lt_LT:lt, ms_MY:ms, mt_MT:mt, my_MM:my, nb_NO:nb, nn_NO:nn, si_LK:si, sk_SK:sk, sw_KE:sw, tg_TJ:tg, th_TH:th, tl_PH:tl, ur_PK:ur.

I think the /l10n/rm-old.sh, /settings/languageCodes.php and /settings/personal.php need an update as well.

On upgrade it would be kind to convert user core.lang preferences as well. Upgrading config/config.php default_language may not be quite as necessary, as it is easy for admin to do.

Btw, I did not understand the rationale why config/config.php default_language overrides the language sent by http client in Accept-Language header. I'd have expected there to be an option to select the default language to use if even http client did not specify any language. But maybe this needs to be raised in separate issue.

[MorrisJobke]

Yes for the second question please open a separate issue.

Sadly there was too much RC work today. I try to fix this tomorrow.

[MorrisJobke]

Thanks for this nice list 👍