KeyError from ttfFontProperty while parsing

[kitchoi]

While importing font_manager on Appveyor, we found a number of similar traceback, pointing towards a failure to parse default system fonts.

The errors look like this:

[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\msjhl.ttc
[00:03:39] Traceback (most recent call last):
[00:03:39]   File "C:\Users\appveyor\.edm\envs\py36-ci\lib\site-packages\kiva\fonttools\font_manager.py", line 1440, in <module>
[00:03:39]     fontManager = pickle_load(_fmcache)
[00:03:39]   File "C:\Users\appveyor\.edm\envs\py36-ci\lib\site-packages\kiva\fonttools\font_manager.py", line 1050, in pickle_load
[00:03:39]     fh = open(filename, 'rb')
[00:03:39] FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\appveyor\\AppData\\Roaming\\Enthought\\kiva\\fontList.cache'
[00:03:39]
[00:03:39] During handling of the above exception, another exception occurred:
[00:03:39]
[00:03:39] Traceback (most recent call last):
[00:03:39]   File "C:\Users\appveyor\.edm\envs\py36-ci\lib\site-packages\kiva\fonttools\font_manager.py", line 717, in createFontList
[00:03:39]     props.append(ttfFontProperty(fpath, font))
[00:03:39]   File "C:\Users\appveyor\.edm\envs\py36-ci\lib\site-packages\kiva\fonttools\font_manager.py", line 525, in ttfFontProperty
[00:03:39]     name = props[(1, 0, 0, 1)].decode()
[00:03:39] KeyError: (1, 0, 0, 1)

There are 50 font files that trigger this error.

From the test log:

[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\msjhl.ttc
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\seguisym.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\leelauib.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\segoeuii.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\segoeuisl.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\msyh.ttc
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\seguisli.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\seguibl.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\segoeuiz.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\seguisbi.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\yugothic.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\segoeuib.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\ebrima.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\seguibli.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\seguili.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\seguiemj.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\simhei.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\segoeui.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\tahomabd.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\seguisb.ttf
[00:03:39] Could not covert font to FontEntry for file c:\windows\fonts\tahoma.ttf
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\msyhl.ttc
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\leelawui.ttf
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\msgothic.ttc
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\yugothib.ttf
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\yumindb.ttf
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\leeluisl.ttf
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\msjhbd.ttc
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\msyhbd.ttc
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\yuminl.ttf
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\segoeuil.ttf
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\ebrimabd.ttf
[00:03:40] Could not covert font to FontEntry for file c:\windows\fonts\msjh.ttc
[00:03:41] Could not covert font to FontEntry for file c:\windows\fonts\yugothil.ttf
[00:03:41] Could not covert font to FontEntry for file c:\windows\fonts\yumin.ttf
[00:03:41] Could not covert font to FontEntry for file c:\windows\fonts\segoewp-black.ttf
[00:03:41] Could not covert font to FontEntry for file c:\windows\fonts\segoewp-bold.ttf
[00:03:41] Could not covert font to FontEntry for file c:\windows\fonts\segoewp-semibold.ttf
[00:03:41] Could not covert font to FontEntry for file c:\windows\fonts\segoewp-light.ttf
[00:03:41] Could not covert font to FontEntry for file c:\windows\fonts\segoewp-semilight.ttf
[00:03:41] Could not covert font to FontEntry for file c:\windows\fonts\segoewp.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\LeelaUIb.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\LeelawUI.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\LeelUIsl.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\SegoeWP-Black.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\SegoeWP-Bold.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\SegoeWP-Light.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\SegoeWP-Semibold.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\SegoeWP-Semilight.ttf
[00:03:41] Could not covert font to FontEntry for file C:\windows\Fonts\SegoeWP.ttf

In other words, this line fails with KeyError, could we have handled this differently?:

enable/kiva/fonttools/font_manager.py

Line 525 in a843b37

name = props[(1, 0, 0, 1)].decode()

[mdickinson]

I'm also seeing this; something seems to have changed on this front recently (possibly elsewhere in ETS).

[JCorson]

I think this is caused by changing from the verbose error handling where all errors were being swallowed to logging the errors. Could be that you want to change the level of those messages from error to debug.

[mdickinson]

@JCorson: Okay, that makes some sense.

Any idea what's actually causing the KeyError: (1, 0, 0, 1)? Is there something to be fixed there?

OT: I do like the idea of "covert fonts" though (from the error message).

[JCorson]

OT: I do like the idea of "covert fonts" though (from the error message).

Well now that we are logging we have blown their cover!

[JCorson]

Any idea what's actually causing the KeyError: (1, 0, 0, 1)? Is there something to be fixed there?

Don't know offhand. If we are expecting (1, 0, 0, 1) to be the key for the font's name, we could at least do a better job of checking and making the error message more helpful. There is likely a better way to handle finding the font name too.

[mdickinson]

Digging in a bit, the first font for which this fails on my (macOS) system is /Library/Fonts/Trattatello.ttf. Its name appears to be there under the key (3, 1, 1033, 1), where the "3" means Microsoft, the "1033" indicates US English, and I'm not sure about the 1st 1. So expecting the name to be found under (1, 0, 0, 1) doesn't seem to be the right thing, in general.

[mdickinson]

Some references:

• Apple: https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6name.html
• MS: https://docs.microsoft.com/en-us/typography/opentype/spec/name

[JCorson]

@mdickinson @corranwebster I can take some time to work on this. Do you have any suggestions for including font files to test against? I have tried the test *.ttf files in fonttools and matplotlib, but they do not show the same error.

[mdickinson]

Do you have /Library/Fonts/Trattatello.ttf? That's the first font that's showing issues for me.

[mdickinson]

Hmm. I think that font may have been installed as a result of doing a local install of MS Office 365.

[JCorson]

Do you have /Library/Fonts/Trattatello.ttf? That's the first font that's showing issues for me.

Yup. I have the font and can reproduce the failure. I was more concerned with including a font file in kiva/fonttools/tests/data and being license compliant.

[mdickinson]

Ah, sorry. No, I don't (knowingly) have a shareable font file that could be used to reproduce.

[JCorson]

So I have been able to get a somewhat working solution locally, but am running into some issues with the encoding of a few font names on OSX:

Raw: b'\xf8\xf2\xf0\xf0\xe4'
Encoding: windows-1255
Decoded: רעננה
Could not covert font to FontEntry for file /Library/Fonts/Raanana.ttc

Raw: b'\xbb\xaa\xce\xc4\xcb\xce\xcc\xe5'
Encoding: KOI8-R
Decoded: ╩╙ндкнлЕ
Could not covert font to FontEntry for file /Library/Fonts/Songti.ttc

Raw: b'Apple SD \xbb\xea\xb5\xb9\xb0\xed\xb5\xf1 Neo'
Encoding: ISO-8859-1
Decoded: Apple SD »êµ¹°íµñ Neo
Could not covert font to FontEntry for file /System/Library/Fonts/AppleSDGothicNeo.ttc

On enable 4.8.0 (OSX), running createFontList(findSystemFonts()) returned 603 fonts, and with the fix I have been working on, we get 739 (haven't checked on Windows or Linux yet).

I determined the above encoding using chardet, but I don't think that is a viable solution given that the package is LGPL.

So I think some of this may come down to the desired UX. Previously all of these errors were completely silent, and so we didn't know that we were failing on so many fonts, which isn't good. What do you think the UX should be for this? We can changing the logging level for these errors so that they are not reported to the console. Given that there is currently no actionable information in the messages, this might be ok.

[mdickinson]

I determined the above encoding using chardet, but I don't think that is a viable solution given that the package is LGPL.

So in theory, I think the other fields in the name table should contain the necessary information for determining the encoding. But writing our own code to handle that seems like a can of worms.