Annotations do not support Unicode characters during I/O roundtrip

[hoechenberger]

Description of the problem

Raw data (and possibly epochs, which I haven't tested) with Annotations that contain Unicode descriptions cannot be saved.

This was initially reported at https://mne.discourse.group/t/saving-filtered-data-and-epochs/6848/3

Steps to reproduce

# %%
import mne

sample_dir = mne.datasets.sample.data_path()
sample_fname = sample_dir / 'MEG' / 'sample' / 'sample_audvis_raw.fif'

raw = mne.io.read_raw_fif(sample_fname, preload=True)
raw.crop(tmax=60)

annots = mne.Annotations(onset=0, duration=1, description='🙃')
raw.set_annotations(annots)

raw.save('/tmp/foo-raw.fif', overwrite=True)

Link to data

No response

Expected results

File is saved

Actual results

Writing /tmp/foo-raw.fif
Traceback (most recent call last):
  File "/private/tmp/mwe.py", line 13, in <module>
    raw.save('/tmp/foo-raw.fif', overwrite=True)
  File "<decorator-gen-242>", line 12, in save
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 1691, in save
    _write_raw(
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 2604, in _write_raw
    cals = _start_writing_raw(
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 2872, in _start_writing_raw
    _write_annotations(fid, annotations)
  File "/Users/hoechenberger/Development/mne-python/mne/annotations.py", line 1084, in _write_annotations
    write_name_list_sanitized(
  File "/Users/hoechenberger/Development/mne-python/mne/io/write.py", line 149, in write_name_list_sanitized
    write_string(fid, kind, _safe_name_list(lst, "write", name))
  File "/Users/hoechenberger/Development/mne-python/mne/io/write.py", line 130, in write_string
    str_data = str(data).encode("latin1")
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f643' in position 0: ordinal not in range(256)

Additional information

Since Unicode Annotations can be set, we should be able to write them. Hence I'm reporting this as a bug and not a feature request.

Proposals by @cbrnr, @drammock, and me:

• prevent using non-latin1 characters in Annotations (@hoechenberger) – but this is actually not acceptable in 2023 (@drammock, and I fully agree with that)
• amend FIFF definition to allow Unicode character storage (@cbrnr, @drammock)
• provide HDF5 export for all affected file types (@drammock)
• Maybe we can sneak in the Unicode via an ASCII encoding, think base64, but this sounds rather ugly (@hoechenberger)

[larsoner]

This problem is not unique to annotations and is a property of any FIF string at the moment:

>>> ave = mne.read_evokeds(mne.datasets.sample.data_path() / "MEG" / "sample" / "sample_audvis-ave.fif")[0]
>>> ave.comment
'Left Auditory'
>>> ave.comment = "Left 🙃"
>>> ave.save("test-ave.fif")
Traceback (most recent call last):
...
  File "/Users/larsoner/python/mne-python/mne/evoked.py", line 1902, in _write_evokeds
    write_string(fid, FIFF.FIFF_COMMENT, e.comment)
  File "/Users/larsoner/python/mne-python/mne/_fiff/write.py", line 131, in write_string
    str_data = str(data).encode("latin1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f643' in position 5: ordinal not in range(256)

I don't think it will be trivial to add support to FIF and would also likely break any other software that works with FIF to add such support. Hence I think we should go with option (1) for now and maybe someday implement (2) across the FIF string read/write routines.

[larsoner]

... also latin-1 is not a subset of utf-8 so backward compat here would be nearly impossible it seems -- writing utf-8 then reading with latin-1 software (MEGIN or old MNE or MNE-MATLAB, etc.) can get garbled stuff, and vice-versa (e.g., mne-tools/fiff-constants@82aaf64 rendering funny on GitHub by latin-1 encoding the superscript 3). So a compatible path forward is not trivial :(

[hoechenberger]

What about this workaround:

• provide HDF5 export for all affected file types (@drammock)

We've resorted to using HDF5 to overcome FIFF limitations in several other places, too.

When the user sets non-latin-1 strings, we can emit a warning that from now on, only HDF5 is allowed for an I/O round-trip, and raise an exception if they try save as FIFF. Or – raise another warning in that case, and force-convert the strings to latin-1 as good as we can.

[larsoner]

To me adding support for writing annot to HDF5 isn't really desirable because you still can't do raw.set_annotations(annot); raw.save(...) and have it work. So it's weird to have this partial support. And having raw.save('...-raw.h5') doesn't make sense because it's not portable to other software, which I think is important for raw data saving. (HDF5 in principle is readable everywhere but it's not a known/defined format for M/EEG data import in any other package.)