Non-breaking space characters are truncated by `Unicode.Transform.LatinAscii.transform/1`

[arcanemachine]

When using Unicode.Transform.LatinAscii.transform/1 to transform strings that contain "non-breaking space" characters (e.g.  ), the non-breaking spaces are being truncated:

iex> Unicode.Transform.LatinAscii.transform("There and Back Again")
"There andBack Again"

Is this intentional behavior, or is this something that should be handled differently by this function? (I would have assumed that these characters would be converted to a regular space character.)

Thanks

[kipcole9]

Ouch, thats very unlikely to be intentional. I'll look into this over the next couple of hours.

[kipcole9]

Bug reproduced and confirmed. There are rules for handling Unicode spaces that should replace them with ASCII spaces and I need to work out why they aren't. I'm suspect I'm not parsing the hex form correctly.

Transformation rules for Unicode spaces:

#
# Spaces
#
\u00A0 → ' ' ; # 00A0;NO-BREAK SPACE
\u2002 → ' ' ; # 2002;EN SPACE (compat)
\u2003 → ' ' ; # 2003;EM SPACE (compat)
\u2004 → ' ' ; # 2004;THREE-PER-EM SPACE (compat)
\u2005 → ' ' ; # 2005;FOUR-PER-EM SPACE (compat)
\u2006 → ' ' ; # 2006;SIX-PER-EM SPACE (compat)
\u2007 → ' ' ; # 2007;FIGURE SPACE (compat)
\u2008 → ' ' ; # 2008;PUNCTUATION SPACE (compat)
\u2009 → ' ' ; # 2009;THIN SPACE (compat)
\u200A → ' ' ; # 200A;HAIR SPACE (compat)
\u205F → ' ' ; # 205F;MEDIUM MATHEMATICAL SPACE (compat)
\u3000 → ' ' ; # 3000;IDEOGRAPHIC SPACE (from ‹character-fallback›)

[kipcole9]

Nope, it's the rules generator. Basically I generate Elixir code from the .xml rules files. In this case, the generate rule is:

  # 00A0;NO-BREAK SPACE
  replace("\u00A0", "")

Which is clearly wrong. It should be:

  # 00A0;NO-BREAK SPACE
  replace("\u00A0", " ")

Now to work out what (no more commentary until a fix is done)

[kipcole9]

I have pushed a commit that fixes this bug. It was a spurious String.trim/2 function call. May I ask you to try it out from Github before I publish and update?

{:unicode_transform, github: "elixir-unicode/unicode_transform"}

[arcanemachine]

Working well on my end.

Thank you for the incredibly quick fix, Kip! Much appreciated.

[kipcole9]

I've published unicode_transform version 0.4.0 with the following changelog entry:

Bug Fixes

• Fix rules generator to ensure that a transform to an ASCII space is not trimmed (thereby causing an empty string). Thanks to @arcanemachine for the report. Closes Non-breaking space characters are truncated by Unicode.Transform.LatinAscii.transform/1 #2.

Enhancements

• Updates to CLDR 47 Transform rules (most of which remain unimplemented in this release)

Thanks again for opening the issue. Glad it wasn't too tricky to get a resolution quickly. I really like this library - even though it's not yet a complete implementation. I'll try and get back to it in the new year. Its a nice feeling when you revisit code you've not work on for a couple of years and it still feels like its solid.