Make System.OsString.Posix.fromBytes and Systen.OsString.Windows.encodeFS total

[Bodigrim]

Currently we have symmetric types:

System.OsString.Posix.fromBytes :: MonadThrow m => ByteString -> m PosixString
System.OsString.Windows.fromBytes :: MonadThrow m => ByteString -> m WindowsString

System.OsString.Posix.encodeFS :: String -> IO PosixString
System.OsString.Windows.encodeFS :: String -> IO WindowsString

It's not uncommon to work on a single-platform application or an area, which utilises only one flavour of OsString on both platforms. Recent work on tar is a prominent example, where PosixString is used as an internal representation even on Windows.

IMO it would be better to break the symmetry and provide

System.OsString.Posix.fromBytes :: ByteString -> PosixString
System.OsString.Windows.encodeFS :: String -> WindowsString

The lack of total conversions is the biggest friction point in my experience so far; if we can alleviate it at least in some scenarios, it's worth doing.

[hasufell]

This would be a breaking change.

[hasufell]

How would we migrate or deprecate? This is invasive and will require another major bump for two packages, as well as patches for directory at least, potentially with CPP.

[Bodigrim]

I'm not attached to names. Maybe just add two new functions:

System.OsString.Posix.fromByteString :: ByteString -> PosixString

and, say,

System.OsString.Windows.fromString :: String -> WindowsString

The latter function suggests that there could exist a well-behaved instance IsString WindowsString...

[hasufell]

It's probably no secret that I am no friend of IsString class: it has bugs and unclear semantics. So I'd rather avoid it, otherwise people will get hopes that OsString may get it.

Instead, GHC SC has to come up with a version of OverloadedStrings that is not broken.

[Bodigrim]

Fair enough. I'm not wed to any particular names, anything with pure type signature will do.

[hasufell]

#11

[BurningWitness]

It's probably no secret that I am no friend of IsString class: it has bugs and unclear semantics. So I'd rather avoid it, otherwise people will get hopes that OsString may get it.

Would IsString OsString be about as unsafe as IsString Text currently is or am I missing some massive usage caveat?

[hasufell]

It's probably no secret that I am no friend of IsString class: it has bugs and unclear semantics. So I'd rather avoid it, otherwise people will get hopes that OsString may get it.

Would IsString OsString be about as unsafe as IsString Text currently is or am I missing some massive usage caveat?

There are two caveats:

1. OsString is not guaranteed to be unicode (unlike Text). So the IsString instance conveys a false sense of "this type is unicode-ish" (although the osstr quasi-quoter encodes as UTF-8/16, but it's more explicit, easier to document and can reject surrogates)
2. Yes, it has the surrogate problem as well

[BurningWitness]

OsString is not guaranteed to be unicode (unlike Text).

But encoding itself is still deterministic on a given platform, so it is possible to make a total conversion function that encodes surrogates into either other surrogates or replacement characters, right?

[hasufell]

OsString is not guaranteed to be unicode (unlike Text).

But encoding itself is still deterministic on a given platform, so it is possible to make a total conversion function that encodes surrogates into either other surrogates or replacement characters, right?

I'm not sure what definition of "total" you are proposing, but encode . decode == id does not hold.

What does "other surrogates" mean? Proper UTF-8 does not allow surrogate range. If you're proposing PEP-383, then the unicode consortium has already voiced its displeasure: https://unicode.org/L2/L2009/09236-pep383-problems.html

[BurningWitness]

The point I'm leading to is that IsString, while being a runtime-evaluated codec typeclass (bad in three different ways, I fully agree), is overwhelmingly only used for hardcoding strings. So in spirit the encode . decode == id rule only matters in regards to encoding valid Unicode points, handling invalid ones may as well be considered undefined behavior (though for performance reasons it should probably encode invalid points into other invalid points as it sees convenient).

The current lack of an IsString OsString instance means that instead of a pleasant "honk" I have to input either [osstr|honk|] or unsafeEncodeUtf "honk", and it gets even stupider with lists. There is no upside to this, other than perhaps confusing newcomers, who most probably don't even know what UTF surrogates are.

So ideally in a proper API I would expect:

• An IsString instance with documentation stating usafe limits and that manual fromString use is never needed;

• unsafeEncodeUtf in a clearly unsafe module (perhaps System.OsString.Unsafe) and matching the behavior of fromString.

• (optional) encodeUtf/decodeUtf should return either Maybes or meaningful Eithers. The current exception format I assume is text cargo cult; a simple "character offset at which the error occurred" format would be far more useful.

[hasufell]

I have to re-iterate:

• the quasiquoter is safe, clearly documented and can even be made more explicit if need be (e.g. by renaming to osstrUnicode or exposing it through a .Unicode module)

• we can also provide ascii and other quasi quoter variants => this approach is clearly superior
• IsString is a misdesigned broken class that the community should stop using
• OverloadedStrings as an extension most certainly could be fixed or redesigned... I have no problem with string literals! Maybe it should be called OverloadedUnicodeStrings.
• the fact that users have to look for how to construct OsString is a good thing and means people are starting to become aware of the intricacies that surround strings and especially filepaths

I have written two large blog posts about Haskell filepath and Haskell strings:

• https://hasufell.github.io/posts/2024-05-07-ultimate-string-guide.html
• https://hasufell.github.io/posts/2022-06-29-fixing-haskell-filepaths.html

The whole point of AFPP/OsPath/OsString was to be more correct than String (yes, in those pesky little corner cases too). Providing an IsString instance is a blow to that vision. People can still use String and FilePath if they don't care that much about correctness or write orphan instances how they see fit.

Since ByteString instance of IsString is also problematic, we already have incentive of fixing OverloadedStrings. I'm not on GHC SC, but I can surely help promote and support such attempts.

[BurningWitness]

I know all the points you stated and I agree with all of them (except the "clearly superior" thing, else I wouldn't be writing this), but the community at large, including me, will only stop using IsString when better IsString arrives. Knowing how slow and painful GHC updates are, we're not going to see anything like that before year 2030. Until that day, every user of this library will be forced to bear with the nine-character-wide Template-Haskell-ridden quasiquoter overhead that doesn't seem to do anything and isn't even highlit properly in the editor.