should we attempt to URL canonicalize patterns?

[wanderview]

The URL() constructor does a number of things to canonicalize the input string. For example:

• Illegal characters are automatically URL encoded. For most components this is percent encoding, but for hostname it appears to use IDNA encoding, etc.
• Some characters, like {, normally get percent encoded by URL encoding, but URLPattern uses it as one of its special characters.
• For some protocols (http/https/etc) the pathname is required to consist of at least /. The slash is added if missing.
• For some protocols (http/https/etc) the pathname is flattened to remove .. and ..
• If the port matches the default port for a given protocol, then it is coerced to the empty string.
• Probably many others...

So the question is can we and should we apply these transformations to component patterns. My original intent was to try to do so, but I have identified a number of problems:

1. Often the transformations are not safe within the custom regexp pattern groupings. For example, if you try to percent encode a unicode character within a regexp [a-z] character list it won't work correctly since each character in the percent encoding is considered independently. Similar difficulties arise with the .. and . flattening since those characters may appear within a regexp. In general we don't want to have to duplicate the regexp parser in URLPattern, so solving this seems quite difficult.
2. A URLPattern may often not have a fixed protocol. This makes it difficult to apply transformations that are conditional on protocol.
3. Developers may confused that some characters are URL encoded, but others are not.

Therefore I'm leaning towards not applying any canonicalization or automatic encoding for patterns. For example, if a non-ascii character is included in the pattern we would throw. In contrast, however, URL values passed to test() and exec() would be fully URL canonicalized. Therefore developers would be required to write patterns to match canonical URLs, but we would not fully enforce or automatically help with that at URLPattern construction time.

I intend to implement the above to start, but if it becomes a problem we could fall back to the a half measure. The URL canonicalization could be applied to patterns, but only outside of custom regexp groups. If you include a unicode character within a custom regexp then URLPattern would throw, but otherwise would be automatically URL encoded. Of course, pattern special characters like { would still need to be exempted from encoding, so it would not be quite equivalent to URL constructor behavior.

Generally it will be easier to move from the "no canonicalization" approach to "canonicalize outside of regexp" without breaking existing patterns.

Thoughts?

[asutherland]

"no canonicalization" with aggressive throwing seems most straightforward and thereby understandable/document-able/reason-about-able.

[annevk]

While understandable in a way, it would make the design rather Latin-script centric. I suppose we also have that problem with the Access-Control-Allow-Origin header, but generally there are no such constraints when it comes to using URLs.

[wanderview]

Well, we could try the URL encoding of characters outside of the custom regexp groups. To better handle special characters like {, we could make an escaped \{ result in a percent encoded {.

It would still be a bit inconsistent and magical in places, but maybe ergonomic enough?

For first implementation, though, I'm going to start with no canonicalization in patterns.

[asutherland]

My motivating scenario is a developer struggling to get a given URLPattern to work but the problem is obscured because of canonicalization and the user assumes the problem is in the pattern match parts of the string rather than the constant parts because they're expecting the pattern matching magic but not the canonicalization magic. Specifically, situations like copying and pasting picking up fancy text styling, for example the pattern prefix "/raw–source" which would canonicalize to "/foo%E2%80%93bar" where "/raw-source" was what was intended.

I assume the pre-canonicalized version would never be exposed, so maybe this is fine as long as people don't use the idiom (new URLPattern(...)).test("foo") from the developer console (which would not pretty print the URLPattern and reveal the canonicalization versus using a temporary). Also, I suppose this is exactly the kind of thing where devtools would do well to detect what's going on and active some kind of specialized tool that breaks down exactly what's going on under the hood.

[wanderview]

Yea, I expect we will expose any canonicalized patterns on the URLPattern either directly or via an object stringifier.

[wanderview]

I realized one bad ergonomic result of not supporting . flattening. Many developers start relative URLs with ./. The dot ultimately gets flattened after the path is appended to the base. If patterns don't work the same way then I expect we will be creating a decent sized footgun.

[wanderview]

I found another canonicalization. If a port number matches the default port for a protocol, then its coerced to the empty string. I'll add this to the list at the top of this issue.

This is one we could try to handle for a pattern. If the pattern parses as the default number, then do the same. It basically would only work for number literals and would not work if any pattern syntax is included.

[annevk]

You can only normalize that if there is a scheme though, right? Otherwise you could only do that during matching it seems.

[wanderview]

Correct. Default ports would only be canonicalized to empty string if the protocol is unambiguously known. So a protocol of exactly "https", etc.

I would note this also applies to the input of test() and exec() when using structured input like #27. If you do p.test({ port: '80' }) the port is not canonicalized to default string because we don't know the protocol. I don't expect setting a port, but not a protocol, to be very common, though.

[wanderview]

Note, I opened a github "discussion" to collect community feedback on this issue. See #37.

[wanderview]

I have canonicalization prototyped for patterns now. What I settled on was running flat strings from that pattern through the browser's relevant component canonicalization routine.

For example, the pattern { pathname: '/foo/../bar/:name.html' } would canonicalize /foo/../bar, /, and .html separately. So in this example you end up with a pattern that matches a pathname /bar/something.html correctly.

It is possible for pattern authors to break this normalization by splitting strings up with groupings an what not. For example, /foo/{..}/:name.html would prevent the canonicalization routine from flatting the .. directory structure since it doesn't see the /../ sequence. While a bit unfortunate, I think its reasonable to accept this limitation and put this in the "don't do that" category.

Those kinds of corner cases aside, I think this will make it more convenient for pattern authors than having to manually do canonicalization in advance for patterns themselves.

I should also mention that "run the url component canonicalization" routine on each part of the pattern is also not quite true. I have a couple of exceptions. For example, I don't apply the part about adding a leading / that chromium's canonicalizer tries to add. I also don't do the bit for port where it flattens the default port to the empty string. We can't do that on partial parts of the input, but instead of have to apply it to the whole pattern, etc.

[wanderview]

Oh, a side note. One advantage of running the canonicalizer on the post-parse output of the pattern is that we can re-use the existing canonicalization routine without worrying about conflicting with special pattern syntax. All that syntax has been parsed out already. Similarly, escaped characters have either been treated specially already or flattened into the pattern string so they will be encoded if necessary; e.g. { is a special character in patterns, but \{ should be encoded as %7B.