need regular expressions for constraining dataset IDs, parameter names

[jbfaden]

I don't think there's a regular expression, or really any tight spec, for what constitutes a valid ID or parameter. I'd suggest valid C variable names, but we are already beyond having that. Also, SPASE IDs are used in some places, so this should be supported as well.

To get the conversation started, I'd suggest:
[a-zA-Z_]+[a-zA-Z0-9_/:-+]*

where plus is understood to be an alias for spaces when options are displayed.

[rweigel]

I think this was discussed before. I can't find it at https://github.com/hapi-server/data-specification/wiki/telecon-notes or in a search on the issues for this repository, however.

The verifier warns if there is no match to [_a-zA-Z][_a-zA-Z0-9]{1,30}, which is my interpretation of a c identifier based on some research (there is no standard, but this seems the least common denominator). Probably the verifier should give a reason for why this is desired and the fact that it is a warning acknowledges that some may not want to do it.

I suspect that decided that we could not constrain dataset IDs beyond what is allowed in a URL parameter. The logic is that clients need to decode the parameters anyway and also that some servers will want to use Unicode characters in the ID (some systems do already internally, again from what I recall). So if a client wants to create a file or directory with a name based on the ID, they need to find a regexp that converts it to an allowed file or directory name. I recall doing this for the fist attempt at a Python client, but probably a MD5 hash of the dataset ID should be used.

[jbfaden]

That's right for a C (or Java or Python) identifier. I think this spec needs to have some limitations, because otherwise someone is going to write a client in a language that doesn't support unicode and things will break. I'd also suggest there be strong conventions like replacing + with spaces, and perhaps displaying unicode characters escaped in the URL. So if the catalog says there should be a dataset called %C3%BC this could be displayed as ü to make things more readable. (This is the UTF-8 encoding). (https://stackoverflow.com/questions/19515616/how-to-handle-special-characters-in-url-as-parameter-values)

[rweigel]

We already have a limitation - the URL specification (assuming HTTP libraries implement it - probably they do for security reasons).

[jbfaden]

I always have trouble with this stuff. The URL package is limited to some characters, for sure. So maybe this ticket should address the question, how should spaces and unicode characters be represented in URLs. Since + and " " are already equivalent in many systems, I think space is handled. Since there's a clear encoding of any ASCII character, I think this is also handled. So the question remains for unicode characters. If I want to set up a server with the dataset "ΔV" should this be allowed and how should the dataset ID be encoded in URLs? How about "温度" (Japanese "temperature")?

[rweigel]

According to https://en.wikipedia.org/wiki/Percent-encoding#Current_standard

The generic URI syntax recommends that new URI schemes that provide for the representation of character data in a URI should, in effect, represent characters from the unreserved set without translation, and should convert all other characters to bytes according to UTF-8, and then percent-encode those values. This suggestion was introduced in January 2005 with the publication of RFC 3986.

• and space are addressed on this page. They must be percent-encoded. Quite often I see people use space in filenames that they post on the web. Most browsers will take an invalid space that you enter in their address field and add the percent-encoded space before making a valid request to the server. I think that I've seen people pass around URLs with symbols before too and I think the browser does the conversion for you. See this guy's complaint https://perishablepress.com/stop-using-unsafe-characters-in-urls/ about this practice. See also https://stackoverflow.com/questions/695438/what-are-the-safe-characters-for-making-urls

温度 and ΔV are addressed by the above excerpt. So I suppose we could cite RFC 3986.

[jbfaden]

I agree, and thanks for finding those. I've been looking for something like https://perishablepress.com/stop-using-unsafe-characters-in-urls/ for a long time, to simply explain what should and shouldn't be there.

[jvandegriff]

related to unicode thoughts in issue #96

[jvandegriff]

need to distinguish between parameters (which are more strict) and IDs, which are already showing up as SPASE Ids, etc;

this could just be a few lines in the spec about positive statements: if you match this expression, it will work (but it still might work if it does not match)

[jvandegriff]

probably need to support URL syntax, especially for DOIs and also for SPASE IDs