Proposal for URI Template support

[darrelmiller]

This is a proposal to enable OpenAPI to support many of the features of URI Templates defined by RFC 6570, without adopting the URI Template syntax.

Overview

Open API already supports some of the URI Template capabilities but it does so with a simplified path template syntax and then parameter objects that describe the parameters.

In summary, the proposal is to add the following properties to the parameter object:

• escape (Boolean, default true)
• separator (Char)
• explode (Boolean, default false)

Below is a description of the URI Template features and how these new parameter properties would work.

Details

RFC 6570 defines four levels of support.

Level 1: /bar/{foo}/baz

Open API already supports Level 1 templates

Level 2: {[+|#]foo}
This level adds support for fragment parameters and unescaped parameter values.
By adding the fragment option to the in property of the we could support fragment parameters. However, as fragment parameters are not sent from client to server, I'm not sure there is any value in adding this support.

Adding an escape Boolean property to the parameter object would allow OpenAPI parameters to specify whether parameter values be escaped. This should be an optional property with a default of true.

{
  "name": "callbackurl",
  "in": "query",
  "description": "Url to be called when event occurs",
  "required": true,
  "escape" : false,
  "schema": {
    "type": "string"
  }
}

Level 3: {[+|#|.|/|;|?|&]foo,bar}

This level adds a bunch of separator characters that are prefixed in front of the parameter value. By adding an optional separator property to the parameter object we can add this capability to OpenAPI. The value the ? and & parameters are debatable because we already known which parameters are in the query string.
The ability to have a list of parameter names in a var spec is just a syntax shortcut and I don't believe adds any capability that OpenAPI doesn't already have.

{
  "name": "colour",
  "in": "path",
  "description": "Optional Colour of car",
  "required": false,
  "separator" : ";",
  "schema": {
    "type": "string"
  }
}

With a path like

/cars/{make}/{model}{colour}/{year}

an output path might be

/cars/Honda/accord;red/2010 

Level 4: {[+|#|.|/|;|?|&]foo,bar[:n|*]]}

This level adds two suffix options and the support for values that are lists and maps. The :nsuffix option that allows you to just take the first few characters off a parameter value seems unnecessary to me.
The explode operator is useful if we accept supporting maps and lists as parameter values. Should map and list be added as two native parameter types? Adding an explode Boolean property to the parameter object would be sufficient to add this support.

{
  "name": "colour",
  "in": "path",
  "description": "Optional Colour of car",
  "required": true,
  "separator" : ";",
   "explode" : true,
  "schema": {
    "type": "string"
  }
}

With this parameter and assuming the path defined in the level 3 example and a parameter value of ["red","green","blue"] then you would get:

     /cars/Honda/accord;colour=red;colour=green;colour=blue/2010 

Constraints

There are certain capabilities in URI Templates that make it difficult for tooling to do the types of things that users are used to doing with OpenAPI. In order to limit the pain, there are some constraints that might be worth adding to these new parameter object properties:

• Escape = false cannot be used in path parameters. This prevents the introduction of additional path segments due to the presence of forward slashes in parameter values. This makes mapping of URLs to operations difficult.
• For similar reasons, should the / separator be allowed in path parameters?
• Explode operator can only be true for parameters of type array and object.

[ePaul]

One side effect of having query parameters in the path string (i.e. level 3) is that it possibly allows having different operations for the same "resource" (i.e. path without query parameters), thereby solving parts of #164. (Doesn't mean I argue for this, just saying that it actually adds capability.)

[darrelmiller]

@ePaul Yeah, I've done something like /customers{;view} before to enable /customers;detailed and /customers;simple

[dilipkrish]

If I may make a suggestion, I think its easier to think of a url as a path template as-is and not re-construct it from attributes on parameters (separators etc.). When you have a non-trivial set of parameters with each specifying separators they conflate the notion of templates into parameters. It is a concept thats at a higher level than a parameter, for e.g. take name and age as query parameters {?name,age} what would the separator for age be? I think its easier for the operation to determine that than let the operation specify it.

IMO I think it should work just like how templates work with path variables.

Another benefit of doing it this way is also that it allows for unique paths that can be exercised independently rather that the work around today where you have one path -> {union or all the parameter combinations}. By that I mean we can have the following

method	path	params
customers by email	GET /customers{?email}	email
customers by name	GET /customers{?firstName,lastName}	firstName, lastName

The current solution is to model it this way

method	path	params
customers by email OR name	GET /customers	email, firstName, lastName

Plus this is already supported in springfox and here is the corresponding swagger-ui with RFC6570 support.

I really don't think the levels really matter in general. However, the spec could specify a subset coming from a contract-first perspective for e.g. GET /customers{#email} would be a weird contract path. There is an inherent impedance mismatch between what you an specify and what server frameworks implement. Its similar to the contract first fidelity of xml schema which never translated cleanly to code e.g allOf, oneOf etc.

[darrelmiller]

@dilipkrish To answer your question about the age parameter. You don't need to specify a separator for it. We already know using in:query that it is a query string parameter and therefore know the separation rules. The only reason that URI templates need the ? separator is because there is no distinction in RFC 6570 between parameters in the authority/path/query. This actually becomes a major problem when trying to match URIs to templates and extract parameter values. I have a bunch of open issues in my URI Templates library because the varspec doesn't have all the information I need to be able to extract parameters correctly.
URI Templates are mostly awesome for generating URLs from templates, but they have a bunch of problems when trying to go from URL instance to matching template.

Yes, I agree we need to address the desire to have distinct resources that only differ by query parameters. For the longest time I was in agreement that pushing complete URI templates into the path was the correct solution. However, I no longer think so. I've spent quite a bit of time looking at API routing mechanisms and the most efficient solution is always to match the path first and then match query parameters as a distinct step. If you don't do that, then you end up with problems like failing to match when query parameters are provided in a different order than the definition. When you live in the hypermedia world, this doesn't happen, but in the rest of the world it does.

I've been considering a solution that looks something like this...

paths:
   /customers:
       get-by-email:
          parameters:
              email: ...
          responses: ...
       get-by-name:
          parameters:
              firstname: ...
              lastname: ...
          responses: ...

The operation key now allows a -{something} suffix to allow distinct resources with the same path. This provides backward compatibility with 2.0 operations and routing tooling gets to continue to use the path key and HTTP method to limit to a set of operations. Where multiple operations are matched we then need to match the query string to defined parameters. This is a new process that we would need to define some rules for.

Interestingly, in the tables you created to explain your scenario, you created a column called method and included customers by name and customers by email. In my example, I'm simply allowing methods to be get-by-name and get-by-email. It doesn't sound like our approaches are too far apart :-)

[dilipkrish]

The only reason that URI templates need the ? separator is because there is no distinction in RFC 6570 between parameters in the authority/path/query.

Its also to distinguish between forms continuation. e.g. constant/defaulted parameters GET customers?state=TX{&firstName,lastName}

If you don't do that, then you end up with problems like failing to match when query parameters are provided in a different order than the definition. When you live in the hypermedia world, this doesn't happen, but in the rest of the world it does.

Url templates is a solved problem, at least it has a comprehensive test suite. Tactically tho' this can be solved by sorting the parameters when building the templates. For backwards compatibility with existing swagger endpoints and providing determinism in template expansion, what I've been doing is

• stripping the query parameters
• stripping the fragments
• sorting the parameters and re-constituting up the template from the parameters.

While its essentially moving characters from part of the (OAI) object graph to another, its not the same thing IMHO. Im thinking of downstream possibilities that it opens up rather than just the ability to describe operations. To list a few reasons why I prefer not having that indirection,

• Using templates when matching link relations. When the server responds with a templated link (HAL kind) we want to be able to lookup the definition for that operation easily. Without that, it puts the burden of de-structuring these url templates by hand for lookup at runtime on the clients. Its easier for clients to just match the url template than match the stem of the url and search for candidates based on query parameters it infers from the url template. Hope that makes sense.
• For a contract first approach its easier to describe the path template directly, rather than describe what constitutes it. Its visually apparent what we're trying to do.
• It especially breaks down when you have islands of templates e.g.

endpoint	params	expansion
GET /customers?state=TX{&firstName,lastName}	state (always TX), firstName, lastName	GET /customers?state=TX&firstName=ABC&lastName=DEF
GET customers{?state}&latlong={coodinates}	state, coordinates := ("98.0", "91.0")	GET /customers?state=TX&latlong=98.0,91.0

• Not to get philosophical about it but it moves from describing http endpoints to describing remote method calls (ala WSDL). Which is why I was careful not to make it look like a method name.
• Lastly (most importantly :-)) It makes it a tad harder for code-first approaches like springfox, which has already solved this problem.

[darrelmiller]

@dilipkrish

Its also to distinguish between forms continuation. e.g. constant/defaulted parameters GET customers?state=TX{&firstName,lastName}

Arguably we can already make that distinction because we have the required attribute.

Url templates is a solved problem, at least it has a comprehensive test suite.

Only for going from template to URL instance, not the other way around. RFC 6570 explicitly calls this out

Some URI Templates can be used in reverse for the purpose of variable
matching: comparing the template to a fully formed URI in order to
extract the variable parts from that URI and assign them to the named
variables. Variable matching only works well if the template
expressions are delimited by the beginning or end of the URI or by
characters that cannot be part of the expansion, such as reserved
characters surrounding a simple string expression. In general,
regular expression languages are better suited for variable matching.

While its essentially moving characters from part of the (OAI) object graph to another, its not the same thing IMHO

There is definitely an elegance to URI Templates (once you get past some of the weird operator syntax). However, I think it is easier for tools to glue things together than to split them apart. The code to create a URI Template out of the component pieces in an OpenAPI document is massively simpler than extracting parameter information from a URI Template.

Allowing query parameter templates in the path template introduces the obvious problem that it is no longer just a path. Do we really want to rename path?

If a template indicates a query parameter is optional, but the parameter definition says the opposite, which wins? Do parsers need to detect that as a syntax error?

And URI templates themselves are not without limitations. Parameter names cannot include hyphens without them be escaped. And there is no way to express an optional query parameter that should not be escaped.

As an author of a URI Template library, I would love to see their use everywhere, but I'm not convinced that they are the best fit for the OpenAPI model.

[dilipkrish]

Only for going from template to URL instance, not the other way around. RFC 6570 explicitly calls this out

Wasn't what I was implying. Take this example of a HAL resource interaction.

Curl

   GET /orders HTTP/1.1
   Host: example.org
   Accept: application/hal+json

   HTTP/1.1 200 OK
   Content-Type: application/hal+json

Response body

   {
     "_links": {
       "self": { "href": "/orders" },
       "next": { "href": "/orders?page=2" },
       "find": { "href": "/orders{?id}", "templated": true }
     },
     "_embedded": {
       "orders": [{
           "_links": {
             "self": { "href": "/orders/123" },
             "basket": { "href": "/baskets/98712" },
             "customer": { "href": "/customers/7809" }
           },
           "total": 30.00,
           "currency": "USD",
           "status": "shipped",
         },{
           "_links": {
             "self": { "href": "/orders/124" },
             "basket": { "href": "/baskets/97213" },
             "customer": { "href": "/customers/12369" }
           },
           "total": 20.00,
           "currency": "USD",
           "status": "processing"
       }]
     },
     "currentlyProcessing": 14,
     "shippedToday": 20
   }

I will be convenient to lookup the definition of the find operation defined in the linked relationships using the path /orders{?id}. Without the full url template, the clients will have a much more difficult job matching the definition (1. find the /orders path, 2. extract the parameter(s) required from the uri template (regex) and match the different "methods" with those parameter(s) deterministically), wouldn't you agree?

Ofcourse the solution for #688 might make it simpler, but on its own, this solution would enable richer possibilities for downstream effects.

Allowing query parameter templates in the path template introduces the obvious problem that it is no longer just a path. Do we really want to rename path?

Don't follow what that means. Path already supports a template minus templates for query parameters. Are u suggesting URI templates with query strings in the template are no longer paths?

Also, we're not trying to implement the uri templates exhaustively, an 80/20 solution would suffice IMO. Its not very different from OAI supporting json schema, but not every nuance it can describe. It goes back to your post about open vs closed world specification.

[fehguy]

This has been clarified by #804