Does anyone actually use JSON Hyper-schema

[awwright]

There's a lot of broken features defined in JSON Hyper-Schema, I want to ask implementors how much I can be allowed to "break" (i.e. make compliant with normative references).

Mostly things like quirks about how it defines URI templates, uses "rel", and uses "method".

Anyone?

[handrews]

From my point of view, I'd be happy to reboot the whole hyper-schema approach while preserving some specific elements. In my past work, we picked bits and pieces from JSON Hyper-Schema to assemble into an alternate approach that got broad buy-in where Hyper-Schema as defined did not.

That has a bit more to do with what Hyper-Schema should look like in v6, but it does indicate (for me, at least) that the v5 cleanup can be pretty aggressive since we should expect significant changes for v6 anyway.

[Anthropic]

@awwright I use it for defining a link with rel of "options" to then pull in the options for a select.

Totally agree that it doesn't seem to be the best spec of the bunch and more than happy to re-work if there are changes. But just wanted to submit a use case for consideration.

[jdesrosiers]

I use Hyper-Schema for all of my JSON based REST APIs (although I haven't had the opportunity in while). In fact, I would have very little interest in JSON Schema if it weren't for JSON Hyper-Schema. As far as I have been able to find, it is the only standard capable of doing hypermedia APIs in JSON properly (1).

In REST, resources should be self descriptive giving you everything you need to know about what it is and what you can do with it. On the web this takes the form of links and forms. Many JSON standards out there can describe links, but only JSON Hyper-Schema can do something like a form. If you are familiar with the Richardson Maturity Model (2), all the other options are designed for Level 2 REST APIs. JSON Hyper-Schema is the only Level 3 option around.

I created a little library a while back that allows you to create fully functional CRUD APIs by writing only Hyper-Schemas (3). These APIs can be used without any out-of-band knowledge using Jsonary's (4) generic Hyper-Schema browser. This library allowed me to walk through the workflow while designing APIs. Nothing else is capable of that.

This was a much more long winded response than I planned on, but my ultimate purpose is to say that I am strongly against changing the Hyper-Schema approach. There are certainly things that can be cleaned up, refactored, or extended, but I'll be keeping an eye on any Hyper-Schema proposals to ensure that the spirit of Hyper-Schema is not lost.

I'm all for the kind of changes it sounds like @awwright is talking about, but I am concerned about what I think I'm hearing from @handrews. I look forward to reading your proposals.

(1) JSON-LD has Hydra and XML has XHTML, but JSON has only JSON Hyper-Schema
(2) http://martinfowler.com/articles/richardsonMaturityModel.html
(3) https://github.com/jdesrosiers/resourceful
(4) https://github.com/jsonary-js/jsonary

[slurmulon]

We use JSON Hyper-Schema extensively in both our platform API and client (really just JSON Schema in the back-end for validation purposes). I have worked on another enterprise API that was based on the Siren specification, but JSON Hyper-Schema is far superior due to its insane flexibility, its ability to keep things DRY and support for meta schemas ✨. I completely agree with @jdesrosiers that Hyper-Schema is the only JSON-based hypermedia API specification that is truly Restful. My favorite aspect is how it doesn't require you to modify your API responses, it's entirely non-invasive and complementary.

I'm not sure what issues you are having resolving your refs, but we have found Ajv to be very consistent and robust in resolving them. Many libraries, such as is-my-json-valid, expect the user to provide a self-managed object mapping $ref to schema.

The only "limitation" our team has encountered (really just an initial mis-understanding) has to do with entity instances that are required/used for resolving URI Templates into absolute URLs - the issue is that some APIs want to use deeply nested resources that require multiple entities, like so:

/api/v1/a/{uuid}/b/{uuid}

The questions we ran into were "Do we use the same entity instance? Another? If it's from another, how do we specify where the instance should come from?"

Our complications were mostly due to the fact that we designed our schemas to map nearly 1:1 to our API's domain model entities, such that each schema correlated to a single high-level entity (and I believe this is the general idea, but I'm sure there's other opinions on that). We still referenced other schemas with $refs and had normalized schemas, but the general idea was that the links defined in the schema only needed to know about the entity correlating to, well, that schema.

Because of this, it wasn't initially clear that support for working with multiple entity instances when resolving LDO (Link Description Object) URI templates is possible with JSON Pointer references - otherwise the URI template slugs correlate with the defacto entity instance provided (http://json-schema.org/latest/json-schema-hypermedia.html#anchor8). The question you still have to answer in your client, though, is what entity should be used when the reference is say #/foo but the user has multiple Foos to work with in the client - should it be the first, the last, the "selected" one by the user? I think it's fine to allow the application to handle this in whatever way they want, it's just not explicitly talked about in many JSON Hyper-Schema resources/tutorials.

Once our client-side API resolves the URI Template into a URL based on the provided entity instance, we then just construct an HTTP/REST resource caller that automatically follows whatever method is defined in the LDO

Edit: couple typos

[awwright]

@jdesrosiers Great feedback, thanks! As how REST has the code-on-demand constraint (which I take as including scripts and stylesheets), I sort of view JSON Schema as similar to a stylesheet, JSON Schema tells you how to render or how to work with the data you've been given.

One of my outstanding concerns with JSON Hyper-schema is it has a "method" property combined with "href" and "rel". A link is supposed to describe relationships between resources, not necessarily listing things to do. Retrieving information about an associated resource is GET, HEAD, or OPTIONS, changing a resource is PUT or PATCH, executing some remote function is POST.

Can you provide some more examples of how you'd like to approach linking?

I'm re-reviewing the Richardson Maturity Model, which is so great for describing what RESTful really means. I'm also reviewing RESTful Web APIs by Leonard Richardson & Mike Amundsen (the legends of hypermedia protocols).

[handrews]

@jdesrosiers while I would happily reboot hyper-schema, a more incremental approach based on adding rather than changing is entirely possible.

Honestly, I've been saying provocative things about hyper-schema mostly to motivate people to actually speak up. "Resourceful" looks really interesting, I'll have to look into it more deeply. Part of my interests here have been around seeing if anyone has developed best practices for hyper-schema that mitigate the problems we found. If so, I am absolutely not dead-set on changing it.

I agree that JSON Hyper-Schema is the only thing that comes close to RMM Level 3. It is actually what helped me sell RMM Level 3 at Riverbed, but ultimately the team found hyper-schema too confusing to accept.

At a high-level there were three problems, ranging from annoyances to deal-breakers. I was going to wait and file proposed solutions (um... after I figured them out), but since we're seeing some life here, let's just look at the problems. Perhaps someone else figured out a way around these, or can explain what we missed along the way.

This is, of course, my personal opinion and recollection several years down the road, and not the official opinion of my employer at the time (or current employer).

Implicitly defined resources

Hyper-Schema implicitly defines resources by defining links to them. Every schema that has a self link is a resource (relatively explicit), and every link URI that is not otherwise accounted for in a self link is also a resource (it can be confusing to line these all up).

This means there is no straightforward listing of resources anywhere obvious. Unless every resource is accessible from the root, which is not the case in many non-trivial APIs. It also means that with link URI templates defined all over the place, it is easy to produce conflicts, which also plays into the next problem.

This is the problem that only some people considered a deal-breaker. I was not one of them, but there were very strong opinions about this. Our solution was to separate the declaration of resources and links from the schemas. Schemas could be defined inline within the resources, but usually were $ref'd in.

I would like to sort out best practices for making the resources more clear, but am not at all attached to the separate enumeration approach. I can see "fixing" this just by implementing the documentation generator in a way that pulls the resources out and enumerates them clearly in the docs.

Repetition in link definitions

There's a lot of repetition in link definitions, or else functionality left out that should be doable in one step. Simply put, if I can do all the usual CRUD stuff on books and people, and a book has a list of authors, all of the links I define on authors are redundant. They're already defined for people. But I have to re-define them with a slightly different URI template of "/people/{authorId}" instead of "/people/{id}", and re-define the schema non-authoritatively in targetSchema (which can at least be done with $ref). I either need to re-define every person link, or force the caller to first GET the author in order to discover that it can be DELETEd. However, many operations, including DELETE, should not require a GET first, so that's back to duplication.

What makes more sense when both resources are described in the schema is is to be able to define a relationship between books and people with a rel of "author" and rules for how to translate the book instance data into a form that can fill out the self link for "person." e.g. map "author_id" in book to "id" in person (assuming person's self link is something like "/people/{id}").

This approach doesn't technically make anything impossible, but it was considered excessively burdensome. The only way we would have been able to use JSON Hyper-Schema would have been to write a pre-processor that duplicated all of the links out to the proper places.

Our solution to this was just to add a "relation" field that referenced the other resource (it helped to have resources enumerated separately here, but is by no means required), and provided a map from the source schema's fields to the destination schema's self link URI template. This was done by the vars approach seen in issue #52 (extended href templating). Using the same vars mechanism in multiple places made things a lot easier- it is always the way to translate URI parameters when they don't line up perfectly with the URI template syntax.

[edit- removed the bit about href duplication from self as Hyper-Schema actually does that part fine and I just forgot]

Another benefit of this solution was that it removed the need for most targetSchema definitions. Instead, you just looked at the target resource's self link. The relevant security considerations are already described for self links. targetSchema is still available for when you want to link to something outside of JSON Hyper-Schema.

I like this solution very much, but would be happy with anything that satisfied the same requirements of eliminating the duplication and more clearly showing resource-to-resource relations.

Can't describe both URI parameters and a message body

This makes JSON Hyper-Schema unusable for me. There is only one schema in the LDO, which applies to the URI parameters for GET and the body for POST (and presumably for PUT and PATCH although they are not mentioned?). Unless I am totally misreading things (which I could be), this makes it impossible to perform operations with a message body on URIs with query parameters. This means no such operations on filtered collections (a powerful alternative to messy batch processors), or partial representations (selecting a lighter-weight format), or anything else involving non-hierarchical identifiers.

Our solution is the one described in issue #52 (extended href templating). We used URI templates for all query string parameters. URI template variables either mapped to points in the instance schema (by the default action or through vars), indicating that they used the same validation rules, or they were given a schema directly if they were parameters that must come from an external source.

The extended href templating proposal was already in existence, we just adapted it and standardized that the request schema never applies to the URI, it only applies to the request body. Again, without a solution for this, JSON Hyper-Schema cannot adequately describe my APIs.

[handrews]

@awwright : your comment here

One of my outstanding concerns with JSON Hyper-schema is it has a "method" property combined with "href" and "rel". A link is supposed to describe relationships between resources, not necessarily listing things to do. Retrieving information about an associated resource is GET, HEAD, or OPTIONS, changing a resource is PUT or PATCH, executing some remote function is POST.

captures my key concern in 500% less words :-)

Basically, resources are not a first-class concept in JSON Hyper-Schema. They kinda just happen, and a lot of weirdness exists around that.

[awwright]

@slurmulon I'm looking at "rel"... the current draft wants to add four values to the IANA registry and I'm not sure all those are necessary.

As for nested resources, unfortunately I'm not really sure how to handle the more complex URI cases.

Consider, though, if you're listing an absolute URI in a URI Template, then you're sandboxing yourself inside that namespace. It means if I want to store instances of resources on my server, I have to modify the schema so it points to my server instead of yours.

So increasingly, I think the best design pattern is that JSON documents should provide pre-computed absolute URIs or URI references, and minimize computation. The only time I would use an absolute URI in a URI Template would be if it's globally assignable anyways, like "urn:uuid:{uuid}", "ni:///sha-256;{hash_sha_256}" and the like.

The other alternative is we provide some way to let a script compute a value on a JSON instance, including provide a list of link relations. This is certainly within the scope of REST, but it seems excessive.

[handrews]

@awwright : We set a base URI for all of our schemas and then if the template began with $, the dollar sign was replaced with the base URI. This is similar to issue #46 (LDO baseUri) but at the scope of an entire API (a concept which JSON schema also lacks, although making a schema that is all definitions plus a schema for the entry point resource can more or less serve that purpose).

Anyway, this removed all need for absolute URIs for us except when going outside of the API entirely, in which case it was fine. We needed to be able to define URIs relative to the API base without having to repeat the API base (e.g. "/some/api/base/path" in "https://api.example.com/some/api/base/path") all over everything.

[slurmulon]

@awwright Regarding nested resources and how they are currently handled, I stumbled across something like this (yes, with encoding):

/api/v1/user/{ root.json%23%2Fdefinitions%2Fuser }/quotes/{ root.json%23%2Fdefinitions%2Fquote }

I'm unable to find the resource for this, I ran into it months ago and it's very possible that it isn't officially supported (not seeing anything mentioned here: https://github.com/json-schema/json-schema/wiki/href or in the spec). Anyways, it's the closest thing I could find that would allow us to support multiple entity instances (our API team was annoyed with this fact, but I don't think it's a big deal at all). We had to write our own LDO href resolver and we just didn't add support for referring to other entities, particularly because of the complexities around "which one is the right entity?" The schemas define what an entity can be, not which one it is - and in my opinion this is a very good thing, it's just something for the client to consider in its state management/design.

I agree that you should minimize computation with your links, it will generally make everybody's life easier. I think though that it will still be necessary, even for common cases such as /api/v1/user/{uuid} - this prevents the need to modify the API response in any way (:heart:) - other hypermedia specs like Siren do not have this luxury. This strict separation also ensures that the API never needs to concern itself with any sort of state (specifically to answer "which entities should be used to make this link?"), satisfying HTTP and REST.

Even with globally unique identifiers, you still need to ask the question of "which instance do I use for the other entity/entities?" in the case of LDOs with more than one entity reference. With one entity it's trivial, because you can always just associate the schema/schema validator with whatever entity you want. But when there is a nested entity, it must either also be explicitly provided by the dev (increasing domain coupling, nipping away at a clean design) or implicitly provided by some state management system, and that (to me) is the hairy part. But again, I think this is primarily a concern of layers above JSON Hyper-Schema.

Edit: clarity, grammar

[slurmulon]

@handrews

Basically, resources are not a first-class concept in JSON Hyper-Schema. They kinda just happen, and a lot of weirdness exists around that.

I've noticed this as well, what do you think can be done to help alleviate/fix this? What if you say that an LDO method correlates with the resource's OPTIONS when the value is an Array (please excuse the rel here, clearly not sufficient):

{
  "rel": "self",
  "method": "GET",
  "href": "/v1/api/user"
}
{
  "rel": "updates",
  "method": ["POST", "DELETE"],
  "href": "/v1/api/user/{uuid}"
}

Otherwise, if it's a single value, method should be interpreted as the method that should be used (in other words there is only one "usable" method on the resource, so there's no point to provide a collection).

It seems this could help to eliminate redundancy in the LDOs as well since there's no need to duplicate links for additional actions that can be made on a resource. Perhaps the example I provided could be collapsed even further with URI Templates, such that GET, POST and DELETE can exist in the same LDO.

[awwright]

@slurmulon We could have a field similar to HTTP "Allow" that specifies which methods the resource is capable of handling. So if "PUT" is missing, then it's a read-only resource; if POST exists, then it's an executable resource; etc.

@handrews See how many of those concerns have issues filed against them? I should create a milestone for this maybe.

[slurmulon]

@awwright yeah love it, that way method doesn't have two different meanings depending on its value

[awwright]

Unlike jsonschema-core and jsonschema-validation, I'm at something of a loss for how to save jsonschema-hyperschema.

My best idea right now is I'll rewrite major sections of the spec, and won't be afraid to break reverse compatibility with existing implementations. These are just Internet-Drafts, after all, and I think this is warranted because HTML, Atom, HTTP Link, HAL, etc, have much bigger adoption and are far more consistent, and have probably seen better adoption because of their consistency.

These are the following features I'm looking to gently change or preserve, everything else I'll tear out unless someone says they'd like it:

• links (how to extract hyperlinks - list of Link Description Objects)
  • anchor (e.g. "http://example.com/subject")
  • rel (e.g. "self")
  • rev (reverse link, e.g. "author")
  • href (e.g. "http://example.com/object")
  • hreflang (e.g. "en")
  • media (e.g. "screen" "print")
  • title (e.g. "Measurement Levels and Configuration: A Linear Approach")
  • type/mediaType (e.g. "application/json")
  • accept/methods (e.g. "GET, HEAD, OPTIONS, POST, PUT, PATCH")
  • schema
  • enctype
  • schema
  • targetSchema
• media (how to interpret a string as an information resource)
• readOnly (the instance is managed exclusively by the authority, don't bother changing it)

During my survey, I found or came up with these features that I think would be useful:

• deprecation (provides information that an instance shouldn't be used and might be removed in the future)
• base (changes the URI base of an instance to this value, after URI template processing)
• forProperties, forItems, forAdditionalProperties, forAdditionalItems (descend into an array or object, but links still describe the current JSON instance)

I'll create issues for these enhancements.

Please let me know all your opinions!