Generate compose file reference from JSON-schema

[thaJeztah]

This is something I mentioned on a couple of occasions, but I realised I didn't open an issue for this. Opening this ticket to see if this makes sense 😅

The compose-file format has gained a lot of options over the years, and documenting all options has become more difficult because of that.
The current compose-file reference (https://docs.docker.com/compose/compose-file/) is maintained by hand,

• We maintain a single page per major version (1.x, 2.x, 3.x), and document inline where certain options require a specific minor version ("Version 3.3 only")
• (Partial) examples and reference descriptions are intertwined, and while it's great to have an example describing a specific option, it's not always possible to provide an example just for that option (e.g., to describe using a named volume for a service, multiple sections of the compose-file have to be included in the example).
• Some fields are ambiguous, leading to confusion: for example, volumes is used both to define volumes, and to specify which volumes are used by a service (I've frequently seen users copy the wrong example in the wrong section)

Proposal

Given that we maintain a JSON-schema for all versions of the 3.x compose file (https://github.com/docker/cli/tree/master/cli/compose/schema/data), we should investigate if we can generate the reference documentation from those files. Swagger (OpenAPI) is a subset of JSON-schema, so (at least to an extent) we can generate documentation from those files.

Before diving in too deep, we should do some testing: is the generated documentation still useful enough when options that are not supported by swagger/openAPI are left-out?

The JSON-schema website also mentions some tools to generate documentation (which may be worth exploring); http://json-schema.org/implementations.html#documentation-generation, and some others bootprint-json-schema, interagent/prmd

Challenges

There are some challenges that I can think of (likely more);

• Swagger / OpenAPI is designed for APIs; to generate documentation we may have to create some "dummy" API endpoint (e.g. POST/PUT /stack/)
• The script we use to generate the remote API documentation assumes JSON, so generated example requests are shown as JSON: can we make it show as YAML?
• Some options take multiple formats (long form, short form): not sure how easy it is to do this with generated swagger/openAPI docs
• Having per-property information about the minimum schema version (this property is new in version x.y) is useful information; perhaps we can use extension-fields for this? (-x-since: "3.3")
• We should keep examples (but perhaps combine some to make them more "real-world" examples); I think this should be a combination of;
  • Inline examples in the swagger/json-schema file (less risk of getting out of sync with the spec)
  • Separate page(s) in the docs that have examples, and link to those examples from the swagger/json-spec

Quick tests

Using https://github.com/bootprint/bootprint-json-schema (looks really bare-bones)

FROM node:alpine AS build
RUN npm install -g bootprint
RUN npm install -g bootprint-json-schema
RUN bootprint json-schema https://raw.githubusercontent.com/docker/cli/master/cli/compose/schema/data/config_schema_v3.6.json target

FROM nginx:alpine
COPY --from=build /target/ /usr/share/nginx/html/

Using https://github.com/cloudflare/json-schema-tools/tree/master/workspaces/doca (example failed to build);

FROM node:alpine AS build
RUN apk add --no-cache git yarn
RUN git clone https://github.com/cloudflare/json-schema-tools.git
RUN npm install -g lerna doca
RUN cd json-schema-tools && lerna bootstrap
RUN cd /json-schema-tools/workspaces/doca/example-schemas/draft-04 \
 && doca init \
 && cd documentation \
 && yarn install \
 && yarn build

Using https://github.com/interagent/prmd (not really suitable; generates a single markdown, targeted at API's)

FROM ruby:alpine
RUN gem install prmd
RUN mkdir -p schemata \
 && prmd init app  > schemata/app.json \
 && prmd init user > schemata/user.json
RUN echo '{"description": "Hello world prmd API","id": "hello-prmd","links": [{"href": "https://api.hello.com","rel": "self"}],"title": "Hello Prmd"}' > meta.json
RUN prmd combine --meta meta.json schemata/ > schema.json
RUN prmd doc schema.json > schema.md
EOF

[ahh-docker]

@thaJeztah Given this is over 6 months on this, can we schedule some time to scope this out? Thanks.

[ndeloof]

I guess this is obsolete now the compose specification has been created to manage compose file format, documentation and schema