Skip to content

THREESCALE-11817: Use json_schemer for policies validation #4069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mayorova merged 6 commits into from
Jul 8, 2025
Merged

THREESCALE-11817: Use json_schemer for policies validation #4069

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
47fa57d
Use json_schemer for validating policies
mayorova Apr 29, 2025
efb651f
Remove json-schema fork and upgrade the gem
mayorova Apr 30, 2025
51ed054
Use a default policy schema if it's missing in the spec document
mayorova Apr 30, 2025
bfdf2cd
Add backwards compatibility and fix some tests
mayorova Apr 30, 2025
42da4e4
Fix more tests
mayorova Apr 30, 2025
f10c1e0
Refactor
mayorova May 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ gem 'browser'
gem 'diff-lcs', '~> 1.2'
gem 'hiredis-client'
gem 'httpclient', github: '3scale/httpclient', branch: 'ssl-env-cert'
gem 'json-schema', git: 'https://github.com/3scale/json-schema.git'
gem 'json-schema'
gem 'json_schemer'
gem 'local-fastimage_resize', '~> 3.4.0', require: 'fastimage/resize'
gem 'kt-paperclip', '~> 7.2'
Expand Down
24 changes: 10 additions & 14 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,6 @@ GIT
specs:
httpclient (2.8.3)

GIT
remote: https://github.com/3scale/json-schema.git
revision: 26487618a68443e94d623bb585cb464b07d36702
specs:
json-schema (3.0.0)
addressable (>= 2.4)

GIT
remote: https://github.com/3scale/rspec-html-matchers.git
revision: 58b19212afcf66540da729214287d85f586fa7cd
Expand Down Expand Up @@ -163,8 +156,8 @@ GEM
activerecord (>= 3.0)
acts_as_tree (2.9.1)
activerecord (>= 3.0.0)
addressable (2.8.0)
public_suffix (>= 2.0.2, < 5.0)
addressable (2.8.7)
public_suffix (>= 2.0.2, < 7.0)
afm (0.2.2)
after_commit_queue (1.1.0)
rails (>= 3.0)
Expand Down Expand Up @@ -219,7 +212,7 @@ GEM
ransack (~> 4.1)
base64 (0.2.0)
bcrypt (3.1.13)
bigdecimal (3.1.8)
bigdecimal (3.1.9)
binding_of_caller (1.0.0)
debug_inspector (>= 0.0.1)
bootsnap (1.16.0)
Expand Down Expand Up @@ -438,7 +431,10 @@ GEM
railties (>= 4.2.0)
thor (>= 0.14, < 2.0)
json (2.7.1)
json_schemer (2.3.0)
json-schema (5.1.1)
addressable (~> 2.8)
bigdecimal (~> 3.1)
json_schemer (2.4.0)
bigdecimal
hana (~> 1.3)
regexp_parser (~> 2.0)
Expand Down Expand Up @@ -609,7 +605,7 @@ GEM
pry-stack_explorer (0.6.1)
binding_of_caller (~> 1.0)
pry (~> 0.13)
public_suffix (4.0.7)
public_suffix (6.0.1)
raabro (1.4.0)
racc (1.8.1)
rack (2.2.13)
Expand Down Expand Up @@ -708,7 +704,7 @@ GEM
reform-rails (0.2.6)
activemodel (>= 5.0)
reform (>= 2.3.1, < 3.0.0)
regexp_parser (2.6.1)
regexp_parser (2.10.0)
representable (3.0.4)
declarative (< 0.1.0)
declarative-option (< 0.2.0)
Expand Down Expand Up @@ -1034,7 +1030,7 @@ DEPENDENCIES
inherited_resources (~> 1.14.0)
jquery-rails (= 4.6)
json (~> 2.7, >= 2.7.1)
json-schema!
json-schema
json_schemer
jwt (~> 1.5.2)
kt-paperclip (~> 7.2)
Expand Down
3 changes: 1 addition & 2 deletions app/lib/three_scale/json_validator.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ class JSONValidator < ::JSON::Validator
# or add them manually later with ThreeScale::JSONValidator.add_schema(schema)
AUTOLOAD_SCHEMA_FILES = [
'app/lib/three_scale/swagger/schemas/*.schema.json',
'app/lib/three_scale/swagger/schemas/1.2/*.json',
'app/lib/three_scale/policies/schemas/*.schema.json'
'app/lib/three_scale/swagger/schemas/1.2/*.json'
].freeze
private_constant :AUTOLOAD_SCHEMA_FILES

Expand Down
114 changes: 114 additions & 0 deletions app/lib/three_scale/policies/schemas/apicast-policy-v1.1.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
{
"$id": "http://apicast.io/policy-v1.1/schema",
Copy link
Contributor Author

@mayorova mayorova Apr 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current version of the apicast policy schema is v1.1: https://github.com/3scale/APIcast/blob/5c33a1b896545a1421dc4162ea8651f389dcaa6d/gateway/src/apicast/policy/manifest-schema.json

Added in 3scale/APIcast@9e1a5c7

Copy link
Contributor

@jlledom jlledom May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is copied from there?

Copy link
Contributor Author

@mayorova mayorova Jun 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, copied from there and adjusted

Copy link
Contributor

@jlledom jlledom Jun 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem then is we have to keep our copy synced with the original. Would it be possible to download and use the original? Is this the same as the APICAST_REGISTRY_URL variable? For consistency it would be good to use that variable as a single source of truth. Does this make any sense?

Copy link
Contributor Author

@mayorova mayorova Jun 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APICAST_REGISTRY_URL doesn't expose that manifest, AFAIK.

"type": "object",
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"schema": {
"$id": "#/definitions/schema",
"$schema": "http://json-schema.org/draft-07/schema#",
"default": {}
},
"version": {
"$id": "#/definitions/version",
"type": "string",
"title": "The Policy Version",
"description": "A semantic version of a policy.",
"examples": [
"1.3.4",
"0.1"
],
"pattern": "^((\\d+\\.)?(\\d+\\.)?(\\*|\\d+))|builtin$"
}
},
"properties": {
"name": {
"$id": "/properties/name",
"type": "string",
"title": "The Policy Name",
"description": "Name of the policy.",
"examples": [
"Basic Authentication"
],
"minLength": 1
},
"summary": {
"$id": "/properties/summary",
"type": "string",
"title": "The Policy Summary",
"description": "Short description of what the policy does",
"examples": [
"Enables CORS (Cross Origin Resource Sharing) request handling."
],
"maxLength": 75
},
"description": {
"$id": "/properties/description",
"oneOf": [
{ "type": "string",
"minLength": 1 },
{ "type": "array", "items": { "type": "string" },
"minItems": 1
}
],
"title": "The Policy Description",
"description": "Longer description of what the policy does.",
"examples": [
"Extract authentication credentials from the HTTP Authorization header and pass them to 3scale backend.",
[ "Redirect request to different upstream: ", " - based on path", "- set different Host header"]
]
},
"order": {
"$id": "/properties/order",
"type": "object",
"title": "Order restrictions of the policy",
"description": "Specifies before or after which policies the policy should be placed in the chain.",
"properties": {
"before": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"version"
]
},
"description": "The policy should be placed before these ones in the chain."
},
"after": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"version"
]
},
"description": "The policy should be placed after these ones in the chain."
}
}
},
"version": {
"$ref": "#/definitions/version"
},
"configuration": {
"$ref": "#/definitions/schema"
}
},
"required": ["name", "version", "configuration", "summary"]
}
4 changes: 2 additions & 2 deletions ...ale/policies/schemas/manifest.schema.json → ...ies/schemas/apicast-policy-v1.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{
"$id": "http://apicast.io/policy-v1/schema#manifest",
"$id": "http://apicast.io/policy-v1/schema",
Copy link
Contributor Author

@mayorova mayorova Apr 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, having #manifest in the schema ID is problematic...

While it is not prohibited as such in JSON schema, in json_schemer having #manifest messes up with refs resolution.

So, there are relative references in the schema:

{
  "$id": "http://apicast.io/policy-v1.1/schema#manifest",
  ...
  "properties": {
    ...
    "version": {
      "$ref": "#/definitions/version"
    },
    "configuration": {
      "$ref": "#/definitions/schema"
    }
  }
  ...
}

When they are resolved, the "location" of the reference is defined as http://apicast.io/policy-v1.1/schema#/definitions/version (in case of the "version" property).

The library then goes "backwards" and tries to find corresponding schema to resolve the reference, and takes the base URL which is http://apicast.io/policy-v1.1/schema, and then it cannot find it, and fails with JSONSchemer::UnknownRef "http://apicast.io/policy-v1.1/schema".

I guess json-scheme gem has a different logic, that's why it was accepted.

In general, I think it's better to avoid #manifest if it is not necessary.

Copy link
Contributor

@akostadinov akostadinov May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://json-schema.org/understanding-json-schema/structuring

It looks like the fragment #manifests is supposed to point at a specific part of the target schema. Having the fragment means that we know which part of the target schema is important for our interoperability.

Not having it is probably not a big deal.

It is interesting to know whether the fragment is wrong and perhaps was previously just ignored. Or schemer handles that fragment improperly.

"type": "object",
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"schema": {
"$id": "#/definitions/schema",
"$ref": "http://json-schema.org/draft-07/schema#",
"$schema": "http://json-schema.org/draft-07/schema#",
Copy link
Contributor Author

@mayorova mayorova Apr 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This $ref here also was problematic with json_schemer, it causes the following error:

JSONSchemer::UnknownRef "http://json-schema.org/draft-07/schema".

I actually don't know whether it is supposed to work, but in fact, I assume that $schema is what was meant here, because this is the schema that the "configuration" property of the policy must follow.

Copy link
Contributor

@jlledom jlledom May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But doesn't this file have to match the same file in apicast? Or am I wrong?

Copy link
Contributor Author

@mayorova mayorova Jun 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, ideally, yes, they should match. I did open a PR to fix inconsistencies, but actually, I missed this specific entry. So, thanks for catching this!
https://github.com/3scale/APIcast/pull/1550/files

"default": {}
},
"version": {
Expand Down
168 changes: 0 additions & 168 deletions app/lib/three_scale/policies/schemas/json-draft-07.schema.json
View file
Open in desktop
Copy link
Contributor

@jlledom jlledom May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can remove this because it's now integrated in json_schemer?

This file was deleted.

Loading