Complex OpenAPI Validations with `@property`

[virkt25]

Description / Steps to reproduce / Feature proposal

LoopBack's @requestBody decorator is supposed to be capable of validation complex OpenAPI Schemas such as:

{
  type: string,
  minLength: 10,
  format: 'email'
}

This only works when this schema is passed to the spec via the @model decorator (top-down approach) ... vs. the more common route / CLI recommended path of using the @property decorator.

If you add validation properties such as minLength, format, etc. to a property using the @property decorator the validation properties don't even show up in /openapi.json and as such are never validated against the requestBody since those validations aren't a part of the schema.

Current Behavior

• Can't add validation properties using the @property decorator.

Expected Behavior

• Should be able to add model validation properties to a property via the @property decorator.

Acceptance Criteria

• @property() decorator should accept additional validation properties as per the OpenAPI Spec and add them to the schema of the Model (should show in /openapi.json) - see Refactor metaToJsonProperty to accept AJV keywords #2685
• Incoming @requestBody() should honour validation properties added via @property decorator.
• Confirm @model() also works in preserving the validation properties
• Update any relevant docs (if needed)
• Tests

See Reporting Issues for more tips on writing good issues

[bajtos]

+1 for allowing models to provide additional validation rules. I don't think this is as simple as allowing @property decorator to receive arbitrary OpenAPI spec constraints. @property is using LoopBack Definition Language (the schema format used by juggler & LB 3.x).

IMO, we need to find a (generic) solution that will support other transports than just REST/OpenAPI, for example gRPC and GraphQL and ideally allow connectors to setup some of these restrictions at the database level too. For example, maxLength can be leveraged to specify the storage size of a column (maxLen: 5 means the column can be defined as nvarchar(5)).

See also #699

[bajtos]

IMO, this is beyond 4.0 GA scope.

[virkt25]

Well ... shouldn't it be up to the extension implementing GRPC / GraphQL to determine what it wants to do with the metadata. I don't think it's feasible for us to have a common way of supporting all of these protocols.

This issue came from me working on loopback4-example-shopping where we can avoid a lot of custom validation on a per route basis by simply supporting such validations through the @property layer so the information gets passed to Ajv for validation. Some of this may / may not be needed for GRPC / GraphQL (or they might require additional metadata) and it's up to those extensions to decide how they want to deal with the metadata imo.

[bajtos]

Well ... shouldn't it be up to the extension implementing GRPC / GraphQL to determine what it wants to do with the metadata. I don't think it's feasible for us to have a common way of supporting all of these protocols.

This issue came from me working on loopback4-example-shopping where we can avoid a lot of custom validation on a per route basis by simply supporting such validations through the @Property layer so the information gets passed to Ajv for validation. Some of this may / may not be needed for GRPC / GraphQL (or they might require additional metadata) and it's up to those extensions to decide how they want to deal with the metadata imo.

First of all, I see the pain point you have described and agree LB4 should provide an easy solution for that.

My objection against adding REST-specific validations to @property decorator is that @property is not tied to REST transport, it's a generic model-level decorator that should work with any and all transports. IMO, by using JSON-Schema language in @property decorator, we would signal to LB4 users that all such JSON-Schema validation rules are supported by all parts of the framework (ORM, different transports, etc.), which is not true.

For example, gRPC uses Protocol Buffers to specify transport format, translating validations specified as JSON schema to Protocol Buffers may be difficult if possible at all (Protocol Buffer have very minimal validation rules).

I would like to propose to define an extension format allowing @property metadata to include transport-specific validations. This can be similar to what we already have for database-specific configuration (e.g. column mapping).

Then we can define a convention for leveraging the extension format to provide additional JSON-Schema validations in property definition; and finally modify the code translating LB models to JSON Schema to take these extensions into account.

An example showing a jsonSchema extension specifying email format:

{
  type: string,
  minLength: 10,
  // the LDL extension: jsonSchema
  jsonSchema: {
    // JSON-Schema constraints follow
    format: 'email'
  }
}

Optionally, we can also review OpenAPI constraints, pick those that seem to be "universal", promote them to top-level LDL (LoopBack definition language) and implement support for them in juggler/connectors/transports. For example, format: email is a common validation rule that we have already implemented in LoopBack 3.x core:

https://github.com/strongloop/loopback/blob/c9164aed1794defbf7eaeafa797567ee74658507/common/models/user.js#L1398-L1407

function emailValidator(err, done) {
  var value = this.email;
  if (value == null)
    return;
  if (typeof value !== 'string')
    return err('string');
  if (value === '') return;
  if (!isEmail.validate(value))
    return err('email');
}

https://github.com/strongloop/loopback/blob/c9164aed1794defbf7eaeafa797567ee74658507/common/models/user.js#L1254-L1256

    UserModel.validate('email', emailValidator, {
      message: g.f('Must provide a valid email'),
    });

As I am thinking about this more, my conclusion is that for a validation rule to be eligible for including in top-level property definition (in LDL), we must have the validation implemented at juggler level, because that's the place where all data operations must go through. If a validation is not implemented at transport level (e.g. because gRPC/ProtocolBuffers do not allow that), then it's ok, because the values will be still validated by repository (juggler) operations.

On the other hand, if a validation is implemented at transport level only, then we have a problem - code working with models/repositories directly via TS/JS API is effectively bypassing such validation rules! For example, test code or database seeding script can easily created invalid model instances.

In that light, I am actually not sure if JSON-Schema validations applied at REST transport level are actually the proper solution for avoiding a lot of custom validation on a per route basis.

Thoughts?

@raymondfeng Please join our discussion, what's your opinion on this matter?

[virkt25]

Hmm, I see the point you're making. I'm ok if we did the validation in Juggler as long as we do it somewhere. That said I think my biggest concern is some inconsistency within the framework right now where by @requestBody() does the validation against the OpenAPI spec for the Model ... and if someone passes in the spec to @model or overrides it via app.api() and it includes these validations, they will be enforced by the framework but @property won't allow such validations -- my main intent is that we should be consistent with our approach and ideally that empower such validations because we can for free using AJV.

IMO the decorators should support storing the openapi validations and allow other layers of the framework to leverage the metadata to see if they wish to support the validation or not. I get the concern for allowing transport specific validations / juggler validations ... and for that I would propose the following:

• Support decorators storing validations by transport @property({type: 'string', rest: {format: 'email'}}) ... documenting it only applies to REST validation.
• Eventually move validation support from REST layer to juggler and we can eliminate the transport specific validation OR do this now ... and say juggler supports OpenAPI style validations.

[David-Mulder]

@bajtos If my understanding of what you're proposing is correct that means that there would be different extensions to the LDL which would do the validation on the ORM-level regardless of the transport. Is that correct?

Because if that's what you're proposing that sounds like a good idea 👍 .

Right now I am going to take a leap of faith/hope and include the validation in the property decorators according to the

{
    jsonSchema: {}
}

specification and do the validation in the frontend and hope that that syntax will get to be supported later on in loopback 4 itself through some extension that one needs to enable.

Given proper instruction I would be very open to helping contribute to such a solution.