Api v4 doc

[ocket8888]

Includes changes based on the WG discussion on Jan 14^th 2020.

[mitchell852]

I'd love to see support for searching. I.e. the following query params or something along these lines:

&search[value]=foo
&search[columns]=a,b,c,d (to limit the search to certain columns if desired)
&search[regex]=true|false (or maybe just let the search value support globbing?)

and because it's important to know how many records there are before a search (or any filter for that matter) was applied maybe a response like this:

{
	response: []
	alerts: []
	meta: { total: 200, otherfieldTBD: ..., otherfieldTBD: ... }
}

[ocket8888]

"I'd love to see support for searching. I.e. the following query params or something along these lines:"
&search[value]=foo &search[columns]=a,b,c,d (to limit the search to certain columns if desired) &search[regex]=true|false (or maybe just let the search value support globbing?)

That seems extremely complicated. First, I'd vote no regex, that's unlikely to be helpful and clients should largely be in charge of such complex filtering IMO. Having a single value that applies to arbitrary "columns" not only ties the representations directly to table columns in the database - which won't necessarily be true - but also sounds like a nightmare to implement. I also hope this doesn't turn into something that people turn into something like what "Types" are now; e.g. "For this part of Traffic Router, select all the Cache Servers that have a Profile name that matches Foo* - because those are Foo Edge Cache Servers. Tags should ideally fill that role if necessary, IMO.

If you really think this is necessary - I'm not still quite convinced of clients' performance-driven need for that - then what I'd suggest is: object property filtering query parameters support the use of * for globbing in the case of string fields. So if you want all the Cache Servers that have a Profile name that matches Foo* you'd request GET /api/3.0/cache_servers?profile=Foo* HTTP/1.1. This obviously won't work on numeric fields like e.g. "ID" but somewhat less obviously also won't work on timestamps - or at least shouldn't, because that's what Age Filtering is for.

[ocket8888]

"... because it's important to know how many records there are before a search (or any filter for that matter) was applied maybe a response like this"

This 'meta' field for API responses does seem useful, even outside the context of server-side searches. I just have trouble deciding how it should be standardized. It's important because of statistics endpoints, they would ideally use it as well to provide a more consistent form, but I'll need to spend some time digging into all the statistics currently available through the API and what kinds of information they return, what properly belongs in response and what might use "meta". Also kinda want to try to workshop that name.

[jhg03a]

It seems like we're starting to rewrite our help documentation inside the api documentation.

[ocket8888]

"It seems like we're starting to rewrite our help documentation inside the api documentation."

What do you mean?

[jhg03a]

Note this is a global configuration underneath this boolean flag. There's an open issue to allow two DS to vary in the configuration.

[ocket8888]

So you're saying that anonymous blocking is configured on a Traffic Router today, and individual DS settings have no effect?

[jhg03a]

You can't express different settings per DS, it's all or nothing per ds and one config per cdn. apache#4227

[ocket8888]

Well you can't define subsets of anonymous blocking or IP whitelists on a per-Delivery Service basis, but as I understand that issue the simple "check for anonymization vs don't" flag does work as advertised. Correct?

Do you think Delivery Services also need a property to define IP whitelists for the purposes of anonymous blocking and/or geographic blocking? And maybe also a "policy" of what to consider "anonymized"? That second one sounds pretty hard to do without digging into TR implementation details.

[jhg03a]

That's not how I've understood it. Per the docs it's a TR profile setting which per CDN, and it's only a boolean to turn whatever that profile has enabled on or off.

[jhg03a]

Cachability might be a better word here.

[ocket8888]

I'm a little hesitant to use made-up words. Is there any other phrase you think might be more appropriate? Like "Content Caching" or "Caching Type"?

[jhg03a]

I believe we should kill this option all together and force people to actually think about their cachekey. Most HTTP security classes that mention CDNs talk specifically about this concept.

[ocket8888]

Well how would that be implemented today using ATS configuration files? I was under the impression that cache key configuration required a plugin that was totally separate from remap.config

[jhg03a]

We do it today using a combination of raw remap and DS profile/parameters. This is required today for every base origin url field change or you end up breaking existing caches.

[ocket8888]

Wait, so every time you change the Delivery Service's orgServerFqdn you have to change the cache key using raw remap.config lines? Why? And what Parameters?

[jhg03a]

Correct. If you don't manually assert the cachekey config before changing the base origin url field, you effectively invalidate the entire cache of that ds.

[jhg03a]

These list items don't match how our existing range request options work.

[ocket8888]

It matches Range Request Handling on current Delivery Services.

[jhg03a]

Not quite, and there are more options now. We're just hand crafting them into other fields because the field doesn't support them.

[ocket8888]

What other fields?

[jhg03a]

Header rewrites and the raw remap

[ocket8888]

In what ways? I'd like to expand this property to encompass the different handling methods.

[jhg03a]

By static, I assume you mean ANY_MAP. This is not an option anyone without operator should know about because it's capable of harming traffic outside a tenant.

[ocket8888]

Well no, actually I mean Federations. My hope is that ANY_MAP can be done away with entirely, but that's still something to be discussed.

Since the whole code base is open source, though, I see little point in not documenting things just because a non-operator can do harm with them. In 2.2 I can count at least three ways to gain root access on a machine running TO - that wouldn't be safer if they weren't documented. That ought to just be not possible.

[jhg03a]

ANY_MAP is unfortunately still required in dire circumstances. But a general user shouldn't know about it.

[ocket8888]

Okay, but this is a full data model definition, not a help guide for general user.

Like I said, we'll still need to discuss ANY_MAP because I don't actually know what "dire circumstances" necessitate it.

[jhg03a]

Are we finally killing steering off and going to merge its use cases with Client Steering?

[ocket8888]

Yes.

[jhg03a]

What's the plan for steering regex then?

[ocket8888]

Depends: what's that?

Absolutely nothing in any section above 2 (and 2 only as of this past Tuesday) has been discussed. They should be considered very immature works in progress that can and will change without warning as I change my mind and learn more about specific use-cases etc.

[jhg03a]

If this is to replace HOST_REGEXs, what do you intend to do about the other kinds of regexes?

[ocket8888]

Nothing, yet. Still has to be discussed because I don't understand the full use-case set for custom match lists.

[jhg03a]

See above comment about promoting the cachekey concept.

[ocket8888]

This doesn't have anything to do with caching, it's the "Consistent Hashing Query Parameters". Maybe the property needs a different name, but the only other one I could come up with was "Query Parameters Significant for Routing" which is a pretty long name.

[jhg03a]

Or think about it as overlapping subsets of cachekey config that spans both ATS and TR. You'd never configure TR to do one thing while ATS to do another (assuming you weren't off in the weeds of ATS power).

[ocket8888]

Yeah, one would hope those line up.

[jhg03a]

I mostly comment about feeling like the API doc scope is creeping because we're including redefinitions/copypasta from our existing documentation. If we intend to keep the same fields and same concepts, we might want to reference our existing help documents. This would call out places that are specifically different, or allows for an explanation as to what happened.

[jhg03a]

Who is the anticipated audience for this document? Does this entail just patching the API, or adjusting both the API and the underlying postgres datamodel?

[ocket8888]

"I mostly comment about feeling like the API doc scope is creeping because we're including redefinitions/copypasta from our existing documentation. If we intend to keep the same fields and same concepts, we might want to reference our existing help documents. This would call out places that are specifically different, or allows for an explanation as to what happened."

Yes and no. Existing documentation would be superseded and replaced by this documentation, but in most cases the semantic meaning of objects have changed. I don't mean for this to be a full re-telling of ATC concepts (e.g. the "Deep Caching" property of a Delivery Service is described as defining "whether or not 'Deep Caching' is permitted to be used by the Delivery Service") but a full definition of its scope is necessary because many scopes have changed.

[ocket8888]

"Who is the anticipated audience for this document?"

The intended audience is primarily TO API authors, but also TO API consumers - including other ATC component developers - and particularly those who develop user-land clients for the TO API.

"Does this entail just patching the API, or adjusting both the API and the underlying postgres datamodel?"

This defines a data model to which both the API and database must adhere. They need not implement things the same way, but must ultimately express the same concepts herein defined as the data model.

[jhg03a]

One other tidbit to throw in, in addition to the markdown version of this document I'd also like to see a programatic schema in either JSON Schema or Swagger / Open API Spec. I believe that would help with API stability and testability significantly.

[ocket8888]

You know, I had thought about including Typescript interface definitions for the data model, but idk how many people know typescript - or even care since none of our main components use that. The JSON schema thing (which OAS uses too) looks kind of interesting, if verbose. The only thing that concerns me is that given the schema:

{
	"type": "object",
	"title": "SomeType",
	"properties": {
		"foo": {"type": "string"},
		"test": {"type": "string"}
	},
	"required": ["foo"]
}

... this is for some reason considered totally valid:

{
	"foo": "bar",
	"otherProperty": {}
}

... which is totally strange to me. It doesn't seem to have a way - that I can see - to be less permissive. Also seems kinda verbose to express things like "properties that must exist". But if that's the best way to strictly define the model in a machine-readable way, I can probably make do. I'm not going to even think about doing that until the data model is actually defined, though.

For the record, the equivalent in Typescript is simply:

interface SomeType {
	foo: string;
	test?: string;
}

edit: the Typescript interface kind of has the same problem as JSON Schema w/r to permissiveness. Given the above interface definition, you technically can do e.g. const st = {foo: "bar", otherProperty: {}} as SomeType;, but this sort of casts the extraneous fields aside so that later doing e.g. let a = st.otherProperty; is a compile error. And you still can't do const st: SomeType = {foo: "bar", otherProperty: {}};. But that's all pretty implementation-dependent. Which is maybe a problem.