docs: Categorize transforms into functions or runtimes

[binarylogic]

This change moves transforms into 2 categories: functions or runtimes. This helps to separate the 2 for future changes we're making:

1. Runtimes should be able to call functions at some point in the future.
2. Runtimes will be encouraged to replace the entire transform step, versus chaining many transforms together. For example, if a user reaches for the lua runtime we should encourage them to do everything within that runtime (add fields, remove fields, control flow, etc).

[binarylogic]

@Hoverbear I'm curious what you think about this change, especially in the context of the upcoming WASM changes. I'm thinking mostly about:

1. Where do wasm sources and sinks fit? Are those "runtimes" too?
2. What are your thoughts on the second point?

[lukesteensen]

swimlanes is interesting because it's not really a function or a runtime. It's really a macro that expands to multiple function-based pipelines.

[binarylogic]

That's a good point. I think for this exercise the term "component" might be better? I'm really trying to distinguisn between a set of simple components that you connect to form transformation pipelines, and full runtimes.

[lukesteensen]

Yeah that sounds reasonable to me.

[Hoverbear]

@binarylogic

Where do wasm sources and sinks fit? Are those "runtimes" too?

I don't think so. Ideally a wasm module would be able to register itself as a source/sink/function/runtime.

For example, an assemblyscript or lua runtime could be eventually powered by wasm...

I'd almost say a noble goal would be that all ryntimes are wasm modules, but not all wasm modules are runtimes.

Runtimes will be encouraged to replace the entire transform step, versus chaining many transforms together. For example, if a user reaches for the lua runtime we should encourage them to do everything within that runtime (add fields, remove fields, control flow, etc).

Disagree! We should encourage users to use this power to create logical blocks of their pipeline... instead of forcing them to create functional blocks.

This is especially powerful as we get the new swimlanes functionality, as users can route things easily between these blocks!

[binarylogic]

Disagree! We should encourage users to use this power to create logical blocks of their pipeline... instead of forcing them to create functional blocks.

Sure. I can get onboard with this.

This is especially powerful as we get the new swimlanes functionality, as users can route things easily between these blocks!

So this is where my point becomes a little more clear. I like the swimlanes transform for simple routing, but a runtime with if, else, switch, and other advanced control-flow tools, is much more powerful. This is why #2000 is allowing users to specify "lanes" for events.

Given this scenario, where a user reaches for a runtime to control flow, it seems easier to just do the entire transform within the runtime, versus composing a bunch of small components together. I'm not saying a user couldn't do both, but I want to guide a user towards one.

[Hoverbear]

Right, so consider a non-trivial config with, say sink0..5 and source0..5.

Consider this user to be needing to do any or many of different logical tasks as part of the route from given sources to sinks, eg "Parse json", "Do GDPR filtering", "Tag routing data", "Strip unneeded metadata" etc.

It makes sense it they do this logically instead of needing to replicate functionality across transforms.

[binarylogic]

It makes sense it they do this logically instead of needing to replicate functionality across transforms.

I'm not following. Why would a user need to replicate anything? For example, they could maintain a single lua script with functions to perform all of their processing.

[Hoverbear]

So you'd suggest they have all their sources feed to one transform, then have that transform feel to all the sinks? Seems like at that point Vector could operate as a sort of library/proxy at that point no? Then we could totally forget about transforms all together.

[binarylogic]

Yep. In some circumstances that might be the best UX. It would allow them to leverage an entire runtime for control flow, parsing, shaping, and so on. They can break it up into files and manage it just like any other code.

To clarify, I am not suggesting every user do this. This is something a user can graduate to once their pipelines are hard to manage with our basic functions. It could also just be user preference.

[Hoverbear]

Perhaps instead of runtimes the more appropriate term is Languages?

[binarylogic]

I'm going to close this for now. It's not clear if making this distinction makes sense yet. As we mature the runtimes I think we'll get a better understanding around how we want to encourage users to use them. As much as I like the "runtime for your entire pipeline" idea, it might not be practical.