Should we show containers without any connections in the container topology?

[tomwilkie]

Its make sense to ignore processes without connections, but it feels like ignoring containers without any connections is wrong.

[peterbourgon]

The frontend Applications and Containers topologies are built from the report Process topology, whose nodes are endpoints. The concept or entity of a container is emergent from the metadata on those nodes, it's not represented at a low level.

To report containers, processes, or any other emergent entities as first-order objects, i.e. to still show them even if they don't have connections, we would need to add them to the report as a new field. We should be careful to make their unique IDs (keys) exactly the same as the unique IDs produced by MapFuncs.

[tomwilkie]

For sure I understand why it is like it is, and how we might achieve it. Question is should we do it? Do you think its something users will expect?

Given containers have no ui (in general) and no persistent storage (in general), surfacing containers that are not doing any network IO seems useful to me.

[peterbourgon]

I think psuedo nodes should be the same "kind" as the topology we're injecting them into

I like this angle, and agree. Pseudonodes as they exist today probably do expose too much of the underlying data topology, which can be confusing. As a rule I think we should always find a way to keep pseudonodes in the UI, through some kind of intelligent merging and/or labeling, and potentially including visual differentiation.

So in general 👍

[peterbourgon]

I would propose to merge this with #114 and take it as my next issue. Can you think of anything else higher on the list that you don't already claim?

[tomwilkie]

I think #114 was more about grabbing extra stuff to put on the new container report topology. I think making a container report topology using the existing docker process mapper would be pretty straight forward.

How do you intend to merge the process/connection topology with the container topology, to get a container/connection topology?

I think you working on this next would be AWESOME.

[tomwilkie]

Oh feel free to work on #111 if you like!

[peterbourgon]

Neither the work here nor in #114 would result in a new Topolgy in the Report struct. Rather, the work here would require changing the Report struct to something like..

type Report struct {
    Process Topology
    Network Topology
    EntityMetadatas 
}

type EntityMetadatas map[EntityType]EntityMetadata

type EntityMetadata map[string]string

const (
    Origin    EntityType = "origin" // host
    Container            = "container"
    Process              = "process"
)

(Not exactly that, as we need to preserve merge semantics, but you get the idea.)

With this in place, we can get metadata for these emergent entities directly in the probe, and propagate it up to the UI. And with that, we can scrape per-process stats like CPU and mem usage. Make sense?

[tomwilkie]

Hmm I see. Is there a layer of indirection missing in EntityMetadatas? is right now its EntityMetadatas[type][key] = value, but should it be EntityMetadatas[type][node id][key] = value?

And given the topology contains a NodeMetadatas, how is this EntityMetadatas any different from a topology without any edges? Are you proposing we put process and host node NodeMetadatas in EntityMetadatas?

[peterbourgon]

Is there a layer of indirection missing in EntityMetadatas?

Yes, it would need to be EntityMetadatas["container"]["a0b1c2d..."]["description"]["foo bar"]. Good catch.

Are you proposing we put process and host node NodeMetadatas in EntityMetadatas?

Yes. Right now we don't have any per-process metadata in the Report; Report.Process.NodeMetadatas is actually per-endpoint metadata. And host metadata is already separate, in HostMetadatas.

But maybe it's moot, as...

And given the topology contains a NodeMetadatas, how is this EntityMetadatas any different from a topology without any edges?

...I guess it is like a topology without edges, sure. We could make it like that explicitly.

type Report struct {
    Process       Topology // should rename to Endpoint
    Network       Topology // should rename to Interface or Address
    Container     Topology // contains only nodes, no edges (at this stage)
    ActualProcess Topology // likewise
    Host          Topology // will initially contain only 1 host node, merge to more
}

This would have the added benefit of making merging easier. Nice. I think :)

[tomwilkie]

Yes. Right now we don't have any process metadata in the Report; Report.Process.NodeMetadatas is actually per-endpoint metadata.

does Report.Process.NodeMetadatas not also contain the data from process mappers?

And host metadata is already separate, in HostMetadatas.

Yeah this seems weird; why is it not part of the network topology?

But in general, I like this approach. As you say, if you apply the container map fun to process and then merge with containers, you get exactly what we want. Similarly, one could do the same for hosts.

[peterbourgon]

does Report.Process.NodeMetadatas not also contain the data from process mappers?

Yes, but indirectly. The process mapper output is attached to the Report.Process.NodeMetadata, but the key there is an endpoint ID. So if one process has 10 connections, the same set of process metadata is duplicated 10 times in Report.Process.

Why is it not part of the network topology?

Because it's fundamentally not the same thing — Report.Network is keyed on IPs which are many-to-1 with a probe; host metadata is about the host the probe is running on which is 1-to-1 with a probe.

But in general, I like this approach.

Great! I'll create a new issue with all of this a bit more concrete, merge #114 and #115 (this issue) into it, and take it from there.

[tomwilkie]

SGTM

[peterbourgon]

Merged to #120