diff --git a/modules/network-observability-con_filter-network-flows-at-ingestion.adoc b/modules/network-observability-con_filter-network-flows-at-ingestion.adoc new file mode 100644 index 000000000000..97519b968b88 --- /dev/null +++ b/modules/network-observability-con_filter-network-flows-at-ingestion.adoc @@ -0,0 +1,81 @@ +// Module included in the following assemblies: + +// * networking/network_observability/configuring-operators.adoc + +:_mod-docs-content-type: CONCEPT +[id="network-observability-filter-network-flows-at-ingestion_{context}"] += Filter network flows at ingestion + +You can create filters to reduce the number of generated network flows. Filtering network flows can reduce the resource usage of the Network Observability components. + +You can configure the following two different kinds of filters: + +* eBPF agent filters +* Flowlogs-pipeline filters + +[id="ebpf-agent-filters_{context}"] +== eBPF agent filters + +eBPF agent filters maximize performance because they take effect at the earliest stage of the network flows collection process. + +To configure eBPF agent filters with the Network Observability Operator, see "Filtering eBPF flow data using multiple rules". + +[id="flowlogs-pipeline-filters_{context}"] +== Flowlogs-pipeline filters + +Flowlogs-pipeline filters provide greater control over traffic selection because they take effect later in the network flows collection process. They are primarily used to improve data storage. + +Flowlogs-pipeline filters use a simple query language to filter network flow, as shown in the following example: + +[source,terminal] +---- +(srcnamespace="netobserv" OR (srcnamespace="ingress" AND dstnamespace="netobserv")) AND srckind!="service" +---- + +The query language uses the following syntax: + +.Query language syntax +[cols="1,3", options="header"] +|=== +| Category +| Operators + +| Logical boolean operators (not case-sensitive) +| `and`, `or` + +| Comparison operators +| `=` (equals), + +`!=` (not equals), + +`=~` (matches regexp), + +`!~` (not matches regexp), + +`<` / `\<=` (less than or equal to), + +`>` / `>=` (greater than or equal to) + +| Unary operations +| `with(field)` (field is present), + +`without(field)` (field is absent) + +| Parenthesis-based priority +|=== + +You can configure flowlogs-pipeline filters in the `spec.processor.filters` section of the `FlowCollector` resource. For example: + +.Example YAML Flowlogs-pipeline filter +[source,yaml] +---- +apiVersion: flows.netobserv.io/v1beta2 +kind: FlowCollector +metadata: + name: cluster +spec: + namespace: netobserv + agent: + processor: + filters: + - query: | + (SrcK8S_Namespace="netobserv" OR (SrcK8S_Namespace="openshift-ingress" AND DstK8S_Namespace="netobserv")) + outputTarget: Loki <1> + sampling: 10 <2> +---- +<1> Sends matching flows to a specific output, such as Loki, Prometheus, or an external system. When omitted, sends to all configured outputs. +<2> Optional. Applies a sampling ratio to limit the number of matching flows to be stored or exported. For example, if a sampling value is `10`, then 1 matching flow is kept. \ No newline at end of file diff --git a/observability/network_observability/configuring-operator.adoc b/observability/network_observability/configuring-operator.adoc index 258bab64cab8..0554bfb868ba 100644 --- a/observability/network_observability/configuring-operator.adoc +++ b/observability/network_observability/configuring-operator.adoc @@ -24,6 +24,12 @@ include::modules/network-observability-enriched-flows.adoc[leveloffset=+1] * xref:../../observability/network_observability/json-flows-format-reference.adoc#network-observability-flows-format_json_reference[Network flows format reference]. include::modules/network-observability-configuring-FLP-sampling.adoc[leveloffset=+1] +include::modules/network-observability-con_filter-network-flows-at-ingestion.adoc[leveloffset=+1] + +[role="_additional-resources"] +.Additional resources +* xref:../network_observability/observing-network-traffic.adoc#network-observability-filtering-ebpf-rule_nw-observe-network-traffic[Filtering eBPF flow data using multiple rules] + include::modules/network-observability-configuring-quickfilters-flowcollector.adoc[leveloffset=+1] include::modules/network-observability-resource-recommendations.adoc[leveloffset=+1] include::modules/network-observability-resources-table.adoc[leveloffset=+2]