OSDOCS-14601 [NETOBSERV-2053] Flowlogs Flp filtering

modules/network-observability-con_filter-network-flows-at-ingestion.adoc

@@ -0,0 +1,81 @@
+// Module included in the following assemblies:
+
+// * networking/network_observability/configuring-operators.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="network-observability-filter-network-flows-at-ingestion_{context}"]
+= Filter network flows at ingestion
+
+You can create filters to reduce the number of generated network flows. Filtering network flows can reduce the resource usage of the Network Observability components.
+
+You can configure the following two different kinds of filters:
+
+* eBPF agent filters
+* Flowlogs-pipeline filters
+
+[id="ebpf-agent-filters_{context}"]
+== eBPF agent filters
+
+eBPF agent filters maximize performance because they take effect at the earliest stage of the network flows collection process.
+
+To configure eBPF agent filters with the Network Observability Operator, see "Filtering eBPF flow data using multiple rules".
+
+[id="flowlogs-pipeline-filters_{context}"]
+== Flowlogs-pipeline filters
+
+Flowlogs-pipeline filters provide greater control over traffic selection because they take effect later in the network flows collection process. They are primarily used to improve data storage.
+
+Flowlogs-pipeline filters use a simple query language to filter network flow, as shown in the following example:
+
+[source,terminal]
+----
+(srcnamespace="netobserv" OR (srcnamespace="ingress" AND dstnamespace="netobserv")) AND srckind!="service"
+----
+
+The query language uses the following syntax:
+
+.Query language syntax
+[cols="1,3", options="header"]
+|===
+| Category
+| Operators
+
+| Logical boolean operators (not case-sensitive)
+| `and`, `or`
+
+| Comparison operators
+| `=` (equals), +
+`!=` (not equals), +
+`=~` (matches regexp), +
+`!~` (not matches regexp), +
+`<` / `\<=` (less than or equal to), +
+`>` / `>=` (greater than or equal to)
+
+| Unary operations
+| `with(field)` (field is present), +
+`without(field)` (field is absent)
+
+| Parenthesis-based priority
+|===
+
+You can configure flowlogs-pipeline filters in the `spec.processor.filters` section of the `FlowCollector` resource. For example:
+
+.Example YAML Flowlogs-pipeline filter
+[source,yaml]
+----
+apiVersion: flows.netobserv.io/v1beta2
+kind: FlowCollector
+metadata:
+  name: cluster
+spec:
+  namespace: netobserv
+  agent:
+  processor:
+    filters:
+      - query: |
+          (SrcK8S_Namespace="netobserv" OR (SrcK8S_Namespace="openshift-ingress" AND DstK8S_Namespace="netobserv"))
+        outputTarget: Loki  <1>
+        sampling: 10  <2>
+----
+<1> Sends matching flows to a specific output, such as Loki, Prometheus, or an external system. When omitted, sends to all configured outputs.
+<2> Optional. Applies a sampling ratio to limit the number of matching flows to be stored or exported. For example, if a sampling value is `10`, then 1 matching flow is kept.

observability/network_observability/configuring-operator.adoc

@@ -24,6 +24,12 @@ include::modules/network-observability-enriched-flows.adoc[leveloffset=+1]
 * xref:../../observability/network_observability/json-flows-format-reference.adoc#network-observability-flows-format_json_reference[Network flows format reference].
 
 include::modules/network-observability-configuring-FLP-sampling.adoc[leveloffset=+1]
+include::modules/network-observability-con_filter-network-flows-at-ingestion.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+* xref:../network_observability/observing-network-traffic.adoc#network-observability-filtering-ebpf-rule_nw-observe-network-traffic[Filtering eBPF flow data using multiple rules]
+
 include::modules/network-observability-configuring-quickfilters-flowcollector.adoc[leveloffset=+1]
 include::modules/network-observability-resource-recommendations.adoc[leveloffset=+1]
 include::modules/network-observability-resources-table.adoc[leveloffset=+2]