Spike: Open structured event pipeline for the backend

[khvn26]

Our goal is to start tracking product events in the backend to, potentially, increase quality of the data (richer context, tracking accuracy).

Our hypothesis is that we can piggyback on structured logging + OTel's event tracking by introducing an exporter that will push events to our analytics platform (Amplitude), and, later, to other interesting destinations.

Possible outcomes of this spike include:

• A working PoC or an epic detailing out implementation for a PoC.
• An iternal RFC that includes design proposals for event taxonomy and the governance model.

Why now?

• We're working on improving our product analytics stack and data capabilities. We want to learn and evolve together with our customers, and see how we can improve the product analytics space.
• We'd like to try out a process where engineering thinks upfront about what kind of events their business logic should emit. OTel instrumentation model provides excellent developer experience for that — for free.
• Frontend tracking has accuracy issues + events emitted from backend provide richer context (feature flag states, segments, server-side logic)
• An increasing number of customers want (and implement) warehouse-native analytics. We are interested to see how we can contribute to this while continuing to use open standards.
• Amplitude costs are growing - we need better control over what we track.

Considerations

• Platform-agnosticism. Our target is Amplitude but the OTel Collector supports many exporters, allowing customers to route events to any CDP or data warehouse they use.
• Our users need to be able to self-host this, and integrate in their data infrastructure with little friction.
• GDPR compliance from the get-go: pseudonymise/prevent PII, event routing, consent.

What we'd like to validate as part of PoC

Product:

• Contribute to adoption and retentions metrics for 2 of features that we can't track today

Tech:

• Prove we can emit structured events via Python/Django logging that reach OTel Collector
• Demonstrate filtering/routing at Collector level (not application code)
• Confirm events can reach both Amplitude AND a warehouse (e.g., local ClickHouse)

Privacy/Compliance:

• Demonstrate PII removal/hashing at Collector level
• Show how to handle consent status in event pipeline
• Prove data doesn't leave customer infrastructure in self-hosted mode

Developer Experience:

• Single config file to control what events go where
• Hot-reload config without restart
• Clear examples of event taxonomy (technical vs business vs PII)

Non-Goals

• Replace frontend tracking. Backend events are complementary, providing enriched context for existing frontend analytics.
• Track high-volume events like flag evaluations or API calls.
• Implement full data warehouse pipeline. The focus on routing events to existing infrastructure.
• Create new Analytics product offering. This is for internal product analytics to understand our users better, not a customer-facing feature — yet.

[asaphko]

Great spike — the OTel-as-backbone approach is spot on, and the self-hosting + GDPR considerations are well-placed. A few things worth tightening before we move to PoC:

Clarify the primary goal
The spike sits between two objectives: observability and product analytics. These share infrastructure but have different schemas, retention needs, and consumers. Worth being explicit about which is the primary driver because it will affect architecture decisions downstream.

Call out the DWH angle as a product differentiator
The warehouse-native point in "Why now" is underselling something bigger. If Flagsmith natively pipes into customer data stacks, that's not just a cost lever, it's a meaningful platform differentiator worth naming explicitly as a product outcome of this work.

Sharpen the PoC success criteria
The technical validation goals are solid, but let's add one that is outcome-oriented: "we're able to answer questions about adoption and retention of 2 specific features that we can't answer today." (go beyond the 'plumbing' into business value)

Add a taxonomy ownership section to the RFC
Engineering thinking upfront about events is exactly right, but without clarity on who owns the taxonomy and what the playbook is for naming, approving, and deprecating events you'll accumulate analytics debt very quick. The RFC should dedicate a section to this: who decides, how events get reviewed, and what the governance model looks like.

[khvn26]

Thanks @asaphko — addressed your feedback (see updated description), also added non-goals.

Addressing @matthewelwell's feedback from Slack, posting a rough architecture diagram for the PoC and a list of deliverables:

graph TB
    subgraph "structlog-OTel Integration"
        API[Flagsmith API]
        SL[structlog Logger]
        SCHEMA[Event Schema<br/>- event_name<br/>- category<br/>- properties<br/>- privacy_level]
        OTEL_SDK[OTel SDK<br/>Logs Bridge]
        
        API --> SL
        SL --> SCHEMA
        SCHEMA --> OTEL_SDK
    end
    
    subgraph "OTel Collector Pipeline"
        subgraph "Routing & PII Stripping"
            RECV[OTLP Receiver]
            PROC1[Transform Processor<br/>PII Removal/Hashing]
            PROC2[Routing Processor<br/>Filter by event.category]
            PROC3[Batch Processor]
        end
        
        subgraph "Exporters"
            EXP_AMP[Amplitude Exporter<br/>Option A: OTLPHTTP + Transform<br/>Option B: Custom Plugin]
            EXP_CH[ClickHouse Exporter<br/>Built-in]
            EXP_S3[S3 Exporter<br/>Built-in]
        end
        
        RECV --> PROC1
        PROC1 --> PROC2
        PROC2 --> PROC3
        PROC3 --> EXP_AMP
        PROC3 --> EXP_CH
        PROC3 --> EXP_S3
    end
    
    subgraph "Destinations"
        AMP[Amplitude]
        CH[ClickHouse<br/>Self-hosted]
        S3[S3 Backup]
    end
    
    OTEL_SDK -->|OTLP/HTTP| RECV
    EXP_AMP --> AMP
    EXP_CH --> CH
    EXP_S3 --> S3

Loading

Deliverable 1: structlog-OTel Integration

What: Bridge between our existing structlog usage and OpenTelemetry
Output: Python code that converts structured logs to OTel events with proper schema

# Example implementation
class OTelStructlogProcessor:
    def __call__(self, logger, method_name, event_dict):
        # Convert structlog event to OTel LogRecord
        log_record = LogRecord(
            body=event_dict.get("event"),
            attributes=event_dict,
            severity_text=method_name
        )
        otel_logger.emit(log_record)
        return event_dict

Deliverable 2: PII Stripping & Routing Configuration

What: OTel Collector configuration that removes PII and routes events
Output: YAML configuration using standard processors (no custom code needed)

processors:
  transform:
    log_statements:
      - context: log
        statements:
          # Hash PII fields
          - set(attributes["user_hash"], SHA256(attributes["user_id"], "${SALT}"))
          - delete_key(attributes, "user_id")
          - delete_key(attributes, "email")
  
  routing:
    from_attribute: attributes["event.category"]
    table:
      - value: "business"
        exporters: [amplitude, clickhouse]
      - value: "technical"
        exporters: [clickhouse]

Deliverable 3: Amplitude Exporter

What: Configuration to send events to Amplitude
Output: Either standard OTLPHTTP config or custom exporter if needed

# Option A: Standard OTLPHTTP (preferred)
exporters:
  otlphttp/amplitude:
    endpoint: https://api2.amplitude.com/2/httpapi
    headers:
      Authorization: "Bearer ${AMPLITUDE_API_KEY}"

Implementation Approach

Deliverable	Easy Path	Custom Path	Recommendation
1. structlog-OTel	Use OTel Logging instrumentation	Custom processor	Start with instrumentation
2. PII & Routing	Standard transform + routing processors	Custom processor plugin	Standard processors should suffice
3. Amplitude Export	OTLPHTTP + transform	Custom exporter plugin	Try OTLPHTTP first

Glossary

• OTel SDK: Library in our Python code that sends events
• OTel Collector: Standalone service that routes/processes events (this is the "router" in the middle)
• Exporters: Plugins in the Collector that send to specific destinations (Amplitude exporter, ClickHouse exporter, etc.)
• OTLP: The protocol used between SDK and Collector

We can likely achieve everything with configuration only (no custom Collector code).

[khvn26]

Opened #7012.