Real-time sensor telemetry pipeline with anomaly detection and low-latency processing.
PulseStream is a high-performance, concurrent telemetry processing system that simulates real-time sensor data ingestion, processing, and anomaly detection. It is designed to model how modern distributed systems handle continuous data streams with strict latency and reliability requirements.
The system implements a producer-consumer architecture with thread-based concurrency, enabling efficient handling of streaming data under load.
- Real-time ingestion of multi-sensor telemetry data
- Concurrent processing using producer-consumer architecture
- Queue-based decoupling of ingestion and processing layers
- Rule-based anomaly detection on streaming data
- Structured logging with timestamps for traceability
- Runtime metrics for system observability
Sensors → Producer Threads → Queue → Worker Threads → Processing → Metrics + Logs
- Sensors: Simulate telemetry data streams (temperature, speed, battery, vibration)
- Producers: Push data into a shared queue in real time
- Queue: Thread-safe buffer for decoupled communication
- Workers: Process data concurrently
- Processor: Applies anomaly detection logic
- Metrics: Tracks system performance and anomaly counts
- Concurrent programming using threads
- Producer-consumer design pattern
- Real-time data processing pipelines
- Low-latency system design principles
- Observability through logging and metrics
- Fault-tolerant queue consumption (timeout-based handling)
- Sensor streams continuously generate telemetry data.
- Producer threads push data into a shared queue.
- Worker threads consume messages from the queue.
- Each message is processed and checked for anomalies.
- Results are logged, and metrics are updated in real time.
python3 main.py
[PRODUCER] sensor_0 sent data
[12:45:10] OK: sensor_0 data processed
[12:45:11] ALERT: sensor_1 -> high_temperature, low_battery
=== Metrics Summary ===
Total messages processed: 15
Total anomalies detected: 4
PulseStream completed.
- Handles concurrent ingestion and processing using multiple threads
- Minimizes blocking through queue-based decoupling
- Designed for low-latency processing of streaming data
- Scales with additional worker threads
- Python
- threading
- queue
- time
- random
- Latency and throughput benchmarking
- Distributed queue integration (Kafka / Redis Streams)
- Retry mechanisms and failure recovery
- Real-time dashboard for metrics visualization
- Adaptive anomaly detection using ML models
PulseStream reflects core system design principles used in:
- Real-time communication platforms
- IoT and embedded systems
- Vehicle telemetry and monitoring systems
- High-throughput data processing pipelines
Ancy Patel