Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions Makefile

Large diffs are not rendered by default.

44 changes: 44 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,50 @@ We welcome contributions! Please see:
- [FastAPI Integration](examples/fastapi_app/README.md) - Complete REST API example
- [More Examples](examples/) - Additional usage patterns

## 🎯 Running the Examples

The project includes comprehensive examples demonstrating various features and use cases. Each example can be run using the provided Makefile, which automatically handles Cassandra setup if needed.

### Available Examples

Run any example with: `make example-<name>`

- **`make example-basic`** - Basic connection and query execution
- **`make example-streaming`** - Memory-efficient streaming of large result sets with True Async Paging
- **`make example-context-safety`** - Demonstrates proper context manager usage and resource isolation
- **`make example-export-large-table`** - Export large tables to CSV with progress tracking
- **`make example-export-parquet`** - Export data to Parquet format with complex data types
- **`make example-metrics`** - Comprehensive metrics collection and performance monitoring
- **`make example-metrics-simple`** - Basic metrics collection example
- **`make example-realtime`** - Real-time data processing with sliding window analytics
- **`make example-streaming-demo`** - Visual demonstration that streaming doesn't block the event loop

### Running with External Cassandra

If you have Cassandra running elsewhere:

```bash
# Single node
CASSANDRA_CONTACT_POINTS=10.0.0.1 make example-streaming

# Multiple nodes
CASSANDRA_CONTACT_POINTS=10.0.0.1,10.0.0.2,10.0.0.3 make example-streaming

# With custom port
CASSANDRA_CONTACT_POINTS=cassandra.example.com CASSANDRA_PORT=9043 make example-basic
```

### Example Descriptions

- **Basic Example**: Shows fundamental operations like connecting, executing queries, and using prepared statements
- **Streaming Examples**: Demonstrate True Async Paging for processing millions of rows without memory issues
- **Export Examples**: Show how to export Cassandra data to various formats (CSV, Parquet) with progress tracking
- **Metrics Examples**: Illustrate performance monitoring, query tracking, and connection health checking
- **Real-time Processing**: Demonstrates processing time-series IoT data with concurrent operations
- **Context Safety Demo**: Proves that errors in one operation don't affect others when using context managers

Each example includes detailed comments explaining the concepts and best practices. Start with `example-basic` if you're new to the library.

## ⚡ Performance

async-cassandra enables your async Python application to work with Cassandra without blocking the event loop. While it doesn't eliminate the underlying driver's thread pool, it prevents those blocking operations from freezing your entire application. This is crucial for web servers where a blocked event loop means no requests can be processed.
Expand Down
107 changes: 102 additions & 5 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,48 @@

This directory contains working examples demonstrating various features and use cases of async-cassandra.

## Quick Start

### Running Examples with Make

The easiest way to run examples is using the provided Make targets:

```bash
# Run a specific example (automatically starts Cassandra if needed)
make example-streaming
make example-export-csv
make example-export-parquet
make example-realtime
make example-metrics
make example-non-blocking
make example-context

# Run all examples in sequence
make examples-all

# Use external Cassandra cluster
CASSANDRA_CONTACT_POINTS=node1.example.com,node2.example.com make example-streaming
```

### Installing Example Dependencies

Some examples require additional dependencies:

```bash
# Install all example dependencies (including pyarrow for Parquet export)
make install-examples

# Or manually
pip install -r examples/requirements.txt
```

### Environment Variables

All examples support these environment variables:
- `CASSANDRA_CONTACT_POINTS`: Comma-separated list of contact points (default: localhost)
- `CASSANDRA_PORT`: Port number (default: 9042)
- `EXAMPLE_OUTPUT_DIR`: Directory for output files like CSV and Parquet exports (default: examples/exampleoutput)

## Available Examples

### 1. [FastAPI Integration](fastapi_app/)
Expand Down Expand Up @@ -50,10 +92,38 @@ Shows how to export large Cassandra tables to CSV:
**Run:**
```bash
python export_large_table.py
# Exports will be saved in ./exports/ directory
# Exports will be saved in examples/exampleoutput/ directory (default)

# Or with custom output directory:
EXAMPLE_OUTPUT_DIR=/tmp/my-exports python export_large_table.py
```

### 4. [Real-time Data Processing](realtime_processing.py)
### 4. [Export to Parquet Format](export_to_parquet.py)

Advanced example of exporting large Cassandra tables to Parquet format:
- Memory-efficient streaming with page-by-page processing
- Automatic schema inference from Cassandra data types
- Multiple compression options (snappy, gzip, lz4)
- Progress tracking during export
- Handles all Cassandra data types including collections
- Configurable row group sizes for optimization
- Export statistics and performance metrics

**Run:**
```bash
python export_to_parquet.py
# Exports will be saved in examples/exampleoutput/ directory (default)

# Or with custom output directory:
EXAMPLE_OUTPUT_DIR=/tmp/my-parquet-exports python export_to_parquet.py
```

**Note:** Requires PyArrow to be installed:
```bash
pip install pyarrow
```

### 5. [Real-time Data Processing](realtime_processing.py)

Example of processing time-series data in real-time:
- Sliding window analytics
Expand All @@ -67,7 +137,7 @@ Example of processing time-series data in real-time:
python realtime_processing.py
```

### 5. [Metrics Collection](metrics_simple.py)
### 6. [Metrics Collection](metrics_simple.py)

Simple example of metrics collection:
- Query performance tracking
Expand All @@ -80,7 +150,7 @@ Simple example of metrics collection:
python metrics_simple.py
```

### 6. [Advanced Metrics](metrics_example.py)
### 7. [Advanced Metrics](metrics_example.py)

Comprehensive metrics and observability example:
- Multiple metrics collectors setup
Expand All @@ -94,7 +164,21 @@ Comprehensive metrics and observability example:
python metrics_example.py
```

### 7. [Context Manager Safety](context_manager_safety_demo.py)
### 8. [Non-Blocking Streaming Demo](streaming_non_blocking_demo.py)

Visual demonstration that streaming doesn't block the event loop:
- Heartbeat monitoring to detect event loop blocking
- Concurrent queries during streaming
- Visual feedback showing event loop responsiveness
- Performance analysis of concurrent operations
- Proves the async wrapper truly keeps the event loop free

**Run:**
```bash
python streaming_non_blocking_demo.py
```

### 9. [Context Manager Safety](context_manager_safety_demo.py)

Demonstrates proper context manager usage:
- Context manager isolation
Expand All @@ -119,6 +203,19 @@ Production-ready monitoring configurations:
- Connection health status
- Error rates and trends

## Output Files

Examples that generate output files (CSV exports, Parquet exports, etc.) save them to a configurable directory:

- **Default location**: `examples/exampleoutput/`
- **Configure via environment variable**: `EXAMPLE_OUTPUT_DIR=/path/to/output`
- **Git ignored**: All files in the default output directory are ignored by Git (except README.md and .gitignore)
- **Cleanup**: Files are not automatically deleted; clean up manually when needed:
```bash
rm -f examples/exampleoutput/*.csv
rm -f examples/exampleoutput/*.parquet
```

## Prerequisites

All examples require:
Expand Down
Loading
Loading