diff --git a/README.md b/README.md index 40b59e77..a79a3c8f 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,145 @@ -[README.md](https://github.com/user-attachments/files/21972618/README.md) -# AgStream -# Inference Optimization Project +# AgStream - Plant Classification Pipeline For NVIDIA Edge Devices -Welcome! 🚀 -We’re really happy to have you here. This project is your chance to explore how cutting-edge AI models can be made **faster, smarter, and lighter** — the same kind of work that powers self-driving cars, mobile AI apps, and real-time decision systems. +![Plant Classification](docs/images/agstream-hero-image.png) -Think of it like tuning a race car: the model already drives, but now we’ll make it smooth, efficient, and lightning-fast. Along the way you’ll get to experiment, break things, fix them, and share what you’ve learned. Collaboration is key — at the beginning, you’ll be working in pairs, supporting each other, and reviewing each other’s work, just like in real engineering teams. +## 🌾 Product Description +AgStream is a **real-time inference pipeline** for crop and weed classification, optimized for NVIDIA Jetson devices. The system demonstrates end-to-end model optimization from PyTorch → ONNX → TensorRT deployment with **DeepStream 6.4**. -Here’s a real-world connection: the model we’re starting from is accurate, but not yet fast enough for real-time use in practical applications. Engineers often face this exact challenge - they have a working solution, but it’s too slow to be useful in the field. By applying optimizations, exporting to ONNX, and leveraging TensorRT, you’ll experience first-hand how these research models become deployable systems that can actually run on devices like laptops or even Jetsons. +**Key Capabilities:** +- **Real-time Processing:** Classify RTSP video streams with metadata extraction +- **Edge Deployment:** Optimized for NVIDIA Jetson platforms +- **Model Optimization:** PyTorch → ONNX → TensorRT conversion +- **Agricultural Focus:** 83 crop and weed categories from CropAndWeed dataset -Most importantly: don’t stress. This is a safe place to try, fail, and learn. Mistakes are part of the journey, and every one of them makes you stronger. +--- -## Next Steps -- Check out [our collaboration guide](docs/general/collaboration_guide.md)! -- Check out [setup_guide.md](docs/general/setup_guide.md) if you need help running things. -- See the weekly goals in [milestones.md](docs/general/milestones_and_weekly_breakdown.md). -- Ask questions, share insights, and help your teammates — you’re not alone in this! +## 🔧 Hardware & Software Requirements -Good luck, and let’s make this model fly 🚀 +### Target Platform +- **Device:** NVIDIA Jetson Orin Nano (Developer Kit) +- **JetPack:** 6.2 [L4T 36.4.3] +- **DeepStream SDK:** 6.4 (Triton multi-arch) +- **CUDA:** 12.6 +- **TensorRT:** 10.3 +- **Memory:** 8GB RAM +- **OpenCV:** 4.8.0 (GPU support depends on build) +--- -## AgStream - Implement Basic PyTorch Inference Script +## 📊 Performance Metrics -This branch includes the implementation of a PyTorch inference script and related updates. +Evaluation was done using the PyTorch model, with inputs resized to 256×256 (CPU inference). -### Changes Made: -1. **Inference Script**: - - Supports MobileNet and ResNet models. - - Processes images and saves predictions to `outputs/predictions.json`. +### Classification Accuracy (CropAndWeed Dataset) +| Model | 83-Class | Binary (Crop/Weed) | 9-Class | 24-Class | Model Size | +|-------|----------|-------------------|---------|----------|------------| +| MobileNet | 67.2% | 85.2% | 84.6% | 43.6% | 28.3 MB | +| ResNet18 | 67.2% | 82.1% | 81.5% | 41.0% | 135 MB | -2. **Dataset Documentation**: - - Added `docs/data.md` with details about the CropAndWeed dataset. +### Inference Latency (CPU) +| Model | Average Latency | Std Dev | +|-------|----------------|---------| +| MobileNet | 55.1ms | 26.7ms | +| ResNet18 | 84.9ms | 51.4ms | -3. **Model Documentation**: - - Added `docs/models.md` with performance metrics for MobileNet and ResNet. +> ⚠ Accuracy improves with hierarchical classification (fewer classes) -4. **Test Images**: - - Added sample images in `assets/test_images/`. +--- -5. **Pretrained Models**: - - Added pretrained MobileNet and ResNet models in `data/models/`. +## 🚀 Quick Start -6. **Environment Setup**: - - Added `env/Dockerfile` and `env/requirements.txt`. +### 1. Environment Setup -### How to Run: -1. Install dependencies: - ```bash - pip install -r env/requirements.txt - ``` +```bash +bash scripts/run_dev_jetson.sh +``` -2. Run inference: - ```bash - python inference.py - ``` +### 2. Run Pipeline + +```bash +# Start RTSP server (terminal 1) +python src/rtsp/rtsp_server.py + +# Run classification pipeline (terminal 2) +python src/deepstream/pipelines/deepstream_pipeline_cpu.py +# or +python src/deepstream/pipelines/deepstream_pipeline_gpu.py + +# Optional: run metadata extraction +python src/deepstream/pipelines/access_metadata.py +``` + +--- + +## 🧠 Research and Development + +### 1. Model Conversion & Optimization + +```bash +# Export PyTorch to ONNX +python scripts/export_to_onnx.py resnet18 +python scripts/export_to_onnx.py mobilenet +# TensorRT engine generation is automatic +``` + +### 2. Performance Benchmarking + +```bash +python src/deepstream/speed_benchmark.py +``` + +--- + +## 🎯 Pipeline Architecture +**RTSP Stream → H.264 Decode → Video Convert → Stream Mux → AI Inference (TensorRT) → OSD Overlay → JPEG Encode → Frame Output** + +![pipeline schema](docs/images/pipeline-schema.png) + +**Processing Details:** +- Input: 256×256 RGB frames from RTSP +- Normalization: mean=[0.5,0.5,0.5], std=[0.25,0.25,0.25] +- Batch Size: 1 (real-time) +- Precision: FP16 (default; configurable) +--- + +## 📁 Project Structure +* `src/` – Pipeline logic, inference modules, conversion scripts, evaluation +* `models/` – Trained models (PyTorch, ONNX, TensorRT) +* `scripts/` – Execution and export scripts +* `env/` – Environment setup per target (Jetson / CPU) +* `configs/` – Configuration files for pipeline and models +* `assets/` – Test data (images, videos) +* `docs/` – Documentation + +--- + +## 🔬 Technical Details +- Dataset: CropAndWeed (WACV 2023), 83 categories +- Training: PyTorch +- Export: PyTorch → ONNX (opset 17) +- Optimization: ONNX → TensorRT +- Deployment: DeepStream Python API +- Container: nvcr.io/nvidia/deepstream-l4t:6.4-triton-multiarch, Python 3.10, OpenCV 4.11 CUDA + +**Development Focus:** +- Model optimization & performance analysis +- Edge deployment & real-time inference +- End-to-end video processing & metadata extraction +- Benchmarking: latency & throughput + +**Code Quality:** + +```bash +isort src/ && black src/ && flake8 src/ +``` + +**Model Evaluation:** + +```bash +python src/evaluation/run_evaluation.py +python src/evaluation/run_hierarchical_evaluation.py +``` + +--- + +⭐ If you found this project useful, consider giving it a star -3. View results: - - Predictions are saved in `outputs/predictions.json`. diff --git a/docs/images/agstream-hero-image.png b/docs/images/agstream-hero-image.png new file mode 100644 index 00000000..04d5a4be Binary files /dev/null and b/docs/images/agstream-hero-image.png differ diff --git a/docs/images/pipeline-schema.png b/docs/images/pipeline-schema.png new file mode 100644 index 00000000..7a9be99a Binary files /dev/null and b/docs/images/pipeline-schema.png differ