Renewable Energy Data Processing & Analysis Project

This project provides a complete workflow for processing renewable energy data, from raw weather and generation data to comprehensive analysis and simulation dashboards.

📋 Project Overview

This project processes renewable energy data through multiple stages:

1. Data Preparation: Weather data processing and solar generation modeling
2. Data Integration: Combining generation, price, and weather data
3. Analysis & Simulation: Running simulations and creating interactive dashboards

🏗️ Project Structure

V1_infrasure_GPR/
├── README.md                           # This file
├── site_registry.csv                   # Site configuration and portfolio mapping
├── weather_data/                       # Weather data processing
│   ├── weather_processing_script.py    # Weather data processing script
│   ├── run_weather_processing.bat      # Weather processing batch file
│   └── requirements.txt                # Weather processing dependencies
├── generation_data/                    # Solar generation modeling
│   ├── solar_generation_modeling.py    # Generation modeling script
│   ├── run_solar_modeling.bat          # Generation modeling batch file
│   └── requirements.txt                # Generation modeling dependencies
├── resurety_gen_pr_data/               # Data combination
│   ├── combine_data_script.py          # Data combination script
│   ├── run_data_combiner.bat           # Data combination batch file
│   └── requirements.txt                # Data combination dependencies
├── simulation/                         # Simulation and analysis
│   ├── simulation_script.py            # Simulation script
│   ├── run_simulation.bat              # Simulation batch file
│   └── requirements.txt                # Simulation dependencies
├── dashboard/                          # Interactive dashboard
│   ├── dash_app.py                     # Dashboard application
│   ├── run_dashboard.bat               # Dashboard batch file
│   └── requirements.txt                # Dashboard dependencies
└── output/                             # Generated output files
    ├── infrasure_input_data/           # Combined data (CSV format)
    ├── infrasure_input_data_parquet/   # Combined data (Parquet format)
    └── simulation_results/             # Simulation outputs

🚀 Quick Start Guide

Prerequisites

1. Python Installation: Ensure Python 3.8+ is installed
2. Required Packages: Install all dependencies

   pip install pandas numpy pvlib pyarrow plotly dash

Step-by-Step Workflow

📊 STEP 1: Data Preparation

1.1 Weather Data Processing

Purpose: Process raw weather data into hourly format for generation modeling

Location: weather_data/ folder

How to Run:

# Method 1: Batch file (Windows)
cd weather_data
run_weather_processing.bat

# Method 2: Direct Python execution
cd weather_data
python weather_processing_script.py

What it does:

• Processes raw ERA5 weather data
• Converts to hourly format
• Creates site-specific weather files
• Output: [SITE_NAME]_weather_hourly.csv files

1.2 Solar Generation Modeling

Purpose: Model solar generation using weather data and site parameters

Location: generation_data/ folder

How to Run:

# Method 1: Batch file (Windows)
cd generation_data
run_solar_modeling.bat

# Method 2: Direct Python execution
cd generation_data
python solar_generation_modeling.py

What it does:

• Loads weather data and site parameters
• Models solar generation using hybrid approach
• Applies degradation, soiling, and efficiency factors
• Output: [SITE_NAME]_modeled_generation.csv files

🔗 STEP 2: Data Integration

2.1 Combine Data Sources

Purpose: Integrate generation, price, and weather data into unified datasets

Location: resurety_gen_pr_data/ folder

How to Run:

# Method 1: Batch file (Windows)
cd resurety_gen_pr_data
run_data_combiner.bat

# Method 2: Direct Python execution
cd resurety_gen_pr_data
python combine_data_script.py

What it does:

• Matches files across price, generation, and weather folders
• Combines data with flexible date handling options
• Calculates revenue metrics (RT and DA)
• Organizes output by portfolio
• Output: [PORTFOLIO]/[ASSET_NAME]_generation_price_combined.csv or .parquet

📈 STEP 3: Analysis & Simulation

3.1 Run Simulations

Purpose: Perform statistical analysis and simulations on combined data

Location: simulation/ folder

How to Run:

# Method 1: Batch file (Windows)
cd simulation
run_simulation.bat

# Method 2: Direct Python execution
cd simulation
python simulation_script.py

What it does:

• Loads combined data from previous step
• Performs statistical analysis
• Runs Monte Carlo simulations
• Generates risk metrics and forecasts
• Output: Simulation results and reports

3.2 Launch Dashboard

Purpose: Create interactive web dashboard for data visualization

Location: dashboard/ folder

How to Run:

# Method 1: Batch file (Windows)
cd dashboard
run_dashboard.bat

# Method 2: Direct Python execution
cd dashboard
python dash_app.py

What it does:

• Creates interactive web dashboard
• Visualizes generation, prices, and revenue data
• Provides filtering and analysis tools
• Runs on local web server (typically http://localhost:8050)

📁 Input Data Requirements

Required Files Structure:

V1_Input_data_files/
├── site_registry.csv                   # Site configuration
├── weather_data/
│   └── raw_data/                       # Raw ERA5 weather files
├── resurety_gen_pr_data/
│   └── [PRICE_DATA_FILES].csv          # Price data files
└── [OTHER_INPUT_FILES]                 # Additional input files

Site Registry Format:

site_name,portfolio_name,latitude,longitude,dc_capacity_mw,ac_capacity_mw,system_type,start_month,start_year
Albemarle_Beach_Solar,General_Renewable,29.30418,-98.40034,100.0,80.0,fixed,Jan,2020

🔧 Configuration Options

Weather Processing Options:

• Date range selection
• Weather variables selection
• Output format options

Generation Modeling Options:

• Site selection (individual or all sites)
• Degradation model parameters
• Efficiency factor adjustments

Data Combination Options:

• Date handling (fixed year, common date, all data)
• Output format (CSV or Parquet)
• Portfolio organization

Simulation Options:

• Analysis parameters
• Simulation scenarios
• Risk metrics calculation

📊 Output Files

Weather Data Output:

• weather_data/[SITE_NAME]_weather_hourly.csv

Generation Data Output:

• generation_data/[SITE_NAME]_modeled_generation.csv

Combined Data Output:

• infrasure_input_data/[PORTFOLIO]/[ASSET_NAME]_generation_price_combined.csv
• infrasure_input_data_parquet/[PORTFOLIO]/[ASSET_NAME]_generation_price_combined.parquet

Simulation Output:

• simulation_results/ folder with analysis reports

🛠️ Troubleshooting

Common Issues:

1. Missing Dependencies

   pip install pandas numpy pvlib pyarrow plotly dash

2. File Path Issues

   • Ensure all required input files are in correct locations
   • Check file naming conventions
   • Verify site registry format

3. Memory Issues

   • Process smaller datasets for testing
   • Use Parquet format for large files
   • Close other applications to free memory

4. Data Quality Issues

   • Check input file formats
   • Verify date ranges and time zones
   • Review error messages in console output

Performance Tips:

• Use Parquet format for large datasets
• Process sites individually for testing
• Monitor memory usage during processing
• Use batch files for automated execution

📞 Support

For issues or questions:

1. Check the individual README files in each folder
2. Review console output for error messages
3. Verify input file formats and requirements
4. Ensure all dependencies are installed

🔄 Workflow Summary

Raw Weather Data → Weather Processing → Hourly Weather Files
                                                      ↓
Site Parameters → Generation Modeling → Modeled Generation Files
                                                      ↓
Price Data → Data Combination → Combined Data Files
                                                      ↓
Combined Data → Simulation → Analysis Results
                                                      ↓
Analysis Results → Dashboard → Interactive Visualization

📝 Notes

• Each step can be run independently if previous outputs exist
• Batch files provide easy execution on Windows
• Scripts include comprehensive error handling and progress reporting
• Output files are organized by portfolio and asset name
• All scripts support both individual and batch processing modes