🦑 OctoNet Toolbox 🦑

The Ultimate Multi-Modal Human Activity Understanding Toolkit

🎯 Revolutionary Multi-Modal Dataset for Human Activity Understanding

Comprehensive sensor fusion • State-of-the-art benchmarks • Ready-to-use visualization tools

OctoNet Dataset Overview - Multi-modal sensor data visualization

🚀 What's Inside This Toolbox

✨ Comprehensive OctoNet Toolkit ✨

This powerful toolbox provides everything you need to work with the OctoNet dataset:

🎨 Visualization Suite

Interactive dataset exploration tools
Multi-modal data visualization capabilities

⚡ Benchmark Framework

Reproducible benchmark implementations
Benchmark results recordings

🎯 Ready to dive into multi-modal human activity recognition? Let's get started!

🎨 Part 1: Dataset Visualization & Exploration 🎨

Interactive Multi-Modal Data Analysis Suite

💡 💻 Recommended Environment: Run the code in Python Jupyter Notebook demo.ipynb for the best interactive experience!

📁 Dataset Structure Overview

🗂️ Complete Directory Layout:

./dataset
├── mocap_csv_final          # Data: Final motion capture data in CSV format.
├── mocap_pose               # Data: Final motion capture data in npy format.
├── node_1                   # Data: Data related to multi-modal sensor node 1.
├── node_2                   # Data: Data related to multi-modal sensor node 2.
├── node_3                   # Data: Data related to multi-modal sensor node 3.
├── node_4                   # Data: Data related to multi-modal sensor node 4.
├── node_5                   # Data: Data related to multi-modal sensor node 5.
├── imu                      # Data: Inertial measurement unit data.
├── vayyar_pickle            # Data: vayyar mmWave radar data.
└── cut_manual.csv           # Manually curated data cuts.

📊 Dataset Metadata & Statistics

🔍 **📋 Click to View Complete OctoNet Dataset Metadata**

📝 Key Information:

📌 Important Notes:

👥 Gender Classification: Male (M) and Female (F) participants

🏃 Activity Types: PA&F indicates subjects performed both Programmed Aerobics and Freestyle activities

⭐ Special Marking: Asterisk (*) denotes subjects who performed only Programmed Aerobics (no Freestyle)

🏠 Scene Mapping: Scene 1: IDs 1-99, Scene 2: IDs 101-199, Scene 3: IDs 201-299

User (Gender)	Exp ID	Scene 1: Activity IDs	Scene 1: PA&F	Scene 2: Activity IDs	Scene 2: PA&F	Scene 3: Activity IDs	Scene 3: PA&F
1 (M)	1, 11, 101, 201	all 62	✓	1–23		1–23, 57–62	✓*
2 (M)	2, 12, 102, 112, 202	all 62	✓	9–29	✓	9–29
3 (M)	3, 13, 113, 213	all 62	✓		✓		✓
4 (F)	4, 14, 104, 114, 204	all 62	✓	30–56	✓	30–56
5 (M)	5, 15, 115, 215	all 62	✓		✓		✓
6 (F)	6, 16	all 62	✓
7 (M)	7, 17, 117, 217	all 62	✓		✓		✓
8 (M)	8, 18, 108, 118	all 62	✓	24–62	✓	24–62
9 (M)	9	all 62
10 (M)	10, 20, 120, 220	all 62	✓		✓		✓
11 (F)	21		✓
12 (M)	22		✓
13 (F)	23		✓
14 (M)	24		✓
15 (F)	25		✓
16 (F)	26		✓
17 (F)	27		✓
18 (F)	28		✓
19 (F)	29		✓
20 (F)	30, 230		✓				✓
21 (M)	31		✓
22 (M)	32		✓
23 (F)	33		✓
24 (M)	34		✓
25 (M)	35		✓
26 (M)	36		✓
27 (M)	37		✓
28 (F)	38		✓
29 (F)	39		✓
30 (M)	40		✓
31 (M)	41		✓
32 (F)	42		✓
33 (F)	43		✓
34 (F)	44		✓
35 (M)	45		✓
36 (M)	46		✓
37 (M)	47		✓
38 (F)	48		✓
39 (F)	49		✓
40 (M)	111, 211			1–8	✓	1–8	✓
41 (F)	121, 221				✓		✓

⚙️ Environment Setup & Installation ⚙️

Quick Start Guide for OctoNet Development Environment

🐍 Step 1: Create Conda Environment

🔧 Automated Environment Creation:

# Create the OctoNet environment from the provided specification
conda env create -f environment.yaml

# Install additional Python packages
pip install -r requirements.txt

# Activate the environment (uncomment when ready)
# conda activate octonet

💡 💻 Alternative: You can skip the conda environment creation if you're using an existing Python environment with compatible packages.

🚀 Step 2: Launch Jupyter Notebook

📓 Start Interactive Development:

# Launch Jupyter Notebook with the OctoNet environment
jupyter notebook demo.ipynb

🎯 Ready to explore the OctoNet dataset interactively!

📊 Sample Data Selection & Configuration 📊

Intelligent Dataset Loading with Flexible Configuration

🔧 Core Dataset Loading Function

📁 Smart Dataset Management:

In dataset_loader.py, we provide a powerful get_dataset function that enables flexible dataset loading with comprehensive configuration options:

def get_dataset(config, dataset_path="", mocap_downsample_num = None) -> OctonetDataset:
    """
    Args:
        config: config file
        dataset_path: path to the dataset
        mocap_downsample_num: number of downsample for mocap data, could be shadowed by config['mocap_downsample_num']
    Returns:
        OctonetDataset: a dataset object
    """
    ...

⚙️ Complete Configuration Template

🔧 Full Dataset Configuration Options:

config = {
    'exp_list': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 101, 102, 104, 108, 111, 112, 113, 114, 115, 117, 118, 120, 121, 201, 202, 204, 208, 211, 213, 215, 217, 220, 221, 230]
    'activity_list': ['sit', 'walk', 'bow', 'sleep', 'dance', 'jog', 'falldown', 'jump', 'jumpingjack', 'thunmbup'
        'squat', 'lunge', 'turn', 'pushup', 'legraise', 'airdrum', 'boxing', 'shakehead',
        'answerphone', 'eat', 'drink', 'wipeface', 'pickup', 'jumprope', 'moppingfloor',
        'brushhair', 'bicepcurl', 'playphone', 'brushteeth', 'type', 'thumbup',
        'makeoksign', 'makevictorysign', 'drawcircleclockwise', 'drawcirclecounterclockwise',
        'stopsign', 'pullhandin','pushhandaway', 'handwave', 'sweep', 'clap', 'slide',
        'drawzigzag', 'dodge', 'bowling', 'liftupahand', 'tap', 'spreadandpinch', 'drawtriangle',
        'sneeze', 'cough', 'stagger', 'yawn', 'blownose', 'stretchoneself', 'touchface',
        'handshake', 'hug', 'pushsomeone', 'kicksomeone', 'punchsomeone', 'conversation', 'gym', 'freestyle'],  # Specify which activities to filter
    'node_id': [1, 2, 3, 4, 5], 
    'segmentation_flag': True, # whether to include segmentation in the dataset
    'modality': [ 'mmWave', 'IRA', 'uwb', 'ToF', 'polar', 'wifi', 'depthCamera', 'seekThermal','acoustic', 'imu', 'vayyar', 'mocap'] # depthCamera is RGB-D camera
}

🎯 Custom Configuration Example

🔍 Targeted Dataset Selection:

To select a specific subset of the dataset, you can customize the configuration. Here's a practical example:

config = {
    'exp_list': [1],  # select exp 1
    'activity_list': ['dance'],  # select activity 'dance'
    'node_id': [1, 2, 3, 4, 5],  # select all nodes
    'segmentation_flag': True, # data is segmented
    'modality': [ 'mmWave', 'IRA', 'uwb', 'ToF', 'polar', 'wifi', 'depthCamera', 'seekThermal','acoustic', 'imu', 'vayyar', 'mocap'], # select all modalities
    # 'modality': ['polar', 'depthCamera'], # select polar and depthCamera modalities
    # 'mocap_downsample_num': 6 # downsample the mocap data to 6 frames per second
}

💡 📝 Smart Loading: get_dataset intelligently includes all available data that matches your configuration criteria!

🎨 Interactive Visualization Suite

📊 Multi-Modal Data Visualization:

The comprehensive visualization code is provided in demo.ipynb. This interactive notebook will automatically generate:

📈 Figures and plots for data analysis
🎬 Video outputs for temporal data visualization
📁 Organized results in the vis_output folder

# Sample configuration and usage
dataset_path = "dataset"
data_config = {
    'exp_list': [1],  # Specify which experiments to filter
    'activity_list': ['dance'],  
    'node_id': [1, 2, 3, 4, 5], 
    'segmentation_flag': True,
    'modality': [ 'mmWave', 'IRA', 'uwb', 'ToF', 'polar', 'wifi', 'depthCamera', 'seekThermal','acoustic', 'imu', 'vayyar', 'mocap'],
    # 'modality': ['polar', 'depthCamera'],
    # 'mocap_downsample_num': 6
}

# Get the DataLoader
dataset = get_dataset(data_config, dataset_path)
dataloader = get_dataloader(dataset, batch_size=1, shuffle=False, config=data_config)

for batch in dataloader:
    dump_seekthermal_frames_as_png(
        batch, 
        output_dir="validation_seekthermal"
    )
    visualize_seekthermal_and_rgb_mosaic_batch_discard_excess(
        batch,
        output_dir='seekthermal_rgb_mosaic_videos',
        fps_out=8.80
    )
    visualize_3_depth_3_rgb_mosaic_batch_discard_excess(
        batch,
        output_dir='depth_rgb_mosaic_discard',
        fps_out=10
    )
    visualize_4wifi_time_subcarrier_with_camera(
        batch,
        output_dir='wifi_rgb_mosaic_videos',
        fps_out=10.0,
        BW="40MHz"
    )
    visualize_ira_and_rgb_mosaic_batch_downsample_cam(
        batch,
        output_dir='ira_rgb_mosaic_videos',
        fps_out=6.91
    )
    visualize_mocap_and_rgb_mosaic_batch_downsample_mocap(
        batch,
        output_dir='mocap_rgb_mosaic_videos',
        fps_out=10
    )
    visualize_tof_and_rgb_mosaic_batch_downsample_tof(
        batch,
        output_dir='tof_rgb_mosaic_videos',
        fps_out=7.32
    )
    visualize_fmcw_and_rgb_mosaic_batch_raw_fixed_axes(
        batch,
        output_dir='fmcw_rgb_mosaic',
        fps_out=8.81
    )
    visualize_vayyar_txrx_only_and_camera(
        batch,
        output_dir="vayyar_rgb_mosaic",
        fps_out=10.0
    )
    visualize_acoustic_2node_melspectrogram_and_rgb(
        batch,
        output_dir="acoustic_melspec_plus_rgb",
        fps_out=10.0
    )
    visualize_polar_and_camera_batch(
        batch,
        output_dir="polar_hr_plus_rgb",
        fps_out=10.0,
        y_domain=None
    )
    visualize_imu_four_rows_no_zscore(
        batch,
        output_dir="imu_time_features_plus_rgb",
        fps_out=10.0
    )
    visualize_uwb_and_rgb_in_same_row_with_box(
        batch,
        output_dir="uwb_rgb_same_row_with_box",
        fps_out=10.0
    )
    break

⚡ Part 2: Benchmark & Reproducible Results ⚡

State-of-the-Art Performance Evaluation Framework

🏆 Comprehensive benchmarking suite for the OctoNet dataset with reproducible results and performance comparisons.

🚀 Step 1: Environment Setup

📁 Navigate to Benchmark Directory:

# Change to the benchmark directory
cd OctonetBenchmark

🐍 Python Environment Requirements:

📋 Recommended: Python 3.9–3.11 with our pre-configured Conda environment

🔧 Automated Environment Creation:

# Create the benchmark environment from the provided specification
conda env create -f environment.yml

# Activate the environment (name defined in environment.yml)
conda activate octo

# (Optional) Update existing environment after changes
conda env update -f environment.yml --prune

💡 Important Notes:

🎮 GPU Support: The environment.yml includes GPU-enabled PyTorch and CUDA libraries. For CPU-only setups, remove CUDA-related packages (pytorch-cuda, cudnn, cuda-*) and the nvidia channel.

⚡ Performance Tip: If conda solver struggles, try using mamba as a faster drop-in replacement for conda.

📥 Step 2: Dataset Download

🌐 Official Dataset Sources:

📊 Primary Source: Hugging Face Dataset
💻 Code Repository: GitHub Repository

⚡ Automated Download Process:

The dataset provider offers an automated script that downloads 16 chunks, merges, and extracts them seamlessly.

💾 Storage Requirements: ~1.5TB peak disk space during download and extraction

# One-command automated download and extraction
bash -c "$(wget -qO- https://huggingface.co/datasets/hku-aiot/OctoNet/resolve/main/download_octonet.sh)"

📁 Expected Dataset Structure:

After download and extraction, ensure you have the following structure under a directory named dataset (you can choose a different parent path):

./dataset
├── mocap_csv_final          # Data: Final motion capture data in CSV format.
├── mocap_pose               # Data: Final motion capture data in npy format.
├── node_1                   # Data: Data related to multi-modal sensor node 1.
├── node_2                   # Data: Data related to multi-modal sensor node 2.
├── node_3                   # Data: Data related to multi-modal sensor node 3.
├── node_4                   # Data: Data related to multi-modal sensor node 4.
├── node_5                   # Data: Data related to multi-modal sensor node 5.
├── imu                      # Data: Inertial measurement unit data.
├── vayyar_pickle            # Data: vayyar mmWave radar data.
└── cut_manual.csv           # Manually curated data cuts.

🔧 Step 3: Dataset Preparation

📦 Benchmark-Specific Setup:

Rename the dataset directory to octonet and place the dataset helper script Octonet.py inside it. You should end up with:

./octonet
├── mocap_csv_final          # Data: Final motion capture data in CSV format.
├── mocap_pose               # Data: Final motion capture data in npy format.
├── node_1                   # Data: Data related to multi-modal sensor node 1.
├── node_2                   # Data: Data related to multi-modal sensor node 2.
├── node_3                   # Data: Data related to multi-modal sensor node 3.
├── node_4                   # Data: Data related to multi-modal sensor node 4.
├── node_5                   # Data: Data related to multi-modal sensor node 5.
├── imu                      # Data: Inertial measurement unit data.
├── vayyar_pickle            # Data: vayyar mmWave radar data.
├── Octonet.py               # script: contains the dataset PyTorch functions and dataloader
└── cut_manual.csv           # Manually curated data cuts.

⚠️ Important Setup Notes:

📁 Folder Naming: This repository imports dataset utilities via from octonet.Octonet import get_dataset, custom_collate. The folder name must be exactly octonet.

📍 Custom Paths: If you downloaded to a different location, set dataset_path in your chosen configuration file to the absolute path of your octonet folder.

⚙️ Step 4: Configuration Selection

🎯 Choose Your Benchmark Configuration:

To reproduce the paper results, select the corresponding configuration YAML listed in the results table below. Example configurations already included:

📋 Available Configurations:

Configurations/acoustic_denesnet121_10.yaml
Configurations/ira_rf_net_pose.yaml

💡 Usage Tip: Pass the config name without the .yaml suffix to the --config_file flag.

🚀 Step 5: Execute Benchmarks

⚡ Running Your Selected Configuration:

🎯 Training + Testing (Full Pipeline):

python main.py --config_file acoustic_denesnet121_10 --cuda_index 0 --mode 0

🧪 Testing Only (Pre-trained Model):

python main.py --config_file acoustic_denesnet121_10 --cuda_index 0 --mode 1 --pretrained_model /absolute/path/to/weights.pth

🔄 Fine-tuning + Testing:

python main.py --config_file ira_rf_net_pose --cuda_index 0 --mode 2 --pretrained_model /absolute/path/to/weights.pth

💾 Model Saving Configuration:

🔧 To save trained model weights: Change model_save_enable: False to model_save_enable: True in your selected configuration file. Control output paths via trained_model_folder, log_folder, and tensorboard_folder in the same YAML.

📊 Reproducible Results & Performance Tables 📊

Complete Benchmark Results with Configurations, Runs, and Logs

📁 Additional Resources:

💾 Large Log Files: Download the logs folder and place it in the same directory as the 'Configurations' folder.
🔗 Download Link: SharePoint Logs Repository

🏃 Human Activity Recognition Results 🏃

Multi-Modal Performance Comparison

Modality	Protocol	ResNet 10/62	DenseNet 10/62	Swin-T 10/62	RFNet 10/62
RGB	ID	91.5 (±2.6) Config Log Run / 93.4 (±0.9) Config Log Run	93.2 (±2.3) Config Log Run / 91.2 (±1.0) Config Log Run	94.9 (±2.0) Config Log Run / 93.1 (±0.9) Config Log Run	89.7 (±2.8) Config Log Run / 60.9 (±1.8) Config Log Run
	CU	46.0 (±3.4) Log / 12.3 (±0.9) Log	68.2 (±3.2) Log / 24.7 (±1.2) Log	37.0 (±3.3) Log / 7.7 (±0.7) Log	45.0 (±3.4) Log / 9.2 (±0.8) Log
	CS	14.9 (±3.0) Log / 4.1 (±0.7) Log	33.3 (±4.0) Log / 11.3 (±1.1) Log	12.1 (±2.8) Log / 1.7 (±0.4) Log	13.5 (±2.9) Log / 3.1 (±0.6) Log
Depth	ID	89.7 (±2.8) Config Log Run / 86.6 (±1.2) Config Log Run	90.6 (±2.7) Config Log Run / 83.2 (±1.3) Config Log Run	86.3 (±3.2) Config Log Run / 81.7 (±1.4) Config Log Run	87.2 (±3.1) Config Log Run / 40.0 (±1.8) Config Log Run
	CU	41.2 (±3.4) Log / 11.1 (±0.9) Log	64.9 (±3.3) Log / 27.3 (±1.2) Log	46.0 (±3.4) Log / 14.4 (±1.0) Log	45.0 (±3.4) Log / 11.2 (±0.9) Log
	CS	17.7 (±3.2) Log / 3.9 (±0.7) Log	22.7 (±3.5) Log / 12.2 (±1.1) Log	23.4 (±3.6) Log / 4.3 (±0.7) Log	28.4 (±3.8) Log / 4.8 (±0.7) Log
ToF	ID	86.8 (±3.1) Config Log Run / 70.3 (±1.6) Config Log Run	N/A	82.6 (±3.5) Config Log Run / 51.8 (±1.8) Config Log Run	89.3 (±2.8) Config Log Run / 75.9 (±1.5) Config Log Run
	CU	44.5 (±3.4) Log / 11.8 (±0.9) Log	N/A	46.4 (±3.4) Log / 15.3 (±1.0) Log	78.7 (±2.8) Log / 28.3 (±1.2) Log
	CS	25.5 (±3.7) Log / 8.0 (±0.9) Log	N/A	22.7 (±3.5) Log / 4.7 (±0.7) Log	44.7 (±4.2) Log / 18.6 (±1.3) Log
Thermal	ID	90.1 (±2.7) Config Log Run / 85.0 (±1.3) Config Log Run	91.7 (±2.5) Config Log Run / 85.4 (±1.3) Config Log Run	85.1 (±3.2) Config Log Run / 79.2 (±1.5) Config Log Run	47.1 (±4.6) Config Log Run / 28.6 (±1.6) Config Log Run
	CU	50.2 (±3.5) Log / 25.7 (±1.2) Log	64.5 (±3.4) Log / 32.5 (±1.3) Log	46.8 (±3.5) Log / 15.6 (±1.0) Log	15.3 (±2.5) Log / 1.0 (±0.3) Log
	CS	36.9 (±4.1) Log / 13.4 (±1.2) Log	44.0 (±4.2) Log / 21.0 (±1.4) Log	36.2 (±4.1) Log / 10.1 (±1.0) Log	17.7 (±3.2) Log / 2.1 (±0.5) Log
IRA	ID	25.6 (±4.0) Config Log Run / 1.8 (±0.5) Config Log Run	N/A	14.0 (±3.2) Config Log Run / 3.7 (±0.7) Config Log Run	19.0 (±3.6) Config Log Run / 4.2 (±0.7) Config Log Run
	CU	19.9 (±2.8) Log / 2.6 (±0.4) Log	N/A	22.3 (±2.9) Log / 2.8 (±0.4) Log	21.8 (±2.8) Log / 3.2 (±0.5) Log
	CS	18.4 (±3.3) Log / 0.8 (±0.3) Log	N/A	20.6 (±3.4) Log / 3.8 (±0.6) Log	21.3 (±3.5) Log / 2.7 (±0.6) Log
FMCW	ID	39.3 (±4.5) Config Log Run / 24.0 (±1.6) Config Log Run	74.4 (±4.1) Config Log Run / 46.3 (±1.8) Config Log Run	36.8 (±4.5) Config Log Run / 5.0 (±0.8) Config Log Run	38.5 (±4.5) Config Log Run / 12.6 (±1.2) Config Log Run
	CU	27.0 (±3.1) Log / 8.9 (±0.8) Log	44.1 (±3.4) Log / 16.1 (±1.0) Log	24.2 (±3.0) Log / 4.4 (±0.6) Log	26.5 (±3.0) Log / 7.2 (±0.7) Log
	CS	26.0 (±4.3) Log / 5.3 (±1.0) Log	14.4 (±3.5) Log / 7.5 (±1.2) Log	14.4 (±3.5) Log / 3.6 (±0.8) Log	26.0 (±4.3) Log / 4.3 (±0.9) Log
SFCW	ID	30.6 (±4.2) Config Log Run / 9.0 (±1.0) Config Log Run	59.5 (±4.5) Config Log Run / 13.0 (±1.2) Config Log Run	26.4 (±4.0) Config Log Run / 0.9 (±0.3) Config Log Run	28.1 (±4.1) Config Log Run / 5.1 (±0.8) Config Log Run
	CU	12.3 (±2.3) Log / 1.6 (±0.3) Log	4.3 (±1.4) Log / 1.2 (±0.3) Log	7.6 (±1.8) Log / 1.6 (±0.3) Log	13.3 (±2.3) Log / 2.2 (±0.4) Log
	CS	11.3 (±2.7) Log / 2.5 (±0.5) Log	15.6 (±3.1) Log / 1.5 (±0.4) Log	7.8 (±2.3) Log / 1.6 (±0.4) Log	17.0 (±3.2) Log / 1.5 (±0.4) Log
UWB	ID	98.3 (±1.2) Config Log Run / 93.8 (±0.9) Config Log Run	88.4 (±2.9) Config Log Run / 80.1 (±1.4) Config Log Run	100.0 (±0.0) Config Log Run / 90.4 (±1.1) Config Log Run	94.2 (±2.1) Config Log Run / 75.8 (±1.5) Config Log Run
	CU	62.6 (±3.3) Log / 21.5 (±1.1) Log	59.7 (±3.4) Log / 27.4 (±1.2) Log	17.1 (±2.6) Log / 2.7 (±0.4) Log	64.5 (±3.3) Log / 13.5 (±0.9) Log
	CS	27.0 (±3.7) Log / 6.7 (±0.8) Log	20.6 (±3.4) Log / 6.3 (±0.8) Log	21.3 (±3.5) Log / 2.4 (±0.5) Log	12.1 (±2.8) Log / 1.7 (±0.4) Log
WiFi	ID	93.3 (±2.3) Config Log Run / 91.1 (±1.0) Config Log Run	90.8 (±2.6) Config Log Run / 91.0 (±1.0) Config Log Run	91.7 (±2.5) Config Log Run / 92.3 (±1.0) Config Log Run	81.7 (±3.5) Config Log Run / 60.5 (±1.8) Config Log Run
	CU	13.3 (±2.3) Log / 3.4 (±0.5) Log	11.4 (±2.2) Log / 4.8 (±0.6) Log	12.3 (±2.3) Log / 2.3 (±0.4) Log	19.9 (±2.8) Log / 4.3 (±0.6) Log
	CS	19.1 (±3.3) Log / 2.4 (±0.5) Log	11.3 (±2.7) Log / 1.9 (±0.5) Log	13.5 (±2.9) Log / 2.8 (±0.6) Log	11.3 (±2.7) Log / 1.1 (±0.4) Log
Acoustic	ID	40.8 (±4.5) Config Log Run / 45.5 (±1.8) Config Log Run	60.0 (±4.5) Config Log Run / 54.6 (±1.8) Config Log Run	36.7 (±4.4) Config Log Run / 32.1 (±1.7) Config Log Run	29.2 (±4.2) Config Log Run / 19.1 (±1.4) Config Log Run
	CU	37.0 (±3.3) Log / 19.9 (±1.1) Log	42.7 (±3.4) Log / 16.4 (±1.0) Log	27.5 (±3.1) Log / 8.4 (±0.8) Log	20.4 (±2.8) Log / 7.1 (±0.7) Log
	CS	26.2 (±3.7) Log / 9.3 (±1.0) Log	25.5 (±3.7) Log / 8.7 (±1.0) Log	12.8 (±2.8) Log / 1.9 (±0.5) Log	13.5 (±2.9) Log / 5.1 (±0.7) Log
IMU	ID	96.6 (±1.7) Config Log Run / 96.5 (±0.7) Config Log Run	97.4 (±1.5) Config Log Run / 95.7 (±0.7) Config Log Run	98.3 (±1.2) Config Log Run / 95.7 (±0.7) Config Log Run	94.0 (±2.2) Config Log Run / 35.8 (±1.8) Config Log Run
	CU	73.5 (±3.0) Log / 43.9 (±1.4) Log	74.4 (±3.0) Log / 34.6 (±1.3) Log	82.9 (±2.6) Log / 40.8 (±1.3) Log	66.4 (±3.3) Log / 13.8 (±0.9) Log
	CS	62.4 (±4.1) Log / 43.1 (±1.7) Log	62.4 (±4.1) Log / 31.5 (±1.6) Log	47.5 (±4.2) Log / 34.4 (±1.6) Log	54.6 (±4.2) Log / 12.4 (±1.1) Log

🧍 Human Pose Estimation Results 🧍

Multi-Modal Pose Estimation Performance

Modality	Protocol	ResNet	DenseNet	Swin-T	RFNet
RGB	ID	133.3 (±4.4) Config Log Run	147.2 (±5.1) Config Log Run	269.6 (±6.2) Config Log Run	162.8 (±4.6) Config Log Run
	CU	199.8 (±4.2) Log	204.8 (±4.8) Log	286.2 (±5.7) Log1 Log2 Log3 Log4	223.7 (±4.5) Log
	CS	473.9 (±5.0) Log	524.6 (±4.3) Log	273.0 (±7.0) Log	331.8 (±6.5) Log
Depth	ID	131.4 (±4.5) Config Log Run	147.4 (±4.6) Config Log Run	248.2 (±6.5) Config Log Run	194.8 (±5.7) Config Log Run
	CU	197.1 (±4.6) Log	212.5 (±4.6) Log	256.2 (±5.8) Log	230.7 (±5.3) Log
	CS	363.6 (±5.6) Log	436.4 (±5.3) Log	305.1 (±7.0) Log	444.2 (±5.8) Log
ToF	ID	152.5 (±5.2) Config Log Run	N/A	252.1 (±6.0) Config Log Run	162.2 (±5.0) Config Log Run
	CU	205.7 (±5.0) Log	N/A	257.3 (±5.7) Log	193.7 (±4.8) Log
	CS	361.2 (±5.4) Log	N/A	303.9 (±7.0) Log	363.7 (±4.8) Log
Thermal	ID	142.8 (±4.7) Config Log Run	147.0 (±4.9) Config Log Run	259.9 (±5.9) Config Log Run	254.3 (±6.1) Config Log Run
	CU	216.9 (±4.4) Log	222.4 (±4.7) Log	259.9 (±5.8) Log	308.2 (±5.7) Log
	CS	308.8 (±5.9) Log	325.4 (±5.3) Log	313.3 (±6.8) Log	403.4 (±7.0) Log
IRA	ID	244.4 (±6.8) Config Log Run	N/A	261.1 (±6.4) Config Log Run	265.1 (±6.7) Config Log Run
	CU	373.3 (±5.8) Log	N/A	261.1 (±5.9) Log	299.1 (±5.8) Log
	CS	398.8 (±7.2) Log	N/A	313.0 (±6.8) Log	313.4 (±7.3) Log
FMCW	ID	198.5 (±5.7) Config Log Run	185.4 (±5.4) Config Log Run	272.5 (±7.3) Config Log Run	220.9 (±6.0) Config Log Run
	CU	244.0 (±4.9) Log	236.8 (±4.7) Log	263.0 (±6.0) Log	272.0 (±5.1) Log
	CS	369.4 (±10.6) Log	389.8 (±9.8) Log	338.8 (±10.1) Log	328.3 (±10.2) Log
SFCW	ID	206.7 (±6.2) Config Log Run	202.9 (±6.2) Config Log Run	264.2 (±6.4) Config Log Run	270.7 (±6.6) Config Log Run
	CU	314.6 (±5.4) Log	334.4 (±5.4) Log	259.1 (±5.9) Log	408.1 (±23.4) Log
	CS	352.2 (±7.0) Log	408.9 (±7.0) Log	339.7 (±8.9) Log	392.7 (±10.3) Log
UWB	ID	142.4 (±4.8) Config Log Run	158.0 (±5.2) Config Log Run	260.5 (±6.1) Config Log Run	159.5 (±5.0) Config Log Run
	CU	241.2 (±4.8) Log	239.0 (±4.7) Log	261.3 (±5.8) Log	241.6 (±4.6) Log
	CS	310.0 (±6.5) Log	327.6 (±6.6) Log	312.5 (±6.8) Log	295.8 (±6.8) Log
Wi-Fi	ID	147.3 (±4.7) Config Log Run	147.4 (±4.9) Config Log Run	262.2 (±6.0) Config Log Run	186.8 (±5.3) Config Log Run
	CU	270.4 (±5.6) Log	267.8 (±5.8) Log	256.3 (±5.8) Log	274.2 (±5.6) Log
	CS	399.4 (±5.7) Log	322.1 (±6.8) Log	312.8 (±6.8) Log	400.9 (±8.3) Log
Acoustic	ID	258.8 (±6.9) Config Log Run	256.8 (±6.7) Config Log Run	271.2 (±6.7) Config Log Run	243.6 (±6.8) Config Log Run
	CU	304.1 (±5.8) Log	312.8 (±5.8) Log	260.6 (±5.8) Log	291.8 (±5.6) Log
	CS	367.2 (±6.8) Log	441.4 (±6.9) Log	312.0 (±6.8) Log	323.3 (±7.2) Log
IMU	ID	147.9 (±5.0) Config Log Run	159.3 (±5.5) Config Log Run	251.6 (±6.4) Config Log Run	180.9 (±5.3) Config Log Run
	CU	252.9 (±4.9) Log	274.3 (±5.0) Log	259.9 (±5.9) Log	266.3 (±5.0) Log
	CS	289.7 (±6.8) Log	324.0 (±6.7) Log	310.8 (±6.9) Log	328.4 (±6.9) Log

📄 License & Citation 📄

📜 License

⚖️ Open Source License:

This project is licensed under the GPL-3.0 License. See the LICENSE file for complete details.

📚 Citation

🔬 Academic Recognition:

If you find this work useful in your research, please cite our paper:

📝 Citation: Coming soon...

🎉 Thank you for using OctoNet! 🎉

Empowering multi-modal human activity understanding through comprehensive sensor fusion

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
OctonetBenchmark		OctonetBenchmark
dataset		dataset
figs		figs
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
dataset_loader.py		dataset_loader.py
dataset_visualizer.py		dataset_visualizer.py
demo.ipynb		demo.ipynb
download_octonet.sh		download_octonet.sh
environment.yaml		environment.yaml
func_utils.py		func_utils.py
requirements.txt		requirements.txt
shape_converter.py		shape_converter.py

Folders and files

Latest commit

History

Repository files navigation

🦑 OctoNet Toolbox 🦑

The Ultimate Multi-Modal Human Activity Understanding Toolkit

🎯 Revolutionary Multi-Modal Dataset for Human Activity Understanding

🚀 What's Inside This Toolbox

✨ Comprehensive OctoNet Toolkit ✨

🎨 Visualization Suite

⚡ Benchmark Framework

🎨 Part 1: Dataset Visualization & Exploration 🎨

Interactive Multi-Modal Data Analysis Suite

📁 Dataset Structure Overview

📊 Dataset Metadata & Statistics

📝 Key Information:

⚙️ Environment Setup & Installation ⚙️

Quick Start Guide for OctoNet Development Environment

🐍 Step 1: Create Conda Environment

🚀 Step 2: Launch Jupyter Notebook

📊 Sample Data Selection & Configuration 📊

Intelligent Dataset Loading with Flexible Configuration

🔧 Core Dataset Loading Function

⚙️ Complete Configuration Template

🎯 Custom Configuration Example

🎨 Interactive Visualization Suite

⚡ Part 2: Benchmark & Reproducible Results ⚡

State-of-the-Art Performance Evaluation Framework

🚀 Step 1: Environment Setup

📥 Step 2: Dataset Download

🔧 Step 3: Dataset Preparation

⚙️ Step 4: Configuration Selection

🚀 Step 5: Execute Benchmarks

📊 Reproducible Results & Performance Tables 📊

Complete Benchmark Results with Configurations, Runs, and Logs

🏃 Human Activity Recognition Results 🏃

Multi-Modal Performance Comparison

🧍 Human Pose Estimation Results 🧍

Multi-Modal Pose Estimation Performance

📄 License & Citation 📄

📜 License

📚 Citation

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages