Update script for downloading large dataset from Huggingface

[daphne-cornelisse]

No description provided.

[greptile-apps]

Greptile Overview

Greptile Summary

Added new data processing pipeline for downloading and validating WOMD dataset from HuggingFace, with group-based downloading to handle rate limits and parallel validation/extraction.

Key Changes:

• New download_womd_data.py script downloads datasets from EMERGE-lab/GPUDrive repository with configurable limits per split (training/validation/testing), group-based downloads for training data to work around HuggingFace rate limits, and authentication support
• New post_processing.py script validates JSON structure, moves files from group subdirectories to parent directory, deletes invalid files, and uses multiprocessing for parallel validation
• Deleted large Town01_0.json file (107k lines) from repository

Issues Found:

• Critical bug in post_processing.py:106 - file filtering logic will incorrectly exclude valid files if their names contain group directory names
• Two code style improvements needed in download_womd_data.py (bare except clause, unnecessary loop iteration)

Confidence Score: 2/5

• This PR contains a critical logical bug that will cause incorrect file filtering behavior in production
• The file filtering bug in post_processing.py:106 is a definite logic error that will cause files to be incorrectly skipped during processing if their names happen to contain group directory names (e.g., group_1_data.json would be filtered out if a group_1/ directory exists). This needs to be fixed before merging.
• data_utils/womd/post_processing.py requires immediate attention due to the file filtering bug on line 106

Important Files Changed

File Analysis

Filename	Score	Overview
data_utils/womd/download_womd_data.py	3/5	New HuggingFace dataset download script with group-based downloading, rate limit handling, and progress monitoring; has bare except clause and potential off-by-one in loop range
data_utils/womd/post_processing.py	2/5	JSON validation and group extraction script with multiprocessing support; contains logical bug in file filtering that could incorrectly exclude valid files from processing

Sequence Diagram

sequenceDiagram
    participant User
    participant DownloadScript as download_womd_data.py
    participant HF as HuggingFace Hub
    participant FS as File System
    participant PostProcess as post_processing.py
    participant Monitor as monitor_and_alert

    User->>DownloadScript: Run with config
    DownloadScript->>HF: Check authentication token
    alt No token
        HF-->>DownloadScript: No token found
        DownloadScript->>User: Prompt for continue without auth
        User-->>DownloadScript: y/n response
    end
    
    DownloadScript->>DownloadScript: Calculate num_groups_needed
    
    loop For each training group (0 to num_groups_needed+1)
        DownloadScript->>FS: count_files_in_dir(training)
        FS-->>DownloadScript: current_count
        
        alt current_count >= max_training_files
            DownloadScript->>DownloadScript: Break loop
        else
            DownloadScript->>HF: snapshot_download(group_N)
            alt Success
                HF-->>FS: Download files to group_N/
                DownloadScript->>FS: count_files_in_dir(training)
                FS-->>DownloadScript: new_count
            else Rate limit (429)
                HF-->>DownloadScript: Error 429
                DownloadScript->>User: Show rate limit suggestions
                DownloadScript->>DownloadScript: Break loop
            else Group not found (404)
                HF-->>DownloadScript: Error 404
                DownloadScript->>DownloadScript: Break loop
            end
        end
    end
    
    DownloadScript->>Monitor: Start monitoring thread for validation
    activate Monitor
    DownloadScript->>HF: snapshot_download(validation/*)
    loop Monitor validation count
        Monitor->>FS: count_files_in_dir(validation)
        FS-->>Monitor: count
        Monitor->>User: Display progress
        alt count >= max_validation_files
            Monitor->>User: Alert limit reached
            Monitor->>Monitor: Exit thread
        end
    end
    deactivate Monitor
    
    DownloadScript->>HF: snapshot_download(testing/*)
    HF-->>FS: Download testing files
    
    DownloadScript->>User: Display final summary
    
    User->>PostProcess: Run post_processing.py
    PostProcess->>FS: Find group_* directories
    FS-->>PostProcess: List of group dirs
    
    PostProcess->>FS: Collect all JSON files
    FS-->>PostProcess: File list
    
    PostProcess->>PostProcess: Create worker pool
    
    par Parallel processing
        loop For each JSON file
            PostProcess->>PostProcess: is_valid_json_structure(file)
            alt Invalid JSON
                PostProcess->>FS: Delete file
            else Valid JSON and in group dir
                PostProcess->>FS: Move file to parent directory
            end
        end
    end
    
    PostProcess->>FS: Remove empty group directories
    PostProcess->>User: Display statistics

Loading

[greptile-apps]

_{3 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: incorrect file filtering logic - using g.name in str(f) will incorrectly exclude files like group_1_data.json from the main directory if a group_1/ subdirectory exists

Suggested change

	main_dir_files = [f for f in dataset_path.glob("*.json") if not any(g.name in str(f) for g in group_dirs)]
	main_dir_files = [f for f in dataset_path.glob("*.json") if f.parent == dataset_path]

Prompt To Fix With AI

This is a comment left during a code review.
Path: data_utils/womd/post_processing.py
Line: 106:106

Comment:
**logic:** incorrect file filtering logic - using `g.name in str(f)` will incorrectly exclude files like `group_1_data.json` from the main directory if a `group_1/` subdirectory exists

```suggestion
    main_dir_files = [f for f in dataset_path.glob("*.json") if f.parent == dataset_path]
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: bare except: catches all exceptions including KeyboardInterrupt and SystemExit

Suggested change

	except:
	except Exception:

Prompt To Fix With AI

This is a comment left during a code review.
Path: data_utils/womd/download_womd_data.py
Line: 67:67

Comment:
**style:** bare `except:` catches all exceptions including `KeyboardInterrupt` and `SystemExit`

```suggestion
        except Exception:
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: loop iterates to num_groups_needed + 1, potentially downloading one extra group beyond max_training_files limit (the break condition on line 94 prevents excess, but the extra iteration is unnecessary)

Prompt To Fix With AI

This is a comment left during a code review.
Path: data_utils/womd/download_womd_data.py
Line: 91:91

Comment:
**style:** loop iterates to `num_groups_needed + 1`, potentially downloading one extra group beyond `max_training_files` limit (the break condition on line 94 prevents excess, but the extra iteration is unnecessary)

How can I resolve this? If you propose a fix, please make it concise.