astro_cat

astro_cat is a comprehensive toolkit for managing, cataloging, and archiving astrophotography data. It streamlines raw FITS image intake, session organization, processing workflows, and cloud backups—with an intuitive web interface and powerful CLI tools.

NO WARRANTY Please use with caution! I can take no responsibility for data loss, corruption or other issues. I strongly recommend you back up your data before using this! Test with a small dataset first!

🚀 Features

Core Functionality

🔭 FITS Image Cataloging — Automatically scan directories for new FITS files, extract metadata (object, filter, exposure, etc.), and organize into a searchable SQLite database
📁 Session Management — Organize images into imaging sessions with automatic detection and manual override capabilities
🎯 Processing Workflows — Create processing sessions, select appropriate calibration frames, track processed outputs
📝 Obsidian-Compatible Notes — Maintain Markdown session notes that integrate seamlessly with Obsidian vaults
🔍 Advanced Search — Query by object, filter, date, telescope, camera, exposure time, and more

Web Interface

📊 Dashboard — Real-time statistics, storage usage, recent activity
🌐 Session Browser — Browse and manage imaging sessions with detailed file listings
🖼️ File Explorer — Search and filter FITS files across your entire library
📈 Statistics & Charts — Visualize collection growth, equipment usage, integration times
⚙️ Processing Sessions — Create and manage processing workflows through the UI

Command Line Tools

🧰 Comprehensive CLI — Full control over all features via command line
🔄 Batch Operations — Scan, catalog, validate, and migrate files in bulk
📋 List & Query — Generate reports on files, sessions, equipment, and statistics
✅ Validation — Check file integrity, verify metadata consistency

Cloud Backup (Optional)

☁️ S3 Integration — Automated backups to AWS S3 with compression
🔐 Integrity Verification — MD5 checksum validation for all uploads
📦 Lifecycle Management — Automatic transitions to Glacier and Deep Archive
💾 Incremental Backups — Only backup new or changed sessions
📊 Backup Dashboard — Monitor backup status, storage costs, and coverage

📋 Requirements

Python 3.8+
Operating System: Linux, macOS, or Windows
Storage: Local or network-attached storage for FITS files
Optional: AWS account for S3 backup features

⚙️ Installation

1. Clone the Repository

git clone https://github.com/smartc/astro_cat.git
cd astro_cat

2. Create Virtual Environment

python3 -m venv venv

# Activate on Linux/macOS:
source venv/bin/activate

# Or on Windows:
venv\Scripts\activate

3. Install Dependencies

pip install -r requirements.txt

4. Install Frontend Libraries

The web interface requires JavaScript and CSS libraries that are not included in the repository. Run the installer script to download them:

python install_frontend_libs.py

This will download:

Vue.js 3.4.21
Axios 1.6.7
Toast UI Editor (latest)
Tailwind CSS 2.2.19

Note: These libraries are downloaded to static/js/lib/ and static/css/lib/ directories, which are excluded from version control.

5. Verify Installation

python main.py --help

You should see the CLI help menu with available commands.

🛠️ Initial Configuration

1. Configure Paths

First, copy the template configuration file:

cp config.json.template config.json

Then edit config.json to set your file locations:

{
  "paths": {
    "quarantine_dir": "/path/to/quarantine",     # Incoming raw FITS files
    "image_dir": "/path/to/images",              # Organized library
    "processing_dir": "/path/to/processing",     # Processing sessions
    "database_path": "/path/to/fits_catalog.db", # SQLite database
    "notes_dir": "/path/to/session_notes"        # Markdown notes
  },
  "database": {
    "type": "sqlite",
    "connection_string": "sqlite:///{{database_path}}"
  },
  "file_monitoring": {
    "scan_interval_minutes": 30,
    "ignore_newer_than_minutes": 30
  },
  "timezone_offset": -6
}

Key Paths Explained:

quarantine_dir: Where new FITS files arrive (from camera/acquisition software)
image_dir: Organized library structured by date and session
processing_dir: Processing sessions with calibration, intermediate, and final outputs
database_path: SQLite database storing all metadata
notes_dir: Markdown notes for sessions (Obsidian-compatible)

2. Configure Equipment (Optional)

Edit the equipment definition files to match your gear:

cameras.json:

{
  "ZWO ASI2600MM Pro": {
    "sensor_width": 23.5,
    "sensor_height": 15.7,
    "pixel_size": 3.76
  }
}

telescopes.json:

{
  "WO Star71": {
    "aperture": 71,
    "focal_length": 350,
    "type": "Refractor"
  }
}

filters.json:

[
  {
    "raw_name": "Lum",
    "proper_name": "L"
  },
  {
    "raw_name": "Red",
    "proper_name": "R"
  },
  {
    "raw_name": "Green",
    "proper_name": "G"
  },
  {
    "raw_name": "Blue",
    "proper_name": "B"
  }
]

3. Initialize Database

The database and required directories will be created automatically on first run. To verify:

python main.py list imaging-sessions

If the database doesn't exist, it will be created with the proper schema.

🌐 Web Interface Usage

Starting the Web Server

Important: Before starting the web server for the first time, make sure you've installed the frontend libraries:

python install_frontend_libs.py

Then start the server:

python run_web.py

The server will start on http://localhost:8000. Open this URL in your browser.

Environment Variables

Configure the web server using environment variables:

Variable	Default	Description
`ASTROCAT_HOST`	`127.0.0.1`	Main app bind address
`ASTROCAT_PORT`	`8000`	Main app port
`ASTROCAT_BIND_HOST`	`127.0.0.1`	Secondary services bind address
`ASTROCAT_DB_BROWSER_PORT`	`8081`	Database browser (sqlite_web) port
`ASTROCAT_WEBDAV_PORT`	`8082`	WebDAV server port
`ASTROCAT_S3_BACKUP_PORT`	`8083`	S3 backup web interface port

Example - Custom ports:

ASTROCAT_PORT=9000 ASTROCAT_DB_BROWSER_PORT=9081 python run_web.py

Example - Bind to specific interface:

ASTROCAT_HOST=192.168.1.100 python run_web.py

Running Behind a Reverse Proxy (HTTPS)

For production deployments with HTTPS, run AstroCat behind Apache or nginx. The application includes built-in proxy routes that forward requests to secondary services, so your reverse proxy only needs to connect to port 8000.

Proxy Routes:

/db-browser/* → Database browser (sqlite_web on port 8081)
/s3-backup/* → S3 backup manager (port 8083)
/webdav/* → WebDAV server (port 8082)

Apache Configuration:

See docs/apache-reverse-proxy.conf for a complete example. The key configuration:

<VirtualHost *:443>
    ServerName your-server.example.com

    SSLEngine on
    SSLCertificateFile /path/to/certificate.crt
    SSLCertificateKeyFile /path/to/private.key

    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto "https"

    # All requests go through main app
    ProxyPass / http://127.0.0.1:8000/
    ProxyPassReverse / http://127.0.0.1:8000/
</VirtualHost>

Systemd Service with Environment Variables:

Create /etc/systemd/system/astrocat.service:

[Unit]
Description=AstroCat Web Interface
After=network.target

[Service]
Type=simple
User=your-user
WorkingDirectory=/path/to/astro_cat
Environment=ASTROCAT_HOST=127.0.0.1
Environment=ASTROCAT_PORT=8000
ExecStart=/path/to/astro_cat/venv/bin/python run_web.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl enable astrocat
sudo systemctl start astrocat

Note: When running behind a reverse proxy, secondary services bind to 127.0.0.1 by default for security. The ASTROCAT_BIND_HOST variable is rarely needed since the proxy handles external access.

Dashboard

The main dashboard shows:

Collection Statistics: Total files, sessions, integration time
Recent Activity: Latest imaging sessions and processed files
Storage Overview: Disk usage by session, camera, and telescope
Quick Actions: Common tasks and shortcuts

Imaging Sessions

View Sessions:

Click "Imaging Sessions" in the navigation
Browse sessions organized by date
Click a session to view details

Session Details:

List of all files in the session
Summary statistics (integration time, filter breakdown)
Links to processing sessions using these files
Option to edit session notes (Markdown)

Create Processing Session:

From session details, click "Create Processing Session"
Select files to include
Choose calibration frames
Name and save the processing session

File Browser

Search Files:

Click "Files" in navigation
Use filters:
- Object name
- Filter (L, R, G, B, Ha, etc.)
- Date range
- Telescope/Camera
- Frame type (Light, Dark, Flat, Bias)

File Details:

Full FITS metadata
Preview (if supported)
Download options
Associated session information

Processing Sessions

Create Processing Session:

Click "Processing Sessions" → "New Session"
Enter session details:
- Name (e.g., "M31 LRGB Stack")
- Primary target
- Target type (Galaxy, Nebula, etc.)
Select source files from imaging sessions
Save session

Session Workflow:

Select Files: Choose light, dark, flat, bias frames
Process: Use PixInsight or other software (external)
Catalog Results: Add processed files back to the session
Track Progress: Mark files as intermediate or final

Statistics

View detailed analytics:

File Counts: By object, filter, telescope, camera
Integration Time: Total hours, by target, by filter
Collection Growth: Charts showing acquisitions over time
Equipment Usage: Which gear produces the most data

💻 Command Line Usage

Basic Workflow

1. Scan for New Files

# Scan quarantine directory for new FITS files
python main.py scan

2. Catalog Files

# Extract metadata and add to database
python main.py catalog

3. List Imaging Sessions

# Show all imaging sessions
python main.py list imaging-sessions

# Show sessions from specific year
python main.py list imaging-sessions --year 2024

4. View File Details

# List files with filters
python main.py list files --object M31
python main.py list files --filter Ha
python main.py list files --date 2024-11-01

Advanced Commands

Processing Sessions

# Create a processing session
python main.py processing-session create "M31 LRGB" --target M31

# List processing sessions
python main.py list processing-sessions

# Add files to a processing session
python main.py processing-session add-files <session-id> --file-ids 1,2,3

Statistics

# Show collection statistics
python main.py stats summary

# Statistics by object
python main.py stats by-object

# Statistics by filter
python main.py stats by-filter

Validation

# Validate all files in database
python main.py validate all

# Check for missing files
python main.py verify files

# Check database integrity
python main.py verify database

Equipment Management

# List cameras
python main.py list cameras

# List telescopes
python main.py list telescopes

# Show equipment statistics
python main.py stats equipment

CLI Help

Get help for any command:

python main.py --help                    # List all commands
python main.py scan --help               # Scan command options
python main.py processing-session --help # Processing session commands

☁️ S3 Backup Configuration (Optional)

The S3 backup system provides automated, cost-effective cloud storage for your entire FITS library. With lifecycle management, you can store hundreds of gigabytes for as little as $10/year using AWS Glacier Deep Archive.

Quick Start

For detailed setup instructions, see:

S3 Quick Start Guide - Step-by-step setup in 5 steps
S3 Backup README - Comprehensive documentation

Basic Setup

1. AWS Prerequisites

Create an S3 bucket and configure AWS credentials:

# Create S3 bucket
aws s3 mb s3://your-bucket-name --region us-east-1

# Configure credentials (choose one method)
aws configure  # Interactive setup

# OR set environment variables
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key

2. Install Dependencies

pip install boto3

3. Configure S3 Settings

On first startup, s3_config.json is created automatically with S3 disabled. Edit this file to enable S3 backups:

nano s3_config.json  # Edit with your bucket name and region

Required changes in s3_config.json:

{
  "enabled": true,              # Change from false to true
  "aws_region": "us-east-1",    # Your AWS region
  "buckets": {
    "primary": "your-bucket-name"  # Your S3 bucket name
  }
}

4. Test Backup

# Upload a test session
python -m s3_backup.cli upload --session-id YOUR_SESSION_ID

# Check status
python -m s3_backup.cli status

# Verify upload
python -m s3_backup.cli verify --all

Lifecycle Management (Cost Optimization)

Automatically transition backups to low-cost storage with one command:

# Configure lifecycle rules (moves files to Deep Archive after 1 day)
python -m s3_backup.cli configure-lifecycle

# View current lifecycle rules
python -m s3_backup.cli show-lifecycle

# Estimate storage costs
python -m s3_backup.cli estimate-costs

Cost Savings Example:

827 GB in STANDARD storage: ~$19/month
827 GB in DEEP_ARCHIVE: ~$0.82/month
Annual savings: ~$218/year

Common Backup Commands

# Upload all sessions from a specific year
python -m s3_backup.cli upload --year 2024

# Upload all un-backed-up sessions
python -m s3_backup.cli upload

# List backed-up sessions
python -m s3_backup.cli list-sessions

# Verify backup integrity
python -m s3_backup.cli verify --all

# Backup database
python -m s3_backup.cli backup-database

S3 Backup Web Interface

Launch a dedicated web interface for backup management:

python -m s3_backup.web_app

Access at http://localhost:8083 for:

Backup dashboard and statistics
One-click session backups
Storage cost tracking
Upload progress monitoring
Verification and restore operations

Full Documentation

For complete details on:

Lifecycle policy customization
Restore procedures
Cost estimation
Troubleshooting
Advanced configuration

See the comprehensive guides in the s3_backup/ directory.

📂 Directory Structure

Organized Library Layout

/path/to/images/
├── 2024/
│   ├── 2024-11-01_M31/
│   │   ├── Light/
│   │   │   ├── M31_L_180s_001.fits
│   │   │   └── M31_L_180s_002.fits
│   │   ├── Dark/
│   │   ├── Flat/
│   │   └── Bias/
│   └── 2024-11-02_M42/
└── 2023/

Processing Directory Layout

/path/to/processing/
├── M31_LRGB_20241101/
│   ├── calibration/
│   │   ├── master_dark.fits
│   │   └── master_flat.fits
│   ├── intermediate/
│   │   ├── M31_L_stacked.xisf
│   │   └── M31_RGB_stacked.xisf
│   └── final/
│       ├── M31_LRGB.jpg
│       └── M31_LRGB.xisf

Session Notes

/path/to/session_notes/
├── Imaging_Sessions/
│   ├── 2024/
│   │   ├── 20241101_M31.md
│   │   └── 20241102_M42.md
└── Processing_Sessions/
    └── M31_LRGB_20241101.md

🔧 Common Workflows

New Imaging Session Workflow

Acquire images to quarantine directory
Scan for new files: python main.py scan
Catalog metadata: python main.py catalog
Review in web UI: Check session was created correctly
Add notes: Edit session notes in web UI or Obsidian
Optional: Backup to S3

Processing Workflow

Create processing session in web UI
Select source files from imaging sessions
Process externally (PixInsight, etc.)
Add processed files to session directory
Catalog results: Files auto-detected in processing directory
Track as final: Mark final outputs in web UI

Backup Workflow

Review backup status in S3 web UI (localhost:8083)
Select sessions to backup
Monitor upload progress
Verify backups with integrity checks
Optional: Set lifecycle policies for cost savings

🔍 Troubleshooting

Database Issues

Database locked error:

# Close any open connections to the database
# Check for stale lock files
rm -f /path/to/fits_catalog.db-journal

Rebuild statistics:

python scripts/fix_performance_post_migration.py

Check database integrity:

python scripts/diagnose_db_size.py

Performance Issues

Slow queries:

# Analyze database and rebuild indexes
python scripts/fix_performance_post_migration.py

Large database size:

# Check for duplicate indexes
python scripts/diagnose_db_size.py

# Remove duplicates
python scripts/remove_duplicate_indexes.py

Web Interface Issues

Port already in use:

# Change port in run_web.py or specify via environment
PORT=8001 python run_web.py

Can't connect to web interface:

Check firewall settings
Verify server is running: ps aux | grep run_web
Check logs in terminal output

📚 Additional Resources

File Monitoring

For automatic file intake, set up a cron job or systemd service:

# Cron example (every 30 minutes)
*/30 * * * * cd /path/to/astro_cat && ./venv/bin/python main.py scan && ./venv/bin/python main.py catalog

Obsidian Integration

Session notes are stored as Markdown files compatible with Obsidian:

Add notes directory to Obsidian vault
Use [[links]] to reference between sessions
Add tags for organization: #M31 #LRGB #stacked

PixInsight Integration

The file structure supports PixInsight workflows:

Processing sessions organize files by type
Use File Selector to find calibration frames
Export processed files to processing/<session>/final/
Files are auto-cataloged when placed in session directories

🤝 Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch
Make your changes
Submit a pull request

📜 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

Built for the astrophotography community
Optimized for large FITS file collections
Designed to integrate with popular processing tools (PixInsight, etc.)
Inspired by the need for better data management in amateur astrophotography

📞 Support

Issues: https://github.com/smartc/astro_cat/issues
Discussions: https://github.com/smartc/astro_cat/discussions
Documentation: See docs/ directory (coming soon)

Happy imaging! 🔭✨

Name		Name	Last commit message	Last commit date
Latest commit History 301 Commits
cli		cli
docs		docs
migrations		migrations
processed_catalog		processed_catalog
processing		processing
profiles		profiles
s3_backup		s3_backup
scripts		scripts
static		static
web		web
.gitignore		.gitignore
LICENSE		LICENSE
PHASE4_COMPLETE.md		PHASE4_COMPLETE.md
README.md		README.md
THIRD-PARTY-LICENSES.txt		THIRD-PARTY-LICENSES.txt
astro_cat.service		astro_cat.service
astrobin_filters.json		astrobin_filters.json
cameras.json		cameras.json
config.json.template		config.json.template
config.py		config.py
equipment_manager.py		equipment_manager.py
file_monitor.py		file_monitor.py
file_organizer.py		file_organizer.py
file_selector.py		file_selector.py
filters.json		filters.json
fits_processor.py		fits_processor.py
install_frontend_libs.py		install_frontend_libs.py
install_service.sh		install_service.sh
main.py		main.py
manage_profiles.py		manage_profiles.py
models.py		models.py
object_processor.py		object_processor.py
processing_session_manager.py		processing_session_manager.py
requirements.txt		requirements.txt
rescan_extended_metadata.py		rescan_extended_metadata.py
run_web.py		run_web.py
setup.sh		setup.sh
telescopes.json		telescopes.json
test_phase4_models.py		test_phase4_models.py
uninstall_service.sh		uninstall_service.sh
validation.py		validation.py
version.py		version.py
webdav_server.py		webdav_server.py

License

smartc/astro_cat

Folders and files

Latest commit

History

Repository files navigation

astro_cat

🚀 Features

Core Functionality

Web Interface

Command Line Tools

Cloud Backup (Optional)

📋 Requirements

⚙️ Installation

1. Clone the Repository

2. Create Virtual Environment

3. Install Dependencies

4. Install Frontend Libraries

5. Verify Installation

🛠️ Initial Configuration

1. Configure Paths

2. Configure Equipment (Optional)

3. Initialize Database

🌐 Web Interface Usage

Starting the Web Server

Environment Variables

Running Behind a Reverse Proxy (HTTPS)

Dashboard

Imaging Sessions

File Browser

Processing Sessions

Statistics

💻 Command Line Usage

Basic Workflow

Advanced Commands

CLI Help

☁️ S3 Backup Configuration (Optional)

Quick Start

Basic Setup

1. AWS Prerequisites

2. Install Dependencies

3. Configure S3 Settings

4. Test Backup

Lifecycle Management (Cost Optimization)

Common Backup Commands

S3 Backup Web Interface

Full Documentation

📂 Directory Structure

Organized Library Layout

Processing Directory Layout

Session Notes

🔧 Common Workflows

New Imaging Session Workflow

Processing Workflow

Backup Workflow

🔍 Troubleshooting

Database Issues

Performance Issues

Web Interface Issues

📚 Additional Resources

File Monitoring

Obsidian Integration

PixInsight Integration

🤝 Contributing

📜 License

🙏 Acknowledgments

📞 Support

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 3

Uh oh!

Languages

Packages