AI File Organizer Backend

A powerful AI-powered file processing and organization system that can parse, summarize, and organize files using various AI models (OpenAI, Ollama, local embeddings) with semantic search capabilities.

Features

Multi-format File Processing: Parse various file formats (PDF, DOCX, TXT, Markdown, etc.)
AI-Powered Summarization: Generate intelligent summaries using OpenAI or local Ollama models
Semantic Search: Vector-based search using embeddings (OpenAI or local models)
Database Storage: SQLite with vector extensions for efficient storage and retrieval
Flexible AI Providers: Support for OpenAI API, local Ollama, and local embedding models
RESTful API: Complete REST API for all file operations
Hierarchical Organization: Analyze and organize files in folder structures

Quick Start

1. Prerequisites

Python 3.12+
uv (recommended package manager)
Ollama (optional, for local models)
OpenAI API Key (optional, for OpenAI models)

2. Installation

# Clone the repository
git clone <repository-url>
cd backend

# Install dependencies with uv
uv sync

# Or with pip
pip install -r requirements.txt

3. Configuration

Copy the environment template and configure it:

cp .env.template .env

Edit .env file with your settings:

# Database Configuration
BACKEND_DATABASE_PATH="backend/db/app.db"

# AI Provider Selection (auto, ollama, online)
AI_PROVIDER="auto"

# OpenAI Configuration (if using OpenAI)
OPENAI_API_KEY="sk-your-api-key-here"
ONLINE_MODEL="openai/gpt-4o-mini"

# Ollama Configuration (if using local models)
OLLAMA_HOST="http://localhost:11434"
OLLAMA_MODEL="gemma3:4b"

# Embedding Configuration
EMBEDDING_PROVIDER="auto"
EMBEDDING_MODEL="text-embedding-3-small"
LOCAL_EMBEDDING_MODEL="Qwen/Qwen3-Embedding-0.6B"

4. Setup Ollama (Optional)

If you want to use local models:

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Start Ollama service
ollama serve

# Pull a model (in another terminal)
ollama pull gemma3:4b

5. Run the Application

# Start the application
uv run python -m src.app

# Or with uvicorn directly (using configured port from .env)
uv run uvicorn src.app:app --host 127.0.0.1 --port 8347 --reload

The API will be available at: http://localhost:8347 (or the port configured in .env)

6. Test the Setup

# Quick test of core functionality
uv run tests/quick_test.py

# Run comprehensive tests
uv run tests/test_comprehensive_capabilities.py

# Test API health
curl http://localhost:8347/health

7. Run Complete Demo

# Run the complete pipeline demo
uv run tests/demo_pipeline.py

This demo will:

Create test files with different content types
Process files through the AI pipeline
Generate summaries and embeddings
Test semantic search functionality
Display results and statistics

8. Visualize Database

# Launch SQLite database visualizer
uv run src/visualizer/db_visualizer.py

# Create demo database if needed
uv run src/visualizer/db_visualizer.py --create-demo

# Open in read-only mode
uv run src/visualizer/db_visualizer.py --readonly

The database visualizer will be available at http://localhost:8348 (or the port configured in .env)

📚 Documentation

All documentation is available in the /docs folder:

Postman Collection - Complete API testing collection
Postman Guide - Step-by-step testing instructions
Architecture - System architecture documentation
Implementation Plan - Detailed implementation notes
Pipeline Demo Guide - Complete pipeline demonstration guide

🧪 Testing & Demo

Quick Pipeline Demo

Run a comprehensive demonstration of the entire AI file processing pipeline:

uv run test_pipeline_demo.py

This demo will:

✅ Create test files with different content types
✅ Process files through the complete AI pipeline
✅ Generate intelligent summaries and extract keywords
✅ Test semantic search functionality
✅ Display detailed debugging information
✅ Show before/after database states

What You'll See

File Processing: Real-time processing of TXT, JSON, and Markdown files
AI Summarization: Intelligent summaries generated using Ollama/OpenAI
Keyword Extraction: Automatic metadata extraction
Duplicate Detection: Content-based deduplication in action
Database Monitoring: Complete visibility into the processing pipeline
Performance Metrics: Processing times and system statistics

API Documentation

Base URL

http://localhost:8347

(Or the port configured in your .env file)

Authentication

Currently, no authentication is required. In production, implement proper authentication.

Core Endpoints

1. Health Check

GET /health

2. File Operations

Upload and Process File

POST /api/files/process
Content-Type: multipart/form-data

Form Data:
- file: (file upload)
- force: boolean (optional, default: false)
- generate_embeddings: boolean (optional, default: true)

Get File Details

GET /api/files/{file_id}

List All Files

GET /api/files
Query Parameters:
- limit: int (optional, default: 50)
- offset: int (optional, default: 0)

Delete File

DELETE /api/files/{file_id}

3. Folder Operations

Process Folder

POST /api/folders/process
Content-Type: application/json

Body:
{
  "folder_path": "/path/to/folder",
  "recursive": true,
  "force": false,
  "generate_embeddings": true
}

Analyze Folder Hierarchy

POST /api/folders/analyze
Content-Type: application/json

Body:
{
  "folder_path": "/path/to/folder"
}

4. Search Operations

Semantic Search

GET /api/search
Query Parameters:
- query: string (required)
- limit: int (optional, default: 10)
- semantic_weight: float (optional, default: 0.7)
- keyword_weight: float (optional, default: 0.3)

Find Similar Files

GET /api/files/{file_id}/similar
Query Parameters:
- limit: int (optional, default: 10)
- threshold: float (optional, default: 0.8)

Search Suggestions

GET /api/search/suggestions
Query Parameters:
- query: string (required)
- limit: int (optional, default: 5)

5. Statistics and Information

Database Statistics

GET /api/stats

System Information

GET /api/info

Available AI Providers

GET /api/providers

Usage Examples

1. Process a Single File

# Upload and process a document
curl -X POST "http://localhost:8347/api/files/process" \
  -F "file=@/path/to/document.pdf" \
  -F "force=false" \
  -F "generate_embeddings=true"

2. Process a Folder

# Process all files in a folder
curl -X POST "http://localhost:8347/api/folders/process" \
  -H "Content-Type: application/json" \
  -d '{
    "folder_path": "/Users/username/Documents",
    "recursive": true,
    "force": false,
    "generate_embeddings": true
  }'

3. Search Files

# Semantic search
curl "http://localhost:8347/api/search?query=machine%20learning&limit=5"

# Find similar files
curl "http://localhost:8347/api/files/1/similar?limit=5"

4. Get File Information

# Get file details
curl "http://localhost:8347/api/files/1"

# List all files
curl "http://localhost:8347/api/files?limit=10&offset=0"

Environment Variables Reference

Variable	Description	Default
`BACKEND_DATABASE_PATH`	Database file path	`backend/db/app.db`
`AI_PROVIDER`	AI provider selection	`auto`
`OPENAI_API_KEY`	OpenAI API key	-
`ONLINE_MODEL`	Online model name	`openai/gpt-4o-mini`
`OLLAMA_HOST`	Ollama server URL	`http://localhost:11434`
`OLLAMA_MODEL`	Ollama model name	`gemma3:4b`
`EMBEDDING_PROVIDER`	Embedding provider	`auto`
`EMBEDDING_MODEL`	OpenAI embedding model	`text-embedding-3-small`
`LOCAL_EMBEDDING_MODEL`	Local embedding model	`Qwen/Qwen3-Embedding-0.6B`
`LOG_LEVEL`	Logging level	`INFO`
`MAX_FILE_SIZE_MB`	Max file size for processing	`100`
`MAX_CONCURRENT_FILES`	Max concurrent worker threads	`3`
`SERVER_HOST`	Server host address	`127.0.0.1`
`SERVER_PORT`	Server port number	`8347`
`DB_VISUALIZER_PORT`	Database visualizer port	`8348`

Troubleshooting

Common Issues

Database Connection Error
- Check database path permissions
- Ensure SQLite and sqlite-vec are properly installed
Ollama Connection Failed
- Verify Ollama is running: ollama serve
- Check model availability: ollama list
OpenAI API Error
- Verify API key is set correctly
- Check API key permissions and billing
File Processing Failures
- Check file format support
- Verify file permissions and size limits

Testing

# Run all tests
uv run tests/test_comprehensive_capabilities.py

# Test specific functionality
uv run python -c "
from src.parser.parser import FileParser
parser = FileParser()
print('Supported formats:', parser.get_supported_extensions())
"

Development

Project Structure

backend/
├── src/                    # Source code
│   ├── app.py              # Main application
│   ├── db/                 # Database layer
│   ├── models/             # Data models
│   ├── parser/             # File parsing
│   ├── summarizer/         # AI summarization
│   ├── embeddings/         # Vector embeddings
│   ├── search/             # Search functionality
│   ├── processor/          # File processing pipeline
│   ├── organizer/          # File organization
│   └── visualizer/         # Database visualization
├── tests/                  # Test files and demos
├── docs/                   # Documentation
├── requirements.txt        # Dependencies
└── .env.template          # Environment template

Adding New Features

New File Format Support: Extend FileParser class
New AI Provider: Implement BaseSummarizer interface
New Search Method: Extend HybridSearcher class
New API Endpoints: Add routes in app.py

Contributing

Fork the repository
Create a feature branch
Make your changes
Add tests
Submit a pull request

License

[License information here]

Name		Name	Last commit message	Last commit date
Latest commit History 52 Commits
.claude		.claude
acl-style-files-master		acl-style-files-master
docs		docs
logs		logs
main		main
migrations		migrations
tests		tests
ui		ui
.env.template		.env.template
.gitignore		.gitignore
README.md		README.md
pyproject.toml		pyproject.toml
run_api.py		run_api.py
uv.lock		uv.lock

cosmasense/sail2025

Folders and files

Latest commit

History

Repository files navigation