UniLLM

A unified AI gateway system that provides standardized API interfaces for multiple inference engines, while offering enterprise-grade management and monitoring capabilities.

Core Features

• Unified Interface: Provides standard APIs compatible with OpenAI and Anthropic
• Multi-engine Support: Supports mainstream inference frameworks such as vLLM, SGLang, MindIE, Ollama, etc.
• Enterprise-grade Features: Includes complete functionality such as authentication, monitoring, tenant management, and traffic control
• Flexible Extension: Modular design for easy addition of new inference engines and features

Tech Stack

• Python 3.11+
• FastAPI
• PostgreSQL 15+
• Redis 7+
• Elasticsearch 8+ (optional, for log storage)

Quick Start

Prerequisites

• Python 3.11+
• PostgreSQL 15+
• Redis 7+

Install Dependencies

pip install -r requirements.txt

Configure Environment Variables

cp .env.example .env
# Edit the .env file to configure database connection and other settings

Initialize Database

alembic upgrade head

Start Development Server

uvicorn src.main:app --reload --host 0.0.0.0 --port 8000

Access API Documentation

• Swagger UI: http://localhost:8000/docs
• Redoc: http://localhost:8000/redoc

Directory Structure

src/                    # Source code directory
├── api/                # API route layer
├── core/               # Core functionality layer
├── adapters/           # Engine adapter layer
├── models/             # Data model layer
├── database/           # Data access layer
├── config/             # Configuration management
└── main.py             # Application entry point

Development Guide

Code Style

Use Black and isort for code formatting:

black .
isort .

Run Tests

pytest

Run Code Checks

flake8 .
mypy .

Deployment

See Deployment Documentation for details.

License

MIT