diff --git a/ARCHITECTURE_OVERVIEW.md b/ARCHITECTURE_OVERVIEW.md new file mode 100644 index 00000000000..48b680759ef --- /dev/null +++ b/ARCHITECTURE_OVERVIEW.md @@ -0,0 +1,505 @@ +# Huly Platform Architecture Overview + +## Service Overview + +The Huly platform consists of **30+ microservices** working together in a distributed architecture. Services are organized into functional layers for core business logic, data storage, real-time communication, media processing, and supporting features. + +### Core Backend Services + +| Service | Port | Description | +|---------|------|-------------| +| **account** | 3000 | Authentication and user management. Handles login, registration, JWT tokens, workspace membership, and user permissions. | +| **transactor** | 3332 | Core transaction processing engine. Maintains WebSocket connections for real-time updates, processes all data mutations, enforces business logic, and publishes events to the message queue. | +| **workspace** | - | Workspace lifecycle management. Handles workspace creation, initialization, upgrades, and maintenance background jobs. | +| **stats** | 4900 | Metrics collection and aggregation. Collects usage statistics and health metrics from all services for monitoring. | + +### Storage & Data Services + +| Service | Port | Description | +|---------|------|-------------| +| **datalake** | 4030 | Blob storage management with metadata. Handles file uploads, permissions, and coordinates with MinIO for object storage. | +| **hulylake** | 8096 | Storage adapter API. Provides S3-compatible interface for accessing stored objects. | +| **hulykvs** | 8094 | Key-value store service. Fast storage for configuration, preferences, and cached data. | + +### Search & Indexing + +| Service | Port | Description | +|---------|------|-------------| +| **fulltext** | 4702 | Full-text search indexing. Consumes events, extracts content, maintains Elasticsearch index for fast searches. | +| **rekoni** | 4004 | Document intelligence service. Extracts text and structured data from binary documents (PDF, DOC, DOCX, RTF, HTML) for searchability and resume parsing. Supports HeadHunter, LinkedIn, and generic resume formats. | + +### Real-time Services + +| Service | Port | Description | +|---------|------|-------------| +| **collaborator** | 3078 | Real-time document collaboration using Y.js CRDT. Enables simultaneous editing with conflict resolution. | +| **hulypulse** | 8099 | WebSocket notification server. Handles real-time push notifications to connected clients. | +| **hulygun** | - | Event processor. Consumes and processes events from Redpanda for real-time updates. | + +### Media Services + +| Service | Port | Description | +|---------|------|-------------| +| **stream** | 1080 | Video streaming service with HLS transcoding for recording playback. | +| **media** | - | Media processing queue. Handles video/audio conversion and optimization. | +| **preview** | 4040 | Thumbnail and preview generation for images and documents. | + +### Feature Services + +| Service | Port | Description | +|---------|------|-------------| +| **print** | 4005 | PDF generation service for printing documents and reports. | +| **sign** | 4006 | Digital signature service for document signing with certificates. | +| **payment** | 3040 | Payment and billing integration (Polar). Handles subscriptions and payments. | +| **export** | 4009 | Data export service for workspace backups and data extraction. | +| **analytics** | 4017 | Analytics event collection and processing. | +| **process** | - | Workflow automation and process orchestration. | +| **rating** | - | Content rating and quality metrics. | + +### Backup Services + +| Service | Port | Description | +|---------|------|-------------| +| **backup** | - | Automated backup service. Periodically backs up workspace data to object storage. | +| **backup-api** | 4039 | Backup REST API. Provides endpoints for backup management and restoration. | + +### Frontend + +| Service | Port | Description | +|---------|------|-------------| +| **front** | 8087/8088 | Web application server. Serves the Huly UI, handles static assets and client-side routing. | + +### Infrastructure (Databases & Message Queues) + +| Service | Port(s) | Description | +|---------|---------|-------------| +| **cockroach** | 26257, 8089 | **CockroachDB - Primary Application Database**. Stores ALL business data: users, workspaces, documents, transactions, metadata, permissions. Distributed SQL with ACID guarantees. | +| **elastic** | 9200 | Elasticsearch search engine. Stores full-text search indexes managed by fulltext service. | +| **minio** | 9000, 9001 | S3-compatible object storage. Stores binary files, attachments, images, and blobs in buckets (blobs, eu, backups). | +| **redpanda** | 9092, 19092 | Kafka-compatible event streaming. Provides reliable async messaging between services. | +| **redis** | 6379 | In-memory cache and pub/sub. Used by HulyPulse for real-time notifications. | + +### Monitoring & Observability + +| Service | Port | Description | +|---------|------|-------------| +| **jaeger** | 16686, 4318 | Distributed tracing UI and OTLP collector. All services send traces for performance monitoring. | +| **redpanda_console** | 8000 | Kafka management UI for monitoring topics, consumers, and message flow. | + +### Service Communication Patterns + +- **Synchronous (HTTP/WebSocket)**: Client ↔ Front ↔ Backend Services +- **Asynchronous (Events)**: Producers (Transactor, Workspace) → Redpanda → Consumers (Fulltext, Media, Process) +- **Primary Database**: All services → CockroachDB (main application data) +- **Search Index**: Fulltext → Elasticsearch +- **Object Storage**: Services → MinIO (S3 API) +- **Cache/Pub-Sub**: HulyPulse → Redis +- **Real-time Updates**: Client ↔ Transactor (WebSocket), Client ↔ Collaborator (WebSocket) + +--- + +## 1. High-Level System Architecture + +```mermaid +graph TB + subgraph "Client Layer" + Browser[Web Browser] + Desktop[Desktop App] + end + + subgraph "Frontend" + Front[Front Server
:8087] + end + + subgraph "Core Backend Services" + Account[Account
:3000
Auth & Users] + Workspace[Workspace
Management] + Transactor[Transactor
:3332
Transactions] + Stats[Stats
:4900
Metrics] + end + + subgraph "Storage Services" + Datalake[Datalake
:4030
Blob Storage] + Hulylake[Hulylake
:8096
Storage API] + HulyKVS[HulyKVS
:8094
Key-Value] + end + + subgraph "Search & Indexing" + Fulltext[Fulltext
:4702
Search] + Rekoni[Rekoni
:4004
Doc Intelligence] + end + + subgraph "Real-time" + Collaborator[Collaborator
:3078
Doc Sync] + Pulse[HulyPulse
:8099
WebSocket] + Gun[HulyGun
Events] + end + + subgraph "Media Services" + Stream[Stream
:1080
Video] + MediaProc[Media
Processing] + Preview[Preview
:4040
Thumbnails] + end + + subgraph "Feature Services" + Print[Print
:4005] + Sign[Sign
:4006] + Payment[Payment
:3040] + Export[Export
:4009] + Analytics[Analytics
:4017] + Process[Process
Automation] + Rating[Rating
Metrics] + end + + subgraph "Backup Services" + BackupSvc[Backup
Service] + BackupAPI[Backup API
:4039] + end + + subgraph "Primary Database" + CockroachDB[(CockroachDB
:26257
Main Application DB)] + end + + subgraph "Supporting Infrastructure" + Elasticsearch[(Elasticsearch
:9200)] + Minio[(MinIO
:9000)] + Redpanda[Redpanda
:9092
Kafka] + Redis[(Redis
:6379)] + end + + Browser --> Front + Desktop --> Front + + Front --> Account + Front --> Transactor + Front --> Collaborator + Front --> Datalake + Front --> Pulse + + Account --> CockroachDB + Workspace --> CockroachDB + Transactor --> CockroachDB + Transactor --> Redpanda + Transactor --> Fulltext + + Datalake --> CockroachDB + Datalake --> Minio + Hulylake --> CockroachDB + Hulylake --> Minio + HulyKVS --> CockroachDB + + Fulltext --> Elasticsearch + Fulltext --> CockroachDB + Fulltext --> Rekoni + Fulltext --> Redpanda + + Pulse --> Redis + Gun --> Redpanda + + Stream --> Datalake + MediaProc --> Redpanda + Preview --> Datalake + + style Front fill:#4A90E2 + style Account fill:#E24A4A + style Transactor fill:#E24A4A + style CockroachDB fill:#7ED321 + style Redpanda fill:#F5A623 +``` + +--- + + + +## 2. Event-Driven Architecture (Redpanda/Kafka) + +```mermaid +graph LR + subgraph "Event Producers" + Transactor[Transactor
Transactions] + Workspace[Workspace
Workspaces] + Rating[Rating
Ratings] + end + + subgraph "Event Bus" + Redpanda[Redpanda
Kafka Topics
:9092] + end + + subgraph "Event Consumers" + Fulltext[Fulltext
Search Indexing] + Media[Media
Processing] + Process[Process
Automation] + Gun[HulyGun
Event Processor] + BackupSvc2[Backup
Archival] + end + + subgraph "Queue Configuration" + QC["QUEUE_CONFIG
cockroach / redpanda:9092
Region-based routing"] + end + + Transactor -->|Document Events| Redpanda + Transactor -->|User Actions| Redpanda + Workspace -->|Workspace Events| Redpanda + Rating -->|Rating Events| Redpanda + + Redpanda -->|Index Events| Fulltext + Redpanda -->|Media Events| Media + Redpanda -->|Process Events| Process + Redpanda -->|All Events| Gun + Redpanda -->|Backup Events| BackupSvc2 + + QC -.Config.-> Transactor + QC -.Config.-> Workspace + QC -.Config.-> Fulltext + QC -.Config.-> Media + QC -.Config.-> Process + QC -.Config.-> BackupSvc2 + + style Redpanda fill:#F5A623 + style Transactor fill:#E24A4A +``` + +--- + +## 3. Storage Architecture + +```mermaid +graph TB + subgraph "Client Layer" + Client[Client Application] + end + + subgraph "Storage APIs" + Datalake[Datalake :4030
Blob Management] + Hulylake[Hulylake :8096
Storage Adapter] + end + + subgraph "Primary Database" + CockroachDB[(CockroachDB
File Metadata
Permissions
References)] + end + + subgraph "Object Storage" + Minio[(MinIO :9000
S3-Compatible)] + Buckets[Buckets:
• blobs
• eu
• backups] + end + + subgraph "Media Processing" + Preview[Preview :4040
Thumbnails] + Stream[Stream :1080
Video Streaming] + Media[Media
Processing Queue] + end + + Client -->|Upload| Datalake + Client -->|Read| Hulylake + Client -->|Stream Video| Stream + Client -->|Get Preview| Preview + + Datalake -->|Metadata| CockroachDB + Datalake -->|Store Blobs| Minio + + Hulylake -->|Metadata| CockroachDB + Hulylake -->|Access Blobs| Minio + + Minio --> Buckets + + Preview -->|Read| Datalake + Preview -->|Cache| Minio + + Stream -->|Upload| Datalake + Stream -->|Transcode| Media + + Media -->|Store| Minio + + style Datalake fill:#4A90E2 + style Minio fill:#C92A2A + style CockroachDB fill:#7ED321 +``` + +--- + +## 4. Authentication & Authorization Flow + +```mermaid +sequenceDiagram + participant Client + participant Front + participant Account + participant Transactor + participant Workspace + participant CockroachDB + + Client->>Front: Login Request + Front->>Account: Authenticate + Account->>CockroachDB: Verify Credentials + CockroachDB-->>Account: User Record + Account->>Account: Generate JWT Token
(SERVER_SECRET=secret) + Account-->>Front: JWT Token + Front-->>Client: Token + Workspace List + + Client->>Front: Connect to Workspace + Front->>Account: Verify Token + Account-->>Front: Token Valid + User Info + + Front->>Workspace: Get Workspace Info + Workspace->>CockroachDB: Query Workspace + CockroachDB-->>Workspace: Workspace Data + Workspace-->>Front: Workspace Config + + Client->>Transactor: WebSocket Connect
with Token + Transactor->>Account: Verify Token + Account-->>Transactor: User Authorized + Transactor->>CockroachDB: Load User Permissions + Transactor-->>Client: Connected + + Note over Client,CockroachDB: All services share SERVER_SECRET=secret
for internal authentication +``` + +--- + +## Service Summary Table + +| Service | Container | Port | Purpose | Dependencies | +|---------|-----------|------|---------|--------------| +| **Frontend** | | | | | +| front | hardcoreeng/front | 8087/8088 | Web application server | account, transactor, collaborator, datalake | +| **Core** | | | | | +| account | hardcoreeng/account | 3000 | Authentication & user management | cockroach, redpanda, stats | +| transactor | hardcoreeng/transactor | 3332 | Transaction processing (WebSocket) | cockroach, redpanda, fulltext, account | +| workspace | hardcoreeng/workspace | - | Workspace management | cockroach, redpanda, minio, account | +| stats | hardcoreeng/stats | 4900 | Metrics collection | - | +| **Storage** | | | | | +| datalake | hardcoreeng/datalake | 4030 | Blob storage & metadata | cockroach, minio, account | +| hulylake | hardcoreeng/hulylake | 8096 | Storage adapter API | cockroach, minio | +| hulykvs | hardcoreeng/hulykvs | 8094 | Key-value store | cockroach | +| **Search** | | | | | +| fulltext | hardcoreeng/fulltext | 4702 | Full-text search indexing | elasticsearch, cockroach, rekoni, redpanda | +| rekoni | hardcoreeng/rekoni-service | 4004 | Document intelligence | stats | +| **Real-time** | | | | | +| collaborator | hardcoreeng/collaborator | 3078 | Real-time document collaboration | account, datalake, transactor | +| hulypulse | hardcoreeng/hulypulse | 8099 | WebSocket notifications | redis | +| hulygun | hardcoreeng/hulygun | - | Event processor | redpanda, account | +| **Media** | | | | | +| stream | hardcoreeng/stream | 1080 | Video streaming | datalake, redpanda | +| media | hardcoreeng/media | - | Media processing | redpanda, account | +| preview | hardcoreeng/preview | 4040 | Thumbnail generation | datalake | +| **Features** | | | | | +| print | hardcoreeng/print | 4005 | PDF generation | cockroach, minio, account | +| sign | hardcoreeng/sign | 4006 | Digital signatures | cockroach, minio, account | +| payment | hardcoreeng/payment | 3040 | Payment processing | account | +| export | hardcoreeng/export | 4009 | Data export | cockroach, minio, account | +| analytics | hardcoreeng/analytics-collector | 4017 | Analytics collection | account, stats | +| process | hardcoreeng/process | - | Workflow automation | redpanda, account | +| rating | hardcoreeng/rating | - | Content rating | cockroach, redpanda, account | +| **Backup** | | | | | +| backup | hardcoreeng/backup | - | Automated backup | cockroach, minio, account | +| backup-api | hardcoreeng/backup-api | 4039 | Backup REST API | minio, account | +| **Primary Database** | | | | | +| cockroach | cockroachdb/cockroach:latest-v24.3 | 26257, 8089 | **Main application database** - stores users, workspaces, documents, transactions, metadata, permissions | - | +| **Supporting Infrastructure** | | | | | +| elastic | elasticsearch:7.14.2 | 9200 | Search engine for full-text indexes | - | +| minio | minio/minio | 9000, 9001 | Object storage (S3) for files and blobs | - | +| redpanda | redpandadata/redpanda:v24.3.6 | 9092, 19092 | Event streaming (Kafka) for async processing | - | +| redis | redis:8.0.2-alpine3.21 | 6379 | Cache & pub/sub for real-time features | - | +| **Monitoring** | | | | | +| jaeger | jaegertracing/all-in-one | 16686, 4318 | Distributed tracing and performance monitoring | - | +| redpanda_console | redpandadata/console:v2.8.3 | 8000 | Kafka management UI | redpanda | + +--- + +## Environment Variables Summary + +### Common Configuration (Shared by Most Services) +- `SERVER_SECRET` / `SECRET`: `secret` - Shared authentication secret +- `REGION`: `cockroach` - Deployment region identifier +- `ACCOUNTS_URL`: `http://huly.local:3000` - Account service URL +- `STATS_URL`: `http://huly.local:4900` - Metrics collection URL +- `OTEL_EXPORTER_OTLP_ENDPOINT`: `http://jaeger:4318/v1/traces` - Tracing endpoint +- `STORAGE_CONFIG`: `${STORAGE_CONFIG}` - MinIO connection string +- `QUEUE_CONFIG`: `${QUEUE_CONFIG}` - Redpanda/Kafka configuration + +### Database Configuration +- `DB_URL` / `DB_CR_URL`: CockroachDB connection string +- `FULLTEXT_DB_URL`: `http://huly.local:9200` - Elasticsearch URL +- `HULY_DB_CONNECTION`: CockroachDB connection for Huly* services + +### Storage Configuration +- `STORAGE_CONFIG`: MinIO configuration (format: `minio|minio?accessKey=minioadmin&secretKey=minioadmin`) +- `AWS_ENDPOINT_URL`: `http://minio:9000` - S3-compatible endpoint +- `AWS_ACCESS_KEY_ID`: `minioadmin` +- `AWS_SECRET_ACCESS_KEY`: `minioadmin` +- `BUCKETS`: `blobs,eu|http://minio:9000?accessKey=minioadmin&secretKey=minioadmin` - Datalake bucket configuration + +### Queue Configuration +- `QUEUE_CONFIG`: `cockroach|http://redpanda:9092` - Region-based event routing +- `HULY_KAFKA_BOOTSTRAP`: `redpanda:9092` - Kafka bootstrap servers + +### Service URLs (Internal) +- `ACCOUNTS_URL`: `http://huly.local:3000` +- `TRANSACTOR_URL`: `ws://huly.local:3332` +- `FULLTEXT_URL`: `http://huly.local:4702` +- `REKONI_URL`: `http://huly.local:4004` +- `COLLABORATOR_URL`: `ws://huly.local:3078` +- `DATALAKE_URL`: `http://huly.local:4030` +- `HULYLAKE_URL`: `http://huly.local:8096` +- `PULSE_URL`: `ws://huly.local:8099/ws` +- `PREVIEW_URL`: `http://huly.local:4040` +- `STREAM_URL`: `http://huly.local:1080/recording` +- `PAYMENT_URL`: `http://huly.local:3040` +- `PRINT_URL`: `http://huly.local:4005` +- `SIGN_URL`: `http://huly.local:4006` +- `BACKUP_URL`: `http://huly.local:4039/api/backup` +- `AI_BOT_URL`: `http://huly.local:4010` + +### Frontend Configuration +- `FILES_URL`: `http://huly.local:4030/blob/:workspace/:blobId/:filename` - File download URL pattern +- `FRONT_URL`: `http://huly.local:8087` - Frontend base URL +- `BRANDING_URL`: `http://huly.local:8087/branding.json` +- `DESKTOP_UPDATES_URL`: `https://dist.huly.io` + +### Authentication & Security +- `HULY_TOKEN_SECRET`: `secret` - Token signing for Huly services +- `SERVER_SECRET`: `secret` - Service-to-service auth +- `ADMIN_EMAILS`: Admin user emails +- `LAST_NAME_FIRST`: `true` - Name formatting preference + +### Feature Flags +- `COMMUNICATION_API_ENABLED`: `true` +- `COMMUNICATION_TIME_LOGGING_ENABLED`: `true` +- `ENABLE_COMPRESSION`: `true` - Transactor compression + +### Rate Limiting (Transactor) +- `RATE_LIMIT_MAX`: `250` - Requests per window +- `RATE_LIMIT_WINDOW`: `30000` - 30 seconds + +### Workspace Configuration +- `WS_OPERATION`: `all+backup` - Operation mode +- `WORKSPACE_LIMIT_PER_USER`: `10000` +- `REGION_INFO`: `cockroach|CockroachDB` - Available regions + +### Backup Configuration +- `BUCKET_NAME`: `backups` +- `BACKUP_STORAGE`: `${BACKUP_STORAGE_CONFIG}` +- `INTERVAL`: `60` - Backup interval in seconds + +### Redis Configuration (HulyPulse) +- `HULY_REDIS_URLS`: `redis://redis:6379` +- `HULY_BIND_PORT`: `8099` + +### Stream Service +- `STREAM_ENDPOINT_URL`: `datalake://huly.local:4030` +- `STREAM_INSECURE`: `true` +- `STREAM_MAX_PARALLEL_SCALING_COUNT`: `6` + +--- + +## Troubleshooting Guide + +### Service Health Verification + +```bash +# Check all services +docker ps + +# Check individual service logs +docker logs -f [container_id] +``` diff --git a/README.md b/README.md index b16a7a81748..5f72b7871be 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,26 @@ You can find API usage examples in the [Huly examples](https://github.com/hcengi For detailed information about changes, improvements, and bug fixes in each version, see our [Changelog](./changelog.md). +## Versions + +The Huly Platform uses two types of version tags to distinguish between production-ready and development releases: + +- **Production Versions (`v*`)** - Stable releases for end users + - Example: `v0.7.310`, `v0.7.307`, `v0.6.501` + - These versions are recommended for production deployments + - Suitable for self-hosted installations + - Published with release notes on [GitHub Releases](https://github.com/hcengineering/platform/releases) + +- **Development Versions (`s*`)** - Pre-release builds for developers + - Example: `s0.7.313`, `s0.7.292`, `s0.7.288` + - Used for development and testing purposes + - May contain experimental features or bug fixes + - Not recommended for production use + +## Architecture + +For detailed information about the platform architecture, services, and their interactions, see our [Architecture Overview](./ARCHITECTURE_OVERVIEW.md). + ## Table of Contents - [Huly Platform](#huly-platform) @@ -40,6 +60,8 @@ For detailed information about changes, improvements, and bug fixes in each vers - [Activity](#activity) - [API Client](#api-client) - [Changelog](#changelog) + - [Versions](#versions) + - [Architecture](#architecture) - [Table of Contents](#table-of-contents) - [Pre-requisites](#pre-requisites) - [Verification](#verification)