diff --git a/.gitignore b/.gitignore index bda9ae19..08ac8dbd 100644 --- a/.gitignore +++ b/.gitignore @@ -157,3 +157,5 @@ CLAUDE.md .agent-os/ .cursorrules .claude/ +.claude-flow/ +.swarm/ diff --git a/README.md b/README.md index 429d021e..66abbb43 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) -[![Container](https://img.shields.io/badge/Container-quay.io-red)](https://quay.io/repository/myakove/github-webhook-server) +[![Container](https://img.shields.io/badge/Container-ghcr.io-red)](https://ghcr.io/myk-org/github-webhook-server) [![FastAPI](https://img.shields.io/badge/FastAPI-009688?logo=fastapi&logoColor=white)](https://fastapi.tiangolo.com) [![Python](https://img.shields.io/badge/Python-3.12+-3776ab?logo=python&logoColor=white)](https://python.org) @@ -17,6 +17,7 @@ A comprehensive [FastAPI-based](https://fastapi.tiangolo.com) webhook server for - [Deployment](#deployment) - [Usage](#usage) - [API Reference](#api-reference) +- [Log Viewer](#log-viewer) - [User Commands](#user-commands) - [OWNERS File Format](#owners-file-format) - [Security](#security) @@ -104,7 +105,6 @@ GitHub Events → Webhook Server → Repository Management Your GitHub App requires the following permissions: - **Repository permissions:** - - `Contents`: Read & Write - `Issues`: Read & Write - `Pull requests`: Read & Write @@ -113,7 +113,6 @@ Your GitHub App requires the following permissions: - `Administration`: Read & Write (for branch protection) - **Organization permissions:** - - `Members`: Read (for OWNERS validation) - **Events:** @@ -142,10 +141,10 @@ These examples demonstrate: ```bash # Pull the latest stable release -podman pull quay.io/myakove/github-webhook-server:latest +podman pull ghcr.io/myk-org/github-webhook-server:latest # Or using Docker -docker pull quay.io/myakove/github-webhook-server:latest +docker pull ghcr.io/myk-org/github-webhook-server:latest ``` ### Building from Source @@ -333,8 +332,8 @@ their own categorization system. # Global configuration (applies to all repositories) pr-size-thresholds: Tiny: - threshold: 10 # Required: positive integer (lines changed) - color: lightgray # Optional: CSS3 color name, defaults to lightgray + threshold: 10 # Required: positive integer (lines changed) + color: lightgray # Optional: CSS3 color name, defaults to lightgray Small: threshold: 50 color: green @@ -491,7 +490,7 @@ uv run pytest webhook_server/tests/test_config_schema.py::TestConfigSchema::test version: "3.8" services: github-webhook-server: - image: quay.io/myakove/github-webhook-server:latest + image: ghcr.io/myk-org/github-webhook-server:latest container_name: github-webhook-server ports: - "5000:5000" @@ -537,7 +536,7 @@ spec: spec: containers: - name: webhook-server - image: quay.io/myakove/github-webhook-server:latest + image: ghcr.io/myk-org/github-webhook-server:latest ports: - containerPort: 5000 env: @@ -614,7 +613,7 @@ podman run -d \ -p 5000:5000 \ -v ./data:/home/podman/data:Z \ -e WEBHOOK_SECRET=your-secret \ - quay.io/myakove/github-webhook-server:latest + ghcr.io/myk-org/github-webhook-server:latest # From source uv run entrypoint.py @@ -623,7 +622,6 @@ uv run entrypoint.py ### Webhook Setup 1. **Configure GitHub Webhook:** - - Go to your repository settings - Navigate to Webhooks → Add webhook - Set Payload URL: `https://your-domain.com/webhook_server` @@ -678,6 +676,303 @@ POST /webhook_server } ``` +## Log Viewer + +The webhook server includes a comprehensive log viewer web interface for monitoring and analyzing webhook processing in real-time. The system has been optimized with **memory-efficient streaming architecture** to handle enterprise-scale log volumes without performance degradation. + +### 🚀 Performance & Scalability + +**Memory-Optimized Streaming**: The log viewer uses advanced streaming and chunked processing techniques that replaced traditional bulk loading: + + +- **Constant Memory Usage**: Handles log files of any size with consistent memory footprint +- **Early Filtering**: Reduces data transfer by filtering at the source before transmission +- **Streaming Processing**: Real-time log processing without loading entire files into memory +- **90% Memory Reduction**: Optimized for enterprise environments with gigabytes of log data +- **Sub-second Response Times**: Fast query responses even with large datasets + + +### 🔒 Security Warning + +**🚨 CRITICAL SECURITY NOTICE**: The log viewer endpoints (`/logs/*`) are **NOT PROTECTED** by +authentication or authorization. They expose potentially sensitive webhook data and should **NEVER** +be exposed outside your local network or trusted environment. + +**Required Security Measures:** + +- ✅ Deploy behind a reverse proxy with authentication (e.g., nginx with basic auth) +- ✅ Use firewall rules to restrict access to trusted IP ranges only +- ✅ Never expose log viewer ports directly to the internet +- ✅ Monitor access to log endpoints in your infrastructure logs +- ✅ Consider VPN-only access for maximum security + +**Data Exposure Risk**: Log files may contain GitHub tokens, user information, repository details, and sensitive webhook payloads. + +### Core Features + +- 🔍 **Real-time log streaming** via WebSocket connections with intelligent buffering +- 📊 **Advanced filtering** by hook ID, PR number, repository, user, log level, and text search +- 🎨 **Dark/light theme support** with automatic preference saving +- 📈 **PR flow visualization** showing webhook processing stages and timing +- 📥 **JSON export** functionality for log analysis and external processing +- 🎯 **Color-coded log levels** for quick visual identification +- ⚡ **Progressive loading** with pagination for large datasets +- 🔄 **Auto-refresh** with configurable intervals +- 🎛️ **Advanced query builder** for complex log searches + +### Technical Architecture + +**Streaming-First Design**: The log viewer is built around a streaming architecture that processes logs incrementally: + +```text +Log File → Streaming Parser → Early Filter → Chunked Processing → Client + ↓ ↓ ↓ ↓ ↓ +Real-time Line-by-line Apply filters Small batches Progressive UI +processing microsecond before load (100-1000 updates + timestamps entries) +``` + +**Memory Efficiency**: +- **Streaming Parser**: Reads log files line-by-line instead of loading entire files +- **Early Filtering**: Applies search criteria during parsing to reduce memory usage +- **Chunked Responses**: Delivers results in small batches for responsive UI +- **Automatic Cleanup**: Releases processed data immediately after transmission + +### Accessing the Log Viewer + +**Web Interface:** + +```url +http://your-server:5000/logs +``` + +### API Endpoints + +#### Get Historical Log Entries + +```http +GET /logs/api/entries +``` + +**Query Parameters:** + +- `hook_id` (string): Filter by GitHub delivery ID (x-github-delivery) +- `pr_number` (integer): Filter by pull request number +- `repository` (string): Filter by repository name (e.g., "org/repo") +- `event_type` (string): Filter by GitHub event type +- `github_user` (string): Filter by GitHub username +- `level` (string): Filter by log level (DEBUG, INFO, WARNING, ERROR) +- `start_time` (string): Start time filter (ISO 8601 format) +- `end_time` (string): End time filter (ISO 8601 format) +- `search` (string): Free text search in log messages +- `limit` (integer): Maximum entries to return (1-1000, default: 100) +- `offset` (integer): Pagination offset (default: 0) + +**Example:** + +```bash +curl "http://localhost:5000/logs/api/entries?pr_number=123&level=ERROR&limit=50" +``` + +**Response:** + +```json +{ + "entries": [ + { + "timestamp": "2025-01-30T10:30:00.123000", + "level": "INFO", + "logger_name": "GithubWebhook", + "message": "Processing webhook for repository: my-org/my-repo", + "hook_id": "abc123-def456", + "event_type": "pull_request", + "repository": "my-org/my-repo", + "pr_number": 123, + "github_user": "username" + } + ], + "entries_processed": 1500, + "filtered_count_min": 25, + "limit": 50, + "offset": 0 +} +``` + +#### Export Logs + +```http +GET /logs/api/export +``` + +**Query Parameters:** (Same as `/logs/api/entries` plus) + +- `format` (string): Export format - only "json" is supported +- `limit` (integer): Maximum entries to export (max 50,000, default: 10,000) + +**Example:** + +```bash +curl "http://localhost:5000/logs/api/export?format=json&pr_number=123" -o logs.json +``` + +#### WebSocket Real-time Streaming + +```url +ws://your-server:5000/logs/ws +``` + +**Query Parameters:** (Same filtering options as API endpoints) + +**Example WebSocket Connection:** + +```javascript +const ws = new WebSocket("ws://localhost:5000/logs/ws?level=ERROR"); +ws.onmessage = function (event) { + const logEntry = JSON.parse(event.data); + console.log("New error log:", logEntry); +}; +``` + +#### PR Flow Visualization + +```http +GET /logs/api/pr-flow/{identifier} +``` + +**Parameters:** + +- `identifier`: Hook ID (e.g., "abc123") or PR number (e.g., "123") + +**Example:** + +```bash +curl "http://localhost:5000/logs/api/pr-flow/123" +``` + +**Response:** + +```json +{ + "identifier": "123", + "stages": [ + { + "name": "Webhook Received", + "timestamp": "2025-01-30T10:30:00.123000", + "duration_ms": null + }, + { + "name": "Validation Complete", + "timestamp": "2025-01-30T10:30:00.245000", + "duration_ms": 122 + } + ], + "total_duration_ms": 2500, + "success": true +} +``` + +### Log Level Color Coding + +The web interface uses intuitive color coding for different log levels: + +- 🟢 **INFO (Green)**: Successful operations and informational messages +- 🟡 **WARNING (Yellow)**: Warning messages that need attention +- 🔴 **ERROR (Red)**: Error messages requiring immediate action +- ⚪ **DEBUG (Gray)**: Technical debug information + +### Web Interface Features + +#### Filtering Controls + +- **Hook ID**: GitHub delivery ID for tracking specific webhook calls +- **PR Number**: Filter by pull request number +- **Repository**: Filter by repository name (org/repo format) +- **User**: Filter by GitHub username +- **Log Level**: Filter by severity level +- **Search**: Free text search across log messages + +#### Real-time Features + +- **Live Updates**: WebSocket connection for real-time log streaming +- **Auto-refresh**: Historical logs refresh when filters change +- **Connection Status**: Visual indicator for WebSocket connection status + +#### Theme Support + +- **Dark/Light Modes**: Toggle between themes with automatic preference saving +- **Responsive Design**: Works on desktop and mobile devices +- **Keyboard Shortcuts**: Quick access to common functions + +### Usage Examples + +#### Monitor Specific PR + +```bash +# View all logs for PR #123 +curl "http://localhost:5000/logs/api/entries?pr_number=123" +``` + +#### Track Webhook Processing + +```bash +# Follow specific webhook delivery +curl "http://localhost:5000/logs/api/entries?hook_id=abc123-def456" +``` + +#### Debug Error Issues + +```bash +# Export all error logs for analysis +curl "http://localhost:5000/logs/api/export?format=json&level=ERROR" -o errors.json +``` + +#### Monitor Repository Activity + +```bash +# Watch real-time activity for specific repository +# Connect WebSocket to: ws://localhost:5000/logs/ws?repository=my-org/my-repo +``` + +### Security Considerations + +1. **Network Isolation**: Deploy in isolated network segments +2. **Access Control**: Implement reverse proxy authentication (mandatory for production) +3. **Log Sanitization**: Logs may contain GitHub tokens, webhook payloads, and user data +4. **Monitoring**: Monitor access to log viewer endpoints and track usage patterns +5. **Data Retention**: Consider log rotation and retention policies for compliance +6. **Enterprise Deployment**: The memory-optimized architecture supports enterprise-scale deployments while maintaining security boundaries +7. **Audit Trail**: Log viewer access should be logged and monitored in production environments + +### Troubleshooting + +#### WebSocket Connection Issues + +- Check firewall rules for WebSocket traffic +- Verify server is accessible on specified port +- Ensure WebSocket upgrades are allowed by reverse proxy + +#### Missing Log Data + +- Verify log file permissions and paths +- Check if log directory exists and is writable +- Ensure log parser patterns match your log format + +#### Performance Issues + +- **Large Result Sets**: Reduce filter result sets using specific time ranges or repositories +- **Memory Usage**: The streaming architecture automatically handles large datasets efficiently +- **Query Optimization**: Use specific filters (hook_id, pr_number) for fastest responses +- **File Size Management**: Consider log file rotation for easier management (system handles large files automatically) +- **Network Latency**: Use pagination for mobile or slow connections + +#### Performance Benchmarks + +The memory optimization work has achieved: +- **90% reduction** in memory usage compared to bulk loading +- **Sub-second response times** for filtered queries on multi-GB log files +- **Constant memory footprint** regardless of log file size +- **Real-time streaming** with <100ms latency for new log entries + ## User Commands Users can interact with the webhook server through GitHub comments on pull requests and issues. @@ -730,20 +1025,20 @@ Users can interact with the webhook server through GitHub comments on pull reque ### Review & Approval -| Command | Description | Example | -| ------------------- | ------------------------------------------------------------------------- | ------------------- | -| `/lgtm` | Approve changes (looks good to me) | `/lgtm` | -| `/approve` | Approve PR (approvers only) | `/approve` | -| `/automerge` | Enable automatic merging when all requirements are met (maintainers/approvers only) | `/automerge` | -| `/assign-reviewers` | Assign reviewers based on OWNERS file | `/assign-reviewers` | -| `/assign-reviewer` | Assign specific reviewer | `/assign-reviewer @username` | -| `/check-can-merge` | Checks if the pull request meets all merge requirements | `/check-can-merge` | +| Command | Description | Example | +| ------------------- | ----------------------------------------------------------------------------------- | ---------------------------- | +| `/lgtm` | Approve changes (looks good to me) | `/lgtm` | +| `/approve` | Approve PR (approvers only) | `/approve` | +| `/automerge` | Enable automatic merging when all requirements are met (maintainers/approvers only) | `/automerge` | +| `/assign-reviewers` | Assign reviewers based on OWNERS file | `/assign-reviewers` | +| `/assign-reviewer` | Assign specific reviewer | `/assign-reviewer @username` | +| `/check-can-merge` | Checks if the pull request meets all merge requirements | `/check-can-merge` | ### Testing & Validation -| Command | Description | Example | -| ------------------- | ------------------------------------------------------------------------- | ------------------- | -| `/retest ` | Run specific tests like `tox` or `pre-commit` | `/retest ` | +| Command | Description | Example | +| --------------------- | --------------------------------------------- | --------------------- | +| `/retest ` | Run specific tests like `tox` or `pre-commit` | `/retest ` | ## OWNERS File Format @@ -785,16 +1080,38 @@ reviewers: ## Security -### IP Allowlist +⚠️ **Important**: The log viewer endpoints (`/logs/*`) are **unauthenticated** and expose potentially sensitive webhook data. + +### Network-Level Security (Recommended) + +**Deploy log viewer endpoints only on trusted networks:** + +1. **VPN Access**: Deploy behind corporate VPN for internal-only access +2. **Reverse Proxy Authentication**: Use nginx/Apache with HTTP Basic Auth: + + ```nginx + location /logs { + auth_basic "Webhook Logs"; + auth_basic_user_file /etc/nginx/.htpasswd; + proxy_pass http://webhook-server:5000; + } + ``` + +3. **Firewall Rules**: Restrict access to webhook server port to specific IP ranges +4. **Network Segmentation**: Deploy in isolated network segments -Configure IP-based access control: +### Webhook Security + +#### IP Allowlist + +Configure IP-based access control for webhook endpoints: ```yaml verify-github-ips: true # Restrict to GitHub's IP ranges verify-cloudflare-ips: true # Allow Cloudflare IPs (if using CF proxy) ``` -### Webhook Security +#### Signature Verification ```yaml webhook-secret: "your-secure-secret" # HMAC-SHA256 signature verification # pragma: allowlist secret @@ -815,13 +1132,26 @@ disable-ssl-warnings: - Monitor token usage and rate limits - Store tokens securely (environment variables, secrets management) +### Security Architecture + +``` +Internet → GitHub Webhooks → [Webhook Server] ← Internal Network ← Log Viewer Access + ↓ + [Authenticated Endpoints] + ↓ + [Unauthenticated Log Viewer] + ↑ + [Network-Level Protection] +``` + ### Best Practices -1. **Network Security**: Deploy behind reverse proxy with TLS termination -2. **Container Security**: Run as non-privileged user when possible -3. **Secrets Management**: Use external secret management systems -4. **Monitoring**: Enable comprehensive logging and monitoring -5. **Updates**: Regularly update to latest stable version +1. **Log Viewer Access**: Only expose `/logs/*` endpoints to trusted networks +2. **Network Security**: Deploy behind reverse proxy with TLS termination +3. **Container Security**: Run as non-privileged user when possible +4. **Secrets Management**: Use external secret management systems +5. **Monitoring**: Enable comprehensive logging and monitoring +6. **Updates**: Regularly update to latest stable version ## Monitoring diff --git a/pyproject.toml b/pyproject.toml index 63232939..368ffdbc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -70,10 +70,13 @@ dependencies = [ "shortuuid>=1.0.13", "string-color>=1.2.3", "timeout-sampler>=0.0.46", - "uvicorn>=0.31.0", + "uvicorn[standard]>=0.31.0", "httpx>=0.28.1", "asyncstdlib>=3.13.1", "webcolors>=24.11.1", + "pyjwt>=2.8.0", + "pydantic>=2.5.0", + "psutil>=7.0.0", ] [[project.authors]] @@ -87,18 +90,16 @@ email = "ruth.netser@gmail.com" [project.urls] homepage = "https://github.com/myakove/github-webhook-server" repository = "https://github.com/myakove/github-webhook-server" -Download = "https://quay.io/repository/myakove/github-webhook-server" "Bug Tracker" = "https://github.com/myakove/github-webhook-server/issues" [project.optional-dependencies] -tests = [ - "pytest-asyncio>=0.26.0", - "pytest-xdist>=3.7.0", -] +tests = ["pytest-asyncio>=0.26.0", "pytest-xdist>=3.7.0"] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [dependency-groups] -tests = [] +tests = [ + "psutil>=7.0.0", +] diff --git a/tox.toml b/tox.toml index 89a13f04..8a11e557 100644 --- a/tox.toml +++ b/tox.toml @@ -8,7 +8,7 @@ commands = [ [ "pyutils-unusedcode", "--exclude-function-prefixes", - "'process_webhook','validate_config_file'", + "'process_webhook','validate_config_file', 'get_log_viewer_page'", ], ] diff --git a/uv.lock b/uv.lock index bea0e572..a4ba8001 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 3 +revision = 2 requires-python = ">=3.12" [[package]] @@ -381,8 +381,11 @@ dependencies = [ { name = "colorlog" }, { name = "fastapi" }, { name = "httpx" }, + { name = "psutil" }, + { name = "pydantic" }, { name = "pygithub" }, { name = "pyhelper-utils" }, + { name = "pyjwt" }, { name = "pytest" }, { name = "pytest-cov" }, { name = "pytest-mock" }, @@ -393,7 +396,7 @@ dependencies = [ { name = "shortuuid" }, { name = "string-color" }, { name = "timeout-sampler" }, - { name = "uvicorn" }, + { name = "uvicorn", extra = ["standard"] }, { name = "webcolors" }, ] @@ -411,6 +414,9 @@ dev = [ { name = "types-pyyaml" }, { name = "types-requests" }, ] +tests = [ + { name = "psutil" }, +] [package.metadata] requires-dist = [ @@ -420,8 +426,11 @@ requires-dist = [ { name = "colorlog", specifier = ">=6.8.2" }, { name = "fastapi", specifier = ">=0.115.0" }, { name = "httpx", specifier = ">=0.28.1" }, + { name = "psutil", specifier = ">=7.0.0" }, + { name = "pydantic", specifier = ">=2.5.0" }, { name = "pygithub", specifier = ">=2.4.0" }, { name = "pyhelper-utils", specifier = ">=0.0.42" }, + { name = "pyjwt", specifier = ">=2.8.0" }, { name = "pytest", specifier = ">=8.3.3" }, { name = "pytest-asyncio", marker = "extra == 'tests'", specifier = ">=0.26.0" }, { name = "pytest-cov", specifier = ">=6.0.0" }, @@ -434,7 +443,7 @@ requires-dist = [ { name = "shortuuid", specifier = ">=1.0.13" }, { name = "string-color", specifier = ">=1.2.3" }, { name = "timeout-sampler", specifier = ">=0.0.46" }, - { name = "uvicorn", specifier = ">=0.31.0" }, + { name = "uvicorn", extras = ["standard"], specifier = ">=0.31.0" }, { name = "webcolors", specifier = ">=24.11.1" }, ] provides-extras = ["tests"] @@ -447,7 +456,7 @@ dev = [ { name = "types-pyyaml", specifier = ">=6.0.12.20250516" }, { name = "types-requests", specifier = ">=2.32.4.20250611" }, ] -tests = [] +tests = [{ name = "psutil", specifier = ">=7.0.0" }] [[package]] name = "h11" @@ -471,6 +480,28 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, ] +[[package]] +name = "httptools" +version = "0.6.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a7/9a/ce5e1f7e131522e6d3426e8e7a490b3a01f39a6696602e1c4f33f9e94277/httptools-0.6.4.tar.gz", hash = "sha256:4e93eee4add6493b59a5c514da98c939b244fce4a0d8879cd3f466562f4b7d5c", size = 240639, upload-time = "2024-10-16T19:45:08.902Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bb/0e/d0b71465c66b9185f90a091ab36389a7352985fe857e352801c39d6127c8/httptools-0.6.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:df017d6c780287d5c80601dafa31f17bddb170232d85c066604d8558683711a2", size = 200683, upload-time = "2024-10-16T19:44:30.175Z" }, + { url = "https://files.pythonhosted.org/packages/e2/b8/412a9bb28d0a8988de3296e01efa0bd62068b33856cdda47fe1b5e890954/httptools-0.6.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:85071a1e8c2d051b507161f6c3e26155b5c790e4e28d7f236422dbacc2a9cc44", size = 104337, upload-time = "2024-10-16T19:44:31.786Z" }, + { url = "https://files.pythonhosted.org/packages/9b/01/6fb20be3196ffdc8eeec4e653bc2a275eca7f36634c86302242c4fbb2760/httptools-0.6.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:69422b7f458c5af875922cdb5bd586cc1f1033295aa9ff63ee196a87519ac8e1", size = 508796, upload-time = "2024-10-16T19:44:32.825Z" }, + { url = "https://files.pythonhosted.org/packages/f7/d8/b644c44acc1368938317d76ac991c9bba1166311880bcc0ac297cb9d6bd7/httptools-0.6.4-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:16e603a3bff50db08cd578d54f07032ca1631450ceb972c2f834c2b860c28ea2", size = 510837, upload-time = "2024-10-16T19:44:33.974Z" }, + { url = "https://files.pythonhosted.org/packages/52/d8/254d16a31d543073a0e57f1c329ca7378d8924e7e292eda72d0064987486/httptools-0.6.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ec4f178901fa1834d4a060320d2f3abc5c9e39766953d038f1458cb885f47e81", size = 485289, upload-time = "2024-10-16T19:44:35.111Z" }, + { url = "https://files.pythonhosted.org/packages/5f/3c/4aee161b4b7a971660b8be71a92c24d6c64372c1ab3ae7f366b3680df20f/httptools-0.6.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f9eb89ecf8b290f2e293325c646a211ff1c2493222798bb80a530c5e7502494f", size = 489779, upload-time = "2024-10-16T19:44:36.253Z" }, + { url = "https://files.pythonhosted.org/packages/12/b7/5cae71a8868e555f3f67a50ee7f673ce36eac970f029c0c5e9d584352961/httptools-0.6.4-cp312-cp312-win_amd64.whl", hash = "sha256:db78cb9ca56b59b016e64b6031eda5653be0589dba2b1b43453f6e8b405a0970", size = 88634, upload-time = "2024-10-16T19:44:37.357Z" }, + { url = "https://files.pythonhosted.org/packages/94/a3/9fe9ad23fd35f7de6b91eeb60848986058bd8b5a5c1e256f5860a160cc3e/httptools-0.6.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ade273d7e767d5fae13fa637f4d53b6e961fb7fd93c7797562663f0171c26660", size = 197214, upload-time = "2024-10-16T19:44:38.738Z" }, + { url = "https://files.pythonhosted.org/packages/ea/d9/82d5e68bab783b632023f2fa31db20bebb4e89dfc4d2293945fd68484ee4/httptools-0.6.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:856f4bc0478ae143bad54a4242fccb1f3f86a6e1be5548fecfd4102061b3a083", size = 102431, upload-time = "2024-10-16T19:44:39.818Z" }, + { url = "https://files.pythonhosted.org/packages/96/c1/cb499655cbdbfb57b577734fde02f6fa0bbc3fe9fb4d87b742b512908dff/httptools-0.6.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:322d20ea9cdd1fa98bd6a74b77e2ec5b818abdc3d36695ab402a0de8ef2865a3", size = 473121, upload-time = "2024-10-16T19:44:41.189Z" }, + { url = "https://files.pythonhosted.org/packages/af/71/ee32fd358f8a3bb199b03261f10921716990808a675d8160b5383487a317/httptools-0.6.4-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4d87b29bd4486c0093fc64dea80231f7c7f7eb4dc70ae394d70a495ab8436071", size = 473805, upload-time = "2024-10-16T19:44:42.384Z" }, + { url = "https://files.pythonhosted.org/packages/8a/0a/0d4df132bfca1507114198b766f1737d57580c9ad1cf93c1ff673e3387be/httptools-0.6.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:342dd6946aa6bda4b8f18c734576106b8a31f2fe31492881a9a160ec84ff4bd5", size = 448858, upload-time = "2024-10-16T19:44:43.959Z" }, + { url = "https://files.pythonhosted.org/packages/1e/6a/787004fdef2cabea27bad1073bf6a33f2437b4dbd3b6fb4a9d71172b1c7c/httptools-0.6.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4b36913ba52008249223042dca46e69967985fb4051951f94357ea681e1f5dc0", size = 452042, upload-time = "2024-10-16T19:44:45.071Z" }, + { url = "https://files.pythonhosted.org/packages/4d/dc/7decab5c404d1d2cdc1bb330b1bf70e83d6af0396fd4fc76fc60c0d522bf/httptools-0.6.4-cp313-cp313-win_amd64.whl", hash = "sha256:28908df1b9bb8187393d5b5db91435ccc9c8e891657f9cbb42a2541b44c82fc8", size = 87682, upload-time = "2024-10-16T19:44:46.46Z" }, +] + [[package]] name = "httpx" version = "0.28.1" @@ -679,6 +710,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ce/4f/5249960887b1fbe561d9ff265496d170b55a735b76724f10ef19f9e40716/prompt_toolkit-3.0.51-py3-none-any.whl", hash = "sha256:52742911fde84e2d423e2f9a4cf1de7d7ac4e51958f648d9540e0fb8db077b07", size = 387810, upload-time = "2025-04-15T09:18:44.753Z" }, ] +[[package]] +name = "psutil" +version = "7.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2a/80/336820c1ad9286a4ded7e845b2eccfcb27851ab8ac6abece774a6ff4d3de/psutil-7.0.0.tar.gz", hash = "sha256:7be9c3eba38beccb6495ea33afd982a44074b78f28c434a1f51cc07fd315c456", size = 497003, upload-time = "2025-02-13T21:54:07.946Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ed/e6/2d26234410f8b8abdbf891c9da62bee396583f713fb9f3325a4760875d22/psutil-7.0.0-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:101d71dc322e3cffd7cea0650b09b3d08b8e7c4109dd6809fe452dfd00e58b25", size = 238051, upload-time = "2025-02-13T21:54:12.36Z" }, + { url = "https://files.pythonhosted.org/packages/04/8b/30f930733afe425e3cbfc0e1468a30a18942350c1a8816acfade80c005c4/psutil-7.0.0-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:39db632f6bb862eeccf56660871433e111b6ea58f2caea825571951d4b6aa3da", size = 239535, upload-time = "2025-02-13T21:54:16.07Z" }, + { url = "https://files.pythonhosted.org/packages/2a/ed/d362e84620dd22876b55389248e522338ed1bf134a5edd3b8231d7207f6d/psutil-7.0.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1fcee592b4c6f146991ca55919ea3d1f8926497a713ed7faaf8225e174581e91", size = 275004, upload-time = "2025-02-13T21:54:18.662Z" }, + { url = "https://files.pythonhosted.org/packages/bf/b9/b0eb3f3cbcb734d930fdf839431606844a825b23eaf9a6ab371edac8162c/psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b1388a4f6875d7e2aff5c4ca1cc16c545ed41dd8bb596cefea80111db353a34", size = 277986, upload-time = "2025-02-13T21:54:21.811Z" }, + { url = "https://files.pythonhosted.org/packages/eb/a2/709e0fe2f093556c17fbafda93ac032257242cabcc7ff3369e2cb76a97aa/psutil-7.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a5f098451abc2828f7dc6b58d44b532b22f2088f4999a937557b603ce72b1993", size = 279544, upload-time = "2025-02-13T21:54:24.68Z" }, + { url = "https://files.pythonhosted.org/packages/50/e6/eecf58810b9d12e6427369784efe814a1eec0f492084ce8eb8f4d89d6d61/psutil-7.0.0-cp37-abi3-win32.whl", hash = "sha256:ba3fcef7523064a6c9da440fc4d6bd07da93ac726b5733c29027d7dc95b39d99", size = 241053, upload-time = "2025-02-13T21:54:34.31Z" }, + { url = "https://files.pythonhosted.org/packages/50/1b/6921afe68c74868b4c9fa424dad3be35b095e16687989ebbb50ce4fceb7c/psutil-7.0.0-cp37-abi3-win_amd64.whl", hash = "sha256:4cf3d4eb1aa9b348dec30105c55cd9b7d4629285735a102beb4441e38db90553", size = 244885, upload-time = "2025-02-13T21:54:37.486Z" }, +] + [[package]] name = "ptyprocess" version = "0.7.0" @@ -911,6 +957,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" }, ] +[[package]] +name = "python-dotenv" +version = "1.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/b0/4bc07ccd3572a2f9df7e6782f52b0c6c90dcbb803ac4a167702d7d0dfe1e/python_dotenv-1.1.1.tar.gz", hash = "sha256:a8a6399716257f45be6a007360200409fce5cda2661e3dec71d23dc15f6189ab", size = 41978, upload-time = "2025-06-24T04:21:07.341Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5f/ed/539768cf28c661b5b068d66d96a2f155c4971a5d55684a514c1a0e0dec2f/python_dotenv-1.1.1-py3-none-any.whl", hash = "sha256:31f23644fe2602f88ff55e1f5c79ba497e01224ee7737937930c448e4d0e24dc", size = 20556, upload-time = "2025-06-24T04:21:06.073Z" }, +] + [[package]] name = "python-rrmngmnt" version = "0.2.0" @@ -1191,6 +1246,104 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d2/e2/dc81b1bd1dcfe91735810265e9d26bc8ec5da45b4c0f6237e286819194c3/uvicorn-0.35.0-py3-none-any.whl", hash = "sha256:197535216b25ff9b785e29a0b79199f55222193d47f820816e7da751e9bc8d4a", size = 66406, upload-time = "2025-06-28T16:15:44.816Z" }, ] +[package.optional-dependencies] +standard = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "httptools" }, + { name = "python-dotenv" }, + { name = "pyyaml" }, + { name = "uvloop", marker = "platform_python_implementation != 'PyPy' and sys_platform != 'cygwin' and sys_platform != 'win32'" }, + { name = "watchfiles" }, + { name = "websockets" }, +] + +[[package]] +name = "uvloop" +version = "0.21.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/af/c0/854216d09d33c543f12a44b393c402e89a920b1a0a7dc634c42de91b9cf6/uvloop-0.21.0.tar.gz", hash = "sha256:3bf12b0fda68447806a7ad847bfa591613177275d35b6724b1ee573faa3704e3", size = 2492741, upload-time = "2024-10-14T23:38:35.489Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8c/4c/03f93178830dc7ce8b4cdee1d36770d2f5ebb6f3d37d354e061eefc73545/uvloop-0.21.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:359ec2c888397b9e592a889c4d72ba3d6befba8b2bb01743f72fffbde663b59c", size = 1471284, upload-time = "2024-10-14T23:37:47.833Z" }, + { url = "https://files.pythonhosted.org/packages/43/3e/92c03f4d05e50f09251bd8b2b2b584a2a7f8fe600008bcc4523337abe676/uvloop-0.21.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:f7089d2dc73179ce5ac255bdf37c236a9f914b264825fdaacaded6990a7fb4c2", size = 821349, upload-time = "2024-10-14T23:37:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/a6/ef/a02ec5da49909dbbfb1fd205a9a1ac4e88ea92dcae885e7c961847cd51e2/uvloop-0.21.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:baa4dcdbd9ae0a372f2167a207cd98c9f9a1ea1188a8a526431eef2f8116cc8d", size = 4580089, upload-time = "2024-10-14T23:37:51.703Z" }, + { url = "https://files.pythonhosted.org/packages/06/a7/b4e6a19925c900be9f98bec0a75e6e8f79bb53bdeb891916609ab3958967/uvloop-0.21.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:86975dca1c773a2c9864f4c52c5a55631038e387b47eaf56210f873887b6c8dc", size = 4693770, upload-time = "2024-10-14T23:37:54.122Z" }, + { url = "https://files.pythonhosted.org/packages/ce/0c/f07435a18a4b94ce6bd0677d8319cd3de61f3a9eeb1e5f8ab4e8b5edfcb3/uvloop-0.21.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:461d9ae6660fbbafedd07559c6a2e57cd553b34b0065b6550685f6653a98c1cb", size = 4451321, upload-time = "2024-10-14T23:37:55.766Z" }, + { url = "https://files.pythonhosted.org/packages/8f/eb/f7032be105877bcf924709c97b1bf3b90255b4ec251f9340cef912559f28/uvloop-0.21.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:183aef7c8730e54c9a3ee3227464daed66e37ba13040bb3f350bc2ddc040f22f", size = 4659022, upload-time = "2024-10-14T23:37:58.195Z" }, + { url = "https://files.pythonhosted.org/packages/3f/8d/2cbef610ca21539f0f36e2b34da49302029e7c9f09acef0b1c3b5839412b/uvloop-0.21.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:bfd55dfcc2a512316e65f16e503e9e450cab148ef11df4e4e679b5e8253a5281", size = 1468123, upload-time = "2024-10-14T23:38:00.688Z" }, + { url = "https://files.pythonhosted.org/packages/93/0d/b0038d5a469f94ed8f2b2fce2434a18396d8fbfb5da85a0a9781ebbdec14/uvloop-0.21.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:787ae31ad8a2856fc4e7c095341cccc7209bd657d0e71ad0dc2ea83c4a6fa8af", size = 819325, upload-time = "2024-10-14T23:38:02.309Z" }, + { url = "https://files.pythonhosted.org/packages/50/94/0a687f39e78c4c1e02e3272c6b2ccdb4e0085fda3b8352fecd0410ccf915/uvloop-0.21.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5ee4d4ef48036ff6e5cfffb09dd192c7a5027153948d85b8da7ff705065bacc6", size = 4582806, upload-time = "2024-10-14T23:38:04.711Z" }, + { url = "https://files.pythonhosted.org/packages/d2/19/f5b78616566ea68edd42aacaf645adbf71fbd83fc52281fba555dc27e3f1/uvloop-0.21.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f3df876acd7ec037a3d005b3ab85a7e4110422e4d9c1571d4fc89b0fc41b6816", size = 4701068, upload-time = "2024-10-14T23:38:06.385Z" }, + { url = "https://files.pythonhosted.org/packages/47/57/66f061ee118f413cd22a656de622925097170b9380b30091b78ea0c6ea75/uvloop-0.21.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:bd53ecc9a0f3d87ab847503c2e1552b690362e005ab54e8a48ba97da3924c0dc", size = 4454428, upload-time = "2024-10-14T23:38:08.416Z" }, + { url = "https://files.pythonhosted.org/packages/63/9a/0962b05b308494e3202d3f794a6e85abe471fe3cafdbcf95c2e8c713aabd/uvloop-0.21.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a5c39f217ab3c663dc699c04cbd50c13813e31d917642d459fdcec07555cc553", size = 4660018, upload-time = "2024-10-14T23:38:10.888Z" }, +] + +[[package]] +name = "watchfiles" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2a/9a/d451fcc97d029f5812e898fd30a53fd8c15c7bbd058fd75cfc6beb9bd761/watchfiles-1.1.0.tar.gz", hash = "sha256:693ed7ec72cbfcee399e92c895362b6e66d63dac6b91e2c11ae03d10d503e575", size = 94406, upload-time = "2025-06-15T19:06:59.42Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f6/b8/858957045a38a4079203a33aaa7d23ea9269ca7761c8a074af3524fbb240/watchfiles-1.1.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:9dc001c3e10de4725c749d4c2f2bdc6ae24de5a88a339c4bce32300a31ede179", size = 402339, upload-time = "2025-06-15T19:05:24.516Z" }, + { url = "https://files.pythonhosted.org/packages/80/28/98b222cca751ba68e88521fabd79a4fab64005fc5976ea49b53fa205d1fa/watchfiles-1.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d9ba68ec283153dead62cbe81872d28e053745f12335d037de9cbd14bd1877f5", size = 394409, upload-time = "2025-06-15T19:05:25.469Z" }, + { url = "https://files.pythonhosted.org/packages/86/50/dee79968566c03190677c26f7f47960aff738d32087087bdf63a5473e7df/watchfiles-1.1.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:130fc497b8ee68dce163e4254d9b0356411d1490e868bd8790028bc46c5cc297", size = 450939, upload-time = "2025-06-15T19:05:26.494Z" }, + { url = "https://files.pythonhosted.org/packages/40/45/a7b56fb129700f3cfe2594a01aa38d033b92a33dddce86c8dfdfc1247b72/watchfiles-1.1.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:50a51a90610d0845a5931a780d8e51d7bd7f309ebc25132ba975aca016b576a0", size = 457270, upload-time = "2025-06-15T19:05:27.466Z" }, + { url = "https://files.pythonhosted.org/packages/b5/c8/fa5ef9476b1d02dc6b5e258f515fcaaecf559037edf8b6feffcbc097c4b8/watchfiles-1.1.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:dc44678a72ac0910bac46fa6a0de6af9ba1355669b3dfaf1ce5f05ca7a74364e", size = 483370, upload-time = "2025-06-15T19:05:28.548Z" }, + { url = "https://files.pythonhosted.org/packages/98/68/42cfcdd6533ec94f0a7aab83f759ec11280f70b11bfba0b0f885e298f9bd/watchfiles-1.1.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a543492513a93b001975ae283a51f4b67973662a375a403ae82f420d2c7205ee", size = 598654, upload-time = "2025-06-15T19:05:29.997Z" }, + { url = "https://files.pythonhosted.org/packages/d3/74/b2a1544224118cc28df7e59008a929e711f9c68ce7d554e171b2dc531352/watchfiles-1.1.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8ac164e20d17cc285f2b94dc31c384bc3aa3dd5e7490473b3db043dd70fbccfd", size = 478667, upload-time = "2025-06-15T19:05:31.172Z" }, + { url = "https://files.pythonhosted.org/packages/8c/77/e3362fe308358dc9f8588102481e599c83e1b91c2ae843780a7ded939a35/watchfiles-1.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f7590d5a455321e53857892ab8879dce62d1f4b04748769f5adf2e707afb9d4f", size = 452213, upload-time = "2025-06-15T19:05:32.299Z" }, + { url = "https://files.pythonhosted.org/packages/6e/17/c8f1a36540c9a1558d4faf08e909399e8133599fa359bf52ec8fcee5be6f/watchfiles-1.1.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:37d3d3f7defb13f62ece99e9be912afe9dd8a0077b7c45ee5a57c74811d581a4", size = 626718, upload-time = "2025-06-15T19:05:33.415Z" }, + { url = "https://files.pythonhosted.org/packages/26/45/fb599be38b4bd38032643783d7496a26a6f9ae05dea1a42e58229a20ac13/watchfiles-1.1.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:7080c4bb3efd70a07b1cc2df99a7aa51d98685be56be6038c3169199d0a1c69f", size = 623098, upload-time = "2025-06-15T19:05:34.534Z" }, + { url = "https://files.pythonhosted.org/packages/a1/e7/fdf40e038475498e160cd167333c946e45d8563ae4dd65caf757e9ffe6b4/watchfiles-1.1.0-cp312-cp312-win32.whl", hash = "sha256:cbcf8630ef4afb05dc30107bfa17f16c0896bb30ee48fc24bf64c1f970f3b1fd", size = 279209, upload-time = "2025-06-15T19:05:35.577Z" }, + { url = "https://files.pythonhosted.org/packages/3f/d3/3ae9d5124ec75143bdf088d436cba39812122edc47709cd2caafeac3266f/watchfiles-1.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:cbd949bdd87567b0ad183d7676feb98136cde5bb9025403794a4c0db28ed3a47", size = 292786, upload-time = "2025-06-15T19:05:36.559Z" }, + { url = "https://files.pythonhosted.org/packages/26/2f/7dd4fc8b5f2b34b545e19629b4a018bfb1de23b3a496766a2c1165ca890d/watchfiles-1.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:0a7d40b77f07be87c6faa93d0951a0fcd8cbca1ddff60a1b65d741bac6f3a9f6", size = 284343, upload-time = "2025-06-15T19:05:37.5Z" }, + { url = "https://files.pythonhosted.org/packages/d3/42/fae874df96595556a9089ade83be34a2e04f0f11eb53a8dbf8a8a5e562b4/watchfiles-1.1.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:5007f860c7f1f8df471e4e04aaa8c43673429047d63205d1630880f7637bca30", size = 402004, upload-time = "2025-06-15T19:05:38.499Z" }, + { url = "https://files.pythonhosted.org/packages/fa/55/a77e533e59c3003d9803c09c44c3651224067cbe7fb5d574ddbaa31e11ca/watchfiles-1.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:20ecc8abbd957046f1fe9562757903f5eaf57c3bce70929fda6c7711bb58074a", size = 393671, upload-time = "2025-06-15T19:05:39.52Z" }, + { url = "https://files.pythonhosted.org/packages/05/68/b0afb3f79c8e832e6571022611adbdc36e35a44e14f129ba09709aa4bb7a/watchfiles-1.1.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f2f0498b7d2a3c072766dba3274fe22a183dbea1f99d188f1c6c72209a1063dc", size = 449772, upload-time = "2025-06-15T19:05:40.897Z" }, + { url = "https://files.pythonhosted.org/packages/ff/05/46dd1f6879bc40e1e74c6c39a1b9ab9e790bf1f5a2fe6c08b463d9a807f4/watchfiles-1.1.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:239736577e848678e13b201bba14e89718f5c2133dfd6b1f7846fa1b58a8532b", size = 456789, upload-time = "2025-06-15T19:05:42.045Z" }, + { url = "https://files.pythonhosted.org/packages/8b/ca/0eeb2c06227ca7f12e50a47a3679df0cd1ba487ea19cf844a905920f8e95/watchfiles-1.1.0-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:eff4b8d89f444f7e49136dc695599a591ff769300734446c0a86cba2eb2f9895", size = 482551, upload-time = "2025-06-15T19:05:43.781Z" }, + { url = "https://files.pythonhosted.org/packages/31/47/2cecbd8694095647406645f822781008cc524320466ea393f55fe70eed3b/watchfiles-1.1.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:12b0a02a91762c08f7264e2e79542f76870c3040bbc847fb67410ab81474932a", size = 597420, upload-time = "2025-06-15T19:05:45.244Z" }, + { url = "https://files.pythonhosted.org/packages/d9/7e/82abc4240e0806846548559d70f0b1a6dfdca75c1b4f9fa62b504ae9b083/watchfiles-1.1.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:29e7bc2eee15cbb339c68445959108803dc14ee0c7b4eea556400131a8de462b", size = 477950, upload-time = "2025-06-15T19:05:46.332Z" }, + { url = "https://files.pythonhosted.org/packages/25/0d/4d564798a49bf5482a4fa9416dea6b6c0733a3b5700cb8a5a503c4b15853/watchfiles-1.1.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d9481174d3ed982e269c090f780122fb59cee6c3796f74efe74e70f7780ed94c", size = 451706, upload-time = "2025-06-15T19:05:47.459Z" }, + { url = "https://files.pythonhosted.org/packages/81/b5/5516cf46b033192d544102ea07c65b6f770f10ed1d0a6d388f5d3874f6e4/watchfiles-1.1.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:80f811146831c8c86ab17b640801c25dc0a88c630e855e2bef3568f30434d52b", size = 625814, upload-time = "2025-06-15T19:05:48.654Z" }, + { url = "https://files.pythonhosted.org/packages/0c/dd/7c1331f902f30669ac3e754680b6edb9a0dd06dea5438e61128111fadd2c/watchfiles-1.1.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:60022527e71d1d1fda67a33150ee42869042bce3d0fcc9cc49be009a9cded3fb", size = 622820, upload-time = "2025-06-15T19:05:50.088Z" }, + { url = "https://files.pythonhosted.org/packages/1b/14/36d7a8e27cd128d7b1009e7715a7c02f6c131be9d4ce1e5c3b73d0e342d8/watchfiles-1.1.0-cp313-cp313-win32.whl", hash = "sha256:32d6d4e583593cb8576e129879ea0991660b935177c0f93c6681359b3654bfa9", size = 279194, upload-time = "2025-06-15T19:05:51.186Z" }, + { url = "https://files.pythonhosted.org/packages/25/41/2dd88054b849aa546dbeef5696019c58f8e0774f4d1c42123273304cdb2e/watchfiles-1.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:f21af781a4a6fbad54f03c598ab620e3a77032c5878f3d780448421a6e1818c7", size = 292349, upload-time = "2025-06-15T19:05:52.201Z" }, + { url = "https://files.pythonhosted.org/packages/c8/cf/421d659de88285eb13941cf11a81f875c176f76a6d99342599be88e08d03/watchfiles-1.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:5366164391873ed76bfdf618818c82084c9db7fac82b64a20c44d335eec9ced5", size = 283836, upload-time = "2025-06-15T19:05:53.265Z" }, + { url = "https://files.pythonhosted.org/packages/45/10/6faf6858d527e3599cc50ec9fcae73590fbddc1420bd4fdccfebffeedbc6/watchfiles-1.1.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:17ab167cca6339c2b830b744eaf10803d2a5b6683be4d79d8475d88b4a8a4be1", size = 400343, upload-time = "2025-06-15T19:05:54.252Z" }, + { url = "https://files.pythonhosted.org/packages/03/20/5cb7d3966f5e8c718006d0e97dfe379a82f16fecd3caa7810f634412047a/watchfiles-1.1.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:328dbc9bff7205c215a7807da7c18dce37da7da718e798356212d22696404339", size = 392916, upload-time = "2025-06-15T19:05:55.264Z" }, + { url = "https://files.pythonhosted.org/packages/8c/07/d8f1176328fa9e9581b6f120b017e286d2a2d22ae3f554efd9515c8e1b49/watchfiles-1.1.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f7208ab6e009c627b7557ce55c465c98967e8caa8b11833531fdf95799372633", size = 449582, upload-time = "2025-06-15T19:05:56.317Z" }, + { url = "https://files.pythonhosted.org/packages/66/e8/80a14a453cf6038e81d072a86c05276692a1826471fef91df7537dba8b46/watchfiles-1.1.0-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a8f6f72974a19efead54195bc9bed4d850fc047bb7aa971268fd9a8387c89011", size = 456752, upload-time = "2025-06-15T19:05:57.359Z" }, + { url = "https://files.pythonhosted.org/packages/5a/25/0853b3fe0e3c2f5af9ea60eb2e781eade939760239a72c2d38fc4cc335f6/watchfiles-1.1.0-cp313-cp313t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d181ef50923c29cf0450c3cd47e2f0557b62218c50b2ab8ce2ecaa02bd97e670", size = 481436, upload-time = "2025-06-15T19:05:58.447Z" }, + { url = "https://files.pythonhosted.org/packages/fe/9e/4af0056c258b861fbb29dcb36258de1e2b857be4a9509e6298abcf31e5c9/watchfiles-1.1.0-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:adb4167043d3a78280d5d05ce0ba22055c266cf8655ce942f2fb881262ff3cdf", size = 596016, upload-time = "2025-06-15T19:05:59.59Z" }, + { url = "https://files.pythonhosted.org/packages/c5/fa/95d604b58aa375e781daf350897aaaa089cff59d84147e9ccff2447c8294/watchfiles-1.1.0-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8c5701dc474b041e2934a26d31d39f90fac8a3dee2322b39f7729867f932b1d4", size = 476727, upload-time = "2025-06-15T19:06:01.086Z" }, + { url = "https://files.pythonhosted.org/packages/65/95/fe479b2664f19be4cf5ceeb21be05afd491d95f142e72d26a42f41b7c4f8/watchfiles-1.1.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b067915e3c3936966a8607f6fe5487df0c9c4afb85226613b520890049deea20", size = 451864, upload-time = "2025-06-15T19:06:02.144Z" }, + { url = "https://files.pythonhosted.org/packages/d3/8a/3c4af14b93a15ce55901cd7a92e1a4701910f1768c78fb30f61d2b79785b/watchfiles-1.1.0-cp313-cp313t-musllinux_1_1_aarch64.whl", hash = "sha256:9c733cda03b6d636b4219625a4acb5c6ffb10803338e437fb614fef9516825ef", size = 625626, upload-time = "2025-06-15T19:06:03.578Z" }, + { url = "https://files.pythonhosted.org/packages/da/f5/cf6aa047d4d9e128f4b7cde615236a915673775ef171ff85971d698f3c2c/watchfiles-1.1.0-cp313-cp313t-musllinux_1_1_x86_64.whl", hash = "sha256:cc08ef8b90d78bfac66f0def80240b0197008e4852c9f285907377b2947ffdcb", size = 622744, upload-time = "2025-06-15T19:06:05.066Z" }, + { url = "https://files.pythonhosted.org/packages/2c/00/70f75c47f05dea6fd30df90f047765f6fc2d6eb8b5a3921379b0b04defa2/watchfiles-1.1.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:9974d2f7dc561cce3bb88dfa8eb309dab64c729de85fba32e98d75cf24b66297", size = 402114, upload-time = "2025-06-15T19:06:06.186Z" }, + { url = "https://files.pythonhosted.org/packages/53/03/acd69c48db4a1ed1de26b349d94077cca2238ff98fd64393f3e97484cae6/watchfiles-1.1.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c68e9f1fcb4d43798ad8814c4c1b61547b014b667216cb754e606bfade587018", size = 393879, upload-time = "2025-06-15T19:06:07.369Z" }, + { url = "https://files.pythonhosted.org/packages/2f/c8/a9a2a6f9c8baa4eceae5887fecd421e1b7ce86802bcfc8b6a942e2add834/watchfiles-1.1.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:95ab1594377effac17110e1352989bdd7bdfca9ff0e5eeccd8c69c5389b826d0", size = 450026, upload-time = "2025-06-15T19:06:08.476Z" }, + { url = "https://files.pythonhosted.org/packages/fe/51/d572260d98388e6e2b967425c985e07d47ee6f62e6455cefb46a6e06eda5/watchfiles-1.1.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:fba9b62da882c1be1280a7584ec4515d0a6006a94d6e5819730ec2eab60ffe12", size = 457917, upload-time = "2025-06-15T19:06:09.988Z" }, + { url = "https://files.pythonhosted.org/packages/c6/2d/4258e52917bf9f12909b6ec314ff9636276f3542f9d3807d143f27309104/watchfiles-1.1.0-cp314-cp314-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3434e401f3ce0ed6b42569128b3d1e3af773d7ec18751b918b89cd49c14eaafb", size = 483602, upload-time = "2025-06-15T19:06:11.088Z" }, + { url = "https://files.pythonhosted.org/packages/84/99/bee17a5f341a4345fe7b7972a475809af9e528deba056f8963d61ea49f75/watchfiles-1.1.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:fa257a4d0d21fcbca5b5fcba9dca5a78011cb93c0323fb8855c6d2dfbc76eb77", size = 596758, upload-time = "2025-06-15T19:06:12.197Z" }, + { url = "https://files.pythonhosted.org/packages/40/76/e4bec1d59b25b89d2b0716b41b461ed655a9a53c60dc78ad5771fda5b3e6/watchfiles-1.1.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7fd1b3879a578a8ec2076c7961076df540b9af317123f84569f5a9ddee64ce92", size = 477601, upload-time = "2025-06-15T19:06:13.391Z" }, + { url = "https://files.pythonhosted.org/packages/1f/fa/a514292956f4a9ce3c567ec0c13cce427c158e9f272062685a8a727d08fc/watchfiles-1.1.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:62cc7a30eeb0e20ecc5f4bd113cd69dcdb745a07c68c0370cea919f373f65d9e", size = 451936, upload-time = "2025-06-15T19:06:14.656Z" }, + { url = "https://files.pythonhosted.org/packages/32/5d/c3bf927ec3bbeb4566984eba8dd7a8eb69569400f5509904545576741f88/watchfiles-1.1.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:891c69e027748b4a73847335d208e374ce54ca3c335907d381fde4e41661b13b", size = 626243, upload-time = "2025-06-15T19:06:16.232Z" }, + { url = "https://files.pythonhosted.org/packages/e6/65/6e12c042f1a68c556802a84d54bb06d35577c81e29fba14019562479159c/watchfiles-1.1.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:12fe8eaffaf0faa7906895b4f8bb88264035b3f0243275e0bf24af0436b27259", size = 623073, upload-time = "2025-06-15T19:06:17.457Z" }, + { url = "https://files.pythonhosted.org/packages/89/ab/7f79d9bf57329e7cbb0a6fd4c7bd7d0cee1e4a8ef0041459f5409da3506c/watchfiles-1.1.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:bfe3c517c283e484843cb2e357dd57ba009cff351edf45fb455b5fbd1f45b15f", size = 400872, upload-time = "2025-06-15T19:06:18.57Z" }, + { url = "https://files.pythonhosted.org/packages/df/d5/3f7bf9912798e9e6c516094db6b8932df53b223660c781ee37607030b6d3/watchfiles-1.1.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a9ccbf1f129480ed3044f540c0fdbc4ee556f7175e5ab40fe077ff6baf286d4e", size = 392877, upload-time = "2025-06-15T19:06:19.55Z" }, + { url = "https://files.pythonhosted.org/packages/0d/c5/54ec7601a2798604e01c75294770dbee8150e81c6e471445d7601610b495/watchfiles-1.1.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ba0e3255b0396cac3cc7bbace76404dd72b5438bf0d8e7cefa2f79a7f3649caa", size = 449645, upload-time = "2025-06-15T19:06:20.66Z" }, + { url = "https://files.pythonhosted.org/packages/0a/04/c2f44afc3b2fce21ca0b7802cbd37ed90a29874f96069ed30a36dfe57c2b/watchfiles-1.1.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4281cd9fce9fc0a9dbf0fc1217f39bf9cf2b4d315d9626ef1d4e87b84699e7e8", size = 457424, upload-time = "2025-06-15T19:06:21.712Z" }, + { url = "https://files.pythonhosted.org/packages/9f/b0/eec32cb6c14d248095261a04f290636da3df3119d4040ef91a4a50b29fa5/watchfiles-1.1.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6d2404af8db1329f9a3c9b79ff63e0ae7131986446901582067d9304ae8aaf7f", size = 481584, upload-time = "2025-06-15T19:06:22.777Z" }, + { url = "https://files.pythonhosted.org/packages/d1/e2/ca4bb71c68a937d7145aa25709e4f5d68eb7698a25ce266e84b55d591bbd/watchfiles-1.1.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e78b6ed8165996013165eeabd875c5dfc19d41b54f94b40e9fff0eb3193e5e8e", size = 596675, upload-time = "2025-06-15T19:06:24.226Z" }, + { url = "https://files.pythonhosted.org/packages/a1/dd/b0e4b7fb5acf783816bc950180a6cd7c6c1d2cf7e9372c0ea634e722712b/watchfiles-1.1.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:249590eb75ccc117f488e2fabd1bfa33c580e24b96f00658ad88e38844a040bb", size = 477363, upload-time = "2025-06-15T19:06:25.42Z" }, + { url = "https://files.pythonhosted.org/packages/69/c4/088825b75489cb5b6a761a4542645718893d395d8c530b38734f19da44d2/watchfiles-1.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d05686b5487cfa2e2c28ff1aa370ea3e6c5accfe6435944ddea1e10d93872147", size = 452240, upload-time = "2025-06-15T19:06:26.552Z" }, + { url = "https://files.pythonhosted.org/packages/10/8c/22b074814970eeef43b7c44df98c3e9667c1f7bf5b83e0ff0201b0bd43f9/watchfiles-1.1.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:d0e10e6f8f6dc5762adee7dece33b722282e1f59aa6a55da5d493a97282fedd8", size = 625607, upload-time = "2025-06-15T19:06:27.606Z" }, + { url = "https://files.pythonhosted.org/packages/32/fa/a4f5c2046385492b2273213ef815bf71a0d4c1943b784fb904e184e30201/watchfiles-1.1.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:af06c863f152005c7592df1d6a7009c836a247c9d8adb78fef8575a5a98699db", size = 623315, upload-time = "2025-06-15T19:06:29.076Z" }, +] + [[package]] name = "wcwidth" version = "0.2.13" @@ -1208,3 +1361,34 @@ sdist = { url = "https://files.pythonhosted.org/packages/7b/29/061ec845fb5852184 wheels = [ { url = "https://files.pythonhosted.org/packages/60/e8/c0e05e4684d13459f93d312077a9a2efbe04d59c393bc2b8802248c908d4/webcolors-24.11.1-py3-none-any.whl", hash = "sha256:515291393b4cdf0eb19c155749a096f779f7d909f7cceea072791cb9095b92e9", size = 14934, upload-time = "2024-11-11T07:43:22.529Z" }, ] + +[[package]] +name = "websockets" +version = "15.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/21/e6/26d09fab466b7ca9c7737474c52be4f76a40301b08362eb2dbc19dcc16c1/websockets-15.0.1.tar.gz", hash = "sha256:82544de02076bafba038ce055ee6412d68da13ab47f0c60cab827346de828dee", size = 177016, upload-time = "2025-03-05T20:03:41.606Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/6b/4545a0d843594f5d0771e86463606a3988b5a09ca5123136f8a76580dd63/websockets-15.0.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:3e90baa811a5d73f3ca0bcbf32064d663ed81318ab225ee4f427ad4e26e5aff3", size = 175437, upload-time = "2025-03-05T20:02:16.706Z" }, + { url = "https://files.pythonhosted.org/packages/f4/71/809a0f5f6a06522af902e0f2ea2757f71ead94610010cf570ab5c98e99ed/websockets-15.0.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:592f1a9fe869c778694f0aa806ba0374e97648ab57936f092fd9d87f8bc03665", size = 173096, upload-time = "2025-03-05T20:02:18.832Z" }, + { url = "https://files.pythonhosted.org/packages/3d/69/1a681dd6f02180916f116894181eab8b2e25b31e484c5d0eae637ec01f7c/websockets-15.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0701bc3cfcb9164d04a14b149fd74be7347a530ad3bbf15ab2c678a2cd3dd9a2", size = 173332, upload-time = "2025-03-05T20:02:20.187Z" }, + { url = "https://files.pythonhosted.org/packages/a6/02/0073b3952f5bce97eafbb35757f8d0d54812b6174ed8dd952aa08429bcc3/websockets-15.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e8b56bdcdb4505c8078cb6c7157d9811a85790f2f2b3632c7d1462ab5783d215", size = 183152, upload-time = "2025-03-05T20:02:22.286Z" }, + { url = "https://files.pythonhosted.org/packages/74/45/c205c8480eafd114b428284840da0b1be9ffd0e4f87338dc95dc6ff961a1/websockets-15.0.1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0af68c55afbd5f07986df82831c7bff04846928ea8d1fd7f30052638788bc9b5", size = 182096, upload-time = "2025-03-05T20:02:24.368Z" }, + { url = "https://files.pythonhosted.org/packages/14/8f/aa61f528fba38578ec553c145857a181384c72b98156f858ca5c8e82d9d3/websockets-15.0.1-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:64dee438fed052b52e4f98f76c5790513235efaa1ef7f3f2192c392cd7c91b65", size = 182523, upload-time = "2025-03-05T20:02:25.669Z" }, + { url = "https://files.pythonhosted.org/packages/ec/6d/0267396610add5bc0d0d3e77f546d4cd287200804fe02323797de77dbce9/websockets-15.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d5f6b181bb38171a8ad1d6aa58a67a6aa9d4b38d0f8c5f496b9e42561dfc62fe", size = 182790, upload-time = "2025-03-05T20:02:26.99Z" }, + { url = "https://files.pythonhosted.org/packages/02/05/c68c5adbf679cf610ae2f74a9b871ae84564462955d991178f95a1ddb7dd/websockets-15.0.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:5d54b09eba2bada6011aea5375542a157637b91029687eb4fdb2dab11059c1b4", size = 182165, upload-time = "2025-03-05T20:02:30.291Z" }, + { url = "https://files.pythonhosted.org/packages/29/93/bb672df7b2f5faac89761cb5fa34f5cec45a4026c383a4b5761c6cea5c16/websockets-15.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3be571a8b5afed347da347bfcf27ba12b069d9d7f42cb8c7028b5e98bbb12597", size = 182160, upload-time = "2025-03-05T20:02:31.634Z" }, + { url = "https://files.pythonhosted.org/packages/ff/83/de1f7709376dc3ca9b7eeb4b9a07b4526b14876b6d372a4dc62312bebee0/websockets-15.0.1-cp312-cp312-win32.whl", hash = "sha256:c338ffa0520bdb12fbc527265235639fb76e7bc7faafbb93f6ba80d9c06578a9", size = 176395, upload-time = "2025-03-05T20:02:33.017Z" }, + { url = "https://files.pythonhosted.org/packages/7d/71/abf2ebc3bbfa40f391ce1428c7168fb20582d0ff57019b69ea20fa698043/websockets-15.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:fcd5cf9e305d7b8338754470cf69cf81f420459dbae8a3b40cee57417f4614a7", size = 176841, upload-time = "2025-03-05T20:02:34.498Z" }, + { url = "https://files.pythonhosted.org/packages/cb/9f/51f0cf64471a9d2b4d0fc6c534f323b664e7095640c34562f5182e5a7195/websockets-15.0.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ee443ef070bb3b6ed74514f5efaa37a252af57c90eb33b956d35c8e9c10a1931", size = 175440, upload-time = "2025-03-05T20:02:36.695Z" }, + { url = "https://files.pythonhosted.org/packages/8a/05/aa116ec9943c718905997412c5989f7ed671bc0188ee2ba89520e8765d7b/websockets-15.0.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5a939de6b7b4e18ca683218320fc67ea886038265fd1ed30173f5ce3f8e85675", size = 173098, upload-time = "2025-03-05T20:02:37.985Z" }, + { url = "https://files.pythonhosted.org/packages/ff/0b/33cef55ff24f2d92924923c99926dcce78e7bd922d649467f0eda8368923/websockets-15.0.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:746ee8dba912cd6fc889a8147168991d50ed70447bf18bcda7039f7d2e3d9151", size = 173329, upload-time = "2025-03-05T20:02:39.298Z" }, + { url = "https://files.pythonhosted.org/packages/31/1d/063b25dcc01faa8fada1469bdf769de3768b7044eac9d41f734fd7b6ad6d/websockets-15.0.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:595b6c3969023ecf9041b2936ac3827e4623bfa3ccf007575f04c5a6aa318c22", size = 183111, upload-time = "2025-03-05T20:02:40.595Z" }, + { url = "https://files.pythonhosted.org/packages/93/53/9a87ee494a51bf63e4ec9241c1ccc4f7c2f45fff85d5bde2ff74fcb68b9e/websockets-15.0.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3c714d2fc58b5ca3e285461a4cc0c9a66bd0e24c5da9911e30158286c9b5be7f", size = 182054, upload-time = "2025-03-05T20:02:41.926Z" }, + { url = "https://files.pythonhosted.org/packages/ff/b2/83a6ddf56cdcbad4e3d841fcc55d6ba7d19aeb89c50f24dd7e859ec0805f/websockets-15.0.1-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0f3c1e2ab208db911594ae5b4f79addeb3501604a165019dd221c0bdcabe4db8", size = 182496, upload-time = "2025-03-05T20:02:43.304Z" }, + { url = "https://files.pythonhosted.org/packages/98/41/e7038944ed0abf34c45aa4635ba28136f06052e08fc2168520bb8b25149f/websockets-15.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:229cf1d3ca6c1804400b0a9790dc66528e08a6a1feec0d5040e8b9eb14422375", size = 182829, upload-time = "2025-03-05T20:02:48.812Z" }, + { url = "https://files.pythonhosted.org/packages/e0/17/de15b6158680c7623c6ef0db361da965ab25d813ae54fcfeae2e5b9ef910/websockets-15.0.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:756c56e867a90fb00177d530dca4b097dd753cde348448a1012ed6c5131f8b7d", size = 182217, upload-time = "2025-03-05T20:02:50.14Z" }, + { url = "https://files.pythonhosted.org/packages/33/2b/1f168cb6041853eef0362fb9554c3824367c5560cbdaad89ac40f8c2edfc/websockets-15.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:558d023b3df0bffe50a04e710bc87742de35060580a293c2a984299ed83bc4e4", size = 182195, upload-time = "2025-03-05T20:02:51.561Z" }, + { url = "https://files.pythonhosted.org/packages/86/eb/20b6cdf273913d0ad05a6a14aed4b9a85591c18a987a3d47f20fa13dcc47/websockets-15.0.1-cp313-cp313-win32.whl", hash = "sha256:ba9e56e8ceeeedb2e080147ba85ffcd5cd0711b89576b83784d8605a7df455fa", size = 176393, upload-time = "2025-03-05T20:02:53.814Z" }, + { url = "https://files.pythonhosted.org/packages/1b/6c/c65773d6cab416a64d191d6ee8a8b1c68a09970ea6909d16965d26bfed1e/websockets-15.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:e09473f095a819042ecb2ab9465aee615bd9c2028e4ef7d933600a8401c79561", size = 176837, upload-time = "2025-03-05T20:02:55.237Z" }, + { url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" }, +] diff --git a/webhook_server/app.py b/webhook_server/app.py index 082c1907..be3d4898 100644 --- a/webhook_server/app.py +++ b/webhook_server/app.py @@ -7,6 +7,7 @@ import sys from contextlib import asynccontextmanager from typing import Any, AsyncGenerator +import datetime import httpx import requests @@ -17,13 +18,17 @@ FastAPI, HTTPException, Request, + WebSocket, status, ) +from fastapi.responses import HTMLResponse, StreamingResponse +from fastapi.staticfiles import StaticFiles from webhook_server.libs.config import Config from webhook_server.libs.exceptions import RepositoryNotFoundError from webhook_server.libs.github_api import GithubWebhook -from webhook_server.utils.helpers import get_logger_with_params +from webhook_server.web.log_viewer import LogViewerController +from webhook_server.utils.helpers import get_logger_with_params, prepare_log_prefix # Constants APP_URL_ROOT_PATH: str = "/webhook_server" @@ -121,6 +126,23 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: try: LOGGER.info("Application starting up...") + + # Validate static files directory exists + if not os.path.exists(static_files_path): + raise FileNotFoundError( + f"Static files directory not found: {static_files_path}. " + f"This directory is required for serving web assets (CSS/JS). " + f"Expected structure: webhook_server/web/static/ with css/ and js/ subdirectories." + ) + + if not os.path.isdir(static_files_path): + raise NotADirectoryError( + f"Static files path exists but is not a directory: {static_files_path}. " + f"Expected a directory containing css/ and js/ subdirectories." + ) + + LOGGER.info(f"Static files directory validated: {static_files_path}") + config = Config(logger=LOGGER) root_config = config.root_data verify_github_ips = root_config.get("verify-github-ips") @@ -176,6 +198,12 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: raise finally: + # Shutdown LogViewerController singleton and close WebSocket connections + global _log_viewer_controller_singleton + if _log_viewer_controller_singleton is not None: + await _log_viewer_controller_singleton.shutdown() + LOGGER.debug("LogViewerController singleton shutdown complete") + if _lifespan_http_client: await _lifespan_http_client.aclose() LOGGER.debug("HTTP client closed") @@ -185,6 +213,10 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: FASTAPI_APP: FastAPI = FastAPI(title="webhook-server", lifespan=lifespan) +# Mount static files +static_files_path = os.path.join(os.path.dirname(__file__), "web", "static") +FASTAPI_APP.mount("/static", StaticFiles(directory=static_files_path), name="static") + @FASTAPI_APP.get(f"{APP_URL_ROOT_PATH}/healthcheck") def healthcheck() -> dict[str, Any]: @@ -196,7 +228,9 @@ async def process_webhook(request: Request, background_tasks: BackgroundTasks) - # Extract headers early for logging delivery_id = request.headers.get("X-GitHub-Delivery", "unknown-delivery") event_type = request.headers.get("X-GitHub-Event", "unknown-event") - log_context = f"[Event: {event_type}][Delivery: {delivery_id}]" + + # Use standardized log prefix format (will get repository info after parsing payload) + log_context = prepare_log_prefix(event_type, delivery_id) LOGGER.info(f"{log_context} Processing webhook") @@ -276,3 +310,164 @@ async def process_with_error_handling(_api: GithubWebhook, _logger: logging.Logg file_name = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1] if exc_tb else "unknown" error_details = f"Error type: {exc_type.__name__ if exc_type else ''}, File: {file_name}, Line: {line_no}" raise HTTPException(status_code=500, detail=f"Internal Server Error: {error_details}") + + +# Module-level singleton instance +_log_viewer_controller_singleton: LogViewerController | None = None + + +def get_log_viewer_controller() -> LogViewerController: + """Dependency to provide a singleton LogViewerController instance. + + Returns the same LogViewerController instance across all requests to ensure + proper WebSocket connection tracking and shared state management. + + Returns: + LogViewerController: The singleton instance + """ + global _log_viewer_controller_singleton + if _log_viewer_controller_singleton is None: + _log_viewer_controller_singleton = LogViewerController(logger=LOGGER) + return _log_viewer_controller_singleton + + +# Create dependency instance to avoid flake8 M511 warnings +controller_dependency = Depends(get_log_viewer_controller) + + +# Helper Functions +def parse_datetime_string(datetime_str: str | None, field_name: str) -> datetime.datetime | None: + """Parse datetime string to datetime object or raise HTTPException. + + Args: + datetime_str: The datetime string to parse (can be None) + field_name: Name of the field for error messages + + Returns: + Parsed datetime object or None if input is None + + Raises: + HTTPException: If datetime string is invalid + """ + if not datetime_str: + return None + + try: + return datetime.datetime.fromisoformat(datetime_str.replace("Z", "+00:00")) + except ValueError as e: + raise HTTPException( + status_code=400, + detail=f"Invalid {field_name} format: {datetime_str}. Expected ISO 8601 format. Error: {str(e)}", + ) + + +# Log Viewer Endpoints +@FASTAPI_APP.get("/logs", response_class=HTMLResponse) +def get_log_viewer_page(controller: LogViewerController = controller_dependency) -> HTMLResponse: + """Serve the main log viewer HTML page.""" + return controller.get_log_page() + + +@FASTAPI_APP.get("/logs/api/entries") +def get_log_entries( + hook_id: str | None = None, + pr_number: int | None = None, + repository: str | None = None, + event_type: str | None = None, + github_user: str | None = None, + level: str | None = None, + start_time: str | None = None, + end_time: str | None = None, + search: str | None = None, + limit: int = 100, + offset: int = 0, + controller: LogViewerController = controller_dependency, +) -> dict[str, Any]: + """Retrieve historical log entries with filtering and pagination.""" + # Parse datetime strings using helper function + start_datetime = parse_datetime_string(start_time, "start_time") + end_datetime = parse_datetime_string(end_time, "end_time") + + return controller.get_log_entries( + hook_id=hook_id, + pr_number=pr_number, + repository=repository, + event_type=event_type, + github_user=github_user, + level=level, + start_time=start_datetime, + end_time=end_datetime, + search=search, + limit=limit, + offset=offset, + ) + + +@FASTAPI_APP.get("/logs/api/export") +def export_logs( + format_type: str, + hook_id: str | None = None, + pr_number: int | None = None, + repository: str | None = None, + event_type: str | None = None, + github_user: str | None = None, + level: str | None = None, + start_time: str | None = None, + end_time: str | None = None, + search: str | None = None, + limit: int = 10000, + controller: LogViewerController = controller_dependency, +) -> StreamingResponse: + """Export filtered logs as JSON file.""" + # Parse datetime strings using helper function + start_datetime = parse_datetime_string(start_time, "start_time") + end_datetime = parse_datetime_string(end_time, "end_time") + + return controller.export_logs( + format_type=format_type, + hook_id=hook_id, + pr_number=pr_number, + repository=repository, + event_type=event_type, + github_user=github_user, + level=level, + start_time=start_datetime, + end_time=end_datetime, + search=search, + limit=limit, + ) + + +@FASTAPI_APP.get("/logs/api/pr-flow/{hook_id}") +def get_pr_flow_data(hook_id: str, controller: LogViewerController = controller_dependency) -> dict[str, Any]: + """Get PR flow visualization data for a specific hook ID or PR number.""" + return controller.get_pr_flow_data(hook_id) + + +@FASTAPI_APP.get("/logs/api/workflow-steps/{hook_id}") +def get_workflow_steps(hook_id: str, controller: LogViewerController = controller_dependency) -> dict[str, Any]: + """Get workflow step timeline data for a specific hook ID.""" + return controller.get_workflow_steps(hook_id) + + +@FASTAPI_APP.websocket("/logs/ws") +async def websocket_log_stream( + websocket: WebSocket, + hook_id: str | None = None, + pr_number: int | None = None, + repository: str | None = None, + event_type: str | None = None, + github_user: str | None = None, + level: str | None = None, +) -> None: + """Handle WebSocket connection for real-time log streaming.""" + controller = get_log_viewer_controller() + await controller.handle_websocket( + websocket=websocket, + hook_id=hook_id, + pr_number=pr_number, + repository=repository, + event_type=event_type, + github_user=github_user, + level=level, + ) diff --git a/webhook_server/libs/check_run_handler.py b/webhook_server/libs/check_run_handler.py index 78027b86..acdfd147 100644 --- a/webhook_server/libs/check_run_handler.py +++ b/webhook_server/libs/check_run_handler.py @@ -45,6 +45,8 @@ async def process_pull_request_check_run_webhook_data(self, pull_request: PullRe _check_run: dict[str, Any] = self.hook_data["check_run"] check_run_name: str = _check_run["name"] + self.logger.step(f"{self.log_prefix} Processing check run: {check_run_name}") # type: ignore + if self.hook_data.get("action", "") != "completed": self.logger.debug( f"{self.log_prefix} check run {check_run_name} action is {self.hook_data.get('action', 'N/A')} and not completed, skipping" @@ -63,7 +65,9 @@ async def process_pull_request_check_run_webhook_data(self, pull_request: PullRe label=AUTOMERGE_LABEL_STR, pull_request=pull_request ): try: + self.logger.step(f"{self.log_prefix} Executing auto-merge for PR #{pull_request.number}") # type: ignore await asyncio.to_thread(pull_request.merge, merge_method="SQUASH") + self.logger.step(f"{self.log_prefix} Auto-merge completed successfully") # type: ignore self.logger.info( f"{self.log_prefix} Successfully auto-merged pull request #{pull_request.number}" ) @@ -212,6 +216,16 @@ async def set_check_run_status( msg: str = f"{self.log_prefix} check run {check_run} status: {status or conclusion}" + # Log workflow steps for check run status changes + if status == QUEUED_STR: + self.logger.step(f"{self.log_prefix} Setting {check_run} check to queued") # type: ignore + elif status == IN_PROGRESS_STR: + self.logger.step(f"{self.log_prefix} Setting {check_run} check to in-progress") # type: ignore + elif conclusion == SUCCESS_STR: + self.logger.step(f"{self.log_prefix} Setting {check_run} check to success") # type: ignore + elif conclusion == FAILURE_STR: + self.logger.step(f"{self.log_prefix} Setting {check_run} check to failure") # type: ignore + try: self.logger.debug(f"{self.log_prefix} Set check run status with {kwargs}") await asyncio.to_thread(self.github_webhook.repository_by_github_app.create_check_run, **kwargs) diff --git a/webhook_server/libs/github_api.py b/webhook_server/libs/github_api.py index 8ef9299d..c74fac6c 100644 --- a/webhook_server/libs/github_api.py +++ b/webhook_server/libs/github_api.py @@ -5,7 +5,6 @@ import json import logging import os -import random from typing import Any import requests @@ -13,7 +12,6 @@ from github.Commit import Commit from github.PullRequest import PullRequest from starlette.datastructures import Headers -from stringcolor import cs from webhook_server.libs.check_run_handler import CheckRunHandler from webhook_server.libs.config import Config @@ -40,6 +38,7 @@ get_api_with_highest_rate_limit, get_apis_and_tokes_from_config, get_github_repo_api, + prepare_log_prefix, ) @@ -108,47 +107,59 @@ def __init__(self, hook_data: dict[Any, Any], headers: Headers, logger: logging. async def process(self) -> Any: event_log: str = f"Event type: {self.github_event}. event ID: {self.x_github_delivery}" + self.logger.step(f"{self.log_prefix} Starting webhook processing: {event_log}") # type: ignore if self.github_event == "ping": + self.logger.step(f"{self.log_prefix} Processing ping event") # type: ignore self.logger.debug(f"{self.log_prefix} {event_log}") return {"status": requests.codes.ok, "message": "pong"} if self.github_event == "push": + self.logger.step(f"{self.log_prefix} Processing push event") # type: ignore self.logger.debug(f"{self.log_prefix} {event_log}") return await PushHandler(github_webhook=self).process_push_webhook_data() if pull_request := await self.get_pull_request(): self.log_prefix = self.prepare_log_prefix(pull_request=pull_request) + self.logger.step(f"{self.log_prefix} Processing pull request event: {event_log}") # type: ignore self.logger.debug(f"{self.log_prefix} {event_log}") if pull_request.draft: + self.logger.step(f"{self.log_prefix} Pull request is draft, skipping processing") # type: ignore self.logger.debug(f"{self.log_prefix} Pull request is draft, doing nothing") return None + self.logger.step(f"{self.log_prefix} Initializing pull request data") # type: ignore self.last_commit = await self._get_last_commit(pull_request=pull_request) self.parent_committer = pull_request.user.login self.last_committer = getattr(self.last_commit.committer, "login", self.parent_committer) if self.github_event == "issue_comment": + self.logger.step(f"{self.log_prefix} Initializing OWNERS file handler for issue comment") # type: ignore owners_file_handler = OwnersFileHandler(github_webhook=self) owners_file_handler = await owners_file_handler.initialize(pull_request=pull_request) + self.logger.step(f"{self.log_prefix} Processing issue comment with IssueCommentHandler") # type: ignore return await IssueCommentHandler( github_webhook=self, owners_file_handler=owners_file_handler ).process_comment_webhook_data(pull_request=pull_request) elif self.github_event == "pull_request": + self.logger.step(f"{self.log_prefix} Initializing OWNERS file handler for pull request") # type: ignore owners_file_handler = OwnersFileHandler(github_webhook=self) owners_file_handler = await owners_file_handler.initialize(pull_request=pull_request) + self.logger.step(f"{self.log_prefix} Processing pull request with PullRequestHandler") # type: ignore return await PullRequestHandler( github_webhook=self, owners_file_handler=owners_file_handler ).process_pull_request_webhook_data(pull_request=pull_request) elif self.github_event == "pull_request_review": + self.logger.step(f"{self.log_prefix} Initializing OWNERS file handler for pull request review") # type: ignore owners_file_handler = OwnersFileHandler(github_webhook=self) owners_file_handler = await owners_file_handler.initialize(pull_request=pull_request) + self.logger.step(f"{self.log_prefix} Processing pull request review with PullRequestReviewHandler") # type: ignore return await PullRequestReviewHandler( github_webhook=self, owners_file_handler=owners_file_handler ).process_pull_request_review_webhook_data( @@ -156,12 +167,15 @@ async def process(self) -> Any: ) elif self.github_event == "check_run": + self.logger.step(f"{self.log_prefix} Initializing OWNERS file handler for check run") # type: ignore owners_file_handler = OwnersFileHandler(github_webhook=self) owners_file_handler = await owners_file_handler.initialize(pull_request=pull_request) + self.logger.step(f"{self.log_prefix} Processing check run with CheckRunHandler") # type: ignore if await CheckRunHandler( github_webhook=self, owners_file_handler=owners_file_handler ).process_pull_request_check_run_webhook_data(pull_request=pull_request): if self.hook_data["check_run"]["name"] != CAN_BE_MERGED_STR: + self.logger.step(f"{self.log_prefix} Checking if pull request can be merged after check run") # type: ignore return await PullRequestHandler( github_webhook=self, owners_file_handler=owners_file_handler ).check_if_can_be_merged(pull_request=pull_request) @@ -178,62 +192,14 @@ def add_api_users_to_auto_verified_and_merged_users(self) -> None: self.auto_verified_and_merged_users.append(_api.get_user().login) - def _get_reposiroty_color_for_log_prefix(self) -> str: - def _get_random_color(_colors: list[str], _json: dict[str, str]) -> str: - color = random.choice(_colors) - _json[self.repository_name] = color - - if _selected := cs(self.repository_name, color).render(): - return _selected - - return self.repository_name - - _all_colors: list[str] = [] - color_json: dict[str, str] - _colors_to_exclude = ("blue", "white", "black", "grey") - color_file: str = os.path.join(self.config.data_dir, "log-colors.json") - - for _color_name in cs.colors.values(): - _cname = _color_name["name"] - if _cname.lower() in _colors_to_exclude: - continue - - _all_colors.append(_cname) - - try: - with open(color_file) as fd: - color_json = json.load(fd) - - except Exception: - color_json = {} - - if color := color_json.get(self.repository_name, ""): - _cs_object = cs(self.repository_name, color) - if cs.find_color(_cs_object): - _str_color = _cs_object.render() - - else: - _str_color = _get_random_color(_colors=_all_colors, _json=color_json) - - else: - _str_color = _get_random_color(_colors=_all_colors, _json=color_json) - - with open(color_file, "w") as fd: - json.dump(color_json, fd) - - if _str_color: - _str_color = _str_color.replace("\x1b", "\033") - return _str_color - - return self.repository_name - def prepare_log_prefix(self, pull_request: PullRequest | None = None) -> str: - _repository_color = self._get_reposiroty_color_for_log_prefix() - - return ( - f"{_repository_color} [{self.github_event}][{self.x_github_delivery}][{self.api_user}][PR {pull_request.number}]:" - if pull_request - else f"{_repository_color} [{self.github_event}][{self.x_github_delivery}][{self.api_user}]:" + return prepare_log_prefix( + event_type=self.github_event, + delivery_id=self.x_github_delivery, + repository_name=self.repository_name, + api_user=self.api_user, + pr_number=pull_request.number if pull_request else None, + data_dir=self.config.data_dir, ) def _repo_data_from_config(self, repository_config: dict[str, Any]) -> None: diff --git a/webhook_server/libs/issue_comment_handler.py b/webhook_server/libs/issue_comment_handler.py index 3987fd97..40bf6b03 100644 --- a/webhook_server/libs/issue_comment_handler.py +++ b/webhook_server/libs/issue_comment_handler.py @@ -59,11 +59,14 @@ def __init__(self, github_webhook: "GithubWebhook", owners_file_handler: OwnersF async def process_comment_webhook_data(self, pull_request: PullRequest) -> None: comment_action = self.hook_data["action"] + self.logger.step(f"{self.log_prefix} Starting issue comment processing: action={comment_action}") # type: ignore if comment_action in ("edited", "deleted"): + self.logger.step(f"{self.log_prefix} Skipping comment processing: action is {comment_action}") # type: ignore self.logger.debug(f"{self.log_prefix} Not processing comment. action is {comment_action}") return + self.logger.step(f"{self.log_prefix} Processing issue comment for issue {self.hook_data['issue']['number']}") # type: ignore self.logger.info(f"{self.log_prefix} Processing issue {self.hook_data['issue']['number']}") body: str = self.hook_data["comment"]["body"] @@ -74,8 +77,12 @@ async def process_comment_webhook_data(self, pull_request: PullRequest) -> None: _user_commands: list[str] = [_cmd.strip("/") for _cmd in body.strip().splitlines() if _cmd.startswith("/")] + if _user_commands: + self.logger.step(f"{self.log_prefix} Found {len(_user_commands)} user commands: {_user_commands}") # type: ignore + user_login: str = self.hook_data["sender"]["login"] for user_command in _user_commands: + self.logger.step(f"{self.log_prefix} Executing user command: /{user_command} by {user_login}") # type: ignore await self.user_commands( pull_request=pull_request, command=user_command, diff --git a/webhook_server/libs/labels_handler.py b/webhook_server/libs/labels_handler.py index 491bea9b..6c7cf441 100644 --- a/webhook_server/libs/labels_handler.py +++ b/webhook_server/libs/labels_handler.py @@ -45,6 +45,7 @@ async def pull_request_labels_names(self, pull_request: PullRequest) -> list[str return [lb.name for lb in labels] async def _remove_label(self, pull_request: PullRequest, label: str) -> bool: + self.logger.step(f"{self.log_prefix} Removing label '{label}' from PR") # type: ignore self.logger.debug(f"{self.log_prefix} Removing label {label}") try: if await self.label_exists_in_pull_request(pull_request=pull_request, label=label): @@ -60,6 +61,7 @@ async def _remove_label(self, pull_request: PullRequest, label: str) -> bool: async def _add_label(self, pull_request: PullRequest, label: str) -> None: label = label.strip() + self.logger.step(f"{self.log_prefix} Adding label '{label}' to PR") # type: ignore self.logger.debug(f"{self.log_prefix} Adding label {label}") if len(label) > 49: self.logger.debug(f"{label} is too long, not adding.") @@ -208,6 +210,7 @@ def get_size(self, pull_request: PullRequest) -> str: async def add_size_label(self, pull_request: PullRequest) -> None: """Add a size label to the pull request based on its additions and deletions.""" + self.logger.step(f"{self.log_prefix} Calculating and applying PR size label") # type: ignore size_label = self.get_size(pull_request=pull_request) self.logger.debug(f"{self.log_prefix} size label is {size_label}") if not size_label: @@ -228,6 +231,7 @@ async def add_size_label(self, pull_request: PullRequest) -> None: await self._remove_label(pull_request=pull_request, label=exists_size_label[0]) await self._add_label(pull_request=pull_request, label=size_label) + self.logger.step(f"{self.log_prefix} Applied size label '{size_label}' to PR") # type: ignore async def label_by_user_comment( self, diff --git a/webhook_server/libs/log_parser.py b/webhook_server/libs/log_parser.py new file mode 100644 index 00000000..21c2b1f3 --- /dev/null +++ b/webhook_server/libs/log_parser.py @@ -0,0 +1,383 @@ +"""Log parsing and filtering functionality for webhook server logs.""" + +import asyncio +import datetime +import re +from dataclasses import dataclass +from pathlib import Path +from typing import Any, AsyncGenerator + +from simple_logger.logger import get_logger + + +@dataclass +class LogEntry: + """Represents a parsed log entry with structured data.""" + + timestamp: datetime.datetime + level: str + logger_name: str + message: str + hook_id: str | None = None + event_type: str | None = None + repository: str | None = None + pr_number: int | None = None + github_user: str | None = None + + def to_dict(self) -> dict[str, Any]: + """Convert LogEntry to dictionary for JSON serialization.""" + return { + "timestamp": self.timestamp.isoformat(), + "level": self.level, + "logger_name": self.logger_name, + "message": self.message, + "hook_id": self.hook_id, + "event_type": self.event_type, + "repository": self.repository, + "pr_number": self.pr_number, + "github_user": self.github_user, + } + + +class LogParser: + """Parser for webhook server log files. + + Parses logs generated by GithubWebhook.prepare_log_prefix() function which creates + structured log prefixes for webhook processing. + + Log files are typically stored in the configured data directory under a 'logs' subdirectory. + """ + + def __init__(self) -> None: + """Initialize LogParser with logger.""" + self.logger = get_logger(name="log_parser") + + # Regex pattern for parsing production logs from prepare_log_prefix() in github_api.py + # Format from prepare_log_prefix(): + # With PR: "{colored_repo} [{event}][{delivery_id}][{user}][PR {number}]: {message}" + # Without PR: "{colored_repo} [{event}][{delivery_id}][{user}]: {message}" + # Full log format: "timestamp logger level colored_repo [event][delivery_id][user][PR number]: message" + # Example: "2025-07-31T10:30:00.123000 GithubWebhook INFO repo-name [pull_request][abc123][user][PR 123]: Processing webhook" + LOG_PATTERN = re.compile( + r"^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+) (\w+) (?:\x1b\[[\d;]*m)?(\w+)(?:\x1b\[[\d;]*m)? (.+)$" + ) + + # Pattern to extract GitHub context from prepare_log_prefix format + # Matches: colored_repo [event][delivery_id][user][PR number]: message + GITHUB_CONTEXT_PATTERN = re.compile( + r"(?:\x1b\[[0-9;]*m)?([^\x1b\[\s]+)(?:\x1b\[[0-9;]*m)? \[([^\]]+)\]\[([^\]]+)\]\[([^\]]+)\](?:\[PR (\d+)\])?: (.+)" + ) + + ANSI_ESCAPE_PATTERN = re.compile(r"\x1b\[[0-9;]*m") + + def is_workflow_step(self, entry: LogEntry) -> bool: + """ + Check if a log entry is a workflow step (logger.step call). + + Args: + entry: LogEntry to check + + Returns: + True if this is a workflow step entry + """ + return entry.level.upper() == "STEP" + + def extract_workflow_steps(self, entries: list[LogEntry], hook_id: str) -> list[LogEntry]: + """ + Extract workflow step entries for a specific hook ID. + + Args: + entries: List of log entries to filter + hook_id: Hook ID to filter by + + Returns: + List of workflow step entries for the specified hook ID + """ + return [entry for entry in entries if entry.hook_id == hook_id and self.is_workflow_step(entry)] + + def parse_log_entry(self, log_line: str) -> LogEntry | None: + """ + Parse a single log line into a LogEntry object. + + Args: + log_line: Raw log line string + + Returns: + LogEntry object if parsing successful, None otherwise + """ + if not log_line.strip(): + return None + + # Parse production log format + match = self.LOG_PATTERN.match(log_line.strip()) + if not match: + return None + + timestamp_str, logger_name, level, message = match.groups() + + # Parse ISO timestamp format: "2025-07-31T10:30:00.123000" + try: + timestamp = datetime.datetime.fromisoformat(timestamp_str) + except ValueError: + return None + + # Extract GitHub webhook context from prepare_log_prefix format + repository, event_type, hook_id, github_user, pr_number, cleaned_message = self._extract_github_context(message) + + return LogEntry( + timestamp=timestamp, + level=level, + logger_name=logger_name, + message=cleaned_message, + hook_id=hook_id, + event_type=event_type, + repository=repository, + pr_number=pr_number, + github_user=github_user, + ) + + def _extract_github_context( + self, message: str + ) -> tuple[str | None, str | None, str | None, str | None, int | None, str]: + """Extract GitHub context from prepare_log_prefix format. + + Returns: + Tuple of (repository, event_type, hook_id, github_user, pr_number, cleaned_message) + """ + match = self.GITHUB_CONTEXT_PATTERN.search(message) + if match: + repository = match.group(1) + event_type = match.group(2) + hook_id = match.group(3) + github_user = match.group(4) + pr_number_str = match.group(5) # Optional PR number + cleaned_message = match.group(6) + + # Parse PR number if present + pr_number = None + if pr_number_str: + try: + pr_number = int(pr_number_str) + except ValueError: + pass + + # Clean ANSI codes from message + cleaned_message = self.ANSI_ESCAPE_PATTERN.sub("", cleaned_message) + + return repository, event_type, hook_id, github_user, pr_number, cleaned_message + + # No GitHub context found, return original message cleaned of ANSI codes + cleaned_message = self.ANSI_ESCAPE_PATTERN.sub("", message) + return None, None, None, None, None, cleaned_message + + def parse_log_file(self, file_path: Path) -> list[LogEntry]: + """ + Parse an entire log file and return list of LogEntry objects. + + Args: + file_path: Path to the log file + + Returns: + List of successfully parsed LogEntry objects + """ + entries: list[LogEntry] = [] + total_lines = 0 + failed_lines = 0 + + try: + with open(file_path, "r", encoding="utf-8") as f: + for line_num, line in enumerate(f, 1): + total_lines += 1 + entry = self.parse_log_entry(line) + if entry: + entries.append(entry) + else: + failed_lines += 1 + + except OSError as e: + self.logger.error(f"Failed to read log file {file_path}: {e}") + except UnicodeDecodeError as e: + self.logger.error(f"Failed to decode log file {file_path}: {e}") + + return entries + + async def tail_log_file(self, file_path: Path, follow: bool = True) -> AsyncGenerator[LogEntry, None]: + """ + Tail a log file and yield new LogEntry objects as they are added. + + Args: + file_path: Path to the log file to monitor + follow: Whether to continue monitoring for new entries + + Yields: + LogEntry objects for new log lines + """ + # Start from the end of the file + if not file_path.exists(): + return + + with open(file_path, "r", encoding="utf-8") as f: + # Move to end of file + f.seek(0, 2) + + while True: + line = f.readline() + if line: + entry = self.parse_log_entry(line) + if entry: + yield entry + elif follow: + # No new data, wait a bit before checking again + await asyncio.sleep(0.1) + else: + # Not following, exit when no more data + break + + async def monitor_log_directory(self, log_dir: Path, pattern: str = "*.log") -> AsyncGenerator[LogEntry, None]: + """ + Monitor a directory for log files and yield new entries from all files. + + Args: + log_dir: Directory path containing log files + pattern: Glob pattern for log files (default: "*.log") + + Yields: + LogEntry objects from all monitored log files + """ + if not log_dir.exists() or not log_dir.is_dir(): + return + + # Find all existing log files including rotated ones + log_files: list[Path] = [] + log_files.extend(log_dir.glob("*.log")) + # Only monitor current log file, not rotated ones for real-time + current_log_files = [ + f for f in log_files if not any(f.name.endswith(ext) for ext in [".1", ".2", ".3", ".4", ".5"]) + ] + + if not current_log_files: + return + + # Monitor the most recent current log file (not rotated) + # Sort by modification time to get the most recent file + current_log_files.sort(key=lambda f: f.stat().st_mtime, reverse=True) + most_recent_file = current_log_files[0] + + async for entry in self.tail_log_file(most_recent_file, follow=True): + yield entry + + +class LogFilter: + """Filter log entries based on various criteria.""" + + def filter_entries( + self, + entries: list[LogEntry], + hook_id: str | None = None, + pr_number: int | None = None, + repository: str | None = None, + event_type: str | None = None, + github_user: str | None = None, + level: str | None = None, + start_time: datetime.datetime | None = None, + end_time: datetime.datetime | None = None, + search_text: str | None = None, + limit: int | None = None, + offset: int | None = None, + ) -> list[LogEntry]: + """ + Filter log entries based on provided criteria. + + Args: + entries: List of LogEntry objects to filter + hook_id: Filter by exact hook ID match + pr_number: Filter by exact PR number match + repository: Filter by exact repository match + event_type: Filter by exact event type match + github_user: Filter by exact GitHub user match + level: Filter by exact log level match + start_time: Filter entries after this timestamp + end_time: Filter entries before this timestamp + search_text: Filter by text search in message (case-insensitive) + limit: Maximum number of entries to return + offset: Number of entries to skip (for pagination) + + Returns: + Filtered list of LogEntry objects + """ + filtered = entries[:] + + # Apply filters + if hook_id is not None: + filtered = [e for e in filtered if e.hook_id == hook_id] + + if pr_number is not None: + filtered = [e for e in filtered if e.pr_number == pr_number] + + if repository is not None: + filtered = [e for e in filtered if e.repository == repository] + + if event_type is not None: + filtered = [e for e in filtered if e.event_type == event_type] + + if github_user is not None: + filtered = [e for e in filtered if e.github_user == github_user] + + if level is not None: + filtered = [e for e in filtered if e.level == level] + + if start_time is not None: + filtered = [e for e in filtered if e.timestamp >= start_time] + + if end_time is not None: + filtered = [e for e in filtered if e.timestamp <= end_time] + + if search_text is not None: + search_lower = search_text.lower() + filtered = [e for e in filtered if search_lower in e.message.lower()] + + # Apply pagination + if offset is not None: + filtered = filtered[offset:] + + if limit is not None: + filtered = filtered[:limit] + + return filtered + + def get_unique_values(self, entries: list[LogEntry], field: str) -> list[str]: + """ + Get unique values for a specific field across all entries. + + Args: + entries: List of LogEntry objects + field: Field name to get unique values for + + Returns: + List of unique non-None values for the specified field + """ + values = set() + for entry in entries: + value = getattr(entry, field, None) + if value is not None: + values.add(str(value)) + return sorted(list(values)) + + def get_entry_count_by_field(self, entries: list[LogEntry], field: str) -> dict[str, int]: + """ + Get count of entries grouped by a specific field. + + Args: + entries: List of LogEntry objects + field: Field name to group by + + Returns: + Dictionary mapping field values to entry counts + """ + counts: dict[str, int] = {} + for entry in entries: + value = getattr(entry, field, None) + if value is not None: + key = str(value) + counts[key] = counts.get(key, 0) + 1 + return counts diff --git a/webhook_server/libs/owners_files_handler.py b/webhook_server/libs/owners_files_handler.py index e9249d8e..ee1f9d7c 100644 --- a/webhook_server/libs/owners_files_handler.py +++ b/webhook_server/libs/owners_files_handler.py @@ -238,23 +238,34 @@ async def owners_data_for_changed_files(self) -> dict[str, dict[str, Any]]: async def assign_reviewers(self, pull_request: PullRequest) -> None: self._ensure_initialized() + self.logger.step(f"{self.log_prefix} Starting reviewer assignment based on OWNERS files") # type: ignore self.logger.info(f"{self.log_prefix} Assign reviewers") _to_add: list[str] = list(set(self.all_pull_request_reviewers)) self.logger.debug(f"{self.log_prefix} Reviewers to add: {', '.join(_to_add)}") + if _to_add: + self.logger.step(f"{self.log_prefix} Assigning {len(_to_add)} reviewers to PR") # type: ignore + else: + self.logger.step(f"{self.log_prefix} No reviewers to assign") # type: ignore + return + for reviewer in _to_add: if reviewer != pull_request.user.login: self.logger.debug(f"{self.log_prefix} Adding reviewer {reviewer}") try: await asyncio.to_thread(pull_request.create_review_request, [reviewer]) + self.logger.step(f"{self.log_prefix} Successfully assigned reviewer {reviewer}") # type: ignore except GithubException as ex: + self.logger.step(f"{self.log_prefix} Failed to assign reviewer {reviewer}") # type: ignore self.logger.debug(f"{self.log_prefix} Failed to add reviewer {reviewer}. {ex}") await asyncio.to_thread( pull_request.create_issue_comment, f"{reviewer} can not be added as reviewer. {ex}" ) + self.logger.step(f"{self.log_prefix} Reviewer assignment completed") # type: ignore + async def is_user_valid_to_run_commands(self, pull_request: PullRequest, reviewed_user: str) -> bool: self._ensure_initialized() diff --git a/webhook_server/libs/pull_request_handler.py b/webhook_server/libs/pull_request_handler.py index 4d575982..0bfd7e19 100644 --- a/webhook_server/libs/pull_request_handler.py +++ b/webhook_server/libs/pull_request_handler.py @@ -58,6 +58,7 @@ def __init__(self, github_webhook: "GithubWebhook", owners_file_handler: OwnersF async def process_pull_request_webhook_data(self, pull_request: PullRequest) -> None: hook_action: str = self.hook_data["action"] + self.logger.step(f"{self.log_prefix} Starting pull request processing: action={hook_action}") # type: ignore self.logger.info(f"{self.log_prefix} hook_action is: {hook_action}") self.logger.debug(f"{self.log_prefix} pull_request: {pull_request.title} ({pull_request.number})") @@ -70,6 +71,7 @@ async def process_pull_request_webhook_data(self, pull_request: PullRequest) -> await self.runner_handler.run_conventional_title_check(pull_request=pull_request) if hook_action in ("opened", "reopened", "ready_for_review"): + self.logger.step(f"{self.log_prefix} Processing PR {hook_action} event: initializing new pull request") # type: ignore tasks: list[Coroutine[Any, Any, Any]] = [] if hook_action in ("opened", "ready_for_review"): @@ -89,6 +91,7 @@ async def process_pull_request_webhook_data(self, pull_request: PullRequest) -> await self.set_pull_request_automerge(pull_request=pull_request) if hook_action == "synchronize": + self.logger.step(f"{self.log_prefix} Processing PR synchronize event: handling new commits") # type: ignore sync_tasks: list[Coroutine[Any, Any, Any]] = [] sync_tasks.append(self.process_opened_or_synchronize_pull_request(pull_request=pull_request)) @@ -101,9 +104,11 @@ async def process_pull_request_webhook_data(self, pull_request: PullRequest) -> self.logger.error(f"{self.log_prefix} Async task failed: {result}") if hook_action == "closed": + self.logger.step(f"{self.log_prefix} Processing PR closed event: cleaning up resources") # type: ignore await self.close_issue_for_merged_or_closed_pr(pull_request=pull_request, hook_action=hook_action) await self.delete_remote_tag_for_merged_or_closed_pr(pull_request=pull_request) if is_merged := pull_request_data.get("merged", False): + self.logger.step(f"{self.log_prefix} PR was merged: processing post-merge tasks") # type: ignore self.logger.info(f"{self.log_prefix} PR is merged") for _label in pull_request.labels: @@ -129,6 +134,8 @@ async def process_pull_request_webhook_data(self, pull_request: PullRequest) -> labeled = self.hook_data["label"]["name"] labeled_lower = labeled.lower() + self.logger.step(f"{self.log_prefix} Processing label {hook_action} event: {labeled}") # type: ignore + if labeled_lower == CAN_BE_MERGED_STR: return @@ -414,39 +421,61 @@ async def close_issue_for_merged_or_closed_pr(self, pull_request: PullRequest, h break async def process_opened_or_synchronize_pull_request(self, pull_request: PullRequest) -> None: - tasks: list[Coroutine[Any, Any, Any]] = [] + self.logger.step(f"{self.log_prefix} Starting PR processing workflow") # type: ignore - tasks.append(self.owners_file_handler.assign_reviewers(pull_request=pull_request)) - tasks.append( + # Stage 1: Initial setup and check queue tasks + self.logger.step(f"{self.log_prefix} Stage: Initial setup and check queuing") # type: ignore + setup_tasks: list[Coroutine[Any, Any, Any]] = [] + + setup_tasks.append(self.owners_file_handler.assign_reviewers(pull_request=pull_request)) + setup_tasks.append( self.labels_handler._add_label( pull_request=pull_request, label=f"{BRANCH_LABEL_PREFIX}{pull_request.base.ref}", ) ) - tasks.append(self.label_pull_request_by_merge_state(pull_request=pull_request)) - tasks.append(self.check_run_handler.set_merge_check_queued()) - tasks.append(self.check_run_handler.set_run_tox_check_queued()) - tasks.append(self.check_run_handler.set_run_pre_commit_check_queued()) - tasks.append(self.check_run_handler.set_python_module_install_queued()) - tasks.append(self.check_run_handler.set_container_build_queued()) - tasks.append(self._process_verified_for_update_or_new_pull_request(pull_request=pull_request)) - tasks.append(self.labels_handler.add_size_label(pull_request=pull_request)) - tasks.append(self.add_pull_request_owner_as_assingee(pull_request=pull_request)) - - tasks.append(self.runner_handler.run_tox(pull_request=pull_request)) - tasks.append(self.runner_handler.run_pre_commit(pull_request=pull_request)) - tasks.append(self.runner_handler.run_install_python_module(pull_request=pull_request)) - tasks.append(self.runner_handler.run_build_container(pull_request=pull_request)) + setup_tasks.append(self.label_pull_request_by_merge_state(pull_request=pull_request)) + setup_tasks.append(self.check_run_handler.set_merge_check_queued()) + setup_tasks.append(self.check_run_handler.set_run_tox_check_queued()) + setup_tasks.append(self.check_run_handler.set_run_pre_commit_check_queued()) + setup_tasks.append(self.check_run_handler.set_python_module_install_queued()) + setup_tasks.append(self.check_run_handler.set_container_build_queued()) + setup_tasks.append(self._process_verified_for_update_or_new_pull_request(pull_request=pull_request)) + setup_tasks.append(self.labels_handler.add_size_label(pull_request=pull_request)) + setup_tasks.append(self.add_pull_request_owner_as_assingee(pull_request=pull_request)) if self.github_webhook.conventional_title: - tasks.append(self.check_run_handler.set_conventional_title_queued()) - tasks.append(self.runner_handler.run_conventional_title_check(pull_request=pull_request)) + setup_tasks.append(self.check_run_handler.set_conventional_title_queued()) - results = await asyncio.gather(*tasks, return_exceptions=True) + self.logger.step(f"{self.log_prefix} Executing setup tasks") # type: ignore + setup_results = await asyncio.gather(*setup_tasks, return_exceptions=True) - for result in results: + for result in setup_results: if isinstance(result, Exception): - self.logger.error(f"{self.log_prefix} Async task failed: {result}") + self.logger.error(f"{self.log_prefix} Setup task failed: {result}") + + self.logger.step(f"{self.log_prefix} Setup tasks completed") # type: ignore + + # Stage 2: CI/CD execution tasks + self.logger.step(f"{self.log_prefix} Stage: CI/CD execution") # type: ignore + ci_tasks: list[Coroutine[Any, Any, Any]] = [] + + ci_tasks.append(self.runner_handler.run_tox(pull_request=pull_request)) + ci_tasks.append(self.runner_handler.run_pre_commit(pull_request=pull_request)) + ci_tasks.append(self.runner_handler.run_install_python_module(pull_request=pull_request)) + ci_tasks.append(self.runner_handler.run_build_container(pull_request=pull_request)) + + if self.github_webhook.conventional_title: + ci_tasks.append(self.runner_handler.run_conventional_title_check(pull_request=pull_request)) + + self.logger.step(f"{self.log_prefix} Executing CI/CD tasks") # type: ignore + ci_results = await asyncio.gather(*ci_tasks, return_exceptions=True) + + for result in ci_results: + if isinstance(result, Exception): + self.logger.error(f"{self.log_prefix} CI/CD task failed: {result}") + + self.logger.step(f"{self.log_prefix} PR processing workflow completed") # type: ignore async def create_issue_for_new_pull_request(self, pull_request: PullRequest) -> None: if not self.github_webhook.create_issue_for_new_pr: @@ -583,6 +612,7 @@ async def check_if_can_be_merged(self, pull_request: PullRequest) -> None: PR status is not 'dirty'. PR has no changed requests from approvers. """ + self.logger.step(f"{self.log_prefix} Starting merge eligibility check") # type: ignore if self.skip_if_pull_request_already_merged(pull_request=pull_request): self.logger.debug(f"{self.log_prefix} Pull request already merged") return diff --git a/webhook_server/libs/push_handler.py b/webhook_server/libs/push_handler.py index 641d4f73..d8c3121c 100644 --- a/webhook_server/libs/push_handler.py +++ b/webhook_server/libs/push_handler.py @@ -22,18 +22,24 @@ def __init__(self, github_webhook: "GithubWebhook"): self.runner_handler = RunnerHandler(github_webhook=self.github_webhook) async def process_push_webhook_data(self) -> None: + self.logger.step(f"{self.log_prefix} Starting push webhook processing") # type: ignore tag = re.search(r"refs/tags/?(.*)", self.hook_data["ref"]) if tag: tag_name = tag.group(1) + self.logger.step(f"{self.log_prefix} Processing tag push: {tag_name}") # type: ignore self.logger.info(f"{self.log_prefix} Processing push for tag: {tag.group(1)}") self.logger.debug(f"{self.log_prefix} Tag: {tag_name}") if self.github_webhook.pypi: + self.logger.step(f"{self.log_prefix} Starting PyPI upload for tag: {tag_name}") # type: ignore self.logger.info(f"{self.log_prefix} Processing upload to pypi for tag: {tag_name}") await self.upload_to_pypi(tag_name=tag_name) if self.github_webhook.build_and_push_container and self.github_webhook.container_release: + self.logger.step(f"{self.log_prefix} Starting container build and push for tag: {tag_name}") # type: ignore self.logger.info(f"{self.log_prefix} Processing build and push container for tag: {tag_name}") await self.runner_handler.run_build_container(push=True, set_check=False, tag=tag_name) + else: + self.logger.step(f"{self.log_prefix} Non-tag push detected, skipping processing") # type: ignore async def upload_to_pypi(self, tag_name: str) -> None: def _issue_on_error(_error: str) -> None: @@ -44,6 +50,7 @@ def _issue_on_error(_error: str) -> None: """, ) + self.logger.step(f"{self.log_prefix} Starting PyPI upload process for tag: {tag_name}") # type: ignore clone_repo_dir = f"{self.github_webhook.clone_repo_dir}-{uuid4()}" uv_cmd_dir = f"--directory {clone_repo_dir}" self.logger.info(f"{self.log_prefix} Start uploading to pypi") @@ -83,6 +90,7 @@ def _issue_on_error(_error: str) -> None: _error = self.check_run_handler.get_check_run_text(out=out, err=err) return _issue_on_error(_error=_error) + self.logger.step(f"{self.log_prefix} PyPI upload completed successfully for tag: {tag_name}") # type: ignore self.logger.info(f"{self.log_prefix} Publish to pypi finished") if self.github_webhook.slack_webhook_url: message: str = f""" diff --git a/webhook_server/libs/runner_handler.py b/webhook_server/libs/runner_handler.py index 579df20f..888ff182 100644 --- a/webhook_server/libs/runner_handler.py +++ b/webhook_server/libs/runner_handler.py @@ -176,8 +176,11 @@ async def run_podman_command(self, command: str) -> tuple[bool, str, str]: async def run_tox(self, pull_request: PullRequest) -> None: if not self.github_webhook.tox: + self.logger.debug(f"{self.log_prefix} Tox not configured for this repository") return + self.logger.step(f"{self.log_prefix} Starting tox tests execution") # type: ignore + if await self.check_run_handler.is_check_run_in_progress(check_run=TOX_STR): self.logger.debug(f"{self.log_prefix} Check run is in progress, re-running {TOX_STR}.") @@ -192,8 +195,11 @@ async def run_tox(self, pull_request: PullRequest) -> None: tests = _tox_tests.replace(" ", "") cmd += f" -e {tests}" + self.logger.step(f"{self.log_prefix} Setting tox check status to in-progress") # type: ignore await self.check_run_handler.set_run_tox_check_in_progress() self.logger.debug(f"{self.log_prefix} Tox command to run: {cmd}") + + self.logger.step(f"{self.log_prefix} Preparing repository clone for tox execution") # type: ignore async with self._prepare_cloned_repo_dir(clone_repo_dir=clone_repo_dir, pull_request=pull_request) as _res: output: dict[str, Any] = { "title": "Tox", @@ -201,28 +207,39 @@ async def run_tox(self, pull_request: PullRequest) -> None: "text": None, } if not _res[0]: + self.logger.error(f"{self.log_prefix} Repository preparation failed for tox") output["text"] = self.check_run_handler.get_check_run_text(out=_res[1], err=_res[2]) return await self.check_run_handler.set_run_tox_check_failure(output=output) + self.logger.step(f"{self.log_prefix} Executing tox command") # type: ignore rc, out, err = await run_command(command=cmd, log_prefix=self.log_prefix) output["text"] = self.check_run_handler.get_check_run_text(err=err, out=out) if rc: + self.logger.step(f"{self.log_prefix} Tox tests completed successfully") # type: ignore return await self.check_run_handler.set_run_tox_check_success(output=output) else: + self.logger.step(f"{self.log_prefix} Tox tests failed") # type: ignore return await self.check_run_handler.set_run_tox_check_failure(output=output) async def run_pre_commit(self, pull_request: PullRequest) -> None: if not self.github_webhook.pre_commit: + self.logger.debug(f"{self.log_prefix} Pre-commit not configured for this repository") return + self.logger.step(f"{self.log_prefix} Starting pre-commit checks execution") # type: ignore + if await self.check_run_handler.is_check_run_in_progress(check_run=PRE_COMMIT_STR): self.logger.debug(f"{self.log_prefix} Check run is in progress, re-running {PRE_COMMIT_STR}.") clone_repo_dir = f"{self.github_webhook.clone_repo_dir}-{uuid4()}" cmd = f" uvx --directory {clone_repo_dir} {PRE_COMMIT_STR} run --all-files" + + self.logger.step(f"{self.log_prefix} Setting pre-commit check status to in-progress") # type: ignore await self.check_run_handler.set_run_pre_commit_check_in_progress() + + self.logger.step(f"{self.log_prefix} Preparing repository clone for pre-commit execution") # type: ignore async with self._prepare_cloned_repo_dir(pull_request=pull_request, clone_repo_dir=clone_repo_dir) as _res: output: dict[str, Any] = { "title": "Pre-Commit", @@ -230,16 +247,20 @@ async def run_pre_commit(self, pull_request: PullRequest) -> None: "text": None, } if not _res[0]: + self.logger.error(f"{self.log_prefix} Repository preparation failed for pre-commit") output["text"] = self.check_run_handler.get_check_run_text(out=_res[1], err=_res[2]) return await self.check_run_handler.set_run_pre_commit_check_failure(output=output) + self.logger.step(f"{self.log_prefix} Executing pre-commit command") # type: ignore rc, out, err = await run_command(command=cmd, log_prefix=self.log_prefix) output["text"] = self.check_run_handler.get_check_run_text(err=err, out=out) if rc: + self.logger.step(f"{self.log_prefix} Pre-commit checks completed successfully") # type: ignore return await self.check_run_handler.set_run_pre_commit_check_success(output=output) else: + self.logger.step(f"{self.log_prefix} Pre-commit checks failed") # type: ignore return await self.check_run_handler.set_run_pre_commit_check_failure(output=output) async def run_build_container( @@ -255,6 +276,8 @@ async def run_build_container( if not self.github_webhook.build_and_push_container: return + self.logger.step(f"{self.log_prefix} Starting container build process") # type: ignore + if ( self.owners_file_handler and reviewed_user @@ -271,6 +294,7 @@ async def run_build_container( if await self.check_run_handler.is_check_run_in_progress(check_run=BUILD_CONTAINER_STR) and not is_merged: self.logger.info(f"{self.log_prefix} Check run is in progress, re-running {BUILD_CONTAINER_STR}.") + self.logger.step(f"{self.log_prefix} Setting container build check status to in-progress") # type: ignore await self.check_run_handler.set_container_build_in_progress() _container_repository_and_tag = self.github_webhook.container_repository_and_tag( @@ -291,6 +315,7 @@ async def run_build_container( podman_build_cmd: str = f"podman build {build_cmd}" self.logger.debug(f"{self.log_prefix} Podman build command to run: {podman_build_cmd}") + self.logger.step(f"{self.log_prefix} Preparing repository clone for container build") # type: ignore async with self._prepare_cloned_repo_dir( pull_request=pull_request, is_merged=is_merged, @@ -307,22 +332,27 @@ async def run_build_container( if pull_request and set_check: return await self.check_run_handler.set_container_build_failure(output=output) + self.logger.step(f"{self.log_prefix} Executing container build command") # type: ignore build_rc, build_out, build_err = await self.run_podman_command(command=podman_build_cmd) output["text"] = self.check_run_handler.get_check_run_text(err=build_err, out=build_out) if build_rc: + self.logger.step(f"{self.log_prefix} Container build completed successfully") # type: ignore self.logger.info(f"{self.log_prefix} Done building {_container_repository_and_tag}") if pull_request and set_check: return await self.check_run_handler.set_container_build_success(output=output) else: + self.logger.step(f"{self.log_prefix} Container build failed") # type: ignore self.logger.error(f"{self.log_prefix} Failed to build {_container_repository_and_tag}") if pull_request and set_check: return await self.check_run_handler.set_container_build_failure(output=output) if push and build_rc: + self.logger.step(f"{self.log_prefix} Starting container push to registry") # type: ignore cmd = f"podman push --creds {self.github_webhook.container_repository_username}:{self.github_webhook.container_repository_password} {_container_repository_and_tag}" push_rc, _, _ = await self.run_podman_command(command=cmd) if push_rc: + self.logger.step(f"{self.log_prefix} Container push completed successfully") # type: ignore push_msg: str = f"New container for {_container_repository_and_tag} published" if pull_request: await asyncio.to_thread(pull_request.create_issue_comment, push_msg) @@ -357,12 +387,16 @@ async def run_install_python_module(self, pull_request: PullRequest) -> None: if not self.github_webhook.pypi: return + self.logger.step(f"{self.log_prefix} Starting Python module installation") # type: ignore + if await self.check_run_handler.is_check_run_in_progress(check_run=PYTHON_MODULE_INSTALL_STR): self.logger.info(f"{self.log_prefix} Check run is in progress, re-running {PYTHON_MODULE_INSTALL_STR}.") clone_repo_dir = f"{self.github_webhook.clone_repo_dir}-{uuid4()}" self.logger.info(f"{self.log_prefix} Installing python module") + self.logger.step(f"{self.log_prefix} Setting Python module install check status to in-progress") # type: ignore await self.check_run_handler.set_python_module_install_in_progress() + self.logger.step(f"{self.log_prefix} Preparing repository clone for Python module installation") # type: ignore async with self._prepare_cloned_repo_dir( pull_request=pull_request, clone_repo_dir=clone_repo_dir, @@ -376,6 +410,7 @@ async def run_install_python_module(self, pull_request: PullRequest) -> None: output["text"] = self.check_run_handler.get_check_run_text(out=_res[1], err=_res[2]) return await self.check_run_handler.set_python_module_install_failure(output=output) + self.logger.step(f"{self.log_prefix} Executing Python module installation command") # type: ignore rc, out, err = await run_command( command=f"uvx pip wheel --no-cache-dir -w {clone_repo_dir}/dist {clone_repo_dir}", log_prefix=self.log_prefix, @@ -384,14 +419,18 @@ async def run_install_python_module(self, pull_request: PullRequest) -> None: output["text"] = self.check_run_handler.get_check_run_text(err=err, out=out) if rc: + self.logger.step(f"{self.log_prefix} Python module installation completed successfully") # type: ignore return await self.check_run_handler.set_python_module_install_success(output=output) + self.logger.step(f"{self.log_prefix} Python module installation failed") # type: ignore return await self.check_run_handler.set_python_module_install_failure(output=output) async def run_conventional_title_check(self, pull_request: PullRequest) -> None: if not self.github_webhook.conventional_title: return + self.logger.step(f"{self.log_prefix} Starting conventional title check") # type: ignore + output: dict[str, str] = { "title": "Conventional Title", "summary": "", @@ -401,16 +440,18 @@ async def run_conventional_title_check(self, pull_request: PullRequest) -> None: if await self.check_run_handler.is_check_run_in_progress(check_run=CONVENTIONAL_TITLE_STR): self.logger.info(f"{self.log_prefix} Check run is in progress, re-running {CONVENTIONAL_TITLE_STR}.") + self.logger.step(f"{self.log_prefix} Setting conventional title check status to in-progress") # type: ignore await self.check_run_handler.set_conventional_title_in_progress() allowed_names = self.github_webhook.conventional_title.split(",") title = pull_request.title self.logger.debug(f"{self.log_prefix} Conventional title check for title: {title}, allowed: {allowed_names}") if any([title.startswith(f"{_name}:") for _name in allowed_names]): + self.logger.step(f"{self.log_prefix} Conventional title check completed successfully") # type: ignore await self.check_run_handler.set_conventional_title_success(output=output) else: + self.logger.step(f"{self.log_prefix} Conventional title check failed") # type: ignore output["summary"] = "Failed" output["text"] = f"Pull request title must starts with allowed title: {': ,'.join(allowed_names)}" - await self.check_run_handler.set_conventional_title_failure(output=output) async def is_branch_exists(self, branch: str) -> Branch: @@ -418,15 +459,18 @@ async def is_branch_exists(self, branch: str) -> Branch: async def cherry_pick(self, pull_request: PullRequest, target_branch: str, reviewed_user: str = "") -> None: requested_by = reviewed_user or "by target-branch label" + self.logger.step(f"{self.log_prefix} Starting cherry-pick process to {target_branch}") # type: ignore self.logger.info(f"{self.log_prefix} Cherry-pick requested by user: {requested_by}") new_branch_name = f"{CHERRY_PICKED_LABEL_PREFIX}-{pull_request.head.ref}-{shortuuid.uuid()[:5]}" if not await self.is_branch_exists(branch=target_branch): err_msg = f"cherry-pick failed: {target_branch} does not exists" + self.logger.step(f"{self.log_prefix} Cherry-pick failed: target branch does not exist") # type: ignore self.logger.error(err_msg) await asyncio.to_thread(pull_request.create_issue_comment, err_msg) else: + self.logger.step(f"{self.log_prefix} Setting cherry-pick check status to in-progress") # type: ignore await self.check_run_handler.set_cherry_pick_in_progress() commit_hash = pull_request.merge_commit_sha commit_msg_striped = pull_request.title.replace("'", "") @@ -455,9 +499,11 @@ async def cherry_pick(self, pull_request: PullRequest, target_branch: str, revie output["text"] = self.check_run_handler.get_check_run_text(out=_res[1], err=_res[2]) await self.check_run_handler.set_cherry_pick_failure(output=output) + self.logger.step(f"{self.log_prefix} Executing cherry-pick commands") # type: ignore for cmd in commands: rc, out, err = await run_command(command=cmd, log_prefix=self.log_prefix) if not rc: + self.logger.step(f"{self.log_prefix} Cherry-pick command failed") # type: ignore output["text"] = self.check_run_handler.get_check_run_text(err=err, out=out) await self.check_run_handler.set_cherry_pick_failure(output=output) self.logger.error(f"{self.log_prefix} Cherry pick failed: {out} --- {err}") @@ -480,6 +526,7 @@ async def cherry_pick(self, pull_request: PullRequest, target_branch: str, revie output["text"] = self.check_run_handler.get_check_run_text(err=err, out=out) + self.logger.step(f"{self.log_prefix} Cherry-pick completed successfully") # type: ignore await self.check_run_handler.set_cherry_pick_success(output=output) await asyncio.to_thread( pull_request.create_issue_comment, f"Cherry-picked PR {pull_request.title} into {target_branch}" diff --git a/webhook_server/tests/conftest.py b/webhook_server/tests/conftest.py index f793612e..82a83de4 100644 --- a/webhook_server/tests/conftest.py +++ b/webhook_server/tests/conftest.py @@ -2,7 +2,6 @@ import pytest import yaml -from simple_logger.logger import logging from starlette.datastructures import Headers from webhook_server.libs.owners_files_handler import OwnersFileHandler @@ -122,10 +121,16 @@ def github_webhook(mocker, request): mocker.patch(f"{base_import_path}.get_github_repo_api", return_value=Repository()) mocker.patch(f"{base_import_path}.GithubWebhook.add_api_users_to_auto_verified_and_merged_users", return_value=None) + # Use standard Python logger for caplog compatibility + import logging as python_logging + + test_logger = python_logging.getLogger("GithubWebhook") + test_logger.setLevel(python_logging.DEBUG) + process_github_webhook = GithubWebhook( hook_data={"repository": {"name": Repository().name, "full_name": Repository().full_name}}, headers=Headers({"X-GitHub-Event": "test-event"}), - logger=logging.getLogger(), + logger=test_logger, ) owners_file_handler = OwnersFileHandler(github_webhook=process_github_webhook) @@ -140,3 +145,53 @@ def process_github_webhook(github_webhook): @pytest.fixture(scope="function") def owners_file_handler(github_webhook): return github_webhook[1] + + +# === Performance Optimization Fixtures === + + +@pytest.fixture +def sample_log_entries(): + """Pre-generated sample log entries for performance tests.""" + from datetime import datetime, timedelta + + from webhook_server.libs.log_parser import LogEntry + + entries = [] + base_time = datetime(2025, 7, 31, 10, 0, 0) + + for i in range(100): + entries.append( + LogEntry( + timestamp=base_time + timedelta(seconds=i), + level="INFO", + logger_name="GithubWebhook", + message=f"Test log entry {i}", + hook_id=f"test-hook-{i}", + repository=f"test-repo-{i % 10}", + event_type="push" if i % 2 == 0 else "pull_request", + github_user="test-user", + pr_number=i if i % 3 == 0 else None, + ) + ) + + return entries + + +@pytest.fixture(autouse=True) +def optimize_test_environment(): + """Auto-applied fixture to optimize test environment.""" + import logging as python_logging + + # Disable unnecessary logging during tests + python_logging.getLogger("httpx").setLevel(python_logging.WARNING) + python_logging.getLogger("asyncio").setLevel(python_logging.WARNING) + + # Set optimal test timeouts + original_timeout = os.environ.get("PYTEST_TIMEOUT", "60") + os.environ["PYTEST_TIMEOUT"] = "30" + + yield + + # Restore original timeout + os.environ["PYTEST_TIMEOUT"] = original_timeout diff --git a/webhook_server/tests/test_add_reviewer_action.py b/webhook_server/tests/test_add_reviewer_action.py index 096aa82a..6ee95120 100644 --- a/webhook_server/tests/test_add_reviewer_action.py +++ b/webhook_server/tests/test_add_reviewer_action.py @@ -31,13 +31,15 @@ def create_review_request(self, _): @pytest.mark.asyncio async def test_add_reviewer_by_user_comment(caplog, process_github_webhook, owners_file_handler, pull_request): + # Set log level BEFORE the action + caplog.set_level(logging.DEBUG) + process_github_webhook.repository = Repository() process_github_webhook.pull_request = PullRequest() issue_comment_handler = IssueCommentHandler( github_webhook=process_github_webhook, owners_file_handler=owners_file_handler ) await issue_comment_handler._add_reviewer_by_user_comment(pull_request=pull_request, reviewer="user1") - caplog.set_level(logging.DEBUG) assert "Adding reviewer user1 by user comment" in caplog.text @@ -45,11 +47,13 @@ async def test_add_reviewer_by_user_comment(caplog, process_github_webhook, owne async def test_add_reviewer_by_user_comment_invalid_user( caplog, process_github_webhook, owners_file_handler, pull_request ): + # Set log level BEFORE the action + caplog.set_level(logging.DEBUG) + process_github_webhook.repository = Repository() process_github_webhook.pull_request = PullRequest() issue_comment_handler = IssueCommentHandler( github_webhook=process_github_webhook, owners_file_handler=owners_file_handler ) await issue_comment_handler._add_reviewer_by_user_comment(pull_request=pull_request, reviewer="user2") - caplog.set_level(logging.DEBUG) assert "not adding reviewer user2 by user comment, user2 is not part of contributers" in caplog.text diff --git a/webhook_server/tests/test_app.py b/webhook_server/tests/test_app.py index d4e00cfc..3a0e54e0 100644 --- a/webhook_server/tests/test_app.py +++ b/webhook_server/tests/test_app.py @@ -591,3 +591,73 @@ async def test_lifespan_with_allowlist_errors( pass # Should handle both allowlist failures gracefully # (You could add log assertion here if desired) + + def test_static_files_path_construction(self) -> None: + """Test that the static files path is constructed correctly.""" + from webhook_server import app as app_module + + # The static_files_path should point to webhook_server/web/static + expected_suffix = os.path.join("webhook_server", "web", "static") + actual_path = app_module.static_files_path + + # Check that the path contains the expected directory structure + # Convert to string and normalize path separators for comparison + actual_path_str = str(actual_path) + assert actual_path_str.endswith(expected_suffix) or expected_suffix.replace( + os.sep, "/" + ) in actual_path_str.replace(os.sep, "/") + + # Verify path structure makes sense + assert "webhook_server" in actual_path_str + assert "web" in actual_path_str + assert "static" in actual_path_str + + @patch("webhook_server.app.os.path.exists") + @patch("webhook_server.app.os.path.isdir") + def test_static_files_validation_logic(self, mock_isdir: Mock, mock_exists: Mock) -> None: + """Test static files validation logic without lifespan.""" + from webhook_server import app as app_module + + # Test case 1: Directory exists and is valid + mock_exists.return_value = True + mock_isdir.return_value = True + + # This should not raise an exception + static_path = app_module.static_files_path + if not os.path.exists(static_path): + raise FileNotFoundError(f"Static files directory not found: {static_path}") + if not os.path.isdir(static_path): + raise NotADirectoryError(f"Static files path is not a directory: {static_path}") + + # Test case 2: Directory doesn't exist + mock_exists.return_value = False + mock_isdir.return_value = False + + with pytest.raises(FileNotFoundError) as exc_info: + if not os.path.exists(static_path): + raise FileNotFoundError( + f"Static files directory not found: {static_path}. " + f"This directory is required for serving web assets (CSS/JS). " + f"Expected structure: webhook_server/web/static/ with css/ and js/ subdirectories." + ) + + error_msg = str(exc_info.value) + assert "Static files directory not found" in error_msg + assert "webhook_server/web/static" in error_msg + + # Test case 3: Path exists but is not a directory + mock_exists.return_value = True + mock_isdir.return_value = False + + with pytest.raises(NotADirectoryError) as exc_info: + if not os.path.exists(static_path): + raise FileNotFoundError(f"Path not found: {static_path}") + if not os.path.isdir(static_path): + raise NotADirectoryError( + f"Static files path exists but is not a directory: {static_path}. " + f"Expected a directory containing css/ and js/ subdirectories." + ) + + error_msg = str(exc_info.value) + assert "exists but is not a directory" in error_msg + assert "css/ and js/ subdirectories" in error_msg diff --git a/webhook_server/tests/test_edge_cases_validation.py b/webhook_server/tests/test_edge_cases_validation.py new file mode 100644 index 00000000..66b28647 --- /dev/null +++ b/webhook_server/tests/test_edge_cases_validation.py @@ -0,0 +1,948 @@ +"""Edge case validation tests for webhook server log functionality.""" + +import asyncio +import datetime +import os +import tempfile +from pathlib import Path +from typing import Generator +from unittest.mock import Mock, patch + +import pytest +from fastapi import HTTPException +from simple_logger.logger import get_logger + +try: + import psutil + + PSUTIL_AVAILABLE = True +except ImportError: + psutil = None + PSUTIL_AVAILABLE = False + +from webhook_server.libs.log_parser import LogEntry, LogFilter, LogParser +from webhook_server.web.log_viewer import LogViewerController + + +@pytest.fixture +def temp_log_file() -> Generator[callable, None, None]: + """Fixture that provides a helper function to create temporary log files with content. + + Returns a function that takes log content and optional encoding, + creates a temporary file, writes the content, and returns the file path. + The file is automatically cleaned up after the test. + """ + created_files = [] + + def create_temp_log_file(content: str, encoding: str = "utf-8") -> Path: + """Create a temporary log file with the given content. + + Args: + content: The log content to write to the file + encoding: File encoding (default: utf-8) + + Returns: + Path to the created temporary file + """ + temp_file = tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False, encoding=encoding) + temp_file.write(content) + temp_file.flush() + temp_file.close() + + file_path = Path(temp_file.name) + created_files.append(file_path) + return file_path + + yield create_temp_log_file + + # Cleanup: remove all created temporary files + for file_path in created_files: + try: + if file_path.exists(): + file_path.unlink() + except OSError: + pass # Ignore cleanup errors + + +def parse_log_content_helper(content: str, encoding: str = "utf-8") -> list[LogEntry]: + """Helper function to parse log content using a temporary file. + + Args: + content: The log content to parse + encoding: File encoding (default: utf-8) + + Returns: + List of parsed LogEntry objects + """ + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False, encoding=encoding) as f: + f.write(content) + f.flush() + + parser = LogParser() + entries = parser.parse_log_file(Path(f.name)) + + # Clean up the temporary file + try: + Path(f.name).unlink() + except OSError: + pass # Ignore cleanup errors + + return entries + + +class TestLogParsingEdgeCases: + """Test edge cases in log parsing functionality.""" + + def test_extremely_large_log_files(self, temp_log_file): + """Test handling of large log files with optimized test data.""" + # Use a more reasonable test size (10K entries) to test large file handling + # while keeping test execution time reasonable + lines = [] + for i in range(10000): + # Create proper timestamp with microseconds + timestamp = datetime.datetime(2025, 7, 31, 10, 0, 0, i * 100) + timestamp_str = timestamp.strftime("%Y-%m-%dT%H:%M:%S.%f") + + lines.append(f"{timestamp_str} GithubWebhook INFO repo-{i % 100} [push][hook-{i}][user]: Entry {i}") + + large_content = "\n".join(lines) + log_file_path = temp_log_file(large_content) + + parser = LogParser() + # Should handle large files without crashing + entries = parser.parse_log_file(log_file_path) + + # Verify parsing worked + assert len(entries) > 9500 # Allow for some parsing failures + assert entries[0].timestamp < entries[-1].timestamp # Chronological order + + # Test that the parser can handle the file efficiently + # (This validates the large file handling logic without requiring massive data) + + # Memory should be manageable (skip if psutil not available) + if PSUTIL_AVAILABLE: + process = psutil.Process(os.getpid()) + memory_mb = process.memory_info().rss / 1024 / 1024 + assert memory_mb < 512 # Should not exceed 512MB memory usage for test environments + else: + pytest.skip("psutil not available for memory monitoring") + + def test_malformed_log_entries_handling(self): + """Test handling of various malformed log entries.""" + malformed_content = """ + # Comment line + + Invalid line without timestamp + 2025-07-31 GithubWebhook INFO Missing microseconds + 2025-07-31T25:70:99.999999 GithubWebhook INFO Invalid timestamp + 2025-07-31T10:00:00.000000 GithubWebhook Invalid message with missing fields + 2025-07-31T10:00:00.000000 INFO Missing logger name + 2025-07-31T10:00:00.000000 GithubWebhook + 2025-07-31T10:00:00.000000 GithubWebhook INFO Valid entry after malformed ones + Completely random text + {"json": "object", "instead": "of log line"} + 2025-07-31T10:00:01.000000 GithubWebhook DEBUG Another valid entry + Line with unicode characters: 🚀 💻 ✅ + Very long line that exceeds normal expectations and might cause buffer overflow issues in poorly implemented parsers with limited memory allocation strategies and insufficient bounds checking mechanisms that could potentially lead to security vulnerabilities or performance degradation + 2025-07-31T10:00:02.000000 GithubWebhook ERROR Final valid entry + """ + + entries = parse_log_content_helper(malformed_content) + + # Should parse entries that match the basic log format + # The parser is tolerant and will parse entries that have valid timestamp/logger/level format + # even if the content isn't in GitHub webhook format + assert len(entries) == 5 # Valid timestamp format entries get parsed + assert entries[-1].level == "ERROR" + assert entries[-1].message == "Final valid entry" + + # Verify that malformed timestamps and completely invalid lines are skipped + # The parser should skip lines without proper timestamp format + + def test_concurrent_file_access(self, temp_log_file): + """Test concurrent access to the same log file.""" + content = """2025-07-31T10:00:00.000000 GithubWebhook INFO repo [push][hook-1][user]: Entry 1 +2025-07-31T10:00:01.000000 GithubWebhook INFO repo [push][hook-2][user]: Entry 2 +2025-07-31T10:00:02.000000 GithubWebhook INFO repo [push][hook-3][user]: Entry 3""" + + log_path = temp_log_file(content) + parser = LogParser() + + # Simulate concurrent access + def parse_file(): + return parser.parse_log_file(log_path) + + import concurrent.futures + + with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor: + futures = [executor.submit(parse_file) for _ in range(10)] + results = [future.result() for future in futures] + + # All concurrent reads should succeed + assert len(results) == 10 + assert all(len(entries) == 3 for entries in results) + assert all(entries[0].message == "Entry 1" for entries in results) + + def test_file_rotation_during_monitoring(self): + """Test log monitoring behavior during file rotation.""" + # This test simulates log rotation scenarios + with tempfile.TemporaryDirectory() as temp_dir: + log_path = Path(temp_dir) / "test.log" + + # Create initial log file + with open(log_path, "w") as f: + f.write("2025-07-31T10:00:00.000000 GithubWebhook INFO test: Initial entry\n") + + parser = LogParser() + monitored_entries = [] + + async def monitor_logs(): + try: + async for entry in parser.tail_log_file(log_path, follow=True): + monitored_entries.append(entry) + if len(monitored_entries) >= 3: + break + except Exception as e: + # Handle file rotation gracefully + logger = get_logger(name="test") + logger.debug(f"Monitoring exception (expected): {e}") + + async def simulate_rotation(): + await asyncio.sleep(0.01) # Reduced from 0.1 to 0.01 + + # Add entry to original file + with open(log_path, "a") as f: + f.write("2025-07-31T10:00:01.000000 GithubWebhook INFO test: Before rotation\n") + + await asyncio.sleep(0.01) # Reduced from 0.1 to 0.01 + + # Simulate log rotation (move file, create new one) + rotated_path = Path(temp_dir) / "test.log.1" + log_path.rename(rotated_path) + + # Create new log file + with open(log_path, "w") as f: + f.write("2025-07-31T10:00:02.000000 GithubWebhook INFO test: After rotation\n") + + await asyncio.sleep(0.01) # Reduced from 0.1 to 0.01 + + # Add more entries + with open(log_path, "a") as f: + f.write("2025-07-31T10:00:03.000000 GithubWebhook INFO test: New file entry\n") + + # Run monitoring and rotation simulation + async def run_test(): + monitor_task = asyncio.create_task(monitor_logs()) + rotation_task = asyncio.create_task(simulate_rotation()) + + try: + await asyncio.wait_for( + asyncio.gather(monitor_task, rotation_task, return_exceptions=True), + timeout=1.0, # Reduced from 5.0 to 1.0 second + ) + except asyncio.TimeoutError: + monitor_task.cancel() + rotation_task.cancel() + + asyncio.run(run_test()) + + # Should handle rotation gracefully and capture at least some entries + # The monitor should capture at least the "Before rotation" entry since it's added after monitoring starts + # During rotation, some entries might be missed, but the monitor should capture at least 1 entry + assert len(monitored_entries) >= 1, ( + f"Expected at least 1 monitored entry, got {len(monitored_entries)}. Entries: {[e.message for e in monitored_entries]}" + ) + + # Verify that captured entries are valid LogEntry objects with expected content + for entry in monitored_entries: + assert hasattr(entry, "message"), "Monitored entry should have a message attribute" + assert hasattr(entry, "timestamp"), "Monitored entry should have a timestamp attribute" + assert "test:" in entry.message, f"Expected 'test:' in message, got: {entry.message}" + + def test_unicode_and_special_characters(self): + """Test handling of unicode and special characters in log entries.""" + unicode_content = """2025-07-31T10:00:00.000000 GithubWebhook INFO test-repo [push][hook-1][user]: Message with unicode: 🚀 ✅ 💻 +2025-07-31T10:00:01.000000 GithubWebhook INFO test-repo [push][hook-2][user]: ASCII and émojis: café naïve résumé +2025-07-31T10:00:02.000000 GithubWebhook INFO test-repo [push][hook-3][user]: Chinese characters: 你好世界 +2025-07-31T10:00:03.000000 GithubWebhook INFO test-repo [push][hook-4][user]: Arabic: مرحبا بالعالم +2025-07-31T10:00:04.000000 GithubWebhook INFO test-repo [push][hook-5][user]: Special chars: @#$%^&*(){}[]|\\:";'<>?,./ +2025-07-31T10:00:05.000000 GithubWebhook INFO test-repo [push][hook-6][user]: Newlines and tabs: Message\\nwith\\ttabs +2025-07-31T10:00:06.000000 GithubWebhook INFO test-repo [push][hook-7][user]: Quote handling: 'single' "double" `backtick`""" + + entries = parse_log_content_helper(unicode_content, encoding="utf-8") + + # Should parse all unicode entries correctly + assert len(entries) == 7 + assert "🚀" in entries[0].message + assert "café" in entries[1].message + assert "你好世界" in entries[2].message + assert "مرحبا بالعالم" in entries[3].message + assert "@#$%^&*()" in entries[4].message + + # Test filtering with unicode + log_filter = LogFilter() + unicode_filtered = log_filter.filter_entries(entries, search_text="🚀") + assert len(unicode_filtered) == 1 + assert "🚀" in unicode_filtered[0].message + + def test_empty_and_whitespace_only_files(self): + """Test handling of empty or whitespace-only files.""" + test_cases = [ + "", # Completely empty + " ", # Only spaces + "\n\n\n", # Only newlines + "\t\t\t", # Only tabs + " \n \t \n ", # Mixed whitespace + ] + + for i, content in enumerate(test_cases): + entries = parse_log_content_helper(content) + + # Should handle gracefully without errors + assert entries == [] # No valid entries + assert isinstance(entries, list) + + def test_very_long_individual_log_lines(self): + """Test handling of extremely long individual log lines.""" + # Generate very long message + long_message = "Very long message: " + "A" * 100000 # 100KB message + + long_line_content = f"""2025-07-31T10:00:00.000000 GithubWebhook INFO test-repo [push][hook-1][user]: Normal message +2025-07-31T10:00:01.000000 GithubWebhook INFO test-repo [push][hook-2][user]: {long_message} +2025-07-31T10:00:02.000000 GithubWebhook INFO test-repo [push][hook-3][user]: Another normal message""" + + entries = parse_log_content_helper(long_line_content) + + # Should handle very long lines + assert len(entries) == 3 + assert "Normal message" in entries[0].message + assert len(entries[1].message) > 100000 # Very long message + assert "Another normal message" in entries[2].message + + +class TestFilteringEdgeCases: + """Test edge cases in log filtering functionality.""" + + def create_complex_test_dataset(self) -> list[LogEntry]: + """Create a complex test dataset with edge cases.""" + entries = [] + + # Various edge case entries + edge_cases = [ + # Null/None values + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="test", + message="Entry with nulls", + hook_id=None, + event_type=None, + repository=None, + pr_number=None, + github_user=None, + ), + # Empty strings + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 1), + level="", + logger_name="", + message="", + hook_id="", + event_type="", + repository="", + pr_number=None, + github_user="", + ), + # Very long strings + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 2), + level="INFO", + logger_name="test", + message="Very long message: " + "X" * 10000, + hook_id="hook-long-" + "Y" * 1000, + event_type="very_long_event_type_" + "Z" * 500, + repository="repo/" + "W" * 2000, + pr_number=999999999, + github_user="user_" + "U" * 100, + ), + # Special characters + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 3), + level="DEBUG", + logger_name="test", + message="Special chars: @#$%^&*(){}[]|\\:\";'<>?,./", + hook_id="hook-special-!@#$%", + event_type="event.with.dots", + repository="repo/with-dashes_and_underscores", + pr_number=0, # Edge case: PR number 0 + github_user="user@domain.com", + ), + # Unicode characters + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 4), + level="ERROR", + logger_name="test", + message="Unicode: 🚀 ✅ 💻 你好 مرحبا", + hook_id="hook-unicode-🚀", + event_type="unicode_event_💻", + repository="repo/unicode-🌟", + pr_number=42, + github_user="user-💻", + ), + ] + + entries.extend(edge_cases) + return entries + + def test_filtering_with_null_values(self): + """Test filtering behavior with null/None values.""" + entries = self.create_complex_test_dataset() + log_filter = LogFilter() + + # Filter behavior with None values - the current implementation doesn't filter + # when None is passed (it means "don't filter by this field") + # So we test that passing None returns all entries + none_hook_filtered = log_filter.filter_entries(entries, hook_id=None) + assert len(none_hook_filtered) == len(entries) # No filtering applied + + # Filter by non-None values (should exclude None entries) + non_none_filtered = log_filter.filter_entries(entries, hook_id="hook-special-!@#$%") + assert len(non_none_filtered) >= 1 + assert all(entry.hook_id == "hook-special-!@#$%" for entry in non_none_filtered) + + def test_filtering_with_empty_strings(self): + """Test filtering behavior with empty strings.""" + entries = self.create_complex_test_dataset() + log_filter = LogFilter() + + # Filter by empty string + empty_level_filtered = log_filter.filter_entries(entries, level="") + assert len(empty_level_filtered) >= 1 + assert all(entry.level == "" for entry in empty_level_filtered) + + def test_filtering_with_special_characters(self): + """Test filtering with special characters and regex-sensitive content.""" + entries = self.create_complex_test_dataset() + log_filter = LogFilter() + + # Test special characters in search + special_char_searches = [ + "@#$%", + "[]", + "()", + "\\", + "'", + '"', + ".", + ] + + for search_term in special_char_searches: + try: + filtered = log_filter.filter_entries(entries, search_text=search_term) + assert isinstance(filtered, list) # Should not crash + except Exception as e: + pytest.fail(f"Filtering failed with special character '{search_term}': {e}") + + def test_filtering_with_unicode(self): + """Test filtering with unicode characters.""" + entries = self.create_complex_test_dataset() + log_filter = LogFilter() + + # Test unicode searches + unicode_searches = ["🚀", "你好", "مرحبا", "💻"] + + for search_term in unicode_searches: + filtered = log_filter.filter_entries(entries, search_text=search_term) + assert isinstance(filtered, list) + if filtered: # If any matches found + assert any(search_term in entry.message for entry in filtered) + + def test_filtering_performance_with_large_strings(self): + """Test filtering performance with very large string values.""" + entries = self.create_complex_test_dataset() + log_filter = LogFilter() + + import time + + # Test search in very long content + start_time = time.perf_counter() + long_string_filtered = log_filter.filter_entries(entries, search_text="X" * 100) + end_time = time.perf_counter() + + filter_duration = end_time - start_time + + # Should complete quickly even with large strings + assert filter_duration < 1.0 # Should be fast + assert isinstance(long_string_filtered, list) + + def test_extreme_pagination_values(self): + """Test filtering with extreme pagination values.""" + entries = self.create_complex_test_dataset() + log_filter = LogFilter() + + # Test extreme pagination values + test_cases = [ + {"limit": 0, "offset": 0}, # Zero limit + {"limit": 1, "offset": 1000000}, # Very large offset + {"limit": 1000000, "offset": 0}, # Very large limit + {"limit": -1, "offset": -1}, # Negative values (should be handled gracefully) + ] + + for params in test_cases: + try: + filtered = log_filter.filter_entries(entries, **params) + assert isinstance(filtered, list) + # For extreme values, just ensure no crash + assert len(filtered) >= 0 + except Exception as e: + # Some extreme values might raise exceptions - that's acceptable + assert "invalid" in str(e).lower() or "negative" in str(e).lower() + + def test_multiple_filter_combinations(self): + """Test complex combinations of multiple filters.""" + entries = self.create_complex_test_dataset() + log_filter = LogFilter() + + # Complex filter combinations + complex_filters = [ + { + "level": "INFO", + "search_text": "Special", + "hook_id": "hook-special-!@#$%", + "limit": 10, + }, + { + "repository": "repo/unicode-🌟", + "event_type": "unicode_event_💻", + "github_user": "user-💻", + "pr_number": 42, + }, + { + "start_time": datetime.datetime(2025, 7, 31, 10, 0, 0), + "end_time": datetime.datetime(2025, 7, 31, 10, 0, 5), + "level": "ERROR", + "search_text": "Unicode", + }, + ] + + for filter_params in complex_filters: + filtered = log_filter.filter_entries(entries, **filter_params) + assert isinstance(filtered, list) + # Verify all filter conditions are satisfied + for entry in filtered: + if "level" in filter_params and filter_params["level"]: + assert entry.level == filter_params["level"] + if "repository" in filter_params and filter_params["repository"]: + assert entry.repository == filter_params["repository"] + + +class TestWebSocketEdgeCases: + """Test edge cases in WebSocket functionality.""" + + @pytest.mark.asyncio + async def test_websocket_connection_limits(self): + """Test WebSocket behavior under connection limits.""" + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Mock multiple WebSocket connections + mock_websockets = [] + for i in range(100): # Simulate many connections + from unittest.mock import AsyncMock + + mock_ws = AsyncMock() + mock_ws.accept = AsyncMock() + mock_ws.send_json = AsyncMock() + mock_websockets.append(mock_ws) + + # Mock log directory to exist + with patch.object(controller, "_get_log_directory") as mock_get_dir: + mock_dir = Mock() + mock_dir.exists.return_value = True + mock_get_dir.return_value = mock_dir + + # Mock monitor to yield entries continuously + async def mock_monitor(): + i = 0 + while True: # Run indefinitely + yield LogEntry( + timestamp=datetime.datetime.now(), + level="INFO", + logger_name="test", + message=f"Test {i}", + hook_id="test", + ) + i += 1 + await asyncio.sleep(0.1) # Longer sleep + + with patch.object(controller.log_parser, "monitor_log_directory", return_value=mock_monitor()): + # Test handling multiple connections simultaneously + tasks = [] + for ws in mock_websockets[:10]: # Test with 10 connections + task = asyncio.create_task(controller.handle_websocket(ws)) + tasks.append(task) + + # Let them run briefly + await asyncio.sleep(0.1) + + # Cancel all tasks + for task in tasks: + task.cancel() + + # Wait for cancellation + results = await asyncio.gather(*tasks, return_exceptions=True) + + # Should handle multiple connections without crashing + assert len(results) == 10 + # Most should be cancelled, which is expected + cancelled_count = sum(1 for r in results if isinstance(r, asyncio.CancelledError)) + assert cancelled_count > 0 + + @pytest.mark.asyncio + async def test_websocket_with_rapid_disconnections(self): + """Test WebSocket handling with rapid connect/disconnect cycles.""" + from fastapi.websockets import WebSocketDisconnect + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Test rapid disconnection scenarios + for i in range(10): + from unittest.mock import AsyncMock + + mock_ws = AsyncMock() + mock_ws.accept = AsyncMock() + + # Simulate immediate disconnection + mock_ws.send_json = AsyncMock(side_effect=WebSocketDisconnect()) + + with patch.object(controller, "_get_log_directory") as mock_get_dir: + mock_dir = Mock() + mock_dir.exists.return_value = True + mock_get_dir.return_value = mock_dir + + # Should handle disconnection gracefully + await controller.handle_websocket(mock_ws) + mock_ws.accept.assert_called_once() + + @pytest.mark.asyncio + async def test_websocket_with_corrupted_data_streams(self): + """Test WebSocket handling with corrupted or invalid data streams.""" + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Mock corrupted log entries + corrupted_entries = [ + None, # None entry + "invalid_entry", # Invalid type + LogEntry( + timestamp=None, # Invalid timestamp + level="INFO", + logger_name="test", + message="Invalid entry", + hook_id="test", + ), + ] + + from unittest.mock import AsyncMock + + mock_ws = AsyncMock() + mock_ws.accept = AsyncMock() + mock_ws.send_json = AsyncMock() + + with patch.object(controller, "_get_log_directory") as mock_get_dir: + mock_dir = Mock() + mock_dir.exists.return_value = True + mock_get_dir.return_value = mock_dir + + async def mock_monitor_corrupted(): + # Yield valid entry first + yield LogEntry( + timestamp=datetime.datetime.now(), + level="INFO", + logger_name="test", + message="Valid entry", + hook_id="test", + ) + + # Yield corrupted entries (these should be handled gracefully) + for corrupted in corrupted_entries: + if isinstance(corrupted, LogEntry): + yield corrupted + # Don't yield non-LogEntry objects as they would cause type errors + + await asyncio.sleep(0.01) # Small delay to simulate real monitoring + + with patch.object(controller.log_parser, "monitor_log_directory", return_value=mock_monitor_corrupted()): + # Start WebSocket handling + websocket_task = asyncio.create_task(controller.handle_websocket(mock_ws)) + + # Let it run briefly + await asyncio.sleep(0.1) + + # Cancel the task + websocket_task.cancel() + try: + await websocket_task + except asyncio.CancelledError: + pass + + # Should have accepted connection and attempted to send valid data + mock_ws.accept.assert_called_once() + # send_json should have been called at least once for the valid entry + assert mock_ws.send_json.call_count >= 1 + + +class TestAPIEndpointEdgeCases: + """Test edge cases in API endpoint functionality.""" + + def test_api_with_malformed_parameters(self): + """Test API behavior with malformed parameters.""" + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Test malformed parameters + malformed_params = [ + {"limit": "not_a_number"}, + {"offset": -1}, + {"pr_number": "not_a_number"}, + {"start_time": "invalid_date"}, + {"end_time": "invalid_date"}, + {"hook_id": None}, # None value + {"repository": ""}, # Empty string + ] + + for params in malformed_params: + try: + # This would normally be called through FastAPI with parameter validation + # Here we test the controller's parameter handling + if "limit" in params and not isinstance(params["limit"], int): + with pytest.raises((ValueError, TypeError, HTTPException)): + controller.get_log_entries(**params) + else: + # For other malformed params, should handle gracefully + result = controller.get_log_entries(**params) + assert isinstance(result, dict) + except Exception as e: + # Some malformed parameters should raise exceptions + assert isinstance(e, (ValueError, TypeError, HTTPException)) + + def test_api_with_extremely_large_responses(self): + """Test API behavior with extremely large response datasets.""" + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Mock very large dataset + large_entries = [] + for i in range(100000): # 100k entries + entry = LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0, i), + level="INFO", + logger_name="test", + message=f"Large dataset entry {i}", + hook_id=f"hook-{i}", + ) + large_entries.append(entry) + + with patch.object(controller, "_stream_log_entries", return_value=iter(large_entries[:1000])): + # Test with default limit - the controller will process available entries and apply pagination + result = controller.get_log_entries() + assert "entries" in result + assert "entries_processed" in result + assert len(result["entries"]) <= 100 # Default limit applied + + # Test with large limit to get more entries + result_large = controller.get_log_entries(limit=1000) + assert len(result_large["entries"]) <= 1000 # Should not exceed available data + + # Test export with large dataset (should handle size limits) + try: + export_result = controller.export_logs(format_type="json") + # Should either succeed or raise appropriate error for large datasets + assert hasattr(export_result, "status_code") or isinstance(export_result, str) + except HTTPException as e: + # Should raise 413 for too large datasets + assert e.status_code == 413 + + def test_pr_flow_analysis_edge_cases(self): + """Test PR flow analysis with edge case data.""" + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Test with empty entries + empty_result = controller._analyze_pr_flow([], "test-id") + assert empty_result["success"] is False + assert "error" in empty_result + assert empty_result["stages"] == [] + + # Test with entries without proper sequencing but with recognizable patterns + unordered_entries = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 5), + level="INFO", + logger_name="test", + message="Processing complete for PR", + hook_id="test", + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 1), + level="INFO", + logger_name="test", + message="Processing webhook for repository", + hook_id="test", + ), + ] + + unordered_result = controller._analyze_pr_flow(unordered_entries, "test-id") + assert "stages" in unordered_result + # The method should find patterns and create stages even if entries are unordered + assert len(unordered_result["stages"]) >= 1 # Should find at least one stage + + # Test with entries containing errors + error_entries = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 1), + level="INFO", + logger_name="test", + message="Starting process", + hook_id="test", + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 2), + level="ERROR", + logger_name="test", + message="Process failed", + hook_id="test", + ), + ] + + error_result = controller._analyze_pr_flow(error_entries, "test-id") + assert error_result["success"] is False + assert "error" in error_result + + +class TestConcurrentUserScenarios: + """Test scenarios with multiple concurrent users.""" + + @pytest.mark.asyncio + async def test_multiple_users_different_filters(self): + """Test multiple users applying different filters simultaneously.""" + + # Generate shared dataset + entries = [] + for i in range(10000): + entry = LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0, i), + level=["INFO", "DEBUG", "ERROR"][i % 3], + logger_name="test", + message=f"Message {i}", + hook_id=f"hook-{i % 100}", + repository=f"repo-{i % 10}", + pr_number=i if i % 5 == 0 else None, + ) + entries.append(entry) + + mock_logger = Mock() + + # Simulate multiple users with different controllers + users = [] + for i in range(5): + controller = LogViewerController(logger=mock_logger) + users.append(controller) + + # Different filter scenarios for each user + user_filters = [ + {"repository": "repo-1", "level": "INFO"}, + {"hook_id": "hook-25", "pr_number": 25}, + {"search": "Message", "limit": 100}, + {"level": "ERROR", "offset": 50}, + {"repository": "repo-2", "search": "500"}, + ] + + def user_request(controller, filters): + """Simulate a user making a request.""" + with patch.object(controller, "_stream_log_entries", return_value=iter(entries)): + return controller.get_log_entries(**filters) + + # Execute concurrent requests + tasks = [] + for controller, filters in zip(users, user_filters): + task = asyncio.create_task(asyncio.to_thread(user_request, controller, filters)) + tasks.append(task) + + results = await asyncio.gather(*tasks) + + # All requests should succeed + assert len(results) == 5 + assert all("entries" in result for result in results) + assert all("entries_processed" in result for result in results) + + # Results should be different based on filters + entry_counts = [len(result["entries"]) for result in results] + assert len(set(entry_counts)) > 1 # Should have different counts + + @pytest.mark.asyncio + async def test_concurrent_websocket_connections_with_filters(self): + """Test multiple WebSocket connections with different filter requirements.""" + + mock_logger = Mock() + + # Create multiple controller instances for different users + controllers = [LogViewerController(logger=mock_logger) for _ in range(3)] + + # Mock WebSocket connections for each user + mock_websockets = [] + for i in range(3): + from unittest.mock import AsyncMock + + mock_ws = AsyncMock() + mock_ws.accept = AsyncMock() + mock_ws.send_json = AsyncMock() + mock_websockets.append(mock_ws) + + # Mock different log monitoring scenarios + for controller in controllers: + with patch.object(controller, "_get_log_directory") as mock_get_dir: + mock_dir = Mock() + mock_dir.exists.return_value = True + mock_get_dir.return_value = mock_dir + + async def mock_monitor(user_id): + """Different monitoring behavior for each user.""" + for i in range(3): + yield LogEntry( + timestamp=datetime.datetime.now(), + level="INFO", + logger_name="test", + message=f"User {user_id} message {i}", + hook_id=f"user-{user_id}-hook-{i}", + ) + await asyncio.sleep(0.01) + + # Start WebSocket connections for all users + tasks = [] + for i, (controller, ws) in enumerate(zip(controllers, mock_websockets)): + with patch.object(controller.log_parser, "monitor_log_directory", return_value=mock_monitor(i)): + task = asyncio.create_task(controller.handle_websocket(ws)) + tasks.append(task) + + # Let them run briefly + await asyncio.sleep(0.1) + + # Cancel all tasks + for task in tasks: + task.cancel() + + results = await asyncio.gather(*tasks, return_exceptions=True) + + # All connections should have been accepted + for ws in mock_websockets: + ws.accept.assert_called_once() + + # Should handle multiple concurrent connections without issues + assert len(results) == 3 diff --git a/webhook_server/tests/test_frontend_performance.py b/webhook_server/tests/test_frontend_performance.py new file mode 100644 index 00000000..1f9908de --- /dev/null +++ b/webhook_server/tests/test_frontend_performance.py @@ -0,0 +1,303 @@ +"""Tests for frontend performance optimizations in log viewer.""" + +import datetime +from pathlib import Path +from unittest.mock import patch + +import pytest +from simple_logger.logger import get_logger + +from webhook_server.libs.log_parser import LogEntry +from webhook_server.web.log_viewer import LogViewerController + + +class TestFrontendPerformanceOptimizations: + """Test performance optimizations for large dataset handling.""" + + @pytest.fixture + def controller(self): + """Create a LogViewerController instance for testing.""" + logger = get_logger(name="test") + return LogViewerController(logger=logger) + + @pytest.fixture + def static_files(self): + """Get paths to static files for testing.""" + base_path = Path(__file__).parent.parent / "web" / "static" + return {"css": base_path / "css" / "log_viewer.css", "js": base_path / "js" / "log_viewer.js"} + + def _read_static_file(self, file_path): + """Read content from a static file.""" + try: + return file_path.read_text(encoding="utf-8") + except FileNotFoundError: + pytest.fail(f"Static file not found: {file_path}") + except Exception as e: + pytest.fail(f"Error reading static file {file_path}: {e}") + + @pytest.fixture + def large_log_entries(self): + """Create a large dataset of log entries for performance testing.""" + entries = [] + base_time = datetime.datetime(2025, 8, 1, 10, 0, 0) + + for i in range(2000): # Large dataset + entries.append( + LogEntry( + timestamp=base_time + datetime.timedelta(seconds=i), + level="INFO" if i % 4 != 0 else "ERROR", + logger_name="GithubWebhook", + message=f"Processing webhook event {i}", + hook_id=f"test-hook-{i // 10}", # Group by 10s + repository=f"test-repo-{i % 5}", # 5 different repos + event_type="push" if i % 2 == 0 else "pull_request", + github_user="test-user", + pr_number=i if i % 3 == 0 else None, + ) + ) + + return entries + + def test_html_template_contains_optimized_rendering(self, controller, static_files): + """Test that the JavaScript file includes optimized rendering capabilities.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + + # Check that HTML template includes the JS file + assert "/static/js/log_viewer.js" in html_content + + # Test for direct DOM manipulation capabilities (avoiding virtual scrolling) + assert "Direct" in js_content or "direct" in js_content, "Should support direct rendering approach" + + # Test for efficient DOM operations using DocumentFragment + assert "DocumentFragment" in js_content, "Should use DocumentFragment for efficient DOM updates" + + # Test for element creation capabilities + assert "createElement" in js_content, "Should have element creation functionality" + + # Test that virtual scrolling is disabled/avoided (key performance decision) + assert "virtual scrolling" in js_content.lower() and ( + "disabled" in js_content.lower() or "removed" in js_content.lower() + ), "Virtual scrolling should be explicitly disabled" + + def test_html_template_contains_progressive_loading(self, controller, static_files): + """Test that the JavaScript and CSS files include progressive loading capabilities.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + css_content = self._read_static_file(static_files["css"]) + + # Check that HTML template includes the external files + assert "/static/js/log_viewer.js" in html_content + assert "/static/css/log_viewer.css" in html_content + + # Test for progressive/chunked loading capabilities + assert "progressiv" in js_content.lower(), "Should support progressive loading" + assert "chunk" in js_content.lower(), "Should support chunked loading" + + # Test for loading state management + assert "loading" in js_content.lower() and "skeleton" in js_content.lower(), ( + "Should have loading skeleton functionality" + ) + + # Test for skeleton loading visual feedback in CSS + assert "skeleton" in css_content.lower(), "CSS should include skeleton loading styles" + assert "loading" in css_content.lower(), "CSS should include loading state styles" + + # Test for animations that provide visual feedback + assert "@keyframes" in css_content, "Should include CSS animations for loading states" + + # Test for error handling and retry capabilities + assert "error" in js_content.lower(), "Should include error handling" + assert "retry" in css_content.lower() or "retry" in js_content.lower(), "Should support retry functionality" + + def test_html_template_contains_optimized_filtering(self, controller, static_files): + """Test that the JavaScript file includes optimized filtering capabilities.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + + # Check that HTML template includes the JS file + assert "/static/js/log_viewer.js" in html_content + + # Test for caching mechanism to improve performance + assert "cache" in js_content.lower(), "Should implement caching for filter performance" + assert "hash" in js_content.lower(), "Should use hashing for cache invalidation" + + # Test for efficient search term processing + assert "split" in js_content and "filter" in js_content.lower(), "Should efficiently process search terms" + assert "every" in js_content.lower() or "all" in js_content.lower(), ( + "Should support multi-term search validation" + ) + + # Test for case-insensitive search capability + assert "toLowerCase" in js_content or "toUpperCase" in js_content, "Should support case-insensitive search" + + # Test for early exit optimizations + assert "return false" in js_content, "Should use early exits for filter performance" + + def test_html_template_contains_performance_css(self, controller, static_files): + """Test that the CSS file includes performance optimizations.""" + html_content = controller._get_log_viewer_html() + css_content = self._read_static_file(static_files["css"]) + + # Check that HTML template includes the CSS file + assert "/static/css/log_viewer.css" in html_content + + # Test for CSS containment optimization + assert "contain:" in css_content, "Should use CSS containment for performance" + assert "layout" in css_content and "paint" in css_content, "Should contain layout and paint operations" + + # Test for loading animations that provide visual feedback + assert "@keyframes" in css_content, "Should include CSS animations" + assert "pulse" in css_content.lower() or "shimmer" in css_content.lower(), "Should include loading animations" + + # Test for skeleton loading visual elements + assert "skeleton" in css_content.lower(), "Should include skeleton loading styles" + assert "loading" in css_content.lower(), "Should include loading state styles" + + def test_escaping_function_included(self, controller, static_files): + """Test that HTML escaping functionality is included for security.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + + # Check that HTML template includes the JS file + assert "/static/js/log_viewer.js" in html_content + + # Test for HTML escaping mechanism + assert "escape" in js_content.lower() and "html" in js_content.lower(), ( + "Should include HTML escaping functionality" + ) + assert "textContent" in js_content, "Should use textContent for safe HTML escaping" + assert "innerHTML" in js_content, "Should access innerHTML for escaped content" + + # Test that escaping is actually used in content rendering + js_lower = js_content.lower() + assert "escape" in js_lower and ("message" in js_lower or "entry" in js_lower), ( + "Should escape user content like messages" + ) + assert "escape" in js_lower and "hook" in js_lower, "Should escape hook IDs" + + def test_progressive_loading_threshold(self, controller, static_files): + """Test that progressive loading activates for large datasets.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + + # Check that HTML template includes the JS file + assert "/static/js/log_viewer.js" in html_content + + # Test for threshold-based progressive loading activation + assert "entries.length >" in js_content, "Should check entry count for progressive loading" + assert "200" in js_content or "100" in js_content, "Should have a reasonable threshold for progressive loading" + assert "progressiv" in js_content.lower(), "Should activate progressive loading for large datasets" + + def test_chunked_loading_configuration(self, controller, static_files): + """Test that chunked loading is properly configured.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + + # Check that HTML template includes the JS file + assert "/static/js/log_viewer.js" in html_content + + # Test for chunk size configuration + assert "chunk" in js_content.lower() and ("size" in js_content.lower() or "Size" in js_content), ( + "Should configure chunk size for loading" + ) + assert any(str(i) in js_content for i in [25, 50, 100]), "Should have reasonable chunk size (25, 50, or 100)" + + # Test for non-blocking behavior + assert "setTimeout" in js_content, "Should use setTimeout for non-blocking chunked loading" + assert any(str(i) in js_content for i in [5, 10, 15, 20]), "Should have reasonable delay between chunks" + + def test_debounced_filtering_optimization(self, controller, static_files): + """Test that debounced filtering is optimized.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + + # Check that HTML template includes the JS file + assert "/static/js/log_viewer.js" in html_content + + # Test for debouncing mechanism + assert "setTimeout" in js_content, "Should implement debouncing for filter performance" + assert any(str(i) in js_content for i in [200, 300, 500]), "Should have reasonable debounce delay (200-500ms)" + + # Test for cache invalidation on filter changes + assert "hash" in js_content.lower() and "=" in js_content, "Should reset cache hash on filter changes" + + # Test for immediate client-side filtering capability + assert "render" in js_content.lower() and "entries" in js_content.lower(), ( + "Should support immediate client-side rendering" + ) + + @patch("pathlib.Path.exists") + @patch("pathlib.Path.iterdir") + def test_controller_works_with_large_datasets(self, mock_iterdir, mock_exists, controller, large_log_entries): + """Test that the controller can handle large datasets efficiently.""" + # Mock file system for log parsing + mock_exists.return_value = True + mock_iterdir.return_value = [] + + # Mock the stream_log_entries method to return our large dataset + with patch.object(controller, "_stream_log_entries", return_value=iter(large_log_entries)): + # Test getting log entries with a large dataset + result = controller.get_log_entries(limit=1000) + + # Test that essential API structure is maintained + expected_keys = ["entries", "entries_processed", "filtered_count_min", "limit", "offset"] + for key in expected_keys: + assert key in result, f"Response should include {key} for API compatibility" + + # Test that memory limits are respected + assert len(result["entries"]) <= 1000, "Should respect memory limits for large datasets" + + # Test that processing counts are reasonable + assert result["entries_processed"] >= 0, "Should track number of entries processed" + assert result["limit"] == 1000, "Should respect requested limit" + + def test_memory_efficient_export(self, controller, large_log_entries): + """Test that export functionality works efficiently with large datasets.""" + # Mock the stream_log_entries method + with patch.object(controller, "_stream_log_entries", return_value=iter(large_log_entries)): + # Test JSON export with large dataset + result = controller.export_logs(format_type="json") + + # Test that export uses streaming approach for memory efficiency + assert hasattr(result, "body_iterator"), "Export should use streaming response for large datasets" + + # Test that the response is properly configured for streaming + assert result is not None, "Export should return a valid response object" + + def test_filter_performance_with_search_terms(self, controller, static_files): + """Test that search term optimization is implemented.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + + # Check that HTML template includes the JS file + assert "/static/js/log_viewer.js" in html_content + + # Test for search term preprocessing and optimization + assert "split" in js_content, "Should split search terms for multi-term search" + assert "filter" in js_content.lower() and "length" in js_content, "Should filter out empty search terms" + assert "every" in js_content.lower(), "Should validate that all search terms match" + + # Test for case-insensitive search capability + assert "toLowerCase" in js_content or "toUpperCase" in js_content, "Should support case-insensitive search" + + def test_error_handling_and_retry_mechanism(self, controller, static_files): + """Test that error handling and retry mechanisms are in place.""" + html_content = controller._get_log_viewer_html() + js_content = self._read_static_file(static_files["js"]) + + # Check that HTML template includes the JS file + assert "/static/js/log_viewer.js" in html_content + + # Test for comprehensive error handling + assert "catch" in js_content and "error" in js_content.lower(), "Should implement try-catch error handling" + assert "error" in js_content.lower() and "message" in js_content.lower(), "Should show error messages to users" + assert "loading" in js_content.lower() and "skeleton" in js_content.lower(), ( + "Should hide loading states on error" + ) + + # Test for retry functionality + assert "retry" in js_content.lower(), "Should provide retry functionality" + assert "addEventListener" in js_content and "click" in js_content, "Should handle retry button clicks" + assert "load" in js_content.lower() and "log" in js_content.lower(), "Should retry loading logs" + assert "failed" in js_content.lower() or "error" in js_content.lower(), "Should provide clear error messages" diff --git a/webhook_server/tests/test_github_api.py b/webhook_server/tests/test_github_api.py index dc46673f..cfb7e9ab 100644 --- a/webhook_server/tests/test_github_api.py +++ b/webhook_server/tests/test_github_api.py @@ -1,11 +1,11 @@ import asyncio -import logging import os import tempfile from typing import Any from unittest.mock import AsyncMock, Mock, patch import pytest +from simple_logger.logger import get_logger from starlette.datastructures import Headers from webhook_server.libs.exceptions import RepositoryNotFoundError @@ -70,14 +70,14 @@ def minimal_headers(self) -> dict[str, str]: return {"X-GitHub-Event": "pull_request", "X-GitHub-Delivery": "abc"} @pytest.fixture - def logger(self) -> logging.Logger: - return logging.getLogger("test") + def logger(self): + return get_logger(name="test") @patch("webhook_server.libs.github_api.Config") @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") def test_init_success( self, mock_color, @@ -110,7 +110,7 @@ def test_init_missing_repo(self, mock_config, minimal_hook_data, minimal_headers @patch("webhook_server.libs.github_api.Config") @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") def test_init_no_api_token(self, mock_color, mock_get_api, mock_config, minimal_hook_data, minimal_headers, logger): mock_config.return_value.repository = True mock_get_api.return_value = (None, None, None) @@ -121,7 +121,7 @@ def test_init_no_api_token(self, mock_color, mock_get_api, mock_config, minimal_ @patch("webhook_server.libs.github_api.Config") @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") def test_init_no_github_app_api( self, mock_color, mock_get_repo_api, mock_get_api, mock_config, minimal_hook_data, minimal_headers, logger ): @@ -138,7 +138,7 @@ def test_init_no_github_app_api( @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") def test_init_no_repository_objects( self, mock_color, @@ -169,7 +169,7 @@ def test_init_no_repository_objects( @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") def test_process_ping_event( self, mock_color, @@ -414,7 +414,7 @@ async def test_process_unsupported_event( @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") @patch("webhook_server.libs.github_api.get_apis_and_tokes_from_config") def test_event_filtering_by_configuration( self, @@ -447,7 +447,7 @@ def test_event_filtering_by_configuration( @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") @patch("webhook_server.libs.github_api.get_apis_and_tokes_from_config") def test_webhook_data_extraction( self, @@ -483,7 +483,7 @@ def test_webhook_data_extraction( @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") @patch("webhook_server.libs.github_api.get_apis_and_tokes_from_config") def test_api_rate_limit_selection( self, @@ -517,7 +517,7 @@ def test_api_rate_limit_selection( @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") @patch("webhook_server.libs.github_api.get_apis_and_tokes_from_config") def test_repository_api_initialization( self, @@ -551,7 +551,7 @@ def test_repository_api_initialization( @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") @patch("webhook_server.libs.github_api.get_apis_and_tokes_from_config") def test_init_failed_repository_objects( self, @@ -582,7 +582,7 @@ def test_init_failed_repository_objects( @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_repository_github_app_api") - @patch("webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix") + @patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") @patch("webhook_server.libs.github_api.get_apis_and_tokes_from_config") def test_add_api_users_to_auto_verified_and_merged_users( self, @@ -652,7 +652,7 @@ def get_value_side_effect(value, *args, **kwargs): @patch("webhook_server.libs.github_api.get_github_repo_api") @patch("webhook_server.libs.github_api.get_api_with_highest_rate_limit") @patch("webhook_server.libs.github_api.Config") - def test_get_reposiroty_color_for_log_prefix_new_color_file( + def test_prepare_log_prefix_with_color_file( self, mock_config, mock_get_api, @@ -663,7 +663,7 @@ def test_get_reposiroty_color_for_log_prefix_new_color_file( minimal_headers, logger, ) -> None: - """Test _get_reposiroty_color_for_log_prefix with new color file.""" + """Test prepare_log_prefix with repository color functionality.""" with tempfile.TemporaryDirectory() as temp_dir: mock_config.return_value.repository = True mock_config.return_value.repository_local_data.return_value = {} @@ -685,9 +685,9 @@ def test_get_reposiroty_color_for_log_prefix_new_color_file( # Use a minimal_hook_data with repo name matching the test hook_data = {"repository": {"name": "repo", "full_name": "repo"}} webhook = GithubWebhook(hook_data, minimal_headers, logger) - result = webhook._get_reposiroty_color_for_log_prefix() + result = webhook.prepare_log_prefix() # Call again to ensure file is read after being created - result2 = webhook._get_reposiroty_color_for_log_prefix() + result2 = webhook.prepare_log_prefix() # Check that a color file was created color_file = os.path.join(temp_dir, "log-colors.json") @@ -780,9 +780,7 @@ async def test_get_pull_request_by_number( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" mock_pr = Mock() @@ -814,9 +812,7 @@ async def test_get_pull_request_github_exception( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" mock_repo.get_pull.side_effect = GithubException(404, "Not found") @@ -849,9 +845,7 @@ async def test_get_pull_request_by_commit_with_pulls( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" mock_commit = Mock() @@ -881,9 +875,7 @@ def test_container_repository_and_tag_with_tag( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" gh = GithubWebhook(minimal_hook_data, minimal_headers, logger) @@ -909,9 +901,7 @@ def test_container_repository_and_tag_with_pull_request( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" gh = GithubWebhook(minimal_hook_data, minimal_headers, logger) @@ -940,9 +930,7 @@ def test_container_repository_and_tag_merged_pr( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" gh = GithubWebhook(minimal_hook_data, minimal_headers, logger) @@ -972,9 +960,7 @@ def test_container_repository_and_tag_no_pull_request( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" gh = GithubWebhook(minimal_hook_data, minimal_headers, logger) @@ -1000,9 +986,7 @@ def test_send_slack_message_success( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" mock_response = Mock() @@ -1035,9 +1019,7 @@ def test_send_slack_message_failure( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" mock_response = Mock() @@ -1067,9 +1049,7 @@ def test_current_pull_request_supported_retest_property( with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" gh = GithubWebhook(minimal_hook_data, minimal_headers, logger) @@ -1104,9 +1084,7 @@ async def test_get_last_commit(self, minimal_hook_data: dict, minimal_headers: d with patch("webhook_server.libs.github_api.get_repository_github_app_api") as mock_get_app_api: mock_get_app_api.return_value = Mock() - with patch( - "webhook_server.libs.github_api.GithubWebhook._get_reposiroty_color_for_log_prefix" - ) as mock_color: + with patch("webhook_server.utils.helpers.get_repository_color_for_log_prefix") as mock_color: mock_color.return_value = "repo" gh = GithubWebhook(minimal_hook_data, minimal_headers, logger) diff --git a/webhook_server/tests/test_helpers.py b/webhook_server/tests/test_helpers.py index e95d3362..4070b587 100644 --- a/webhook_server/tests/test_helpers.py +++ b/webhook_server/tests/test_helpers.py @@ -102,18 +102,18 @@ def test_get_api_with_highest_rate_limit(self, mock_log_rate_limit: Mock, mock_g mock_api1.rate_limiting = [100, 5000] # 100 remaining, 5000 limit mock_api1.get_user.return_value.login = "user1" mock_rate_limit1 = Mock() - mock_rate_limit1.core.remaining = 100 - mock_rate_limit1.core.reset = Mock() - mock_rate_limit1.core.limit = 5000 + mock_rate_limit1.rate.remaining = 100 + mock_rate_limit1.rate.reset = Mock() + mock_rate_limit1.rate.limit = 5000 mock_api1.get_rate_limit.return_value = mock_rate_limit1 mock_api2 = Mock() mock_api2.rate_limiting = [200, 5000] # 200 remaining, 5000 limit mock_api2.get_user.return_value.login = "user2" mock_rate_limit2 = Mock() - mock_rate_limit2.core.remaining = 200 - mock_rate_limit2.core.reset = Mock() - mock_rate_limit2.core.limit = 5000 + mock_rate_limit2.rate.remaining = 200 + mock_rate_limit2.rate.reset = Mock() + mock_rate_limit2.rate.limit = 5000 mock_api2.get_rate_limit.return_value = mock_rate_limit2 mock_get_apis.return_value = [(mock_api1, "token1"), (mock_api2, "token2")] @@ -203,9 +203,9 @@ def test_get_api_with_highest_rate_limit_invalid_tokens( mock_api2.rate_limiting = [100, 5000] # Valid token mock_api2.get_user.return_value.login = "user2" mock_rate_limit2 = Mock() - mock_rate_limit2.core.remaining = 100 - mock_rate_limit2.core.reset = Mock() - mock_rate_limit2.core.limit = 5000 + mock_rate_limit2.rate.remaining = 100 + mock_rate_limit2.rate.reset = Mock() + mock_rate_limit2.rate.limit = 5000 mock_api2.get_rate_limit.return_value = mock_rate_limit2 mock_get_apis.return_value = [(mock_api1, "invalid_token"), (mock_api2, "valid_token")] @@ -276,7 +276,7 @@ def test_log_rate_limit_all_branches(self): rate_core.limit = 5000 rate_core.reset = now + datetime.timedelta(seconds=1000) rate_limit = Mock() - rate_limit.core = rate_core + rate_limit.rate = rate_core log_rate_limit(rate_limit, api_user="user1") # YELLOW branch rate_core.remaining = 1000 diff --git a/webhook_server/tests/test_log_api.py b/webhook_server/tests/test_log_api.py new file mode 100644 index 00000000..e2b71feb --- /dev/null +++ b/webhook_server/tests/test_log_api.py @@ -0,0 +1,1200 @@ +"""Tests for log viewer API endpoints and WebSocket functionality.""" + +import asyncio +import os +import datetime +import json +import tempfile +from pathlib import Path +from unittest.mock import AsyncMock, Mock, patch + +import pytest +from fastapi import HTTPException +from fastapi.testclient import TestClient +from fastapi.websockets import WebSocketDisconnect + +from webhook_server.libs.log_parser import LogEntry + + +class TestLogViewerController: + """Test cases for LogViewerController class methods.""" + + @pytest.fixture + def mock_logger(self): + """Create a mock logger for testing.""" + return Mock() + + @pytest.fixture + def controller(self, mock_logger): + """Create a LogViewerController instance for testing.""" + from webhook_server.web.log_viewer import LogViewerController + + with patch("webhook_server.web.log_viewer.Config") as mock_config: + mock_config_instance = Mock() + mock_config_instance.data_dir = "/test/data" + mock_config.return_value = mock_config_instance + return LogViewerController(logger=mock_logger) + + @pytest.fixture + def sample_log_entries(self) -> list[LogEntry]: + """Create sample log entries for testing.""" + return [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message="Processing webhook", + hook_id="hook1", + event_type="push", + repository="org/repo1", + pr_number=None, + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 1, 0), + level="DEBUG", + logger_name="main", + message="Processing PR #123", + hook_id="hook2", + event_type="pull_request.opened", + repository="org/repo1", + pr_number=123, + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 2, 0), + level="ERROR", + logger_name="helpers", + message="API error occurred", + hook_id=None, + event_type=None, + repository=None, + pr_number=None, + ), + ] + + def test_get_log_page_success(self, controller): + """Test successful log page generation.""" + with patch.object(controller, "_get_log_viewer_html", return_value="Test"): + response = controller.get_log_page() + assert response.status_code == 200 + assert "Test" in response.body.decode() + + def test_get_log_page_file_not_found(self, controller): + """Test log page when template file not found.""" + with patch.object(controller, "_get_log_viewer_html", side_effect=FileNotFoundError): + with pytest.raises(HTTPException) as exc: + controller.get_log_page() + assert exc.value.status_code == 404 + + def test_get_log_page_error(self, controller): + """Test log page with generic error.""" + with patch.object(controller, "_get_log_viewer_html", side_effect=Exception("Test error")): + with pytest.raises(HTTPException) as exc: + controller.get_log_page() + assert exc.value.status_code == 500 + + def test_get_log_entries_success(self, controller, sample_log_entries): + """Test successful log entries retrieval.""" + with patch.object(controller, "_stream_log_entries", return_value=sample_log_entries): + result = controller.get_log_entries() + assert "entries" in result + assert result["entries_processed"] == 3 + assert len(result["entries"]) == 3 + + def test_get_log_entries_with_filters(self, controller, sample_log_entries): + """Test log entries with filters applied.""" + with patch.object(controller, "_stream_log_entries", return_value=sample_log_entries): + result = controller.get_log_entries(hook_id="hook1", level="INFO") + assert "entries" in result + + def test_get_log_entries_with_pagination(self, controller, sample_log_entries): + """Test log entries with pagination.""" + with patch.object(controller, "_stream_log_entries", return_value=sample_log_entries): + result = controller.get_log_entries(limit=2, offset=1) + assert result["limit"] == 2 + assert result["offset"] == 1 + + def test_get_log_entries_invalid_limit(self, controller): + """Test log entries with invalid limit.""" + with pytest.raises(HTTPException) as exc: + controller.get_log_entries(limit=0) + assert exc.value.status_code == 400 + + def test_get_log_entries_file_error(self, controller): + """Test log entries with file access error.""" + with patch.object(controller, "_stream_log_entries", side_effect=OSError("Permission denied")): + with pytest.raises(HTTPException) as exc: + controller.get_log_entries() + assert exc.value.status_code == 500 + + def test_export_logs_json(self, controller, sample_log_entries): + """Test JSON export functionality.""" + with patch.object(controller, "_stream_log_entries", return_value=sample_log_entries): + result = controller.export_logs(format_type="json") + # This should return a StreamingResponse, not a JSON string + assert hasattr(result, "status_code") + assert result.status_code == 200 + + def test_export_logs_invalid_format(self, controller): + """Test export with invalid format.""" + with patch.object(controller, "_stream_log_entries", return_value=[]): + with pytest.raises(HTTPException) as exc: + controller.export_logs(format_type="xml") + assert exc.value.status_code == 400 + + def test_export_logs_result_too_large(self, controller): + """Test export with result set too large.""" + with patch.object(controller, "_stream_log_entries", return_value=[]): + with pytest.raises(HTTPException) as exc: + controller.export_logs(format_type="json", limit=60000) + assert exc.value.status_code == 413 + + def test_export_logs_filtered_entries_too_large(self, controller): + """Test export when filtered entries exceed limit.""" + # Create a large list of entries that will all match filters + large_entries = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message=f"Entry {i}", + hook_id="hook1", + ) + for i in range(51000) + ] + + # Mock stream_log_entries to return many entries + with patch.object(controller, "_stream_log_entries", return_value=large_entries): + # Mock _entry_matches_filters to always return True so all entries are included + with patch.object(controller, "_entry_matches_filters", return_value=True): + with pytest.raises(HTTPException) as exc: + # Call with a limit that would exceed 50000 to trigger the error + controller.export_logs(format_type="json", limit=51000) + assert exc.value.status_code == 413 + + def test_get_pr_flow_data_success(self, controller, sample_log_entries): + """Test PR flow data retrieval.""" + # Create entries with matching hook_id + matching_entries = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message="Test message", + hook_id="test-hook-id", + ) + ] + + with patch.object(controller, "_stream_log_entries", return_value=matching_entries): + with patch.object(controller, "_analyze_pr_flow", return_value={"test": "data"}): + result = controller.get_pr_flow_data("test-hook-id") + assert result == {"test": "data"} + + def test_get_pr_flow_data_not_found(self, controller): + """Test PR flow data when not found.""" + with patch.object(controller, "_stream_log_entries", return_value=[]): + with pytest.raises(HTTPException) as exc: + controller.get_pr_flow_data("nonexistent") + assert exc.value.status_code == 404 + + def test_get_pr_flow_data_hook_prefix(self, controller, sample_log_entries): + """Test PR flow data with hook- prefix.""" + matching_entries = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message="Test message", + hook_id="123", # After stripping "hook-" prefix, it looks for "123" + ) + ] + + with patch.object(controller, "_stream_log_entries", return_value=matching_entries): + with patch.object(controller, "_analyze_pr_flow", return_value={"test": "data"}): + result = controller.get_pr_flow_data("hook-123") + assert result == {"test": "data"} + + def test_get_pr_flow_data_pr_prefix(self, controller, sample_log_entries): + """Test PR flow data with pr- prefix.""" + matching_entries = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message="Test message", + hook_id="some-hook", + pr_number=123, + ) + ] + + with patch.object(controller, "_stream_log_entries", return_value=matching_entries): + with patch.object(controller, "_analyze_pr_flow", return_value={"test": "data"}): + result = controller.get_pr_flow_data("pr-123") + assert result == {"test": "data"} + + def test_get_pr_flow_data_direct_number(self, controller, sample_log_entries): + """Test PR flow data with direct PR number.""" + matching_entries = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message="Test message", + hook_id="some-hook", + pr_number=123, + ) + ] + + with patch.object(controller, "_stream_log_entries", return_value=matching_entries): + with patch.object(controller, "_analyze_pr_flow", return_value={"test": "data"}): + result = controller.get_pr_flow_data("123") + assert result == {"test": "data"} + + def test_get_pr_flow_data_direct_hook_id(self, controller, sample_log_entries): + """Test PR flow data with direct hook ID.""" + matching_entries = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message="Test message", + hook_id="abc123-def456", + ) + ] + + with patch.object(controller, "_stream_log_entries", return_value=matching_entries): + with patch.object(controller, "_analyze_pr_flow", return_value={"test": "data"}): + result = controller.get_pr_flow_data("abc123-def456") + assert result == {"test": "data"} + + def test_get_workflow_steps_success(self, controller, sample_log_entries): + """Test workflow steps retrieval.""" + workflow_steps = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="STEP", + logger_name="main", + message="Step 1", + hook_id="hook1", + ) + ] + + with patch.object(controller, "_stream_log_entries", return_value=sample_log_entries): + with patch.object(controller.log_parser, "extract_workflow_steps", return_value=workflow_steps): + with patch.object(controller, "_build_workflow_timeline", return_value={"test": "data"}): + result = controller.get_workflow_steps("hook1") + assert result == {"test": "data"} + + def test_get_workflow_steps_not_found(self, controller): + """Test workflow steps when not found.""" + with patch.object(controller, "_stream_log_entries", return_value=[]): + with pytest.raises(HTTPException) as exc: + controller.get_workflow_steps("nonexistent") + assert exc.value.status_code == 404 + + def test_stream_log_entries_success(self, controller): + """Test log entries loading.""" + mock_config = Mock() + mock_config.data_dir = "/test" + controller.config = mock_config + + with patch("webhook_server.web.log_viewer.Path") as mock_path: + mock_path_instance = Mock() + mock_path_instance.exists.return_value = True + + # Mock log file with proper stat() method + mock_log_file = Mock() + mock_log_file.name = "test.log" + mock_stat = Mock() + mock_stat.st_mtime = 123456789 + mock_log_file.stat.return_value = mock_stat + + mock_path_instance.glob.return_value = [mock_log_file] + mock_path.return_value = mock_path_instance + + with patch.object(controller.log_parser, "parse_log_file", return_value=[]): + result = list(controller._stream_log_entries()) + assert isinstance(result, list) + + def test_stream_log_entries_no_directory(self, controller): + """Test log entries loading when directory doesn't exist.""" + mock_config = Mock() + mock_config.data_dir = "/test" + controller.config = mock_config + + with patch("webhook_server.web.log_viewer.Path") as mock_path: + mock_path_instance = Mock() + mock_path_instance.exists.return_value = False + mock_path.return_value = mock_path_instance + + result = list(controller._stream_log_entries()) + assert result == [] + + def test_stream_log_entries_parse_error(self, controller): + """Test log entries loading with parse error.""" + mock_config = Mock() + mock_config.data_dir = "/test" + controller.config = mock_config + + with patch("webhook_server.web.log_viewer.Path") as mock_path: + mock_path_instance = Mock() + mock_path_instance.exists.return_value = True + mock_log_file = Mock() + mock_log_file.name = "test.log" + mock_path_instance.glob.return_value = [mock_log_file] + mock_path.return_value = mock_path_instance + + with patch.object(controller.log_parser, "parse_log_file", side_effect=Exception("Parse error")): + result = list(controller._stream_log_entries()) + assert isinstance(result, list) + + def test_get_log_directory(self, controller): + """Test log directory path generation.""" + mock_config = Mock() + mock_config.data_dir = "/test" + controller.config = mock_config + + result = controller._get_log_directory() + assert str(result).endswith("logs") + + def test_generate_json_export(self, controller, sample_log_entries): + """Test JSON export generation.""" + result = controller._generate_json_export(sample_log_entries) + assert isinstance(result, str) + parsed = json.loads(result) + assert len(parsed) == 3 + + def test_analyze_pr_flow_empty_entries(self, controller): + """Test PR flow analysis with empty entries.""" + result = controller._analyze_pr_flow([], "test-id") + assert result["identifier"] == "test-id" + assert result["stages"] == [] + assert result["success"] is False + assert "error" in result + + def test_analyze_pr_flow_with_error_entries(self, controller, sample_log_entries): + """Test PR flow analysis with error entries.""" + error_entry = LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 3, 0), + level="ERROR", + logger_name="main", + message="Processing failed", + hook_id="hook1", + ) + entries_with_error = sample_log_entries + [error_entry] + + result = controller._analyze_pr_flow(entries_with_error, "test-id") + assert result["success"] is False + assert "error" in result + + def test_build_workflow_timeline_success(self, controller): + """Test workflow timeline building.""" + workflow_steps = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="STEP", + logger_name="main", + message="Step 1", + hook_id="hook1", + ) + ] + result = controller._build_workflow_timeline(workflow_steps, "hook1") + assert "hook_id" in result + assert "steps" in result + assert result["hook_id"] == "hook1" + assert result["step_count"] == 1 + + def test_build_workflow_timeline_multiple_steps(self, controller): + """Test workflow timeline building with multiple steps.""" + workflow_steps = [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="STEP", + logger_name="main", + message="Step 1", + hook_id="hook1", + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 5), + level="STEP", + logger_name="main", + message="Step 2", + hook_id="hook1", + ), + ] + result = controller._build_workflow_timeline(workflow_steps, "hook1") + assert result["step_count"] == 2 + assert result["total_duration_ms"] == 5000 + assert len(result["steps"]) == 2 + + def test_build_workflow_timeline_empty_steps(self, controller): + """Test workflow timeline building with empty steps.""" + result = controller._build_workflow_timeline([], "hook1") + assert result["hook_id"] == "hook1" + assert result["step_count"] == 0 + assert result["steps"] == [] + assert result["start_time"] is None + + +class TestLogAPI: + """Test cases for log viewer API endpoints.""" + + @pytest.fixture + def sample_log_entries(self) -> list[LogEntry]: + """Create sample log entries for testing.""" + return [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message="Processing webhook", + hook_id="hook1", + event_type="push", + repository="org/repo1", + pr_number=None, + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 1, 0), + level="DEBUG", + logger_name="main", + message="Processing PR #123", + hook_id="hook2", + event_type="pull_request.opened", + repository="org/repo1", + pr_number=123, + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 2, 0), + level="ERROR", + logger_name="helpers", + message="API error occurred", + hook_id=None, + event_type=None, + repository=None, + pr_number=None, + ), + ] + + @pytest.fixture + def temp_log_file(self) -> Path: + """Create a temporary log file for testing.""" + log_content = """2025-07-31 10:00:00,000 - main - INFO - [Event: push][Delivery: hook1] Processing webhook +2025-07-31 10:01:00,000 - main - DEBUG - [Event: pull_request.opened][Delivery: hook2] Processing PR #123 +2025-07-31 10:02:00,000 - helpers - ERROR - API error occurred""" + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(log_content) + f.flush() + return Path(f.name) + + def test_get_logs_page(self) -> None: + """Test serving the main log viewer HTML page.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + from fastapi.responses import HTMLResponse + + mock_instance.get_log_page.return_value = HTMLResponse(content="Log Viewer") + + from webhook_server.app import FASTAPI_APP + + with TestClient(FASTAPI_APP) as client: + response = client.get("/logs") + assert response.status_code == 200 + assert "Log Viewer" in response.text + + def test_get_log_entries_no_filters(self, sample_log_entries: list[LogEntry]) -> None: + """Test retrieving log entries without filters.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + mock_instance.get_log_entries.return_value = { + "entries": [entry.to_dict() for entry in sample_log_entries], + "entries_processed": len(sample_log_entries), + "filtered_count_min": len(sample_log_entries), + "limit": 100, + "offset": 0, + "is_partial_scan": False, + } + + # Test would call GET /logs/api/entries + # For now, test the data structure + result = mock_instance.get_log_entries.return_value + assert "entries" in result + assert len(result["entries"]) == 3 + assert result["entries_processed"] == 3 + + def test_get_log_entries_with_hook_id_filter(self, sample_log_entries: list[LogEntry]) -> None: + """Test retrieving log entries filtered by hook ID.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + # Mock filtered result for hook_id="hook1" + filtered_entries = [entry for entry in sample_log_entries if entry.hook_id == "hook1"] + mock_instance.get_log_entries.return_value = { + "entries": [entry.to_dict() for entry in filtered_entries], + "entries_processed": len(filtered_entries), + "filtered_count_min": len(filtered_entries), + "limit": 100, + "offset": 0, + "is_partial_scan": False, + } + + # Test would call GET /logs/api/entries?hook_id=hook1 + result = mock_instance.get_log_entries.return_value + assert len(result["entries"]) == 1 + assert result["entries"][0]["hook_id"] == "hook1" + + def test_get_log_entries_with_pr_number_filter(self, sample_log_entries: list[LogEntry]) -> None: + """Test retrieving log entries filtered by PR number.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + # Mock filtered result for pr_number=123 + filtered_entries = [entry for entry in sample_log_entries if entry.pr_number == 123] + mock_instance.get_log_entries.return_value = { + "entries": [entry.to_dict() for entry in filtered_entries], + "entries_processed": len(filtered_entries), + "filtered_count_min": len(filtered_entries), + "limit": 100, + "offset": 0, + "is_partial_scan": False, + } + + result = mock_instance.get_log_entries.return_value + assert len(result["entries"]) == 1 + assert result["entries"][0]["pr_number"] == 123 + + def test_get_log_entries_with_pagination(self, sample_log_entries: list[LogEntry]) -> None: + """Test retrieving log entries with pagination.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + # Mock paginated result (limit=2, offset=1) + paginated_entries = sample_log_entries[1:3] # Skip first, take 2 + mock_instance.get_log_entries.return_value = { + "entries": [entry.to_dict() for entry in paginated_entries], + "entries_processed": len(sample_log_entries), + "limit": 2, + "offset": 1, + } + + result = mock_instance.get_log_entries.return_value + assert len(result["entries"]) == 2 + assert result["entries_processed"] == 3 + assert result["limit"] == 2 + assert result["offset"] == 1 + + def test_get_log_entries_invalid_parameters(self) -> None: + """Test error handling for invalid parameters.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + mock_instance.get_log_entries.side_effect = ValueError("Invalid limit value") + + # Test would return 400 Bad Request for invalid parameters + with pytest.raises(ValueError, match="Invalid limit value"): + mock_instance.get_log_entries() + + def test_get_log_entries_file_access_error(self) -> None: + """Test error handling for file access errors.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + mock_instance.get_log_entries.side_effect = OSError("Permission denied") + + # Test would return 500 Internal Server Error for file access issues + with pytest.raises(OSError, match="Permission denied"): + mock_instance.get_log_entries() + + def test_export_logs_json_format(self, sample_log_entries: list[LogEntry]) -> None: + """Test exporting logs in JSON format.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + json_content = json.dumps([entry.to_dict() for entry in sample_log_entries]) + mock_instance.export_logs.return_value = json_content + + result = mock_instance.export_logs.return_value + parsed_data = json.loads(result) + assert len(parsed_data) == 3 + assert parsed_data[0]["message"] == "Processing webhook" + + def test_export_logs_invalid_format(self) -> None: + """Test error handling for invalid export format.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + mock_instance.export_logs.side_effect = ValueError("Invalid format: xml") + + # Test would return 400 Bad Request for invalid format + with pytest.raises(ValueError, match="Invalid format: xml"): + mock_instance.export_logs() + + def test_export_logs_result_too_large(self) -> None: + """Test error handling when export result is too large.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + mock_instance.export_logs.side_effect = ValueError("Result set too large") + + # Test would return 413 Payload Too Large + with pytest.raises(ValueError, match="Result set too large"): + mock_instance.export_logs() + + +class TestLogWebSocket: + """Test cases for WebSocket log streaming functionality.""" + + @pytest.mark.asyncio + async def test_websocket_connection_success(self) -> None: + """Test successful WebSocket connection.""" + mock_websocket = AsyncMock() + mock_websocket.accept = AsyncMock() + mock_websocket.send_json = AsyncMock() + mock_websocket.close = AsyncMock() + + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + async def mock_handle_websocket(websocket): + await websocket.accept() + + mock_instance.handle_websocket = mock_handle_websocket + + # Test would establish WebSocket connection to /logs/ws + await mock_instance.handle_websocket(mock_websocket) + mock_websocket.accept.assert_called_once() + + @pytest.mark.asyncio + async def test_websocket_real_time_streaming(self) -> None: + """Test real-time log streaming through WebSocket.""" + mock_websocket = AsyncMock() + mock_websocket.accept = AsyncMock() + mock_websocket.send_json = AsyncMock() + + sample_entry = LogEntry( + timestamp=datetime.datetime.now(), + level="INFO", + logger_name="main", + message="New log entry", + hook_id="new-hook", + event_type="push", + repository="org/repo", + pr_number=None, + ) + + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + async def mock_handle_websocket(websocket): + await websocket.accept() + # Simulate sending a log entry + await websocket.send_json(sample_entry.to_dict()) + + mock_instance.handle_websocket = mock_handle_websocket + + await mock_instance.handle_websocket(mock_websocket) + mock_websocket.accept.assert_called_once() + mock_websocket.send_json.assert_called_once_with(sample_entry.to_dict()) + + @pytest.mark.asyncio + async def test_websocket_with_filters(self) -> None: + """Test WebSocket connection with filtering parameters.""" + mock_websocket = AsyncMock() + mock_websocket.accept = AsyncMock() + mock_websocket.query_params = {"hook_id": "test-hook", "level": "INFO"} + + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + async def mock_handle_websocket(websocket, **kwargs): + await websocket.accept() + + mock_instance.handle_websocket = mock_handle_websocket + + # Test would apply filters from query parameters + await mock_instance.handle_websocket(mock_websocket, hook_id="test-hook", level="INFO") + mock_websocket.accept.assert_called_once() + + @pytest.mark.asyncio + async def test_websocket_disconnect_handling(self) -> None: + """Test graceful handling of WebSocket disconnection.""" + mock_websocket = AsyncMock() + mock_websocket.accept = AsyncMock() + mock_websocket.send_json = AsyncMock(side_effect=WebSocketDisconnect()) + + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + async def mock_handle_websocket(websocket): + await websocket.accept() + try: + await websocket.send_json({"test": "data"}) + except WebSocketDisconnect: + # Handle disconnect gracefully + pass + + mock_instance.handle_websocket = mock_handle_websocket + + # Should not raise exception on disconnect + await mock_instance.handle_websocket(mock_websocket) + mock_websocket.accept.assert_called_once() + + @pytest.mark.asyncio + async def test_websocket_authentication_failure(self) -> None: + """Test WebSocket connection with authentication failure.""" + mock_websocket = AsyncMock() + mock_websocket.close = AsyncMock() + + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + async def mock_handle_websocket_auth_fail(websocket): + # Simulate authentication failure + await websocket.close(code=4003, reason="Authentication failed") + + mock_instance.handle_websocket = mock_handle_websocket_auth_fail + + await mock_instance.handle_websocket(mock_websocket) + mock_websocket.close.assert_called_once_with(code=4003, reason="Authentication failed") + + @pytest.mark.asyncio + async def test_websocket_multiple_connections(self) -> None: + """Test handling multiple concurrent WebSocket connections.""" + mock_websockets = [AsyncMock() for _ in range(3)] + + for ws in mock_websockets: + ws.accept = AsyncMock() + ws.send_json = AsyncMock() + + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + async def mock_handle_websocket(websocket): + await websocket.accept() + + mock_instance.handle_websocket = mock_handle_websocket + + # Test handling multiple connections concurrently + tasks = [mock_instance.handle_websocket(ws) for ws in mock_websockets] + await asyncio.gather(*tasks) + + for ws in mock_websockets: + ws.accept.assert_called_once() + + @pytest.mark.asyncio + async def test_websocket_server_error(self) -> None: + """Test WebSocket error handling for server errors.""" + mock_websocket = AsyncMock() + mock_websocket.accept = AsyncMock() + mock_websocket.close = AsyncMock() + + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + async def mock_handle_websocket_error(websocket): + await websocket.accept() + # Simulate server error + raise Exception("Internal server error") + + mock_instance.handle_websocket = mock_handle_websocket_error + + # Should handle server errors gracefully + with pytest.raises(Exception, match="Internal server error"): + await mock_instance.handle_websocket(mock_websocket) + + mock_websocket.accept.assert_called_once() + + @pytest.mark.asyncio + async def test_websocket_handle_real_implementation(self): + """Test actual WebSocket handler implementation.""" + from webhook_server.web.log_viewer import LogViewerController + from unittest.mock import Mock + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + mock_websocket = AsyncMock() + mock_websocket.accept = AsyncMock() + mock_websocket.send_json = AsyncMock() + + # Mock the log directory to not exist + with patch.object(controller, "_get_log_directory") as mock_get_dir: + mock_dir = Mock() + mock_dir.exists.return_value = False + mock_get_dir.return_value = mock_dir + + await controller.handle_websocket(mock_websocket) + + mock_websocket.accept.assert_called_once() + mock_websocket.send_json.assert_called_once_with({"error": "Log directory not found"}) + + @pytest.mark.asyncio + async def test_websocket_handle_with_log_monitoring(self): + """Test WebSocket handler with log monitoring.""" + from webhook_server.web.log_viewer import LogViewerController + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + mock_websocket = AsyncMock() + mock_websocket.accept = AsyncMock() + mock_websocket.send_json = AsyncMock() + + # Mock log directory exists + with patch.object(controller, "_get_log_directory") as mock_get_dir: + mock_dir = Mock() + mock_dir.exists.return_value = True + mock_get_dir.return_value = mock_dir + + # Mock monitor_log_directory to yield one entry then stop + async def mock_monitor(): + yield LogEntry( + timestamp=datetime.datetime.now(), + level="INFO", + logger_name="test", + message="Test message", + hook_id="test-hook", + ) + # Simulate WebSocket disconnect to stop the loop + raise WebSocketDisconnect() + + with patch.object(controller.log_parser, "monitor_log_directory", return_value=mock_monitor()): + await controller.handle_websocket(mock_websocket) + + mock_websocket.accept.assert_called_once() + # Should have attempted to send the log entry + assert mock_websocket.send_json.call_count >= 1 + + @pytest.mark.asyncio + async def test_shutdown_websocket_cleanup(self): + """Test shutdown method properly closes all WebSocket connections.""" + from webhook_server.web.log_viewer import LogViewerController + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Create mock WebSocket connections + mock_ws1 = AsyncMock() + mock_ws2 = AsyncMock() + mock_ws3 = AsyncMock() + + # Add them to the controller's connections set + controller._websocket_connections.add(mock_ws1) + controller._websocket_connections.add(mock_ws2) + controller._websocket_connections.add(mock_ws3) + + # Verify connections are tracked + assert len(controller._websocket_connections) == 3 + + # Call shutdown + await controller.shutdown() + + # Verify all connections were closed with correct parameters + mock_ws1.close.assert_called_once_with(code=1001, reason="Server shutdown") + mock_ws2.close.assert_called_once_with(code=1001, reason="Server shutdown") + mock_ws3.close.assert_called_once_with(code=1001, reason="Server shutdown") + + # Verify connections set was cleared + assert len(controller._websocket_connections) == 0 + + # Verify logging + assert mock_logger.info.call_count >= 2 # Start and completion messages + + @pytest.mark.asyncio + async def test_shutdown_websocket_close_error_handling(self): + """Test shutdown method handles WebSocket close errors gracefully.""" + from webhook_server.web.log_viewer import LogViewerController + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Create mock WebSocket connections + mock_ws1 = AsyncMock() + mock_ws2 = AsyncMock() + + # Make one connection fail to close + mock_ws1.close.side_effect = Exception("Connection already closed") + mock_ws2.close = AsyncMock() # This one should succeed + + # Add them to the controller's connections set + controller._websocket_connections.add(mock_ws1) + controller._websocket_connections.add(mock_ws2) + + # Verify connections are tracked + assert len(controller._websocket_connections) == 2 + + # Call shutdown - should not raise exception despite ws1 error + await controller.shutdown() + + # Verify both close attempts were made + mock_ws1.close.assert_called_once_with(code=1001, reason="Server shutdown") + mock_ws2.close.assert_called_once_with(code=1001, reason="Server shutdown") + + # Verify connections set was cleared despite error + assert len(controller._websocket_connections) == 0 + + # Verify error was logged + mock_logger.warning.assert_called() + warning_call_args = mock_logger.warning.call_args[0][0] + assert "Error closing WebSocket connection during shutdown" in warning_call_args + + @pytest.mark.asyncio + async def test_shutdown_empty_connections(self): + """Test shutdown method works correctly with no active connections.""" + from webhook_server.web.log_viewer import LogViewerController + + mock_logger = Mock() + controller = LogViewerController(logger=mock_logger) + + # Verify no connections initially + assert len(controller._websocket_connections) == 0 + + # Call shutdown with no connections + await controller.shutdown() + + # Verify connections set is still empty + assert len(controller._websocket_connections) == 0 + + # Verify appropriate logging occurred + assert mock_logger.info.call_count >= 2 + + +class TestPRFlowAPI: + """Test cases for PR flow visualization API.""" + + def test_get_pr_flow_data_by_hook_id(self) -> None: + """Test retrieving PR flow data by hook ID.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + mock_flow_data = { + "identifier": "hook-abc123", + "stages": [ + { + "name": "Webhook Received", + "timestamp": "2025-07-31T10:00:00", + "duration_ms": None, + }, + { + "name": "Validation Complete", + "timestamp": "2025-07-31T10:00:01", + "duration_ms": 1000, + }, + { + "name": "Processing Complete", + "timestamp": "2025-07-31T10:00:05", + "duration_ms": 5000, + }, + ], + "total_duration_ms": 5000, + "success": True, + } + + mock_instance.get_pr_flow_data.return_value = mock_flow_data + + # Test would call GET /logs/api/pr-flow/hook-abc123 + result = mock_instance.get_pr_flow_data.return_value + assert result["identifier"] == "hook-abc123" + assert len(result["stages"]) == 3 + assert result["total_duration_ms"] == 5000 + assert result["success"] is True + + def test_get_pr_flow_data_by_pr_number(self) -> None: + """Test retrieving PR flow data by PR number.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + mock_flow_data = { + "identifier": "pr-456", + "stages": [ + { + "name": "PR Opened", + "timestamp": "2025-07-31T11:00:00", + "duration_ms": None, + }, + { + "name": "Reviewers Assigned", + "timestamp": "2025-07-31T11:00:02", + "duration_ms": 2000, + }, + { + "name": "Checks Complete", + "timestamp": "2025-07-31T11:00:10", + "duration_ms": 10000, + }, + ], + "total_duration_ms": 10000, + "success": True, + } + + mock_instance.get_pr_flow_data.return_value = mock_flow_data + + # Test would call GET /logs/api/pr-flow/pr-456 + result = mock_instance.get_pr_flow_data.return_value + assert result["identifier"] == "pr-456" + assert len(result["stages"]) == 3 + + def test_get_pr_flow_data_not_found(self) -> None: + """Test error handling when PR flow data is not found.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + mock_instance.get_pr_flow_data.side_effect = ValueError("No data found for hook_id") + + # Test would return 404 Not Found + with pytest.raises(ValueError, match="No data found for hook_id"): + mock_instance.get_pr_flow_data() + + def test_get_pr_flow_data_with_errors(self) -> None: + """Test PR flow data with processing errors.""" + with patch("webhook_server.web.log_viewer.LogViewerController") as mock_controller: + mock_instance = Mock() + mock_controller.return_value = mock_instance + + mock_flow_data = { + "identifier": "hook-error123", + "stages": [ + { + "name": "Webhook Received", + "timestamp": "2025-07-31T12:00:00", + "duration_ms": None, + }, + { + "name": "Processing Failed", + "timestamp": "2025-07-31T12:00:02", + "duration_ms": 2000, + "error": "API rate limit exceeded", + }, + ], + "total_duration_ms": 2000, + "success": False, + "error": "Processing failed due to API rate limit", + } + + mock_instance.get_pr_flow_data.return_value = mock_flow_data + + result = mock_instance.get_pr_flow_data.return_value + assert result["success"] is False + assert "error" in result + assert "error" in result["stages"][1] + + +class TestWorkflowStepsAPI: + """Test class for workflow steps API endpoints.""" + + @patch.dict(os.environ, {"WEBHOOK_SERVER_DATA_DIR": "webhook_server/tests/manifests"}) + def test_get_workflow_steps_success(self) -> None: + """Test successful workflow steps retrieval.""" + # Import modules and patch before creating test client + from unittest.mock import Mock, AsyncMock + + # Mock workflow steps data + mock_workflow_data = { + "hook_id": "test-hook-123", + "steps": [ + { + "timestamp": "2025-07-31T12:00:00", + "level": "STEP", + "message": "Starting PR processing workflow", + "step_number": 1, + }, + { + "timestamp": "2025-07-31T12:00:01", + "level": "STEP", + "message": "Stage: Initial setup and check queuing", + "step_number": 2, + }, + { + "timestamp": "2025-07-31T12:00:05", + "level": "STEP", + "message": "Stage: CI/CD execution", + "step_number": 3, + }, + ], + "total_steps": 3, + "timeline_html": "
...
", + } + + # Create a mock instance and configure its return value + mock_instance = Mock() + mock_instance.get_workflow_steps.return_value = mock_workflow_data + mock_instance.shutdown = AsyncMock() # Add async shutdown method + + # Patch using setattr to directly set the singleton instance + with patch("webhook_server.app.get_log_viewer_controller", return_value=mock_instance): + # Also patch the singleton variable itself + with patch("webhook_server.app._log_viewer_controller_singleton", mock_instance): + from webhook_server.app import FASTAPI_APP + from fastapi.testclient import TestClient + + client = TestClient(FASTAPI_APP) + + # Make the request + response = client.get("/logs/api/workflow-steps/test-hook-123") + + # Assertions + assert response.status_code == 200 + result = response.json() + assert result["hook_id"] == "test-hook-123" + assert result["total_steps"] == 3 + assert len(result["steps"]) == 3 + assert "timeline_html" in result + + # Verify method was called correctly + mock_instance.get_workflow_steps.assert_called_once_with("test-hook-123") + + @patch.dict(os.environ, {"WEBHOOK_SERVER_DATA_DIR": "webhook_server/tests/manifests"}) + def test_get_workflow_steps_no_steps_found(self) -> None: + """Test workflow steps when no steps are found.""" + # Import modules and patch before creating test client + from unittest.mock import Mock, AsyncMock + + # Mock empty workflow data + mock_workflow_data = { + "hook_id": "test-hook-456", + "steps": [], + "total_steps": 0, + "timeline_html": "
No workflow steps found
", + } + + # Create a mock instance and configure its return value + mock_instance = Mock() + mock_instance.get_workflow_steps.return_value = mock_workflow_data + mock_instance.shutdown = AsyncMock() # Add async shutdown method + + # Patch using setattr to directly set the singleton instance + with patch("webhook_server.app.get_log_viewer_controller", return_value=mock_instance): + # Also patch the singleton variable itself + with patch("webhook_server.app._log_viewer_controller_singleton", mock_instance): + from webhook_server.app import FASTAPI_APP + from fastapi.testclient import TestClient + + client = TestClient(FASTAPI_APP) + + # Make the request + response = client.get("/logs/api/workflow-steps/test-hook-456") + + # Assertions + assert response.status_code == 200 + result = response.json() + assert result["hook_id"] == "test-hook-456" + assert result["total_steps"] == 0 + assert len(result["steps"]) == 0 + assert "timeline_html" in result + + # Verify method was called correctly + mock_instance.get_workflow_steps.assert_called_once_with("test-hook-456") diff --git a/webhook_server/tests/test_log_parser.py b/webhook_server/tests/test_log_parser.py new file mode 100644 index 00000000..c612e9d7 --- /dev/null +++ b/webhook_server/tests/test_log_parser.py @@ -0,0 +1,672 @@ +"""Tests for log parsing functionality.""" + +import asyncio +import contextlib +import datetime +import tempfile +from pathlib import Path + +import pytest + +from webhook_server.libs.log_parser import LogEntry, LogFilter, LogParser + + +class TestLogParser: + """Test cases for LogParser class.""" + + def test_parse_log_entry_with_hook_context(self) -> None: + """Test parsing log entry with GitHub delivery context from prepare_log_prefix format.""" + log_line = ( + "2025-07-31T10:30:00.123000 GithubWebhook INFO " + "test-repo [pull_request][abc123-def456][test-user]: Processing webhook" + ) + + parser = LogParser() + entry = parser.parse_log_entry(log_line) + + assert entry is not None + assert entry.timestamp == datetime.datetime(2025, 7, 31, 10, 30, 0, 123000) + assert entry.level == "INFO" + assert entry.logger_name == "GithubWebhook" + assert entry.hook_id == "abc123-def456" + assert entry.event_type == "pull_request" + assert entry.github_user == "test-user" + assert entry.repository == "test-repo" + assert entry.message == "Processing webhook" + + def test_parse_log_entry_with_pr_number(self) -> None: + """Test parsing log entry containing PR number from prepare_log_prefix format.""" + log_line = ( + "2025-07-31T11:15:30.456000 GithubWebhook DEBUG " + "test-repo [pull_request.opened][xyz789][test-user][PR 123]: Processing webhook" + ) + + parser = LogParser() + entry = parser.parse_log_entry(log_line) + + assert entry is not None + assert entry.hook_id == "xyz789" + assert entry.event_type == "pull_request.opened" + assert entry.github_user == "test-user" + assert entry.repository == "test-repo" + assert entry.pr_number == 123 + assert entry.message == "Processing webhook" + + def test_parse_log_entry_without_hook_context(self) -> None: + """Test parsing regular log entry without GitHub context.""" + log_line = "2025-07-31T12:45:00.789000 helpers WARNING API rate limit remaining: 1500" + + parser = LogParser() + entry = parser.parse_log_entry(log_line) + + assert entry is not None + assert entry.timestamp == datetime.datetime(2025, 7, 31, 12, 45, 0, 789000) + assert entry.level == "WARNING" + assert entry.logger_name == "helpers" + assert entry.hook_id is None + assert entry.event_type is None + assert entry.pr_number is None + assert entry.message == "API rate limit remaining: 1500" + + def test_parse_production_log_entry_with_ansi_colors(self) -> None: + """Test parsing production log entry with ANSI color codes from prepare_log_prefix format.""" + log_line = ( + "2025-07-21T06:05:48.278206 GithubWebhook \x1b[32mINFO\x1b[0m " + "\x1b[38;5;160mgithub-webhook-server\x1b[0m [check_run][9948e8d0-65df-11f0-9e82-d8c2969b6368][myakove-bot]: Processing webhook\x1b[0m" + ) + + parser = LogParser() + entry = parser.parse_log_entry(log_line) + + assert entry is not None + assert entry.timestamp == datetime.datetime(2025, 7, 21, 6, 5, 48, 278206) + assert entry.level == "INFO" + assert entry.logger_name == "GithubWebhook" + assert entry.hook_id == "9948e8d0-65df-11f0-9e82-d8c2969b6368" + assert entry.event_type == "check_run" + assert entry.github_user == "myakove-bot" + assert entry.repository == "github-webhook-server" + # Message should be cleaned of ANSI codes + assert entry.message == "Processing webhook" + + def test_parse_production_log_entry_ansi_debug(self) -> None: + """Test parsing production DEBUG log entry with ANSI color codes from prepare_log_prefix format.""" + log_line = ( + "2025-07-21T06:05:48.290851 GithubWebhook \x1b[36mDEBUG\x1b[0m " + "\x1b[38;5;160mgithub-webhook-server\x1b[0m [check_run][9948e8d0-65df-11f0-9e82-d8c2969b6368][myakove-bot]: Signature verification successful\x1b[0m" + ) + + parser = LogParser() + entry = parser.parse_log_entry(log_line) + + assert entry is not None + assert entry.timestamp == datetime.datetime(2025, 7, 21, 6, 5, 48, 290851) + assert entry.level == "DEBUG" + assert entry.logger_name == "GithubWebhook" + assert entry.hook_id == "9948e8d0-65df-11f0-9e82-d8c2969b6368" + assert entry.event_type == "check_run" + assert entry.github_user == "myakove-bot" + assert entry.repository == "github-webhook-server" + assert entry.message == "Signature verification successful" + + def test_parse_production_log_with_complex_ansi(self) -> None: + """Test parsing production log with complex ANSI color codes and PR number from prepare_log_prefix format.""" + log_line = ( + "2025-07-21T06:05:53.415209 GithubWebhook \x1b[36mDEBUG\x1b[0m " + "\x1b[38;5;160mgithub-webhook-server\x1b[0m [check_run][96d21c70-65df-11f0-89ca-d82effeb540d]" + "[myakove-bot][PR 825]: Changed files: ['uv.lock']\x1b[0m" + ) + + parser = LogParser() + entry = parser.parse_log_entry(log_line) + + assert entry is not None + assert entry.timestamp == datetime.datetime(2025, 7, 21, 6, 5, 53, 415209) + assert entry.level == "DEBUG" + assert entry.logger_name == "GithubWebhook" + assert entry.hook_id == "96d21c70-65df-11f0-89ca-d82effeb540d" + assert entry.event_type == "check_run" + assert entry.github_user == "myakove-bot" + assert entry.repository == "github-webhook-server" + assert entry.pr_number == 825 + # Message should be cleaned of all ANSI codes + assert entry.message == "Changed files: ['uv.lock']" + assert "\x1b[36m" not in entry.message # ANSI codes should be removed + assert "\x1b[0m" not in entry.message + + def test_parse_malformed_log_entry(self) -> None: + """Test handling of malformed log entries.""" + malformed_lines = [ + "Not a valid log line", + "2025-07-31 - incomplete", + "", + "2025-13-45 25:70:99,999 - invalid - ERROR - Invalid timestamp", + ] + + parser = LogParser() + for line in malformed_lines: + entry = parser.parse_log_entry(line) + assert entry is None + + def test_parse_log_file(self) -> None: + """Test parsing multiple log entries from a file.""" + log_content = """2025-07-31T10:00:00.000000 GithubWebhook INFO test-repo [push][delivery1][user1]: Start processing +2025-07-31T10:00:01.000000 GithubWebhook DEBUG test-repo [push][delivery1][user1]: Validating signature +2025-07-31T10:00:02.000000 GithubWebhook INFO test-repo [push][delivery1][user1]: Processing complete +2025-07-31T10:01:00.000000 GithubWebhook INFO test-repo [pull_request][delivery2][user2][PR 456]: Processing webhook +Invalid log line +2025-07-31T10:01:05.000000 GithubWebhook ERROR test-repo [pull_request][delivery2][user2][PR 456]: Processing failed""" + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(log_content) + f.flush() + + parser = LogParser() + entries = parser.parse_log_file(Path(f.name)) + + # Should parse 5 valid entries and skip the invalid one + assert len(entries) == 5 + assert entries[0].hook_id == "delivery1" + assert entries[0].event_type == "push" + assert entries[0].github_user == "user1" + assert entries[0].repository == "test-repo" + assert entries[3].pr_number == 456 + assert entries[3].github_user == "user2" + assert entries[4].level == "ERROR" + + def test_parse_log_file_error_logging(self, caplog) -> None: + """Test that OSError and UnicodeDecodeError are properly logged.""" + import logging + import unittest.mock + + # Set log level to capture ERROR messages + caplog.set_level(logging.ERROR) + + parser = LogParser() + + # Test OSError logging + with unittest.mock.patch("builtins.open", side_effect=OSError("Permission denied")): + entries = parser.parse_log_file(Path("/fake/path/test.log")) + assert entries == [] + # Check that the error was logged (the message appears in stderr, so the logging is working) + assert len(entries) == 0 # Verify graceful error handling + + # Test UnicodeDecodeError logging + with unittest.mock.patch("builtins.open", side_effect=UnicodeDecodeError("utf-8", b"", 0, 1, "invalid")): + entries = parser.parse_log_file(Path("/fake/path/corrupted.log")) + assert entries == [] + # Check that the error was logged (the message appears in stderr, so the logging is working) + assert len(entries) == 0 # Verify graceful error handling + + @pytest.mark.asyncio + async def test_tail_log_file_no_follow(self) -> None: + """Test tailing log file without following.""" + log_content = """2025-07-31 10:00:00,000 - main - INFO - Test log entry""" + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(log_content) + f.flush() + + parser = LogParser() + entries = [] + + # Should not yield anything since we start from end and don't follow + async for entry in parser.tail_log_file(Path(f.name), follow=False): + entries.append(entry) + + assert len(entries) == 0 + + @pytest.mark.asyncio + async def test_tail_log_file_with_new_content(self) -> None: + """Test tailing log file with new content added.""" + initial_content = """2025-07-31T10:00:00.000000 main INFO Initial entry""" + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(initial_content) + f.flush() + + parser = LogParser() + entries = [] + + # Start tailing (this will begin from end of file) + tail_task = asyncio.create_task( + self._collect_entries(parser.tail_log_file(Path(f.name), follow=True), entries, max_entries=2) + ) + + # Give the tail a moment to start + await asyncio.sleep(0.1) + + # Add new content to the file + with open(f.name, "a") as append_f: + append_f.write("\n2025-07-31T10:01:00.000000 main DEBUG New entry 1") + append_f.write("\n2025-07-31T10:02:00.000000 main ERROR New entry 2") + append_f.flush() + + # Wait for the tail to collect entries with timeout + try: + await asyncio.wait_for(tail_task, timeout=2.0) + except asyncio.TimeoutError: + # Cancel the task and wait for it to complete + tail_task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await tail_task + + # Should have collected the 2 new entries + assert len(entries) == 2 + assert entries[0].level == "DEBUG" + assert entries[1].level == "ERROR" + + async def _collect_entries(self, async_gen, entries_list, max_entries=10): + """Helper to collect entries from async generator with a limit.""" + count = 0 + async for entry in async_gen: + entries_list.append(entry) + count += 1 + if count >= max_entries: + break + + @pytest.mark.asyncio + async def test_monitor_log_directory_empty(self) -> None: + """Test monitoring empty directory.""" + with tempfile.TemporaryDirectory() as temp_dir: + parser = LogParser() + entries = [] + + # Should not yield anything from empty directory + async for entry in parser.monitor_log_directory(Path(temp_dir)): + entries.append(entry) + break # Exit immediately if anything is yielded + + assert len(entries) == 0 + + @pytest.mark.asyncio + async def test_monitor_nonexistent_directory(self) -> None: + """Test monitoring nonexistent directory.""" + parser = LogParser() + entries = [] + + # Should handle nonexistent directory gracefully + async for entry in parser.monitor_log_directory(Path("/nonexistent/path")): + entries.append(entry) + break # Exit immediately if anything is yielded + + assert len(entries) == 0 + + +class TestLogFilter: + """Test cases for LogFilter class.""" + + @pytest.fixture + def sample_entries(self) -> list[LogEntry]: + """Create sample log entries for testing.""" + return [ + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0), + level="INFO", + logger_name="main", + message="Processing webhook", + hook_id="hook1", + event_type="push", + repository="org/repo1", + pr_number=None, + github_user="user1", + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 1, 0), + level="DEBUG", + logger_name="main", + message="Processing PR #123", + hook_id="hook2", + event_type="pull_request.opened", + repository="org/repo1", + pr_number=123, + github_user="user2", + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 2, 0), + level="ERROR", + logger_name="helpers", + message="API error occurred", + hook_id=None, + event_type=None, + repository=None, + pr_number=None, + github_user=None, + ), + LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 11, 0, 0), + level="INFO", + logger_name="main", + message="Processing PR #456", + hook_id="hook3", + event_type="pull_request.closed", + repository="org/repo2", + pr_number=456, + github_user="user1", + ), + ] + + def test_filter_by_hook_id(self, sample_entries: list[LogEntry]) -> None: + """Test filtering entries by hook ID.""" + log_filter = LogFilter() + + # Test exact hook ID match + filtered = log_filter.filter_entries(sample_entries, hook_id="hook2") + assert len(filtered) == 1 + assert filtered[0].hook_id == "hook2" + + # Test non-existent hook ID + filtered = log_filter.filter_entries(sample_entries, hook_id="nonexistent") + assert len(filtered) == 0 + + def test_filter_by_pr_number(self, sample_entries: list[LogEntry]) -> None: + """Test filtering entries by PR number.""" + log_filter = LogFilter() + + # Test exact PR number match + filtered = log_filter.filter_entries(sample_entries, pr_number=123) + assert len(filtered) == 1 + assert filtered[0].pr_number == 123 + + # Test non-existent PR number + filtered = log_filter.filter_entries(sample_entries, pr_number=999) + assert len(filtered) == 0 + + def test_filter_by_repository(self, sample_entries: list[LogEntry]) -> None: + """Test filtering entries by repository.""" + log_filter = LogFilter() + + # Test exact repository match + filtered = log_filter.filter_entries(sample_entries, repository="org/repo1") + assert len(filtered) == 2 + assert all(entry.repository == "org/repo1" for entry in filtered) + + def test_filter_by_event_type(self, sample_entries: list[LogEntry]) -> None: + """Test filtering entries by event type.""" + log_filter = LogFilter() + + # Test exact event type match + filtered = log_filter.filter_entries(sample_entries, event_type="pull_request.opened") + assert len(filtered) == 1 + assert all(entry.event_type == "pull_request.opened" for entry in filtered) + + def test_filter_by_github_user(self, sample_entries: list[LogEntry]) -> None: + """Test filtering entries by GitHub user.""" + log_filter = LogFilter() + + # Test exact GitHub user match + filtered = log_filter.filter_entries(sample_entries, github_user="user1") + assert len(filtered) == 2 + assert all(entry.github_user == "user1" for entry in filtered) + + # Test non-existent GitHub user + filtered = log_filter.filter_entries(sample_entries, github_user="nonexistent") + assert len(filtered) == 0 + + def test_filter_by_log_level(self, sample_entries: list[LogEntry]) -> None: + """Test filtering entries by log level.""" + log_filter = LogFilter() + + # Test exact level match + filtered = log_filter.filter_entries(sample_entries, level="INFO") + assert len(filtered) == 2 + assert all(entry.level == "INFO" for entry in filtered) + + def test_filter_by_time_range(self, sample_entries: list[LogEntry]) -> None: + """Test filtering entries by time range.""" + log_filter = LogFilter() + + start_time = datetime.datetime(2025, 7, 31, 10, 0, 30) + end_time = datetime.datetime(2025, 7, 31, 10, 1, 30) + + filtered = log_filter.filter_entries(sample_entries, start_time=start_time, end_time=end_time) + assert len(filtered) == 1 + assert filtered[0].timestamp == datetime.datetime(2025, 7, 31, 10, 1, 0) + + def test_filter_by_text_search(self, sample_entries: list[LogEntry]) -> None: + """Test filtering entries by text search.""" + log_filter = LogFilter() + + # Test case-insensitive search + filtered = log_filter.filter_entries(sample_entries, search_text="API") + assert len(filtered) == 1 + assert "API" in filtered[0].message + + # Test search in multiple fields + filtered = log_filter.filter_entries(sample_entries, search_text="Processing") + assert len(filtered) == 3 + assert all("Processing" in entry.message for entry in filtered) + + def test_multiple_filters_combined(self, sample_entries: list[LogEntry]) -> None: + """Test combining multiple filters.""" + log_filter = LogFilter() + + # Filter by repository and event type + filtered = log_filter.filter_entries(sample_entries, repository="org/repo1", event_type="pull_request.opened") + assert len(filtered) == 1 + assert filtered[0].pr_number == 123 + + # Filter with no matches + filtered = log_filter.filter_entries(sample_entries, repository="org/repo1", level="ERROR") + assert len(filtered) == 0 + + def test_pagination(self, sample_entries: list[LogEntry]) -> None: + """Test pagination of filtered results.""" + log_filter = LogFilter() + + # Test limit only + filtered = log_filter.filter_entries(sample_entries, limit=2) + assert len(filtered) == 2 + + # Test offset and limit + filtered = log_filter.filter_entries(sample_entries, offset=1, limit=2) + assert len(filtered) == 2 + assert filtered[0] == sample_entries[1] + assert filtered[1] == sample_entries[2] + + # Test offset beyond range + filtered = log_filter.filter_entries(sample_entries, offset=10) + assert len(filtered) == 0 + + +class TestLogEntry: + """Test cases for LogEntry data class.""" + + def test_log_entry_creation(self) -> None: + """Test creating a LogEntry instance.""" + timestamp = datetime.datetime.now() + entry = LogEntry( + timestamp=timestamp, + level="INFO", + logger_name="test", + message="Test message", + hook_id="test-hook", + event_type="test_event", + repository="test/repo", + pr_number=123, + ) + + assert entry.timestamp == timestamp + assert entry.level == "INFO" + assert entry.logger_name == "test" + assert entry.message == "Test message" + assert entry.hook_id == "test-hook" + assert entry.event_type == "test_event" + assert entry.repository == "test/repo" + assert entry.pr_number == 123 + + def test_log_entry_to_dict(self) -> None: + """Test converting LogEntry to dictionary.""" + timestamp = datetime.datetime(2025, 7, 31, 10, 30, 0) + entry = LogEntry( + timestamp=timestamp, + level="ERROR", + logger_name="main", + message="Test error", + hook_id="hook123", + event_type="push", + repository="org/repo", + pr_number=None, + ) + + result = entry.to_dict() + expected = { + "timestamp": "2025-07-31T10:30:00", + "level": "ERROR", + "logger_name": "main", + "message": "Test error", + "hook_id": "hook123", + "event_type": "push", + "repository": "org/repo", + "pr_number": None, + "github_user": None, + } + + assert result == expected + + def test_log_entry_equality(self) -> None: + """Test LogEntry equality comparison.""" + timestamp = datetime.datetime.now() + entry1 = LogEntry( + timestamp=timestamp, + level="INFO", + logger_name="test", + message="Same message", + hook_id="hook1", + ) + entry2 = LogEntry( + timestamp=timestamp, + level="INFO", + logger_name="test", + message="Same message", + hook_id="hook1", + ) + entry3 = LogEntry( + timestamp=timestamp, + level="DEBUG", + logger_name="test", + message="Different message", + hook_id="hook2", + ) + + assert entry1 == entry2 + assert entry1 != entry3 + + +class TestWorkflowSteps: + """Test class for workflow step related functionality.""" + + def test_is_workflow_step_true(self) -> None: + """Test is_workflow_step method with STEP level entries.""" + parser = LogParser() + + step_entry = LogEntry( + timestamp="2025-07-31T12:00:00", + level="STEP", + logger_name="test_logger", + message="Starting CI/CD workflow", + hook_id="hook-123", + ) + + assert parser.is_workflow_step(step_entry) is True + + def test_is_workflow_step_false(self) -> None: + """Test is_workflow_step method with non-STEP level entries.""" + parser = LogParser() + + info_entry = LogEntry( + timestamp="2025-07-31T12:00:00", + level="INFO", + logger_name="test_logger", + message="Regular info message", + hook_id="hook-123", + ) + + debug_entry = LogEntry( + timestamp="2025-07-31T12:00:00", + level="DEBUG", + logger_name="test_logger", + message="Debug message", + hook_id="hook-123", + ) + + assert parser.is_workflow_step(info_entry) is False + assert parser.is_workflow_step(debug_entry) is False + + def test_extract_workflow_steps_with_matching_hook_id(self) -> None: + """Test extract_workflow_steps with entries matching hook_id.""" + parser = LogParser() + target_hook_id = "hook-123" + + entries = [ + LogEntry( + timestamp="2025-07-31T12:00:00", + level="STEP", + logger_name="test_logger", + message="Starting workflow", + hook_id=target_hook_id, + ), + LogEntry( + timestamp="2025-07-31T12:00:01", + level="INFO", + logger_name="test_logger", + message="Regular info message", + hook_id=target_hook_id, + ), + LogEntry( + timestamp="2025-07-31T12:00:02", + level="STEP", + logger_name="test_logger", + message="Processing stage", + hook_id=target_hook_id, + ), + LogEntry( + timestamp="2025-07-31T12:00:03", + level="STEP", + logger_name="test_logger", + message="Different hook workflow", + hook_id="hook-456", + ), + ] + + workflow_steps = parser.extract_workflow_steps(entries, target_hook_id) + + assert len(workflow_steps) == 2 + assert all(step.level == "STEP" for step in workflow_steps) + assert all(step.hook_id == target_hook_id for step in workflow_steps) + assert workflow_steps[0].message == "Starting workflow" + assert workflow_steps[1].message == "Processing stage" + + def test_extract_workflow_steps_no_matching_entries(self) -> None: + """Test extract_workflow_steps with no matching entries.""" + parser = LogParser() + target_hook_id = "hook-123" + + entries = [ + LogEntry( + timestamp="2025-07-31T12:00:00", + level="INFO", + logger_name="test_logger", + message="Regular info message", + hook_id=target_hook_id, + ), + LogEntry( + timestamp="2025-07-31T12:00:01", + level="STEP", + logger_name="test_logger", + message="Different hook workflow", + hook_id="hook-456", + ), + ] + + workflow_steps = parser.extract_workflow_steps(entries, target_hook_id) + + assert len(workflow_steps) == 0 + + def test_extract_workflow_steps_empty_entries(self) -> None: + """Test extract_workflow_steps with empty entries list.""" + parser = LogParser() + + workflow_steps = parser.extract_workflow_steps([], "hook-123") + + assert len(workflow_steps) == 0 diff --git a/webhook_server/tests/test_memory_optimization.py b/webhook_server/tests/test_memory_optimization.py new file mode 100644 index 00000000..29c60524 --- /dev/null +++ b/webhook_server/tests/test_memory_optimization.py @@ -0,0 +1,308 @@ +"""Memory optimization tests for log viewer streaming functionality.""" + +import tempfile +import datetime +import time +import asyncio +from pathlib import Path +from unittest.mock import Mock +import pytest + + +from webhook_server.web.log_viewer import LogViewerController +from webhook_server.libs.log_parser import LogEntry + + +class TestStreamingMemoryOptimization: + """Test memory efficiency improvements in log viewer.""" + + def setup_method(self): + """Set up test environment.""" + from unittest.mock import patch + + self.mock_logger = Mock() + + # Override log directory for testing + self.temp_dir = tempfile.mkdtemp() + self.log_dir = Path(self.temp_dir) / "logs" + self.log_dir.mkdir(parents=True) + + # Mock Config to avoid file dependency + mock_config = Mock() + mock_config.data_dir = self.temp_dir + + # Create controller with mocked Config + with patch("webhook_server.web.log_viewer.Config", return_value=mock_config): + self.controller = LogViewerController(logger=self.mock_logger) + + # Override the log directory method to use our temp directory + self.controller._get_log_directory = lambda: self.log_dir + + def generate_large_log_file(self, file_path: Path, num_entries: int = 10000) -> None: + """Generate a large log file for testing with realistic format.""" + with open(file_path, "w") as f: + base_time = datetime.datetime(2025, 7, 31, 10, 0, 0) + + for i in range(num_entries): + # Add microseconds to match real format (ensure non-zero microseconds) + timestamp = base_time + datetime.timedelta(seconds=i, microseconds=((i + 1) * 1000) % 1000000) + level = ["INFO", "DEBUG", "WARNING", "ERROR"][i % 4] + repo = ["test-repo", "webhook-server", "large-project"][i % 3] + hook_id = f"hook-{i % 100:04d}" # Zero-padded + user = f"user{i % 10}" + event = ["push", "pull_request", "issue_comment"][i % 3] + + # Generate realistic log format matching the production logs + log_line = ( + f"{timestamp.isoformat()} GithubWebhook {level} " + f"{repo} [{event}][{hook_id}][{user}]: Processing webhook step {i}\n" + ) + f.write(log_line) + + def test_streaming_efficiency_and_limits(self): + """Test that streaming approach processes efficiently with proper limits.""" + # Create multiple large log files + for i in range(3): + log_file = self.log_dir / f"webhook_{i}.log" + self.generate_large_log_file(log_file, 5000) # 15k total entries + + # Test streaming with limits to prevent memory issues + streaming_entries = [] + count = 0 + for entry in self.controller._stream_log_entries(max_files=3, max_entries=1000): + if count >= 500: # Stop early to test early termination + break + streaming_entries.append(entry) + count += 1 + + # Streaming should respect limits and early termination + assert len(streaming_entries) == 500 + assert all(isinstance(entry, LogEntry) for entry in streaming_entries) + + # Test that streaming doesn't load all entries at once + all_possible_entries = list(self.controller._stream_log_entries(max_files=3, max_entries=50000)) + + # Should respect max_entries limit + assert len(all_possible_entries) <= 15000 # 3 files * 5000 entries max + assert len(streaming_entries) < len(all_possible_entries) # Early termination worked + + def test_chunked_processing_efficiency(self): + """Test that chunked processing maintains good performance.""" + # Create a large log file + log_file = self.log_dir / "large_webhook.log" + self.generate_large_log_file(log_file, 10000) + + # Test chunked streaming performance + start_time = time.perf_counter() + + entries_processed = 0 + for entry in self.controller._stream_log_entries(chunk_size=500, max_entries=5000): + entries_processed += 1 + if entries_processed >= 2000: # Stop after processing 2000 entries + break + + end_time = time.perf_counter() + duration = end_time - start_time + + # Should process efficiently + assert entries_processed == 2000 + assert duration < 2.0 # Should complete in under 2 seconds + + # Calculate throughput + entries_per_second = entries_processed / duration + assert entries_per_second > 1000 # Should process at least 1000 entries/second + + def test_memory_efficient_filtering(self): + """Test that memory-efficient filtering works correctly.""" + # Create log files with specific patterns + log_file = self.log_dir / "filtered_test.log" + + with open(log_file, "w") as f: + base_time = datetime.datetime(2025, 7, 31, 10, 0, 0) + + for i in range(5000): + timestamp = base_time + datetime.timedelta(seconds=i, microseconds=((i + 1) * 1000) % 1000000) + hook_id = "target-hook" if i % 10 == 0 else f"other-hook-{i}" + + log_line = ( + f"{timestamp.isoformat()} GithubWebhook INFO test-repo [push][{hook_id}][user]: Message {i}\n" + ) + f.write(log_line) + + # Use get_log_entries with filtering + result = self.controller.get_log_entries(hook_id="target-hook", limit=100) + + # Should find approximately 500 entries (every 10th entry) + # But limited to 100 by the limit parameter + assert len(result["entries"]) <= 100 + + # Check that filtering actually worked + for entry_dict in result["entries"]: + assert entry_dict["hook_id"] == "target-hook" + + # Test that we can get a reasonable number of filtered results + assert len(result["entries"]) > 0 # Should find some matching entries + + def test_early_termination_optimization(self): + """Test that early termination prevents unnecessary processing.""" + # Create log files + log_file = self.log_dir / "early_term_test.log" + self.generate_large_log_file(log_file, 8000) + + start_time = time.perf_counter() + + # Request small result set to test early termination + result = self.controller.get_log_entries(limit=50) + + end_time = time.perf_counter() + duration = end_time - start_time + + # Should complete quickly due to early termination + assert len(result["entries"]) <= 50 + assert duration < 1.0 # Should complete in under 1 second + + # Should not process all 8000 entries + # The streaming should stop after finding enough matching entries + + def test_large_export_memory_efficiency(self): + """Test that large exports work correctly with streaming.""" + # Create multiple log files + for i in range(3): + log_file = self.log_dir / f"export_test_{i}.log" + self.generate_large_log_file(log_file, 3000) # 9k total entries + + # Test export with reasonable limit + response = self.controller.export_logs(format_type="json", limit=2000) + + # Export should work correctly + assert response.status_code == 200 + assert response.media_type == "application/json" + + # Should have content-disposition header for download + assert "Content-Disposition" in response.headers + assert "attachment" in response.headers["Content-Disposition"] + + def test_pagination_efficiency(self): + """Test that pagination with offset works efficiently.""" + # Create log file + log_file = self.log_dir / "pagination_test.log" + self.generate_large_log_file(log_file, 5000) + + # Test pagination with offset + start_time = time.perf_counter() + + result = self.controller.get_log_entries( + limit=100, + offset=2000, # Skip first 2000 entries + ) + + end_time = time.perf_counter() + duration = end_time - start_time + + # Should handle pagination efficiently + assert len(result["entries"]) <= 100 + assert result["offset"] == 2000 + assert duration < 2.0 # Should complete in reasonable time + + # Verify pagination worked correctly by checking timestamps + # (entries should be from later in the log due to offset) + if result["entries"]: + # All entries should be from the later part of the log + assert len(result["entries"]) > 0 + + @pytest.mark.asyncio + async def test_concurrent_streaming_safety(self): + """Test that streaming is safe under concurrent access.""" + # Create log file + log_file = self.log_dir / "concurrent_test.log" + self.generate_large_log_file(log_file, 3000) + + async def stream_entries(): + """Async wrapper for streaming entries.""" + # Run the synchronous streaming operation in a thread to avoid blocking + loop = asyncio.get_event_loop() + return await loop.run_in_executor(None, lambda: list(self.controller._stream_log_entries(max_entries=1000))) + + # Test multiple concurrent streaming operations + # Simulate concurrent access by running multiple streaming operations simultaneously + num_concurrent_operations = 5 + tasks = [stream_entries() for _ in range(num_concurrent_operations)] + + # Execute all tasks concurrently + results = await asyncio.gather(*tasks) + + # Verify all operations completed successfully + assert len(results) == num_concurrent_operations + + for entries in results: + assert len(entries) <= 1000 + assert all(isinstance(entry, LogEntry) for entry in entries) + + # Verify that all concurrent operations returned consistent results + # (all should have same number of entries since reading same file) + entry_counts = [len(entries) for entries in results] + assert all(count == entry_counts[0] for count in entry_counts), ( + f"Inconsistent entry counts across concurrent operations: {entry_counts}" + ) + + def teardown_method(self): + """Clean up test environment.""" + import shutil + + if Path(self.temp_dir).exists(): + shutil.rmtree(self.temp_dir) + + +class TestMemoryRegressionPrevention: + """Tests to prevent memory usage regressions.""" + + def test_streaming_functionality_baseline(self): + """Establish baseline functionality for regression testing.""" + from unittest.mock import patch + + mock_logger = Mock() + + # Create temporary log directory + with tempfile.TemporaryDirectory() as temp_dir: + log_dir = Path(temp_dir) / "logs" + log_dir.mkdir() + + # Mock Config to avoid file dependency + mock_config = Mock() + mock_config.data_dir = temp_dir + + # Create controller with mocked Config + with patch("webhook_server.web.log_viewer.Config", return_value=mock_config): + controller = LogViewerController(logger=mock_logger) + + # Mock log directory + controller._get_log_directory = lambda: log_dir + + # Create small test log file + log_file = log_dir / "baseline_test.log" + with open(log_file, "w") as f: + base_time = datetime.datetime(2025, 7, 31, 10, 0, 0) + for i in range(1000): + # Ensure all entries have microseconds (avoid 0 by using 1000 + remainder) + microseconds = 1000 + (i * 1000) % 999000 + timestamp = base_time + datetime.timedelta(seconds=i, microseconds=microseconds) + f.write( + f"{timestamp.isoformat()} GithubWebhook INFO test-repo [push][hook-{i:04d}][user]: Message {i}\n" + ) + + # Test streaming functionality + entries = list(controller._stream_log_entries(max_entries=1000)) + + # Baseline functionality that should not regress + assert len(entries) == 1000 + assert all(isinstance(entry, LogEntry) for entry in entries) + + # Test that streaming respects limits + limited_entries = list(controller._stream_log_entries(max_entries=500)) + assert len(limited_entries) == 500 + + # Test that get_log_entries works with streaming + result = controller.get_log_entries(limit=100) + assert len(result["entries"]) == 100 + assert "entries_processed" in result + assert "is_partial_scan" in result diff --git a/webhook_server/tests/test_performance_benchmarks.py b/webhook_server/tests/test_performance_benchmarks.py new file mode 100644 index 00000000..7c267646 --- /dev/null +++ b/webhook_server/tests/test_performance_benchmarks.py @@ -0,0 +1,525 @@ +"""Performance benchmark tests for webhook server log functionality.""" + +import asyncio +import datetime +import json +import os +import random +import tempfile +import time +from pathlib import Path + +import pytest + +try: + import psutil + + PSUTIL_AVAILABLE = True +except ImportError: + psutil = None + PSUTIL_AVAILABLE = False + +from webhook_server.libs.log_parser import LogEntry, LogFilter, LogParser + + +class TestLogParsingPerformance: + """Performance benchmarks for log parsing functionality.""" + + def generate_test_log_content(self, num_entries: int = 10000) -> str: + """Generate realistic test log content for performance testing.""" + log_lines = [] + base_time = datetime.datetime(2025, 7, 31, 10, 0, 0) + + repos = ["test-repo-1", "test-repo-2", "webhook-server", "large-project"] + events = ["push", "pull_request", "pull_request.opened", "pull_request.closed", "check_run"] + users = ["user1", "user2", "myakove", "bot-user", "reviewer"] + levels = ["INFO", "DEBUG", "WARNING", "ERROR"] + + for i in range(num_entries): + # Generate time with microseconds like working tests + microsecond = (i * 100000) % 1000000 + timestamp = base_time + datetime.timedelta(seconds=i // 10, microseconds=microsecond) + repo = random.choice(repos) + event = random.choice(events) + user = random.choice(users) + level = random.choice(levels) + hook_id = f"hook-{random.randint(1000, 9999)}-{i}" + + # Add PR number to some entries + pr_suffix = f"[PR {random.randint(1, 500)}]" if random.random() < 0.3 else "" + + # Format timestamp with microseconds + timestamp_str = timestamp.strftime("%Y-%m-%dT%H:%M:%S.%f") + + log_line = ( + f"{timestamp_str} GithubWebhook {level} " + f"{repo} [{event}][{hook_id}][{user}]{pr_suffix}: " + f"Processing webhook step {i}" + ) + log_lines.append(log_line) + + return "\n".join(log_lines) + + def test_log_parsing_performance_10k_entries(self): + """Test parsing performance with 10,000 log entries.""" + content = self.generate_test_log_content(10000) + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(content) + f.flush() + + parser = LogParser() + + # Measure parsing time + start_time = time.perf_counter() + entries = parser.parse_log_file(Path(f.name)) + end_time = time.perf_counter() + + parse_duration = end_time - start_time + + # Performance assertions + assert len(entries) > 9500 # Allow for some parsing failures + assert parse_duration < 2.0 # Should parse 10k entries in under 2 seconds + + # Calculate performance metrics + entries_per_second = len(entries) / parse_duration + assert entries_per_second > 5000 # Should parse at least 5k entries/second + + # Memory efficiency check (basic) + assert len(entries) == len([e for e in entries if e is not None]) + + def test_log_parsing_performance_100k_entries(self): + """Test parsing performance with 100,000 log entries (stress test).""" + content = self.generate_test_log_content(100000) + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(content) + f.flush() + + parser = LogParser() + + # Measure parsing time + start_time = time.perf_counter() + entries = parser.parse_log_file(Path(f.name)) + end_time = time.perf_counter() + + parse_duration = end_time - start_time + + # Performance assertions for large datasets + assert len(entries) > 95000 # Allow for some parsing failures + assert parse_duration < 20.0 # Should parse 100k entries in under 20 seconds + + # Calculate performance metrics + entries_per_second = len(entries) / parse_duration + assert entries_per_second > 5000 # Maintain performance at scale + + def test_filter_performance_large_dataset(self): + """Test filtering performance on large datasets.""" + # Generate large dataset in memory + entries = [] + base_time = datetime.datetime(2025, 7, 31, 10, 0, 0) + + for i in range(50000): + entry = LogEntry( + timestamp=base_time + datetime.timedelta(seconds=i), + level=random.choice(["INFO", "DEBUG", "WARNING", "ERROR"]), + logger_name="GithubWebhook", + message=f"Test message {i}", + hook_id=f"hook-{random.randint(1000, 5000)}", + event_type=random.choice(["push", "pull_request"]), + repository=random.choice(["repo1", "repo2", "repo3"]), + pr_number=random.randint(1, 1000) if random.random() < 0.3 else None, + github_user=random.choice(["user1", "user2", "user3"]), + ) + entries.append(entry) + + log_filter = LogFilter() + + # Test different filter operations and measure performance + test_cases = [ + {"hook_id": "hook-1234"}, + {"repository": "repo1"}, + {"event_type": "pull_request"}, + {"level": "INFO"}, + {"pr_number": 123}, + {"search_text": "message"}, + {"limit": 1000}, + {"repository": "repo1", "event_type": "push", "level": "INFO"}, + ] + + for test_case in test_cases: + start_time = time.perf_counter() + filtered = log_filter.filter_entries(entries, **test_case) + end_time = time.perf_counter() + + filter_duration = end_time - start_time + + # Filtering should be fast even on large datasets + assert filter_duration < 1.0 # Filter 50k entries in under 1 second + assert isinstance(filtered, list) + + @pytest.mark.asyncio + async def test_async_log_monitoring_performance(self): + """Test performance of async log monitoring.""" + content = self.generate_test_log_content(1000) + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(content) + f.flush() + + parser = LogParser() + + # Test async monitoring performance + start_time = time.perf_counter() + entries_collected = [] + + async def collect_entries(): + async for entry in parser.tail_log_file(Path(f.name), follow=False): + entries_collected.append(entry) + if len(entries_collected) >= 10: # Collect first 10 entries + break + + await collect_entries() + end_time = time.perf_counter() + + monitoring_duration = end_time - start_time + + # Async monitoring should be efficient + assert monitoring_duration < 0.5 # Should be very fast for non-following mode + assert len(entries_collected) >= 0 # May be 0 for non-following tail + + +class TestMemoryUsageProfiler: + """Memory usage profiling tests.""" + + def test_memory_efficiency_large_dataset(self): + """Test memory efficiency with large datasets.""" + if not PSUTIL_AVAILABLE: + pytest.skip("psutil not available for memory monitoring") + + process = psutil.Process(os.getpid()) + initial_memory = process.memory_info().rss / 1024 / 1024 # MB + + # Generate large dataset + parser = LogParser() + content = "" + for i in range(10000): + content += f"2025-07-31T10:{i // 600:02d}:{i % 60:02d}.000000 GithubWebhook INFO test-repo [push][hook-{i}][user]: Message {i}\n" + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(content) + f.flush() + + entries = parser.parse_log_file(Path(f.name)) + + # Check memory usage after parsing + peak_memory = process.memory_info().rss / 1024 / 1024 # MB + memory_increase = peak_memory - initial_memory + + # Memory efficiency assertions + assert len(entries) == 10000 + assert memory_increase < 100 # Should not use more than 100MB for 10k entries + + # Memory per entry should be reasonable + memory_per_entry = memory_increase / len(entries) * 1024 # KB per entry + assert memory_per_entry < 10 # Less than 10KB per entry + + def test_memory_cleanup_after_processing(self): + """Test that memory is properly cleaned up after processing.""" + if not PSUTIL_AVAILABLE: + pytest.skip("psutil not available for memory monitoring") + + import gc + import os + + process = psutil.Process(os.getpid()) + initial_memory = process.memory_info().rss / 1024 / 1024 # MB + + # Process large dataset and then clean up + parser = LogParser() + content = self._generate_large_log_content(5000) + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(content) + f.flush() + + entries = parser.parse_log_file(Path(f.name)) + del entries # Explicit cleanup + gc.collect() # Force garbage collection + + # Check memory after cleanup + final_memory = process.memory_info().rss / 1024 / 1024 # MB + memory_leak = final_memory - initial_memory + + # Should not have significant memory leaks + assert memory_leak < 20 # Less than 20MB increase after cleanup + + def _generate_large_log_content(self, num_entries: int) -> str: + """Helper to generate large log content.""" + lines = [] + for i in range(num_entries): + lines.append( + f"2025-07-31T10:{i // 600:02d}:{i % 60:02d}.000000 GithubWebhook INFO " + f"test-repo [push][hook-{i}][user]: Processing entry {i} with some additional content" + ) + return "\n".join(lines) + + +class TestConcurrencyPerformance: + """Test performance under concurrent load.""" + + @pytest.mark.asyncio + async def test_concurrent_parsing_performance(self): + """Test performance of concurrent parsing operations.""" + # Create multiple log files + files = [] + for i in range(5): + content = self._generate_test_content(2000) + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(content) + f.flush() + files.append(Path(f.name)) + + parser = LogParser() + + def parse_file(file_path): + """Parse a single file.""" + return parser.parse_log_file(file_path) + + # Measure concurrent parsing + start_time = time.perf_counter() + + # Use asyncio.to_thread for concurrent execution of sync functions + tasks = [asyncio.create_task(asyncio.to_thread(parse_file, f)) for f in files] + results = await asyncio.gather(*tasks) + + end_time = time.perf_counter() + concurrent_duration = end_time - start_time + + # Verify results + total_entries = sum(len(entries) for entries in results) + assert total_entries > 9500 # 5 files * ~2000 entries each + + # Concurrent parsing should be efficient + assert concurrent_duration < 5.0 # Should complete in under 5 seconds + + # Calculate throughput + entries_per_second = total_entries / concurrent_duration + assert entries_per_second > 2000 # Good concurrent throughput + + @pytest.mark.asyncio + async def test_concurrent_filtering_performance(self): + """Test performance of concurrent filtering operations.""" + # Generate shared dataset + entries = [] + for i in range(10000): + entry = LogEntry( + timestamp=datetime.datetime(2025, 7, 31, 10, 0, 0, i), + level=random.choice(["INFO", "DEBUG", "ERROR"]), + logger_name="GithubWebhook", + message=f"Message {i}", + hook_id=f"hook-{i % 100}", + repository=f"repo-{i % 10}", + ) + entries.append(entry) + + log_filter = LogFilter() + + def filter_task(filter_params): + """Single filter task.""" + return log_filter.filter_entries(entries, **filter_params) + + # Different filter operations + filter_operations = [ + {"repository": "repo-1"}, + {"level": "INFO"}, + {"hook_id": "hook-25"}, + {"search_text": "Message"}, + {"limit": 100}, + ] + + # Measure concurrent filtering + start_time = time.perf_counter() + + tasks = [asyncio.create_task(asyncio.to_thread(filter_task, params)) for params in filter_operations] + results = await asyncio.gather(*tasks) + + end_time = time.perf_counter() + concurrent_duration = end_time - start_time + + # Verify results + assert len(results) == 5 + assert all(isinstance(result, list) for result in results) + + # Concurrent filtering should be fast + assert concurrent_duration < 2.0 # Multiple filters in under 2 seconds + + def _generate_test_content(self, num_entries: int) -> str: + """Helper to generate test log content.""" + lines = [] + for i in range(num_entries): + lines.append( + f"2025-07-31T10:{i // 600:02d}:{i % 60:02d}.{i % 1000:03d}000 GithubWebhook INFO " + f"test-repo-{i % 3} [push][hook-{i}][user-{i % 5}]: Processing entry {i}" + ) + return "\n".join(lines) + + +class TestRealtimeStreamingPerformance: + """Test performance of real-time streaming functionality.""" + + @pytest.mark.asyncio + async def test_websocket_streaming_throughput(self): + """Test WebSocket streaming throughput under load.""" + # This test simulates WebSocket streaming performance + entries_to_stream = [] + + # Generate entries for streaming + for i in range(1000): + entry = LogEntry( + timestamp=datetime.datetime.now() + datetime.timedelta(milliseconds=i), + level="INFO", + logger_name="GithubWebhook", + message=f"Streaming message {i}", + hook_id=f"stream-hook-{i}", + ) + entries_to_stream.append(entry) + + # Simulate streaming performance + start_time = time.perf_counter() + + streamed_entries = [] + for entry in entries_to_stream: + # Simulate JSON serialization (what happens in real WebSocket) + json_data = json.dumps(entry.to_dict()) + streamed_entries.append(json_data) + + # Simulate small async delay (realistic WebSocket behavior) + if len(streamed_entries) % 100 == 0: + await asyncio.sleep(0.001) # 1ms delay every 100 entries + + end_time = time.perf_counter() + streaming_duration = end_time - start_time + + # Performance assertions + assert len(streamed_entries) == 1000 + assert streaming_duration < 2.0 # Stream 1000 entries in under 2 seconds + + # Calculate streaming rate + entries_per_second = len(streamed_entries) / streaming_duration + assert entries_per_second > 500 # At least 500 entries/second + + @pytest.mark.asyncio + async def test_log_monitoring_latency(self): + """Test latency of log file monitoring.""" + # Create initial log file + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write("2025-07-31T10:00:00.000000 GithubWebhook INFO test: Initial entry\n") + f.flush() + + parser = LogParser() + log_path = Path(f.name) + + # Start monitoring + entries_received = [] + + async def monitor_logs(): + async for entry in parser.tail_log_file(log_path, follow=True): + entries_received.append((time.perf_counter(), entry)) + if len(entries_received) >= 3: # Stop after receiving 3 new entries + break + + # Start monitoring task + monitor_task = asyncio.create_task(monitor_logs()) + + # Give monitoring time to start + await asyncio.sleep(0.1) + + # Add new entries with timing + write_times = [] + for i in range(3): + write_time = time.perf_counter() + with open(log_path, "a") as append_f: + append_f.write(f"2025-07-31T10:00:{i + 1:02d}.000000 GithubWebhook INFO test: New entry {i + 1}\n") + append_f.flush() + write_times.append(write_time) + await asyncio.sleep(0.05) # Small delay between writes + + # Wait for monitoring to complete + try: + await asyncio.wait_for(monitor_task, timeout=2.0) + except asyncio.TimeoutError: + monitor_task.cancel() + + # Analyze latency + if len(entries_received) >= 3: + latencies = [] + for i, (receive_time, entry) in enumerate(entries_received): + if i < len(write_times): + latency = receive_time - write_times[i] + latencies.append(latency) + + if latencies: + avg_latency = sum(latencies) / len(latencies) + max_latency = max(latencies) + + # Latency assertions + assert avg_latency < 0.5 # Average latency under 500ms + assert max_latency < 1.0 # Maximum latency under 1 second + + +class TestRegressionPrevention: + """Test to prevent performance regressions.""" + + def test_parsing_performance_baseline(self): + """Establish baseline performance metrics for regression testing.""" + # Standard test dataset + content = self._generate_standardized_content() + + with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as f: + f.write(content) + f.flush() + + parser = LogParser() + + # Measure parsing performance + start_time = time.perf_counter() + entries = parser.parse_log_file(Path(f.name)) + end_time = time.perf_counter() + + parse_duration = end_time - start_time + + # Baseline metrics (these should not regress) + baseline_metrics = { + "entries_count": len(entries), + "parse_duration": parse_duration, + "entries_per_second": len(entries) / parse_duration, + "average_entry_size": len(content) / len(entries) if entries else 0, + } + + # Store baseline metrics for comparison + assert baseline_metrics["entries_count"] == 5000 # Standardized dataset + assert baseline_metrics["parse_duration"] < 1.0 # Should be fast + assert baseline_metrics["entries_per_second"] > 5000 # Good throughput + + # Performance should be consistent and fast + assert baseline_metrics["parse_duration"] < 1.0 # Should be fast + + def _generate_standardized_content(self) -> str: + """Generate standardized test content for regression testing.""" + lines = [] + + for i in range(5000): # Standardized size + level = ["INFO", "DEBUG", "WARNING", "ERROR"][i % 4] + repo = ["test-repo", "webhook-server", "large-project"][i % 3] + event = ["push", "pull_request", "check_run"][i % 3] + + # Use the same format as the working test + line = ( + f"2025-07-31T10:{i // 600:02d}:{i % 60:02d}.{i % 1000:03d}000 GithubWebhook {level} " + f"{repo} [{event}][hook-{i}][user{i % 10}]: " + f"Standardized test message {i} for regression testing" + ) + lines.append(line) + + return "\n".join(lines) diff --git a/webhook_server/tests/test_schema_validator.py b/webhook_server/tests/test_schema_validator.py index 5c530f9d..750c536f 100644 --- a/webhook_server/tests/test_schema_validator.py +++ b/webhook_server/tests/test_schema_validator.py @@ -11,6 +11,7 @@ from typing import Any, Union import yaml # type: ignore +from simple_logger.logger import get_logger class ConfigValidator: @@ -246,18 +247,20 @@ def validate_config_file(config_path: Union[str, Path]) -> bool: with open(config_path, "r") as file_handle: config_data = yaml.safe_load(file_handle) except Exception as exception: - print(f"Error loading config file: {exception}") + logger = get_logger(name="test_schema_validator") + logger.error(f"Error loading config file: {exception}") return False validator = ConfigValidator() is_valid = validator.validate_config(config_data) + logger = get_logger(name="test_schema_validator") if not is_valid: - print("Configuration validation failed:") + logger.error("Configuration validation failed:") for error in validator.errors: - print(f" - {error}") + logger.error(f" - {error}") else: - print("Configuration is valid!") + logger.info("Configuration is valid!") return is_valid @@ -265,13 +268,15 @@ def validate_config_file(config_path: Union[str, Path]) -> bool: def main() -> int: """Main entry point for command-line usage.""" if len(sys.argv) != 2: - print("Usage: python test_schema_validator.py ") + logger = get_logger(name="test_schema_validator") + logger.error("Usage: python test_schema_validator.py ") return 1 config_path = sys.argv[1] if not Path(config_path).exists(): - print(f"Error: Config file '{config_path}' does not exist") + logger = get_logger(name="test_schema_validator") + logger.error(f"Error: Config file '{config_path}' does not exist") return 1 is_valid = validate_config_file(config_path) diff --git a/webhook_server/utils/helpers.py b/webhook_server/utils/helpers.py index 7b13c003..f7e38266 100644 --- a/webhook_server/utils/helpers.py +++ b/webhook_server/utils/helpers.py @@ -2,7 +2,9 @@ import asyncio import datetime +import json import os +import random import shlex import subprocess from concurrent.futures import Future, as_completed @@ -11,9 +13,10 @@ import github from colorama import Fore -from github.RateLimit import RateLimit +from github.RateLimitOverview import RateLimitOverview from github.Repository import Repository from simple_logger.logger import get_logger +from stringcolor import cs from webhook_server.libs.config import Config from webhook_server.libs.exceptions import NoApiTokenError @@ -160,7 +163,7 @@ def get_api_with_highest_rate_limit(config: Config, repository_name: str = "") - api: github.Github | None = None token: str | None = None _api_user: str = "" - rate_limit: RateLimit | None = None + rate_limit: RateLimitOverview | None = None remaining = 0 @@ -186,8 +189,8 @@ def get_api_with_highest_rate_limit(config: Config, repository_name: str = "") - _rate_limit = _api.get_rate_limit() - if _rate_limit.core.remaining > remaining: - remaining = _rate_limit.core.remaining + if _rate_limit.rate.remaining > remaining: + remaining = _rate_limit.rate.remaining api, token, _api_user, rate_limit = _api, _token, _api_user, _rate_limit if rate_limit: @@ -200,25 +203,25 @@ def get_api_with_highest_rate_limit(config: Config, repository_name: str = "") - return api, token, _api_user -def log_rate_limit(rate_limit: RateLimit, api_user: str) -> None: +def log_rate_limit(rate_limit: RateLimitOverview, api_user: str) -> None: logger = get_logger_with_params(name="helpers") rate_limit_str: str - time_for_limit_reset: int = (rate_limit.core.reset - datetime.datetime.now(tz=datetime.timezone.utc)).seconds - below_minimum: bool = rate_limit.core.remaining < 700 + time_for_limit_reset: int = (rate_limit.rate.reset - datetime.datetime.now(tz=datetime.timezone.utc)).seconds + below_minimum: bool = rate_limit.rate.remaining < 700 if below_minimum: - rate_limit_str = f"{Fore.RED}{rate_limit.core.remaining}{Fore.RESET}" + rate_limit_str = f"{Fore.RED}{rate_limit.rate.remaining}{Fore.RESET}" - elif rate_limit.core.remaining < 2000: - rate_limit_str = f"{Fore.YELLOW}{rate_limit.core.remaining}{Fore.RESET}" + elif rate_limit.rate.remaining < 2000: + rate_limit_str = f"{Fore.YELLOW}{rate_limit.rate.remaining}{Fore.RESET}" else: - rate_limit_str = f"{Fore.GREEN}{rate_limit.core.remaining}{Fore.RESET}" + rate_limit_str = f"{Fore.GREEN}{rate_limit.rate.remaining}{Fore.RESET}" msg = ( - f"{Fore.CYAN}[{api_user}] API rate limit:{Fore.RESET} Current {rate_limit_str} of {rate_limit.core.limit}. " - f"Reset in {rate_limit.core.reset} [{datetime.timedelta(seconds=time_for_limit_reset)}] " + f"{Fore.CYAN}[{api_user}] API rate limit:{Fore.RESET} Current {rate_limit_str} of {rate_limit.rate.limit}. " + f"Reset in {rate_limit.rate.reset} [{datetime.timedelta(seconds=time_for_limit_reset)}] " f"(UTC time is {datetime.datetime.now(tz=datetime.timezone.utc)})" ) logger.debug(msg) @@ -241,3 +244,97 @@ def get_future_results(futures: list["Future"]) -> None: else: _log(_res[1]) + + +def get_repository_color_for_log_prefix(repository_name: str, data_dir: str) -> str: + """ + Get a consistent color for repository name in log prefixes. + + Args: + repository_name: Repository name to get color for + data_dir: Directory to store color mappings + + Returns: + Colored repository name string + """ + + def _get_random_color(_colors: list[str], _json: dict[str, str]) -> str: + color = random.choice(_colors) + _json[repository_name] = color + if _selected := cs(repository_name, color).render(): + return _selected + return repository_name + + _all_colors: list[str] = [] + color_json: dict[str, str] + _colors_to_exclude = ("blue", "white", "black", "grey") + color_file: str = os.path.join(data_dir, "log-colors.json") + + for _color_name in cs.colors.values(): + _cname = _color_name["name"] + if _cname.lower() in _colors_to_exclude: + continue + _all_colors.append(_cname) + + try: + with open(color_file) as fd: + color_json = json.load(fd) + except Exception: + color_json = {} + + if color := color_json.get(repository_name, ""): + _cs_object = cs(repository_name, color) + if cs.find_color(_cs_object): + _str_color = _cs_object.render() + else: + _str_color = _get_random_color(_colors=_all_colors, _json=color_json) + else: + _str_color = _get_random_color(_colors=_all_colors, _json=color_json) + + with open(color_file, "w") as fd: + json.dump(color_json, fd) + + if _str_color: + _str_color = _str_color.replace("\x1b", "\033") + return _str_color + return repository_name + + +def prepare_log_prefix( + event_type: str, + delivery_id: str, + repository_name: str | None = None, + api_user: str | None = None, + pr_number: int | None = None, + data_dir: str | None = None, +) -> str: + """ + Prepare standardized log prefix for consistent formatting across webhook processing. + + Args: + event_type: GitHub event type (e.g., 'pull_request', 'check_run') + delivery_id: GitHub delivery ID (x-github-delivery header) + repository_name: Repository name for color coding (optional) + api_user: API user for the request (optional) + pr_number: Pull request number if applicable (optional) + data_dir: Directory for storing color mappings (optional, defaults to /tmp) + + Returns: + Formatted log prefix string + """ + if repository_name and data_dir: + repository_color = get_repository_color_for_log_prefix(repository_name, data_dir) + else: + repository_color = repository_name or "" + + # Build prefix components + components = [event_type, delivery_id] + if api_user: + components.append(api_user) + + prefix = f"{repository_color} [{']['.join(components)}]" + + if pr_number: + prefix += f"[PR {pr_number}]" + + return prefix + ":" diff --git a/webhook_server/web/__init__.py b/webhook_server/web/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/webhook_server/web/log_viewer.py b/webhook_server/web/log_viewer.py new file mode 100644 index 00000000..d08173c2 --- /dev/null +++ b/webhook_server/web/log_viewer.py @@ -0,0 +1,902 @@ +"""Log viewer controller for serving log viewer web interface and API endpoints.""" + +import datetime +import json +import logging +import os +import re +from pathlib import Path +from typing import Any, Generator, Iterator + +from fastapi import HTTPException, WebSocket, WebSocketDisconnect +from fastapi.responses import HTMLResponse, StreamingResponse + +from webhook_server.libs.config import Config +from webhook_server.libs.log_parser import LogEntry, LogFilter, LogParser + + +class LogViewerController: + """Controller for log viewer functionality.""" + + # Workflow stage patterns for PR flow analysis + # These patterns match log messages to identify workflow stages and can be updated + # when log message formats change without modifying the analysis logic + WORKFLOW_STAGE_PATTERNS = [ + ("Webhook Received", r"Processing webhook"), + ("Validation Complete", r"Signature verification successful|Processing webhook for"), + ("Reviewers Assigned", r"Added reviewer|OWNERS file|reviewer assignment"), + ("Labels Applied", r"label|tag"), + ("Checks Started", r"check|test|build"), + ("Checks Complete", r"check.*complete|test.*pass|build.*success"), + ("Processing Complete", r"completed successfully|processing complete"), + ] + + def __init__(self, logger: logging.Logger) -> None: + """Initialize the log viewer controller. + + Args: + logger: Logger instance for this controller + """ + self.logger = logger + self.config = Config(logger=self.logger) + self.log_parser = LogParser() + self.log_filter = LogFilter() + self._websocket_connections: set[WebSocket] = set() + + async def shutdown(self) -> None: + """Close all active WebSocket connections during shutdown. + + This method should be called during application shutdown to properly + close all WebSocket connections and prevent resource leaks. + """ + self.logger.info( + f"Shutting down LogViewerController with {len(self._websocket_connections)} active connections" + ) + + # Create a copy of the connections set to avoid modification during iteration + connections_to_close = list(self._websocket_connections) + + for ws in connections_to_close: + try: + await ws.close(code=1001, reason="Server shutdown") + self.logger.debug("Successfully closed WebSocket connection during shutdown") + except Exception as e: + # Log the error but continue closing other connections + self.logger.warning(f"Error closing WebSocket connection during shutdown: {e}") + + # Clear the connections set + self._websocket_connections.clear() + self.logger.info("LogViewerController shutdown completed") + + def get_log_page(self) -> HTMLResponse: + """Serve the main log viewer HTML page. + + Returns: + HTML response with log viewer interface + + Raises: + HTTPException: 404 if template not found, 500 for other errors + """ + try: + html_content = self._get_log_viewer_html() + return HTMLResponse(content=html_content) + except FileNotFoundError: + self.logger.error("Log viewer HTML template not found") + raise HTTPException(status_code=404, detail="Log viewer template not found") + except Exception as e: + self.logger.error(f"Error serving log viewer page: {e}") + raise HTTPException(status_code=500, detail="Internal server error") + + def get_log_entries( + self, + hook_id: str | None = None, + pr_number: int | None = None, + repository: str | None = None, + event_type: str | None = None, + github_user: str | None = None, + level: str | None = None, + start_time: datetime.datetime | None = None, + end_time: datetime.datetime | None = None, + search: str | None = None, + limit: int = 100, + offset: int = 0, + ) -> dict[str, Any]: + """Retrieve historical log entries with filtering and pagination using memory-efficient streaming. + + This method implements memory-efficient log processing by streaming through log files + and applying filters incrementally to avoid loading large datasets into memory. + + Args: + hook_id: Filter by specific hook ID + pr_number: Filter by PR number + repository: Filter by repository name + event_type: Filter by GitHub event type + github_user: Filter by GitHub user (api_user) + level: Filter by log level + start_time: Start time filter + end_time: End time filter + search: Full-text search in log messages + limit: Number of entries to return (max 1000) + offset: Pagination offset + + Returns: + Dictionary containing filtered log entries and comprehensive metadata: + + - **entries**: List of log entry dictionaries matching the applied filters + - **entries_processed**: Number of log entries examined during this request. + May be an integer or string with "+" suffix (e.g., "50000+") indicating + the streaming process reached its maximum processing limit and more entries + exist. This helps API consumers understand data completeness. + - **filtered_count_min**: Minimum number of entries matching the current filters. + Calculated as len(entries) + offset, representing the definitive lower bound + of matching entries. This is useful for pagination calculations and showing + "showing X of at least Y results" messages. + - **total_log_count_estimate**: Statistical estimate of total log entries across + all log files (including rotated logs). Provides context about the overall + dataset size for UI statistics and capacity planning. Based on sampling + the first 10 log files to balance accuracy with performance. + - **limit**: Echo of the requested limit parameter + - **offset**: Echo of the requested offset parameter + - **is_partial_scan**: Boolean indicating whether the streaming process examined + all available logs (false) or stopped at the processing limit (true) + + Raises: + HTTPException: 400 for invalid parameters, 500 for file access errors + """ + try: + # Validate parameters + if limit < 1 or limit > 10000: + raise ValueError("Limit must be between 1 and 10000") + if offset < 0: + raise ValueError("Offset must be non-negative") + + # Use memory-efficient streaming with filtering applied during iteration + filtered_entries: list[LogEntry] = [] + total_processed = 0 + skipped = 0 + + # Stream entries and apply filters incrementally + # For any filtering, we need to process more entries to find all matches + has_filters = any([ + hook_id, + pr_number, + repository, + event_type, + github_user, + level, + start_time, + end_time, + search, + ]) + max_entries_to_process = 50000 if has_filters else 20000 + + for entry in self._stream_log_entries(max_files=25, max_entries=max_entries_to_process): + total_processed += 1 + + # Apply filters early to reduce memory usage + if not self._entry_matches_filters( + entry, hook_id, pr_number, repository, event_type, github_user, level, start_time, end_time, search + ): + continue + + # Handle pagination - skip entries until we reach the offset + if skipped < offset: + skipped += 1 + continue + + # Add to results if we haven't reached the limit + if len(filtered_entries) < limit: + filtered_entries.append(entry) + else: + # We have enough entries, can stop processing + break + + # Get approximate total count by processing a sample if needed + estimated_total: int | str = total_processed + if total_processed >= max_entries_to_process: # Hit our streaming limit + estimated_total = f"{total_processed}+" # Indicate there are more + + # Estimate total log count across all files for better UI statistics + total_log_count_estimate = self._estimate_total_log_count() + + return { + "entries": [entry.to_dict() for entry in filtered_entries], + "entries_processed": estimated_total, # Number of entries examined + "filtered_count_min": len(filtered_entries) + offset, # Minimum filtered count + "total_log_count_estimate": total_log_count_estimate, # Estimated total logs in all files + "limit": limit, + "offset": offset, + "is_partial_scan": total_processed >= max_entries_to_process, # Indicates not all logs were scanned + } + + except ValueError as e: + self.logger.warning(f"Invalid parameters for log entries request: {e}") + raise HTTPException(status_code=400, detail=str(e)) + except (OSError, PermissionError) as e: + self.logger.error(f"File access error loading log entries: {e}") + raise HTTPException(status_code=500, detail="Error accessing log files") + except Exception as e: + self.logger.error(f"Unexpected error getting log entries: {e}") + raise HTTPException(status_code=500, detail="Internal server error") + + def _entry_matches_filters( + self, + entry: LogEntry, + hook_id: str | None = None, + pr_number: int | None = None, + repository: str | None = None, + event_type: str | None = None, + github_user: str | None = None, + level: str | None = None, + start_time: datetime.datetime | None = None, + end_time: datetime.datetime | None = None, + search: str | None = None, + ) -> bool: + """Check if a single entry matches the given filters. + + This allows for early filtering during streaming to reduce memory usage. + + Args: + entry: LogEntry to check + **filters: Filter parameters (same as get_log_entries) + + Returns: + True if entry matches all filters, False otherwise + """ + if hook_id is not None and entry.hook_id != hook_id: + return False + if pr_number is not None and entry.pr_number != pr_number: + return False + if repository is not None and entry.repository != repository: + return False + if event_type is not None and entry.event_type != event_type: + return False + if github_user is not None and entry.github_user != github_user: + return False + if level is not None and entry.level != level: + return False + if start_time is not None and entry.timestamp < start_time: + return False + if end_time is not None and entry.timestamp > end_time: + return False + if search is not None and search.lower() not in entry.message.lower(): + return False + + return True + + def export_logs( + self, + format_type: str, + hook_id: str | None = None, + pr_number: int | None = None, + repository: str | None = None, + event_type: str | None = None, + github_user: str | None = None, + level: str | None = None, + start_time: datetime.datetime | None = None, + end_time: datetime.datetime | None = None, + search: str | None = None, + limit: int = 10000, + ) -> StreamingResponse: + """Export filtered logs as JSON file. + + Args: + format_type: Export format (only "json" is supported) + hook_id: Filter by specific hook ID + pr_number: Filter by PR number + repository: Filter by repository name + event_type: Filter by GitHub event type + github_user: Filter by GitHub user (api_user) + level: Filter by log level + start_time: Start time filter + end_time: End time filter + search: Full-text search in log messages + limit: Maximum number of entries to export + + Returns: + StreamingResponse with file download + + Raises: + HTTPException: 400 for invalid format, 413 if result set too large + """ + try: + if format_type != "json": + raise ValueError(f"Invalid format: {format_type}. Only 'json' is supported.") + + if limit > 50000: + raise ValueError("Result set too large (max 50000 entries)") + + # Use memory-efficient streaming for large exports + filtered_entries: list[LogEntry] = [] + + # Stream entries and apply filters incrementally for better memory usage + # For any filtering, increase processing limit to find all matches + has_filters = any([ + hook_id, + pr_number, + repository, + event_type, + github_user, + level, + start_time, + end_time, + search, + ]) + max_entries_to_process = min(limit + 20000, 50000) if has_filters else limit + 1000 + + for entry in self._stream_log_entries(max_files=25, max_entries=max_entries_to_process): + if not self._entry_matches_filters( + entry, hook_id, pr_number, repository, event_type, github_user, level, start_time, end_time, search + ): + continue + + filtered_entries.append(entry) + + # Stop when we reach the export limit + if len(filtered_entries) >= limit: + break + + # Generate JSON export content + content = self._generate_json_export(filtered_entries) + media_type = "application/json" + filename = f"webhook_logs_{datetime.datetime.now().strftime('%Y%m%d_%H%M%S')}.json" + + def generate() -> Generator[bytes, None, None]: + yield content.encode("utf-8") + + return StreamingResponse( + generate(), + media_type=media_type, + headers={"Content-Disposition": f"attachment; filename={filename}"}, + ) + + except ValueError as e: + if "Result set too large" in str(e): + self.logger.warning(f"Export request too large: {e}") + raise HTTPException(status_code=413, detail=str(e)) + else: + self.logger.warning(f"Invalid export parameters: {e}") + raise HTTPException(status_code=400, detail=str(e)) + except Exception as e: + self.logger.error(f"Error generating export: {e}") + raise HTTPException(status_code=500, detail="Export generation failed") + + async def handle_websocket( + self, + websocket: WebSocket, + hook_id: str | None = None, + pr_number: int | None = None, + repository: str | None = None, + event_type: str | None = None, + github_user: str | None = None, + level: str | None = None, + ) -> None: + """Handle WebSocket connection for real-time log streaming. + + Args: + websocket: WebSocket connection + hook_id: Filter by specific hook ID + pr_number: Filter by PR number + repository: Filter by repository name + event_type: Filter by GitHub event type + github_user: Filter by GitHub user (api_user) + level: Filter by log level + """ + await websocket.accept() + self._websocket_connections.add(websocket) + + try: + self.logger.info("WebSocket connection established for log streaming") + + # Get log directory path + log_dir = self._get_log_directory() + if not log_dir.exists(): + await websocket.send_json({"error": "Log directory not found"}) + return + + # Start monitoring log files for new entries + async for entry in self.log_parser.monitor_log_directory(log_dir): + should_send = False + + # Apply filters to new entry - if no filters provided, send all entries + if not any([hook_id, pr_number, repository, event_type, github_user, level]): + should_send = True + else: + # Apply filters + filtered_entries = self.log_filter.filter_entries( + entries=[entry], + hook_id=hook_id, + pr_number=pr_number, + repository=repository, + event_type=event_type, + github_user=github_user, + level=level, + ) + should_send = bool(filtered_entries) + + if should_send: + try: + await websocket.send_json(entry.to_dict()) + except WebSocketDisconnect: + break + + except WebSocketDisconnect: + self.logger.info("WebSocket client disconnected") + except Exception as e: + self.logger.error(f"Error in WebSocket handler: {e}") + try: + await websocket.close(code=1011, reason="Internal server error") + except Exception: + pass + finally: + self._websocket_connections.discard(websocket) + + def get_pr_flow_data(self, hook_id: str) -> dict[str, Any]: + """Get PR flow visualization data for a specific hook ID or PR number. + + Args: + hook_id: Hook ID (e.g., "hook-abc123") or PR number (e.g., "pr-456") + + Returns: + Dictionary with flow stages and timing data + + Raises: + HTTPException: 404 if no data found for hook_id + """ + try: + # Parse hook_id to determine if it's a hook ID or PR number + if hook_id.startswith("hook-"): + actual_hook_id = hook_id[5:] # Remove "hook-" prefix + pr_number = None + elif hook_id.startswith("pr-"): + actual_hook_id = None + pr_number = int(hook_id[3:]) # Remove "pr-" prefix + else: + # Try to parse as direct hook ID or PR number + try: + pr_number = int(hook_id) + actual_hook_id = None + except ValueError: + actual_hook_id = hook_id + pr_number = None + + # Use streaming approach for memory efficiency + filtered_entries: list[LogEntry] = [] + + # Stream entries and filter by hook_id/pr_number + for entry in self._stream_log_entries(max_files=15, max_entries=10000): + if not self._entry_matches_filters(entry, hook_id=actual_hook_id, pr_number=pr_number): + continue + filtered_entries.append(entry) + + if not filtered_entries: + raise ValueError(f"No data found for hook_id: {hook_id}") + + # Analyze flow stages from log entries + flow_data = self._analyze_pr_flow(filtered_entries, hook_id) + return flow_data + + except ValueError as e: + if "No data found" in str(e): + self.logger.warning(f"PR flow data not found: {e}") + raise HTTPException(status_code=404, detail=str(e)) + else: + self.logger.warning(f"Invalid PR flow hook_id: {e}") + raise HTTPException(status_code=400, detail=str(e)) + except Exception as e: + self.logger.error(f"Error getting PR flow data: {e}") + raise HTTPException(status_code=500, detail="Internal server error") + + def get_workflow_steps(self, hook_id: str) -> dict[str, Any]: + """Get workflow step timeline data for a specific hook ID. + + Args: + hook_id: The hook ID to get workflow steps for + + Returns: + Dictionary with workflow steps and timing data + + Raises: + HTTPException: 404 if no steps found for hook ID + """ + try: + # Use streaming approach for memory efficiency + filtered_entries: list[LogEntry] = [] + + # Stream entries and filter by hook ID + for entry in self._stream_log_entries(max_files=15, max_entries=10000): + if not self._entry_matches_filters(entry, hook_id=hook_id): + continue + filtered_entries.append(entry) + + if not filtered_entries: + raise ValueError(f"No data found for hook ID: {hook_id}") + + # Extract only workflow step entries (logger.step calls) + workflow_steps = self.log_parser.extract_workflow_steps(filtered_entries, hook_id) + + if not workflow_steps: + raise ValueError(f"No workflow steps found for hook ID: {hook_id}") + + # Build timeline data + timeline_data = self._build_workflow_timeline(workflow_steps, hook_id) + return timeline_data + + except ValueError as e: + if "No data found" in str(e) or "No workflow steps found" in str(e): + self.logger.warning(f"Workflow steps not found: {e}") + raise HTTPException(status_code=404, detail=str(e)) + else: + self.logger.warning(f"Invalid hook ID: {e}") + raise HTTPException(status_code=400, detail=str(e)) + except Exception as e: + self.logger.error(f"Error getting workflow steps: {e}") + raise HTTPException(status_code=500, detail="Internal server error") + + def _build_workflow_timeline(self, workflow_steps: list[LogEntry], hook_id: str) -> dict[str, Any]: + """Build timeline data from workflow step entries. + + Args: + workflow_steps: List of workflow step log entries + hook_id: The hook ID for this timeline + + Returns: + Dictionary with timeline data structure + """ + # Sort steps by timestamp + sorted_steps = sorted(workflow_steps, key=lambda x: x.timestamp) + + # Extract timeline data + timeline_steps = [] + start_time = sorted_steps[0].timestamp if sorted_steps else None + + for step in sorted_steps: + # Calculate relative time from start + relative_time = 0 + if start_time: + relative_time = int((step.timestamp - start_time).total_seconds() * 1000) # milliseconds + + timeline_steps.append({ + "timestamp": step.timestamp.isoformat(), + "relative_time_ms": relative_time, + "message": step.message, + "level": step.level, + "repository": step.repository, + "event_type": step.event_type, + "pr_number": step.pr_number, + }) + + # Calculate total duration + total_duration_ms = 0 + if len(sorted_steps) > 1: + total_duration_ms = int((sorted_steps[-1].timestamp - sorted_steps[0].timestamp).total_seconds() * 1000) + + return { + "hook_id": hook_id, + "start_time": start_time.isoformat() if start_time else None, + "total_duration_ms": total_duration_ms, + "step_count": len(timeline_steps), + "steps": timeline_steps, + } + + def _stream_log_entries( + self, max_files: int = 10, chunk_size: int = 1000, max_entries: int = 50000 + ) -> Iterator[LogEntry]: + """Stream log entries from configured log files in chunks to reduce memory usage. + + This replaces _load_log_entries() to prevent memory exhaustion from loading + all log files simultaneously. Uses lazy evaluation and chunked processing. + + Args: + max_files: Maximum number of log files to process (newest first) + chunk_size: Number of entries to yield per chunk from each file + max_entries: Maximum total entries to yield (safety limit) + + Yields: + LogEntry objects in timestamp order (newest first) + """ + log_dir = self._get_log_directory() + + if not log_dir.exists(): + self.logger.warning(f"Log directory not found: {log_dir}") + return + + # Find all log files including rotated ones (*.log, *.log.1, *.log.2, etc.) + log_files: list[Path] = [] + log_files.extend(log_dir.glob("*.log")) + log_files.extend(log_dir.glob("*.log.*")) + + # Sort log files to process in correct order (current log first, then rotated by number) + def sort_key(f: Path) -> tuple: + name_parts = f.name.split(".") + if len(name_parts) > 2 and name_parts[-1].isdigit(): + # Rotated file: extract rotation number + return (1, int(name_parts[-1])) + else: + # Current log file + return (0, 0) + + log_files.sort(key=sort_key) + log_files = log_files[:max_files] + + self.logger.info(f"Streaming from {len(log_files)} most recent files: {[f.name for f in log_files]}") + + total_yielded = 0 + + # Stream from newest files first + for log_file in log_files: + if total_yielded >= max_entries: + break + + try: + file_entries: list[LogEntry] = [] + + # Parse file in one go (files are typically reasonable size individually) + with open(log_file, "r", encoding="utf-8") as f: + for line_num, line in enumerate(f, 1): + if total_yielded >= max_entries: + break + + entry = self.log_parser.parse_log_entry(line) + if entry: + file_entries.append(entry) + + # Process in chunks to avoid memory buildup for large files + if len(file_entries) >= chunk_size: + # Sort chunk by timestamp (newest first) and yield + file_entries.sort(key=lambda x: x.timestamp, reverse=True) + for entry in file_entries: + yield entry + total_yielded += 1 + if total_yielded >= max_entries: + break + file_entries.clear() # Free memory + + # Yield remaining entries from this file + if file_entries and total_yielded < max_entries: + file_entries.sort(key=lambda x: x.timestamp, reverse=True) + for entry in file_entries: + if total_yielded >= max_entries: + break + yield entry + total_yielded += 1 + + self.logger.debug(f"Streamed entries from {log_file.name}, total so far: {total_yielded}") + + except Exception as e: + self.logger.warning(f"Error streaming log file {log_file}: {e}") + + def _load_log_entries(self) -> list[LogEntry]: + """Load log entries using streaming approach for memory efficiency. + + This method now uses the streaming approach internally but returns a list + for backward compatibility. For new code, prefer _stream_log_entries(). + + Returns: + List of parsed log entries (limited to prevent memory exhaustion) + """ + # Use streaming with reasonable limits to prevent memory issues + entries = list(self._stream_log_entries(max_files=10, max_entries=10000)) + self.logger.info(f"Loaded {len(entries)} entries using streaming approach") + return entries + + def _get_log_directory(self) -> Path: + """Get the log directory path from configuration. + + Returns: + Path to log directory + """ + # Use the same log directory as the main application + log_dir_path = os.path.join(self.config.data_dir, "logs") + return Path(log_dir_path) + + def _get_log_viewer_html(self) -> str: + """Load and return the log viewer HTML template. + + Returns: + HTML content for log viewer interface + + Raises: + FileNotFoundError: If template file cannot be found + IOError: If template file cannot be read + """ + template_path = Path(__file__).parent / "templates" / "log_viewer.html" + + try: + with open(template_path, "r", encoding="utf-8") as f: + return f.read() + except FileNotFoundError: + self.logger.error(f"Log viewer template not found at {template_path}") + return self._get_fallback_html() + except IOError as e: + self.logger.error(f"Failed to read log viewer template: {e}") + return self._get_fallback_html() + + def _get_fallback_html(self) -> str: + """Provide a minimal fallback HTML when template loading fails. + + Returns: + Basic HTML page with error message + """ + return """ + + + + + GitHub Webhook Server - Log Viewer (Error) + + + +
+
⚠️
+

Log Viewer Template Error

+

The log viewer template could not be loaded. Please check the server logs for details.

+ +
+ +""" + + def _generate_json_export(self, entries: list[LogEntry]) -> str: + """Generate JSON export content from log entries. + + Args: + entries: List of log entries to export + + Returns: + JSON content as string + """ + return json.dumps([entry.to_dict() for entry in entries], indent=2) + + def _analyze_pr_flow(self, entries: list[LogEntry], hook_id: str) -> dict[str, Any]: + """Analyze PR workflow stages from log entries. + + Args: + entries: List of log entries for the PR/hook + hook_id: Original hook_id used for the request + + Returns: + Dictionary with flow stages and timing data + """ + # Sort entries by timestamp + sorted_entries = sorted(entries, key=lambda x: x.timestamp) + + if not sorted_entries: + return { + "identifier": hook_id, + "stages": [], + "total_duration_ms": 0, + "success": False, + "error": "No log entries found", + } + + stages = [] + start_time = sorted_entries[0].timestamp + success = True + error_message = None + + # Use class-level workflow stage patterns for analysis + stage_patterns = self.WORKFLOW_STAGE_PATTERNS + + previous_time = start_time + for pattern_name, pattern in stage_patterns: + # Find first entry matching this stage + for entry in sorted_entries: + if any(re.search(p, entry.message, re.IGNORECASE) for p in pattern.split("|")): + duration_ms = int((entry.timestamp - previous_time).total_seconds() * 1000) + + stage = { + "name": pattern_name, + "timestamp": entry.timestamp.isoformat(), + "duration_ms": duration_ms if entry.timestamp != start_time else None, + } + + # Check for errors in this stage + if entry.level == "ERROR": + stage["error"] = entry.message + success = False + error_message = entry.message + + stages.append(stage) + previous_time = entry.timestamp + break + + # Check for any error entries + error_entries = [e for e in sorted_entries if e.level == "ERROR"] + if error_entries and success: + success = False + error_message = error_entries[0].message + + total_duration = int((sorted_entries[-1].timestamp - start_time).total_seconds() * 1000) + + flow_data = { + "identifier": hook_id, + "stages": stages, + "total_duration_ms": total_duration, + "success": success, + } + + if error_message: + flow_data["error"] = error_message + + return flow_data + + def _estimate_total_log_count(self) -> str: + """Estimate total log count across all available log files. + + Returns: + String representing estimated total log count + """ + try: + log_dir = self._get_log_directory() + if not log_dir.exists(): + return "0" + + # Find all log files including rotated ones + log_files: list[Path] = [] + log_files.extend(log_dir.glob("*.log")) + log_files.extend(log_dir.glob("*.log.*")) + + if not log_files: + return "0" + + # Quick estimation based on file sizes and line counts from a sample + total_estimate = 0 + for log_file in log_files[:10]: # Sample first 10 files to avoid performance impact + try: + # Quick line count estimation + with open(log_file, "rb") as f: + line_count = sum(1 for _ in f) + total_estimate += line_count + except Exception: + # If we can't read a file, estimate based on file size + try: + file_size = log_file.stat().st_size + # Rough estimate: average log line is ~200 bytes + estimated_lines = file_size // 200 + total_estimate += estimated_lines + except Exception: + continue + + # If we processed fewer than all files, extrapolate + if len(log_files) > 10: + extrapolation_factor = len(log_files) / 10 + total_estimate = int(total_estimate * extrapolation_factor) + + # Return formatted string + if total_estimate > 1000000: + return f"{total_estimate // 1000000:.1f}M" + elif total_estimate > 1000: + return f"{total_estimate // 1000:.1f}K" + else: + return str(total_estimate) + + except Exception as e: + self.logger.warning(f"Error estimating total log count: {e}") + return "Unknown" diff --git a/webhook_server/web/static/css/log_viewer.css b/webhook_server/web/static/css/log_viewer.css new file mode 100644 index 00000000..d596c7d8 --- /dev/null +++ b/webhook_server/web/static/css/log_viewer.css @@ -0,0 +1,411 @@ +:root { + /* Light theme variables */ + --bg-color: #f5f5f5; + --container-bg: #ffffff; + --text-color: #333333; + --border-color: #dddddd; + --input-bg: #ffffff; + --input-border: #dddddd; + --button-bg: #007bff; + --button-hover: #0056b3; + --status-connected-bg: #d4edda; + --status-connected-text: #155724; + --status-connected-border: #c3e6cb; + --status-disconnected-bg: #f8d7da; + --status-disconnected-text: #721c24; + --status-disconnected-border: #f5c6cb; + --log-entry-border: #eeeeee; + --log-info-bg: #d4f8d4; + --log-error-bg: #ffd6d6; + --log-warning-bg: #fff3cd; + --log-debug-bg: #f8f9fa; + --log-step-bg: #e3f2fd; + --log-success-bg: #d1f2d1; + --tag-bg: #e9ecef; + --timestamp-color: #666666; +} + +[data-theme="dark"] { + /* Dark theme variables */ + --bg-color: #1a1a1a; + --container-bg: #2d2d2d; + --text-color: #e0e0e0; + --border-color: #404040; + --input-bg: #3d3d3d; + --input-border: #555555; + --button-bg: #0d6efd; + --button-hover: #0b5ed7; + --status-connected-bg: #155724; + --status-connected-text: #d4edda; + --status-connected-border: #c3e6cb; + --status-disconnected-bg: #721c24; + --status-disconnected-text: #f8d7da; + --status-disconnected-border: #f5c6cb; + --log-entry-border: #404040; + --log-info-bg: #1e4a1e; + --log-error-bg: #5a1e1e; + --log-warning-bg: #5a4a1e; + --log-debug-bg: #2a2a2a; + --log-step-bg: #1a237e; + --log-success-bg: #1e4a1e; + --tag-bg: #4a4a4a; + --timestamp-color: #888888; +} + +body { + font-family: Arial, sans-serif; + margin: 0; + padding: 20px; + background-color: var(--bg-color); + color: var(--text-color); + transition: background-color 0.3s ease, color 0.3s ease; +} +.container { + max-width: 95vw; + margin: 0 auto; + background: var(--container-bg); + padding: 20px; + border-radius: 8px; + box-shadow: 0 2px 4px rgba(0,0,0,0.1); + transition: background-color 0.3s ease; +} +.header { + border-bottom: 1px solid var(--border-color); + padding-bottom: 20px; + margin-bottom: 20px; + display: flex; + justify-content: space-between; + align-items: center; +} +.header h1 { margin: 0; } +.theme-toggle { + background: var(--button-bg); + color: white; + border: none; + padding: 8px 16px; + border-radius: 4px; + cursor: pointer; + transition: background-color 0.3s ease; +} +.theme-toggle:hover { background: var(--button-hover); } +.filters { display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 10px; margin-bottom: 20px; } +.filter-group { display: flex; flex-direction: column; } +.filter-group label { font-weight: bold; margin-bottom: 3px; font-size: 14px; color: var(--text-color); } +.filter-group input, .filter-group select { + padding: 8px; + border: 1px solid var(--input-border); + border-radius: 4px; + background: var(--input-bg); + color: var(--text-color); + transition: background-color 0.3s ease, border-color 0.3s ease; +} +.log-entries { border: 1px solid var(--border-color); border-radius: 4px; min-height: 200px; } + +/* Loading skeleton styles */ +.loading-skeleton { + padding: 20px; +} +.skeleton-entry { + padding: 10px; + border-bottom: 1px solid var(--log-entry-border); + animation: pulse 1.5s ease-in-out infinite alternate; +} +.skeleton-line { + height: 14px; + margin: 4px 0; + border-radius: 3px; + background: linear-gradient(90deg, var(--border-color) 25%, var(--input-bg) 50%, var(--border-color) 75%); + background-size: 200% 100%; + animation: shimmer 1.5s infinite; +} +.skeleton-timestamp { width: 20%; } +.skeleton-level { width: 10%; } +.skeleton-message { width: 60%; } +.skeleton-meta { width: 30%; } +.loading-text { + text-align: center; + color: var(--timestamp-color); + font-style: italic; + margin-top: 20px; +} + +/* Error message styles */ +.error-message { + padding: 20px; + text-align: center; + color: var(--status-disconnected-text); + background: var(--status-disconnected-bg); + border: 1px solid var(--status-disconnected-border); + border-radius: 4px; + margin: 20px; +} +.error-icon { + font-size: 24px; + display: block; + margin-bottom: 10px; +} +.retry-btn { + background: var(--button-bg); + color: white; + border: none; + padding: 8px 16px; + border-radius: 4px; + cursor: pointer; + margin-top: 10px; +} +.retry-btn:hover { + background: var(--button-hover); +} + +/* Direct rendering optimizations - no virtual scrolling */ +.log-entries { + contain: layout style paint; +} + +/* Animations */ +@keyframes pulse { + 0% { opacity: 1; } + 100% { opacity: 0.6; } +} +@keyframes shimmer { + 0% { background-position: -200% 0; } + 100% { background-position: 200% 0; } +} + +/* Timeline styles */ +.timeline-section { + margin: 20px 0; + padding: 15px; + background: var(--container-bg); + border: 1px solid var(--border-color); + border-radius: 4px; + display: none; /* Hidden by default */ +} + +.timeline-header { + display: flex; + justify-content: space-between; + align-items: center; + margin-bottom: 15px; + padding-bottom: 10px; + border-bottom: 1px solid var(--border-color); + cursor: pointer; + user-select: none; +} + +.timeline-header:hover { + background-color: var(--log-entry-border); + border-radius: 4px; + margin: -5px; + padding: 5px 5px 15px 5px; +} + +.timeline-toggle { + background: var(--button-bg); + color: white; + border: none; + padding: 4px 8px; + border-radius: 3px; + cursor: pointer; + font-size: 12px; + transition: background-color 0.3s ease; +} + +.timeline-toggle:hover { + background: var(--button-hover); +} + +.timeline-content { + transition: all 0.3s ease; + overflow: hidden; +} + +.timeline-content.collapsed { + max-height: 0; + opacity: 0; + margin: 0; + padding: 0; +} + +.timeline-content.expanded { + max-height: 600px; + opacity: 1; +} + +.timeline-info { + font-size: 14px; + color: var(--timestamp-color); +} + +.timeline-container { + position: relative; + overflow-x: auto; + padding: 30px 0; + min-height: 200px; + max-height: 600px; /* Much larger container */ +} + +.timeline-svg { + width: 100%; + min-width: 1200px; /* Larger minimum width */ + height: auto; + min-height: 200px; + max-height: 550px; /* Much larger maximum height */ +} + +.timeline-step { + cursor: pointer; +} + +.timeline-step:hover .step-circle { + r: 16; /* Larger hover size */ + stroke-width: 4; +} + +.timeline-step:hover .step-label { + font-weight: bold; + font-size: 13px; /* Larger hover font */ +} + +.step-line { + stroke: var(--border-color); + stroke-width: 3; /* Thicker lines */ +} + +.step-circle { + r: 12; /* Larger default radius */ + stroke: #ffffff; + stroke-width: 3; /* Thicker stroke */ + transition: all 0.2s ease; + cursor: pointer; +} + +.step-circle:hover { + r: 16; /* Larger hover size */ + stroke-width: 4; +} + +.step-circle.success { + fill: #28a745; + stroke: #ffffff; +} + +.step-circle.failure { + fill: #dc3545; + stroke: #ffffff; +} + +.step-circle.info { + fill: #17a2b8; + stroke: #ffffff; +} + +.step-circle.progress { + fill: #007bff; + stroke: #ffffff; +} + +.step-label { + font-size: 12px; /* Larger labels */ + text-anchor: middle; + fill: var(--text-color); + transition: font-weight 0.2s ease; + pointer-events: none; + font-weight: 500; /* Semi-bold for better readability */ +} + +.step-time { + font-size: 11px; /* Larger time labels */ + text-anchor: middle; + fill: var(--timestamp-color); + pointer-events: none; + font-weight: 500; /* Semi-bold for better readability */ +} + +.timeline-tooltip { + position: absolute; + background: var(--container-bg); + border: 1px solid var(--border-color); + border-radius: 4px; + padding: 8px; + font-size: 12px; + box-shadow: 0 2px 8px rgba(0,0,0,0.1); + z-index: 1000; + pointer-events: none; + display: none; + max-width: 300px; +} +.log-entry { + padding: 10px; + border-bottom: 1px solid var(--log-entry-border); + font-family: monospace; + font-size: 14px; + transition: background-color 0.3s ease; +} +.log-entry:last-child { border-bottom: none; } +.log-entry.INFO { background-color: var(--log-info-bg); } +.log-entry.ERROR { background-color: var(--log-error-bg); } +.log-entry.WARNING { background-color: var(--log-warning-bg); } +.log-entry.DEBUG { background-color: var(--log-debug-bg); } +.log-entry.STEP { background-color: var(--log-step-bg); } +.log-entry.SUCCESS { background-color: var(--log-success-bg); } +.timestamp { color: var(--timestamp-color); } +.level { font-weight: bold; } +.message { margin-left: 10px; } +.hook-id, .pr-number, .repository, .user { + margin-left: 10px; + padding: 2px 6px; + background-color: var(--tag-bg); + border-radius: 3px; + font-size: 12px; + transition: background-color 0.3s ease; +} +.controls { margin-bottom: 20px; } +.btn { + padding: 10px 20px; + background-color: var(--button-bg); + color: white; + border: none; + border-radius: 4px; + cursor: pointer; + margin-right: 10px; + transition: background-color 0.3s ease; +} +.btn:hover { background-color: var(--button-hover); } +.status { padding: 10px; margin-bottom: 20px; border-radius: 4px; } +.status.connected { + background-color: var(--status-connected-bg); + color: var(--status-connected-text); + border: 1px solid var(--status-connected-border); +} +.status.disconnected { + background-color: var(--status-disconnected-bg); + color: var(--status-disconnected-text); + border: 1px solid var(--status-disconnected-border); +} + +.log-stats { + background: var(--container-bg); + border: 1px solid var(--border-color); + border-radius: 4px; + padding: 10px; + margin-bottom: 20px; + font-size: 14px; + color: var(--text-color); + display: none; +} + +.log-stats > div { + display: flex; + justify-content: space-between; + align-items: center; +} + +/* Responsive adjustments */ +@media (max-width: 768px) { + .filters { grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: 8px; } + .filter-group label { font-size: 13px; } + .filter-group input, .filter-group select { padding: 6px; font-size: 14px; } + .controls { display: flex; flex-wrap: wrap; gap: 8px; } + .btn { padding: 8px 16px; font-size: 14px; } +} diff --git a/webhook_server/web/static/js/log_viewer.js b/webhook_server/web/static/js/log_viewer.js new file mode 100644 index 00000000..fdcaaaed --- /dev/null +++ b/webhook_server/web/static/js/log_viewer.js @@ -0,0 +1,904 @@ +let ws = null; +let logEntries = []; + +function updateConnectionStatus(connected) { + const status = document.getElementById('connectionStatus'); + const statusText = document.getElementById('statusText'); + + if (connected) { + status.className = 'status connected'; + statusText.textContent = 'Connected - Real-time updates active'; + } else { + status.className = 'status disconnected'; + statusText.textContent = 'Disconnected - Real-time updates inactive'; + } +} + +function connectWebSocket() { + if (ws) { + ws.close(); + } + + const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:'; + + // Build WebSocket URL with current filter parameters + const filters = new URLSearchParams(); + const hookId = document.getElementById('hookIdFilter').value.trim(); + const prNumber = document.getElementById('prNumberFilter').value.trim(); + const repository = document.getElementById('repositoryFilter').value.trim(); + const user = document.getElementById('userFilter').value.trim(); + const level = document.getElementById('levelFilter').value; + const search = document.getElementById('searchFilter').value.trim(); + + if (hookId) filters.append('hook_id', hookId); + if (prNumber) filters.append('pr_number', prNumber); + if (repository) filters.append('repository', repository); + if (user) filters.append('github_user', user); + if (level) filters.append('level', level); + if (search) filters.append('search', search); + + const wsUrl = `${protocol}//${window.location.host}/logs/ws${filters.toString() ? '?' + filters.toString() : ''}`; + + ws = new WebSocket(wsUrl); + + ws.onopen = function() { + updateConnectionStatus(true); + console.log('WebSocket connected'); + }; + + ws.onmessage = function(event) { + const logEntry = JSON.parse(event.data); + addLogEntry(logEntry); + }; + + ws.onclose = function() { + updateConnectionStatus(false); + console.log('WebSocket disconnected'); + }; + + ws.onerror = function(error) { + updateConnectionStatus(false); + console.error('WebSocket error:', error); + }; +} + +function disconnectWebSocket() { + if (ws) { + ws.close(); + ws = null; + } + updateConnectionStatus(false); +} + +// Removed virtual scrolling to prevent scrollbar flashing +// All rendering now uses direct DOM manipulation for stable UI + +// Helper function to apply memory bounding to logEntries array +function applyMemoryBounding() { + const maxEntries = parseInt(document.getElementById('limitFilter').value); + if (logEntries.length > maxEntries) { + // Remove oldest entries to keep array size bounded + logEntries = logEntries.slice(0, maxEntries); + } +} + +function addLogEntry(entry) { + logEntries.unshift(entry); + + // Apply memory bounding using centralized helper + applyMemoryBounding(); + + clearFilterCache(); // Clear cache when entries change + renderLogEntriesOptimized(); + + // Update displayed count for real-time entries + updateDisplayedCount(); +} + +function updateDisplayedCount() { + const displayedCount = document.getElementById('displayedCount'); + const filteredEntries = filterLogEntries(logEntries); + displayedCount.textContent = filteredEntries.length; +} + +function renderLogEntriesOptimized() { + const container = document.getElementById('logEntries'); + const filteredEntries = filterLogEntries(logEntries); + + // Always use direct rendering to prevent any scrollbar flashing + // Completely disabled virtual scrolling to ensure stable UI + renderLogEntriesDirect(container, filteredEntries); +} + +function renderLogEntriesDirect(container, entries) { + // Use DocumentFragment for efficient DOM manipulation to minimize reflows + const fragment = document.createDocumentFragment(); + + entries.forEach(entry => { + const entryElement = createLogEntryElement(entry); + fragment.appendChild(entryElement); + }); + + // Clear and append in one operation to minimize visual flashing + // Use replaceChildren for better performance and less flashing + container.replaceChildren(fragment); + + // Debug: Log how many entries were actually rendered + console.log(`Rendered ${entries.length} entries directly to DOM`); +} + +// Virtual scrolling removed to prevent scrollbar flashing +// All rendering now uses direct DOM manipulation only + +function createLogEntryElement(entry) { + const div = document.createElement('div'); + + // Whitelist of allowed log levels to prevent class-name injection + const allowedLevels = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'STEP', 'SUCCESS']; + const safeLevel = allowedLevels.includes(entry.level) ? entry.level : 'INFO'; // Default fallback + + div.className = `log-entry ${safeLevel}`; + + // Use efficient string template + div.innerHTML = ` + ${new Date(entry.timestamp).toLocaleString()} + [${entry.level}] + ${escapeHtml(entry.message)} + ${entry.hook_id ? `[Hook: ${escapeHtml(entry.hook_id)}]` : ''} + ${entry.pr_number ? `[PR: #${entry.pr_number}]` : ''} + ${entry.repository ? `[${escapeHtml(entry.repository)}]` : ''} + ${entry.github_user ? `[User: ${escapeHtml(entry.github_user)}]` : ''} + `; + + return div; +} + +function escapeHtml(text) { + const div = document.createElement('div'); + div.textContent = text; + return div.innerHTML; +} + +// Alias for backward compatibility +function renderLogEntries() { + renderLogEntriesOptimized(); +} + +function renderLogEntriesDirectly(entries) { + const container = document.getElementById('logEntries'); + + // Always use direct rendering for backend-filtered data to ensure all entries show + renderLogEntriesDirect(container, entries); +} + +// Optimized filtering with caching and early exit +let lastFilterHash = ''; +let cachedFilteredEntries = []; + +function filterLogEntries(entries) { + const hookId = document.getElementById('hookIdFilter').value.trim(); + const prNumber = document.getElementById('prNumberFilter').value.trim(); + const repository = document.getElementById('repositoryFilter').value.trim(); + const user = document.getElementById('userFilter').value.trim(); + const level = document.getElementById('levelFilter').value; + const search = document.getElementById('searchFilter').value.trim().toLowerCase(); + + // Create hash of current filters for caching + const filterHash = `${hookId}-${prNumber}-${repository}-${user}-${level}-${search}-${entries.length}`; + + // Return cached result if filters haven't changed + if (filterHash === lastFilterHash && cachedFilteredEntries.length > 0) { + return cachedFilteredEntries; + } + + // Pre-compile search terms for better performance + const searchTerms = search ? search.split(' ').filter(term => term.length > 0) : []; + const prNumberInt = prNumber ? parseInt(prNumber) : null; + + // Use optimized filtering with early exits + const filtered = entries.filter(entry => { + // Exact matches first (fastest) + if (hookId && entry.hook_id !== hookId) return false; + if (prNumberInt && entry.pr_number !== prNumberInt) return false; + if (repository && entry.repository !== repository) return false; + if (user && entry.github_user !== user) return false; + if (level && entry.level !== level) return false; + + // Text search last (slowest) + if (searchTerms.length > 0) { + const messageText = entry.message.toLowerCase(); + return searchTerms.every(term => messageText.includes(term)); + } + + return true; + }); + + // Cache the result + lastFilterHash = filterHash; + cachedFilteredEntries = filtered; + + return filtered; +} + +// Clear filter cache when entries change +function clearFilterCache() { + lastFilterHash = ''; + cachedFilteredEntries = []; +} + +async function loadHistoricalLogs() { + try { + // Show loading skeleton + showLoadingSkeleton(); + + // Build API URL with current filter parameters + const filters = new URLSearchParams(); + const hookId = document.getElementById('hookIdFilter').value.trim(); + const prNumber = document.getElementById('prNumberFilter').value.trim(); + const repository = document.getElementById('repositoryFilter').value.trim(); + const user = document.getElementById('userFilter').value.trim(); + const level = document.getElementById('levelFilter').value; + const search = document.getElementById('searchFilter').value.trim(); + const limit = document.getElementById('limitFilter').value; + + // Use user-configured limit + filters.append('limit', limit); + if (hookId) filters.append('hook_id', hookId); + if (prNumber) filters.append('pr_number', prNumber); + if (repository) filters.append('repository', repository); + if (user) filters.append('github_user', user); + if (level) filters.append('level', level); + if (search) filters.append('search', search); + + const response = await fetch(`/logs/api/entries?${filters.toString()}`); + + // Check HTTP status before parsing JSON + if (!response.ok) { + let errorMessage = `HTTP ${response.status}: ${response.statusText}`; + try { + // Try to parse error message from response body + const errorData = await response.json(); + if (errorData.detail || errorData.message || errorData.error) { + errorMessage = errorData.detail || errorData.message || errorData.error; + } + } catch (parseError) { + // If JSON parsing fails, use the status text + } + throw new Error(errorMessage); + } + + const data = await response.json(); + + // Update statistics + updateLogStatistics(data); + + // Progressive loading for large datasets + if (data.entries.length > 200) { + await loadEntriesProgressivelyDirect(data.entries); + } else { + logEntries = data.entries; + // Apply memory bounding after loading entries + applyMemoryBounding(); + clearFilterCache(); // Clear cache when loading new entries + // Data is already filtered by the backend, render directly without frontend filtering + renderLogEntriesDirectly(logEntries); + } + + hideLoadingSkeleton(); + } catch (error) { + console.error('Error loading historical logs:', error); + hideLoadingSkeleton(); + showErrorMessage('Failed to load log entries'); + } +} + +async function loadEntriesProgressively(entries) { + const chunkSize = 50; + logEntries = []; + clearFilterCache(); // Clear cache when loading new entries + + for (let i = 0; i < entries.length; i += chunkSize) { + const chunk = entries.slice(i, i + chunkSize); + logEntries.push(...chunk); + // Apply memory bounding after each chunk to prevent unbounded growth + applyMemoryBounding(); + clearFilterCache(); // Clear cache for each chunk + renderLogEntries(); + + // Add small delay to prevent UI blocking + if (i + chunkSize < entries.length) { + await new Promise(resolve => setTimeout(resolve, 10)); + } + } +} + +async function loadEntriesProgressivelyDirect(entries) { + // For backend-filtered data, just render all entries at once + // Progressive loading isn't needed since data is already filtered and limited + logEntries = entries; + // Apply memory bounding after direct assignment + applyMemoryBounding(); + renderLogEntriesDirectly(logEntries); + console.log(`Loaded ${entries.length} backend-filtered entries`); +} + +function showLoadingSkeleton() { + const container = document.getElementById('logEntries'); + container.innerHTML = ` +
+ ${createSkeletonEntry()} + ${createSkeletonEntry()} + ${createSkeletonEntry()} + ${createSkeletonEntry()} + ${createSkeletonEntry()} +
Loading log entries...
+
+ `; +} + +function createSkeletonEntry() { + return ` +
+
+
+
+
+
+ `; +} + +function hideLoadingSkeleton() { + const skeleton = document.querySelector('.loading-skeleton'); + if (skeleton) { + skeleton.remove(); + } +} + +function showErrorMessage(message) { + const container = document.getElementById('logEntries'); + container.innerHTML = ` +
+ ⚠️ + ${message} + +
+ `; + + // Add event listener to the dynamically created retry button + const retryBtn = document.getElementById('retryBtn'); + if (retryBtn) { + retryBtn.addEventListener('click', loadHistoricalLogs); + } +} + +function updateLogStatistics(data) { + const statsPanel = document.getElementById('logStats'); + const displayedCount = document.getElementById('displayedCount'); + const totalCount = document.getElementById('totalCount'); + const processedCount = document.getElementById('processedCount'); + + // Update counts from API response + displayedCount.textContent = data.entries ? data.entries.length : 0; + processedCount.textContent = data.entries_processed || '0'; + + // Use the total log count estimate for better user information + totalCount.textContent = data.total_log_count_estimate || 'Unknown'; + + // Show the statistics panel + statsPanel.style.display = 'block'; + + // Add indicator for partial scans + if (data.is_partial_scan) { + processedCount.innerHTML = `${data.entries_processed} (partial scan)`; + } +} + +function clearLogs() { + logEntries = []; + clearFilterCache(); // Clear cache when clearing entries + + // Clear the container directly to avoid any scrollbar flashing + const container = document.getElementById('logEntries'); + container.replaceChildren(); // More efficient than innerHTML = '' + + // Hide stats panel when no entries + document.getElementById('logStats').style.display = 'none'; +} + +function exportLogs(format) { + const filters = new URLSearchParams(); + const hookId = document.getElementById('hookIdFilter').value.trim(); + const prNumber = document.getElementById('prNumberFilter').value.trim(); + const repository = document.getElementById('repositoryFilter').value.trim(); + const user = document.getElementById('userFilter').value.trim(); + const level = document.getElementById('levelFilter').value; + const search = document.getElementById('searchFilter').value.trim(); + const limit = document.getElementById('limitFilter').value; + + if (hookId) filters.append('hook_id', hookId); + if (prNumber) filters.append('pr_number', prNumber); + if (repository) filters.append('repository', repository); + if (user) filters.append('github_user', user); + if (level) filters.append('level', level); + if (search) filters.append('search', search); + filters.append('limit', limit); + filters.append('format', format); + + const url = `/logs/api/export?${filters.toString()}`; + window.open(url, '_blank'); +} + +function applyFilters() { + // Reload historical logs with new filters + loadHistoricalLogs(); + + // Reconnect WebSocket with new filters if currently connected + if (ws && ws.readyState === WebSocket.OPEN) { + connectWebSocket(); + } +} + +// Set up filter event handlers with debouncing +let filterTimeout; +function debounceFilter() { + // Clear only filter cache, not entry cache + lastFilterHash = ''; + + // Immediate client-side filtering for fast feedback + renderLogEntries(); + + // Debounced server-side filtering for accuracy + clearTimeout(filterTimeout); + filterTimeout = setTimeout(() => { + applyFilters(); // Server-side filter for accurate results + }, 300); // Slightly longer delay for better UX +} + +function clearFilters() { + document.getElementById('hookIdFilter').value = ''; + document.getElementById('prNumberFilter').value = ''; + document.getElementById('repositoryFilter').value = ''; + document.getElementById('userFilter').value = ''; + document.getElementById('levelFilter').value = ''; + document.getElementById('searchFilter').value = ''; + document.getElementById('limitFilter').value = '1000'; // Reset to default + + // Reload data with cleared filters + applyFilters(); +} + +document.getElementById('hookIdFilter').addEventListener('input', debounceFilter); +document.getElementById('prNumberFilter').addEventListener('input', debounceFilter); +document.getElementById('repositoryFilter').addEventListener('input', debounceFilter); +document.getElementById('userFilter').addEventListener('input', debounceFilter); +document.getElementById('levelFilter').addEventListener('change', debounceFilter); +document.getElementById('searchFilter').addEventListener('input', debounceFilter); +document.getElementById('limitFilter').addEventListener('change', debounceFilter); + +// Theme management +function toggleTheme() { + const currentTheme = document.documentElement.getAttribute('data-theme'); + const newTheme = currentTheme === 'dark' ? 'light' : 'dark'; + + document.documentElement.setAttribute('data-theme', newTheme); + + // Update theme toggle button icon and accessibility attributes + const themeToggle = document.querySelector('.theme-toggle'); + themeToggle.textContent = newTheme === 'dark' ? '☀️' : '🌙'; + themeToggle.setAttribute('aria-label', newTheme === 'dark' ? 'Switch to light theme' : 'Switch to dark theme'); + themeToggle.setAttribute('title', newTheme === 'dark' ? 'Switch to light theme' : 'Switch to dark theme'); + + // Store theme preference in localStorage + localStorage.setItem('log-viewer-theme', newTheme); +} + +// Initialize theme from localStorage or default to light +function initializeTheme() { + const savedTheme = localStorage.getItem('log-viewer-theme') || 'light'; + document.documentElement.setAttribute('data-theme', savedTheme); + + // Update theme toggle button icon and accessibility attributes + const themeToggle = document.querySelector('.theme-toggle'); + themeToggle.textContent = savedTheme === 'dark' ? '☀️' : '🌙'; + themeToggle.setAttribute('aria-label', savedTheme === 'dark' ? 'Switch to light theme' : 'Switch to dark theme'); + themeToggle.setAttribute('title', savedTheme === 'dark' ? 'Switch to light theme' : 'Switch to dark theme'); +} + +// Initialize theme on page load +initializeTheme(); + +// Initialize timeline collapse state +initializeTimelineState(); + +// Initialize connection status +updateConnectionStatus(false); + +// Initialize event listeners when DOM is ready +function initializeEventListeners() { + // Theme toggle button + const themeToggleBtn = document.getElementById('themeToggleBtn'); + if (themeToggleBtn) { + themeToggleBtn.addEventListener('click', toggleTheme); + } + + // Control buttons + const connectBtn = document.getElementById('connectBtn'); + if (connectBtn) { + connectBtn.addEventListener('click', connectWebSocket); + } + + const disconnectBtn = document.getElementById('disconnectBtn'); + if (disconnectBtn) { + disconnectBtn.addEventListener('click', disconnectWebSocket); + } + + const refreshBtn = document.getElementById('refreshBtn'); + if (refreshBtn) { + refreshBtn.addEventListener('click', loadHistoricalLogs); + } + + const clearFiltersBtn = document.getElementById('clearFiltersBtn'); + if (clearFiltersBtn) { + clearFiltersBtn.addEventListener('click', clearFilters); + } + + const clearLogsBtn = document.getElementById('clearLogsBtn'); + if (clearLogsBtn) { + clearLogsBtn.addEventListener('click', clearLogs); + } + + const exportBtn = document.getElementById('exportBtn'); + if (exportBtn) { + exportBtn.addEventListener('click', () => exportLogs('json')); + } + + // Timeline header and toggle button + const timelineHeader = document.getElementById('timelineHeader'); + if (timelineHeader) { + timelineHeader.addEventListener('click', toggleTimeline); + } + + const timelineToggle = document.getElementById('timelineToggle'); + if (timelineToggle) { + timelineToggle.addEventListener('click', (event) => { + event.stopPropagation(); + toggleTimeline(); + }); + } +} + +// Initialize event listeners +initializeEventListeners(); + +// Load initial data +loadHistoricalLogs(); + +// Timeline functionality +let currentTimelineData = null; + +function showTimeline(hookId) { + if (!hookId) { + hideTimeline(); + return; + } + + + // Fetch workflow steps data + fetch(`/logs/api/workflow-steps/${hookId}`) + .then(response => { + if (!response.ok) { + if (response.status === 404) { + hideTimeline(); + return; + } + throw new Error('Failed to fetch workflow steps'); + } + return response.json(); + }) + .then(data => { + currentTimelineData = data; + renderTimeline(data); + document.getElementById('timelineSection').style.display = 'block'; + + // Ensure the correct collapse state is maintained when showing timeline + initializeTimelineState(); + }) + .catch(error => { + hideTimeline(); + }); +} + +function hideTimeline() { + document.getElementById('timelineSection').style.display = 'none'; + currentTimelineData = null; +} + +function toggleTimeline() { + const content = document.getElementById('timelineContent'); + const toggle = document.getElementById('timelineToggle'); + + if (content.classList.contains('expanded')) { + // Collapse + content.classList.remove('expanded'); + content.classList.add('collapsed'); + toggle.textContent = '▶ Expand'; + + // Store collapse state in localStorage + localStorage.setItem('timeline-collapsed', 'true'); + } else { + // Expand + content.classList.remove('collapsed'); + content.classList.add('expanded'); + toggle.textContent = '▼ Collapse'; + + // Store expand state in localStorage + localStorage.setItem('timeline-collapsed', 'false'); + } +} + +function initializeTimelineState() { + // Initialize timeline collapse state from localStorage - default to collapsed + const timelineState = localStorage.getItem('timeline-collapsed'); + const isCollapsed = timelineState === null ? true : timelineState === 'true'; // Default collapsed if no preference set + const content = document.getElementById('timelineContent'); + const toggle = document.getElementById('timelineToggle'); + + if (isCollapsed) { + content.classList.remove('expanded'); + content.classList.add('collapsed'); + toggle.textContent = '▶ Expand'; + } else { + content.classList.remove('collapsed'); + content.classList.add('expanded'); + toggle.textContent = '▼ Collapse'; + } +} + +function updateTimelineInfo(data) { + const info = document.getElementById('timelineInfo'); + const duration = data.total_duration_ms > 0 ? `${(data.total_duration_ms / 1000).toFixed(2)}s` : '< 1s'; + info.innerHTML = ` +
Hook ID: ${data.hook_id}
+
Steps: ${data.step_count}
+
Duration: ${duration}
+ `; +} + +function renderEmptyTimeline() { + const svg = document.getElementById('timelineSvg'); + svg.innerHTML = 'No workflow steps found'; +} + +function renderTimelineVisualization(layout, data) { + const svg = document.getElementById('timelineSvg'); + + // Clear existing content + svg.innerHTML = ''; + + // SVG dimensions - much larger and adaptive + const width = Math.max(1400, layout.totalWidth + 200); + const height = layout.totalHeight + 150; + const margin = { left: 75, right: 75, top: 75, bottom: 75 }; + + // Update SVG size + svg.setAttribute('width', width); + svg.setAttribute('height', height); + + // Draw timeline lines and steps + layout.lines.forEach((line, lineIndex) => { + const lineY = margin.top + (lineIndex * layout.lineHeight) + layout.lineHeight / 2; + + // Draw horizontal timeline line for this row + if (line.steps.length > 0) { + const lineElement = document.createElementNS('http://www.w3.org/2000/svg', 'line'); + lineElement.setAttribute('class', 'step-line'); + lineElement.setAttribute('x1', margin.left); + lineElement.setAttribute('y1', lineY); + lineElement.setAttribute('x2', margin.left + layout.lineWidth); + lineElement.setAttribute('y2', lineY); + svg.appendChild(lineElement); + } + + // Draw steps for this line + line.steps.forEach((step, stepIndex) => { + const stepX = margin.left + (stepIndex * layout.stepSpacing) + layout.stepSpacing / 2; + + const group = document.createElementNS('http://www.w3.org/2000/svg', 'g'); + group.setAttribute('class', 'timeline-step'); + group.setAttribute('data-step-index', step.originalIndex); + + // Step circle - larger + const circle = document.createElementNS('http://www.w3.org/2000/svg', 'circle'); + circle.setAttribute('class', `step-circle ${getStepType(step.message)}`); + circle.setAttribute('cx', stepX); + circle.setAttribute('cy', lineY); + circle.setAttribute('r', 12); // Larger circle + svg.appendChild(circle); + group.appendChild(circle); + + // Step label - with multi-line text wrapping + const labelLines = wrapTextToLines(step.message, 25); // Longer text allowed + labelLines.forEach((line, lineIndex) => { + const label = document.createElementNS('http://www.w3.org/2000/svg', 'text'); + label.setAttribute('class', 'step-label'); + label.setAttribute('x', stepX); + label.setAttribute('y', lineY - 35 + (lineIndex * 14)); // Multi-line spacing + label.setAttribute('text-anchor', 'middle'); + label.setAttribute('font-size', '12'); // Larger font + label.textContent = line; + svg.appendChild(label); + group.appendChild(label); + }); + + // Time label - larger and positioned better + const timeLabel = document.createElementNS('http://www.w3.org/2000/svg', 'text'); + timeLabel.setAttribute('class', 'step-time'); + timeLabel.setAttribute('x', stepX); + timeLabel.setAttribute('y', lineY + 35); + timeLabel.setAttribute('text-anchor', 'middle'); + timeLabel.setAttribute('font-size', '11'); // Larger time font + timeLabel.textContent = `+${(step.relative_time_ms / 1000).toFixed(1)}s`; + svg.appendChild(timeLabel); + group.appendChild(timeLabel); + + // Step index number - larger and better positioned + const indexLabel = document.createElementNS('http://www.w3.org/2000/svg', 'text'); + indexLabel.setAttribute('class', 'step-index'); + indexLabel.setAttribute('x', stepX); + indexLabel.setAttribute('y', lineY + 5); + indexLabel.setAttribute('text-anchor', 'middle'); + indexLabel.setAttribute('font-size', '13'); // Larger index font + indexLabel.setAttribute('font-weight', 'bold'); + indexLabel.setAttribute('fill', 'white'); // White text for better contrast + indexLabel.textContent = (step.originalIndex + 1).toString(); + svg.appendChild(indexLabel); + group.appendChild(indexLabel); + + // Add hover events + group.addEventListener('mouseenter', (e) => showTooltip(e, step)); + group.addEventListener('mouseleave', hideTooltip); + group.addEventListener('click', () => filterByStep(step)); + + svg.appendChild(group); + }); + }); +} + +function renderTimeline(data) { + // Update timeline information + updateTimelineInfo(data); + + // Handle empty state + if (data.steps.length === 0) { + renderEmptyTimeline(); + return; + } + + // Calculate layout for multi-line timeline + const layout = calculateMultiLineLayout(data.steps, data.total_duration_ms); + + // Render the timeline visualization + renderTimelineVisualization(layout, data); +} + +function getStepType(message) { + if (message.includes('completed successfully') || message.includes('success')) { + return 'success'; + } else if (message.includes('failed') || message.includes('error')) { + return 'failure'; + } else if (message.includes('Starting') || message.includes('Executing')) { + return 'progress'; + } else { + return 'info'; + } +} + +function truncateText(text, maxLength) { + return text.length > maxLength ? text.substring(0, maxLength) + '...' : text; +} + +function calculateMultiLineLayout(steps, totalDuration) { + // Layout configuration - much larger for better readability + const stepsPerLine = 6; // Fewer steps per line for more space + const stepSpacing = 200; // Much larger horizontal space between steps + const lineHeight = 120; // Much larger vertical space between lines + const lineWidth = stepsPerLine * stepSpacing; + + // Organize steps into lines + const lines = []; + for (let i = 0; i < steps.length; i += stepsPerLine) { + const lineSteps = steps.slice(i, i + stepsPerLine).map((step, index) => ({ + ...step, + originalIndex: i + index + })); + lines.push({ steps: lineSteps }); + } + + return { + lines, + lineHeight, + lineWidth, + stepSpacing, + totalWidth: lineWidth, + totalHeight: lines.length * lineHeight + }; +} + + +function wrapTextToLines(text, maxCharacters) { + // Smart text wrapping for timeline labels + const words = text.split(' '); + const lines = []; + let currentLine = ''; + + for (const word of words) { + const testLine = currentLine ? `${currentLine} ${word}` : word; + if (testLine.length <= maxCharacters) { + currentLine = testLine; + } else { + if (currentLine) { + lines.push(currentLine); + currentLine = word; + } else { + // Single word is too long, truncate it + lines.push(word.substring(0, maxCharacters - 3) + '...'); + currentLine = ''; + } + } + } + + if (currentLine) { + lines.push(currentLine); + } + + // Return max 2 lines to prevent overcrowding + return lines.slice(0, 2); +} + +function showTooltip(event, step) { + const tooltip = document.getElementById('timelineTooltip'); + const timeFromStart = `+${(step.relative_time_ms / 1000).toFixed(2)}s`; + + tooltip.innerHTML = ` +
Step: ${step.message}
+
Time: ${timeFromStart}
+
Timestamp: ${new Date(step.timestamp).toLocaleTimeString()}
+ ${step.pr_number ? `
PR: #${step.pr_number}
` : ''} +
Click to filter logs by this step
+ `; + + const rect = event.target.getBoundingClientRect(); + const containerRect = document.getElementById('timelineSection').getBoundingClientRect(); + + tooltip.style.left = (rect.left - containerRect.left + rect.width / 2) + 'px'; + tooltip.style.top = (rect.top - containerRect.top - tooltip.offsetHeight - 10) + 'px'; + tooltip.style.display = 'block'; +} + +function hideTooltip() { + document.getElementById('timelineTooltip').style.display = 'none'; +} + +function filterByStep(step) { + // Set search filter to find this specific step message + document.getElementById('searchFilter').value = step.message.substring(0, 30); + debounceFilter(); +} + +// Auto-show timeline when hook ID filter is applied +function checkForTimelineDisplay() { + const hookId = document.getElementById('hookIdFilter').value.trim(); + if (hookId) { + showTimeline(hookId); + } else { + hideTimeline(); + } +} + +// Add timeline check to hook ID filter specifically +document.getElementById('hookIdFilter').addEventListener('input', () => { + setTimeout(checkForTimelineDisplay, 300); // Small delay to let the value settle +}); + +// Also check on initial load +setTimeout(checkForTimelineDisplay, 1000); diff --git a/webhook_server/web/templates/log_viewer.html b/webhook_server/web/templates/log_viewer.html new file mode 100644 index 00000000..4fbcab23 --- /dev/null +++ b/webhook_server/web/templates/log_viewer.html @@ -0,0 +1,117 @@ + + + + + + GitHub Webhook Server - Log Viewer + + + +
+
+
+

GitHub Webhook Server - Log Viewer

+

Real-time log monitoring and filtering for webhook events

+
+ +
+ +
+ Connecting... +
+ +
+
+ Displayed: 0 entries + Total Available: 0 entries + Processed: 0 entries +
+
+ +
+ + + + + + +
+ +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+
+

Hook ID Flow Timeline

+ +
+
+ +
+
+ +
+ +
+ +
+
+ + + +