Docs/operations manual #117
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This improves efficiency when processing I/O-bound workloads by: - Adding CONSERVER_WORKERS env var to spawn multiple worker processes that independently consume from Redis queues (Phase 1) - Adding CONSERVER_PARALLEL_STORAGE env var to enable concurrent writes to multiple storage backends using ThreadPoolExecutor (Phase 2) Features: - Workers are automatically restarted if they die unexpectedly - Graceful shutdown signals all workers to complete current work - Redis BLPOP provides atomic distribution of vCons to workers - Single-worker mode (default) preserves original behavior
Tests cover: - Worker count configuration from environment variables - Parallel storage enabled/disabled settings - ThreadPoolExecutor behavior with multiple storage backends - Parallel vs sequential timing comparison - Error handling when individual storage backends fail - Signal handling for graceful shutdown - Worker naming conventions and process management
Memory optimization improvements for multi-worker mode: - Add CONSERVER_START_METHOD env var to choose fork/spawn/forkserver - "fork": Copy-on-write memory sharing (Unix), lower memory - "spawn": Fresh interpreter per worker, safer but higher memory - "forkserver": Hybrid approach - Default: platform default (fork on Unix, spawn on Windows) - Move module imports from main() to worker_loop() - Defers heavy imports until worker starts - With 'spawn', each worker only loads what it needs - Reduces base memory footprint significantly - Add tests for start method configuration - Update documentation in example_config.yml
Document the new multi-worker and parallel storage capabilities: - 02_CORE_FUNCTIONALITY.md: Add multi-worker architecture and parallel storage sections, update processing flow description - 06_DEPLOYMENT_OPERATIONS.md: Add CONSERVER_WORKERS, CONSERVER_PARALLEL_STORAGE, CONSERVER_START_METHOD env vars, update scaling strategies with memory optimization guidance - 09_ARCHITECTURE_DESIGN.md: Update architecture diagram to show worker processes and parallel storage, add detailed scalability design section - 10_ROADMAP_FUTURE.md: Update to v1.1, mark parallel processing features as completed - README.md: Add parallel processing to core capabilities, include worker configuration example
Create MkDocs-based documentation covering: - Getting Started: introduction, requirements, quick start - Installation: Docker, automated script, manual, Kubernetes, upgrading - Configuration: environment variables, YAML config, chains, auth, workers - Operations: API reference, monitoring, logging, troubleshooting, DLQ, backup - Extending: creating links, storage adapters, tracers, dynamic modules - Reference: link docs, storage adapter docs, CLI reference Documentation is structured for hosting on GitHub Pages or ReadTheDocs.
- Change color palette to blue grey with light blue accents - Add Inter font for text and JetBrains Mono for code - Enable additional navigation features (instant, tracking) - Add header autohide and code annotations
|
You have run out of free Bugbot PR reviews for this billing cycle. This will reset on January 28. To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request adds comprehensive documentation for vCon Server configuration and extensibility. It introduces three new guides: a general configuration overview, an in-depth authentication guide, and a dynamic modules guide. These documents provide detailed instructions, best practices, troubleshooting steps, and visual diagrams to help users configure, secure, and extend the vCon Server.
Configuration Documentation
docs/configuration/index.mdwith a full configuration guide, including environment variables, YAML structure, common setups, and troubleshooting.Authentication Documentation
docs/configuration/authentication.mddetailing global API token setup, ingress-specific authentication, token management, security best practices, and troubleshooting. Includes mermaid diagrams for authentication flows.Dynamic Modules Documentation
docs/extending/dynamic-modules.mddescribing how to dynamically install and use modules from PyPI or GitHub, including configuration examples, module creation, version pinning, troubleshooting, and security considerations.Best Practices and Troubleshooting
Visual Aids