feat: Add MegatronBackend
#545
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced MegatronBackend for managing model services and training processes. - Added MegatronService for handling training jobs and OpenAI server interactions. - Created yes-no-maybe-megatron.py for orchestrating model training with prompts. - Included setup script for environment configuration and dependencies. - Implemented training logic in train.py to facilitate distributed training with LoRA support.
- Reformatted command construction for better readability. - Updated optimizer state path assignment for clarity. - Rearranged import statements for consistency and organization.
- Added a reset_lora_parameters method to initialize LoRA weights with Kaiming and zero initialization. - Improved assertion messages for clarity in various sections of the LoRA class. - Refactored loading logic to utilize the new reset method for better parameter handling. - Enhanced code readability by restructuring assertions and method calls.
- Restructured assertions in the LoRA class for better clarity and consistency. - Enhanced error messages to provide more informative feedback. - Improved code readability by consolidating assertion statements.
- Included the Docker image ID for PyTorch version 2.9.0 with CUDA 12.8 and cuDNN 9 in skypilot-config.yaml. - This addition enhances the configuration for better compatibility with specific model training requirements.
- Added logic to create a custom sudo command if not available, ensuring script compatibility. - Implemented checks for essential packages (git, curl, tmux) and automated their installation if missing. - Updated the installation process for 'uv' to use a script from the official source, improving reliability.
…nt handling and LoRA configuration - Updated LocalBackend to copy current checkpoints instead of renaming, ensuring data integrity during training steps. - Refactored MegatronService to ensure identity LoRA creation and configuration management, enhancing model training reliability. - Improved offloading and reloading of model parameters to optimize memory usage during training. - Enhanced error handling and logging for better debugging and user feedback.
- Introduced _get_optimizer_state_path method to streamline optimizer state path management. - Refactored optimizer state path assignment to ensure consistent directory creation and handling. - Improved code clarity and organization within the MegatronService class.
- Added "megatron.**" to allowed unresolved imports in pyproject.toml for better dependency management. - Refactored code in LocalBackend and MegatronService for improved readability and consistency, including assertion formatting and path handling. - Enhanced clarity in the handling of inputs and outputs in training logic.
- Updated _default_lora_adapter_config method to return a LoraConfig instance for improved type safety and clarity. - Refactored _create_identity_lora method to utilize the updated configuration structure. - Improved JSON serialization of LoRA configuration by using asdict for better compatibility. - Cleaned up import statements for consistency and removed unnecessary imports.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.