[MISC] Fix Config Runtime

[YouNeedCryDear]

What type of PR is this?

BUG
Cleanup

What this PR does / why we need it:

Fix and cleanup config runtimes

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

[gemini-code-assist]

Summary of Changes

Hello @YouNeedCryDear, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on fixing and cleaning up configuration runtimes, specifically addressing issues related to port configurations, model versions, and the addition of router configurations for improved service discovery and routing. It also includes updates to sample configurations to align with the runtime changes and introduces a new sample for Llama 3 3-70B instruct.

Highlights

• Configuration Updates: Updates several SRT runtime configurations to use safetensors version 1.0.0 and adjusts the runtime name for llama-3-2-90b-vision-instruct.
• Port Change: Modifies the default port for the sglang server in deepseek-rdma-pd-rt.yaml from 30000 to 8080.
• Router Configuration: Adds a router configuration to deepseek-rdma-pd-rt.yaml, including image, resource limits, ports, command, and environment variables for service discovery.
• Sample Updates: Updates the deepseek-v3-pd.yaml sample to use the correct runtime name, namespace, and adjusts the number of replicas for decoder and adds router replicas. Also, updates the llama-4-maverick-17b-128e-instruct-fp8.yaml sample to include router replicas and reduces engine and decoder replicas.
• New Sample: Adds a new sample llama3-3-70b-instruct.yaml with namespace, InferenceService, model, engine, runtime, and decoder configurations.
• Resource Removal: Removes meta/Llama-3.3-70B-instruct-FP8-Dynamic.yaml from kustomization.yaml.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request includes several important fixes and cleanups, particularly in the configuration of runtimes and inference services. The changes standardize port numbers, update model format versions, and introduce a router configuration for improved traffic management. It's recommended to verify the replica counts and router configurations to ensure optimal performance.