[BUG] Fix Inference Service Status Propagation

[YouNeedCryDear]

What type of PR is this?

/kind bug
/kind cleanup

What this PR does / why we need it:

1. Fix inference service status propagation.
   a. Previous the status will be ready regardless of the other components.
   b. Right now the status of inference service will wait for all the components to be ready then change to ready.
2. Fix sgl runtimes, possibly due to previous rebase error
3. Remove unused FP8 models and runtimes

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

No

[gemini-code-assist]

Summary of Changes

Hello @YouNeedCryDear, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on enhancing the robustness and accuracy of Inference Service status reporting by ensuring that the service's readiness is contingent on all its deployed components being ready. Additionally, I've refined SGLang runtime configurations and cleaned up deprecated model and runtime definitions, improving overall system clarity and maintainability.

Highlights

• Inference Service Status Propagation Fix: I've addressed a bug where the Inference Service's overall ready status was not accurately reflecting the readiness of all its constituent components. Now, the service will only report as ready once all configured components (engine, decoder, and router) are themselves ready, providing a more reliable status indication.
• SGLang Runtime Configuration Updates: I've updated several SGLang runtime configurations. This includes standardizing the server port to 8080 across various SGLang runtime definitions and introducing a new routerConfig section for the deepseek-rdma-pd-rt runtime, which enables the deployment of a dedicated router component.
• Model and Runtime Cleanup: I've removed references to unused FP8 model configurations from kustomization.yaml and renamed one runtime (srt-llama-3-2-90b-vision-instruct-fp8 to srt-llama-3-2-90b-vision-instruct) to reflect the removal of the FP8 suffix, streamlining the available configurations.
• Sample Inference Service Adjustments: I've updated existing sample Inference Service definitions to align with the new component-based architecture, specifically adjusting replica counts for engines and adding explicit decoder and router sections where appropriate. A new sample for llama-3-3-70b-instruct has also been added.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request addresses a critical bug in Inference Service status propagation by introducing logic to aggregate the readiness of all components (engine, decoder, router). It also includes important cleanups, such as removing unused FP8 models and standardizing configuration values. My review identified a high-severity issue related to API versioning that needs to be addressed to ensure the code compiles and functions correctly.

[YouNeedCryDear]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request effectively fixes the inference service status propagation by aggregating the readiness of all components. The changes also include several cleanups, such as updating runtime configurations and removing unused models. The core logic changes in the Go controller are solid. I've added a couple of comments on the sample YAML files to ensure they are correct and usable.