feat: Add benchmarks module with tokenledger integration#290
feat: Add benchmarks module with tokenledger integration#290KooshaPari wants to merge 654 commits intorouter-for-me:mainfrom
Conversation
…-20260223 merge: absorb heliosharness parity branch functional delta
fix: resolve executor compile regressions
…-20260223 chore: merge remaining branch deltas (dependabot + CP2K queue docs)
Antigravity quota display shows 100% because no Google Cloud quota API is integrated. Unlike GitHub Copilot which has quota endpoints, Antigravity would require Google Cloud API integration. This is a complex feature requiring external API integration.
Add util package with safe logging helpers to mask passwords, tokens, and secrets in logs.
- Removed duplicate pkg/llmproxy/config package - Updated all imports to use internal/config - Fixed type mismatch errors between config packages - Build now succeeds
…ravity quota - Remove build-errors.log artifact - Update README and docs config - Clean up translator files - Remove pkg/llmproxy/config/config.go (consolidated to internal/config)
- Add benchmarks client with caching - Add unified store with fallback to hardcoded values - Maintain backward compatibility with existing pareto router
- Add benchmarks.UnifiedBenchmarkStore to ParetoRouter - Use dynamic benchmarks with hardcoded fallback - Maintain backward compatibility
security: Add safe logging utility
…chestration feat: Integrate benchmarks into ParetoRouter
# Conflicts: # .gitignore # .worktrees/config/m/config-build/active/pkg/llmproxy/config/config.go # .worktrees/config/m/config-build/active/pkg/llmproxy/config/oauth_model_alias_migration.go # README.md
chore: merge full-sdk type unification (layer 3)
chore: merge full-sdk/test-cleanup stack into main
Summary of ChangesHello @KooshaPari, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands the project's capabilities by integrating a new benchmarks module with Tokenledger for dynamic data. It also introduces a robust set of CI/CD workflows and scripts to enhance code quality, security, and release processes. A new CLI tool, Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Ignored Files
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new benchmarks module with tokenledger integration, along with numerous CI/CD scripts, GitHub workflows, and configuration files. The changes enhance the project's automation, security, and extensibility. However, there are several areas that require attention, including resolving Git merge conflicts, addressing typos, improving clarity in documentation, and updating CLI usage messages. A significant concern is the use of future Go versions and highly unstable golang.org/x dependency versions, which could lead to compatibility issues and build failures. Additionally, some API responses and configuration fields could be streamlined for better consistency and user experience.
| github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect | ||
| golang.org/x/arch v0.8.0 // indirect | ||
| golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect | ||
| golang.org/x/sys v0.38.0 // indirect |
There was a problem hiding this comment.
The indirect dependency golang.org/x/tools is pinned to a very high version (v0.38.0). This version is significantly higher than current stable releases (e.g., v0.21.0). This strongly suggests that this is a future or unstable version, which can lead to compatibility issues, unexpected behavior, or build failures. It's critical to use stable, officially released versions of these core libraries.
| gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= | ||
| gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= | ||
| gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= | ||
| gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= |
There was a problem hiding this comment.
The indirect dependency golang.org/x/tools is pinned to a very high version (v0.38.0). This version is significantly higher than current stable releases (e.g., v0.21.0). This strongly suggests that this is a future or unstable version, which can lead to compatibility issues, unexpected behavior, or build failures. It's critical to use stable, officially released versions of these core libraries.
| @@ -0,0 +1,112 @@ | |||
| module github.com/router-for-me/CLIProxyAPI/v6 | |||
|
|
|||
| go 1.26.0 | |||
There was a problem hiding this comment.
The go.mod file specifies go 1.26.0. As of the current date, Go 1.26.0 is a future version and not yet released. Using a future Go version can lead to build failures or unexpected behavior in current environments. It's recommended to use a stable, currently released version of Go.
| golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY= | ||
| golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546/go.mod h1:j/pmGrbnkbPtQfxEe5D0VQhZC6qKbfKifgD0oM7sR70= | ||
| golang.org/x/mod v0.29.0 h1:HV8lRxZC4l2cr3Zq1LvtOsi/ThTgWnUk/y64QSs8GwA= | ||
| golang.org/x/mod v0.29.0/go.mod h1:NyhrlYXJ2H4eJiRy/WDBO6HMqZQ6q9nk4JzS3NuCK+w= | ||
| golang.org/x/net v0.47.0 h1:Mx+4dIFzqraBXUugkia1OOvlD6LemFo1ALMHjrXDOhY= | ||
| golang.org/x/net v0.47.0/go.mod h1:/jNxtkgq5yWUGYkaZGqo27cfGZ1c5Nen03aYrrKpVRU= | ||
| golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI= | ||
| golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU= | ||
| golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I= | ||
| golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= | ||
| golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= | ||
| golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= | ||
| golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= | ||
| golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= | ||
| golang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc= | ||
| golang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= | ||
| golang.org/x/term v0.37.0 h1:8EGAD0qCmHYZg6J17DvsMy9/wJ7/D/4pV/wfnld5lTU= | ||
| golang.org/x/term v0.37.0/go.mod h1:5pB4lxRNYYVZuTLmy8oR2BH8dflOR+IbTYFD8fi3254= | ||
| golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM= | ||
| golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM= | ||
| golang.org/x/tools v0.38.0 h1:Hx2Xv8hISq8Lm16jvBZ2VQf+RLmbd7wVUsALibYI/IQ= |
There was a problem hiding this comment.
Several golang.org/x dependencies (crypto, net, oauth2, sync, term, text) are pinned to unusually high version numbers (e.g., v0.45.0, v0.47.0). These versions are significantly higher than current stable releases (e.g., crypto is around v0.25.0). This strongly suggests that these are future or unstable versions, which can lead to compatibility issues, unexpected behavior, or build failures. It's critical to use stable, officially released versions of these core libraries.
| github.com/stretchr/testify v1.11.1 | ||
| github.com/tidwall/gjson v1.18.0 | ||
| github.com/tidwall/sjson v1.2.5 | ||
| github.com/tiktoken-go/tokenizer v0.7.0 | ||
| golang.org/x/crypto v0.45.0 | ||
| golang.org/x/net v0.47.0 |
There was a problem hiding this comment.
Several golang.org/x dependencies (crypto, net, oauth2, sync, term, text) are pinned to unusually high version numbers (e.g., v0.45.0, v0.47.0). These versions are significantly higher than current stable releases (e.g., crypto is around v0.25.0). This strongly suggests that these are future or unstable versions, which can lead to compatibility issues, unexpected behavior, or build failures. It's critical to use stable, officially released versions of these core libraries.
| return | ||
| } | ||
|
|
||
| authIndex := firstNonEmptyString(body.AuthIndexSnake, body.AuthIndexCamel, body.AuthIndexPascal) |
There was a problem hiding this comment.
The firstNonEmptyString helper is used here to extract auth_index from potentially three different fields. While functional, this pattern confirms the redundancy of having AuthIndexSnake, AuthIndexCamel, and AuthIndexPascal in the apiCallRequest struct. Consolidating to a single field with a custom unmarshaler would be cleaner.
| func (h *Handler) PatchGeminiKey(c *gin.Context) { | ||
| type geminiKeyPatch struct { | ||
| APIKey *string `json:"api-key"` | ||
| Prefix *string `json:"prefix"` | ||
| BaseURL *string `json:"base-url"` | ||
| ProxyURL *string `json:"proxy-url"` |
There was a problem hiding this comment.
The geminiKeyPatch struct is missing the Models field from config.GeminiKey. This means the models field cannot be patched via this endpoint, leading to incomplete functionality. Please add the Models field to geminiKeyPatch.
type geminiKeyPatch struct {
APIKey *string `json:"api-key"`
Prefix *string `json:"prefix"`
BaseURL *string `json:"base-url"`
ProxyURL *string `json:"proxy-url"`
Headers *map[string]string `json:"headers"`
Models *[]config.GeminiModel `json:"models"`
ExcludedModels *[]string `json:"excluded-models"`
}| func (h *Handler) PatchOAuthModelAlias(c *gin.Context) { | ||
| var body struct { | ||
| Provider *string `json:"provider"` | ||
| Channel *string `json:"channel"` |
There was a problem hiding this comment.
The PatchOAuthModelAlias request body includes redundant fields for the channel (Provider, Channel). It's better to pick one naming convention (e.g., channel) and stick to it, or use a custom unmarshaler to handle multiple formats gracefully without exposing them as separate fields. This adds unnecessary complexity and can lead to confusion.
| channelRaw := "" | ||
| if body.Channel != nil { | ||
| channelRaw = *body.Channel | ||
| } else if body.Provider != nil { | ||
| channelRaw = *body.Provider | ||
| } |
There was a problem hiding this comment.
This logic handles both Channel and Provider fields to determine the target channel. While functional, it confirms the redundancy of having both fields in the request body. Consolidating to a single field (e.g., channel) would simplify the API and reduce potential confusion.
| func (h *Handler) DeleteOAuthModelAlias(c *gin.Context) { | ||
| channel := strings.ToLower(strings.TrimSpace(c.Query("channel"))) | ||
| if channel == "" { | ||
| channel = strings.ToLower(strings.TrimSpace(c.Query("provider"))) |
There was a problem hiding this comment.
This logic handles both channel and provider query parameters. While functional, it confirms the redundancy of supporting both. Consolidating to a single query parameter (e.g., channel) would simplify the API and reduce potential confusion.
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
This comment was marked as spam.
This comment was marked as spam.
3 participants
Summary
Adds benchmarks module to cliproxy++ for tokenledger integration:
Features
Testing