Skip to content

fix(backend): implement weighted load balancing in selectModel #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pescn merged 1 commit into from
Jan 19, 2026
Merged

fix(backend): implement weighted load balancing in selectModel #49

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions backend/src/utils/api-helpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ export function extractUpstreamHeaders(

/**
* Select the best model/provider combination based on target provider and weights
* Uses weighted random selection for load balancing across multiple providers
*/
export function selectModel(
modelsWithProviders: ModelWithProvider[],
Expand All @@ -105,8 +106,33 @@ export function selectModel(
}
}

// TODO: implement weighted load balancing
return candidates[0] || null;
// Single candidate, return directly
if (candidates.length === 1) {
// oxlint-disable-next-line no-unnecessary-type-assertion
return candidates[0]!; // TypeScript needs assertion here
Comment on lines +111 to +112
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The non-null assertion ! appears to be unnecessary here. The if (candidates.length === 1) check on line 110 guarantees that candidates[0] is defined. The oxlint-disable-next-line comment also suggests that a linter agrees the assertion is not needed. You can simplify the code by removing the assertion and the related comments.

Suggested change
// oxlint-disable-next-line no-unnecessary-type-assertion
return candidates[0]!; // TypeScript needs assertion here
return candidates[0];

}

// Weighted random selection for load balancing
const totalWeight = candidates.reduce((sum, c) => sum + c.model.weight, 0);
const random = Math.random() * totalWeight;

let cumulative = 0;
for (const candidate of candidates) {
cumulative += candidate.model.weight;
if (random < cumulative) {
logger.debug("Selected model via weighted random", {
modelId: candidate.model.id,
providerId: candidate.provider.id,
providerName: candidate.provider.name,
weight: candidate.model.weight,
totalWeight,
});
return candidate;
}
}

// Fallback (should not happen)
return candidates[0] ?? null;
}

// =============================================================================
Expand Down