feat: GitHub Copilot Enterprise/Organization Metrics Support#5
feat: GitHub Copilot Enterprise/Organization Metrics Support#5IgorWarzocha wants to merge 4 commits intomasterfrom
Conversation
Add enterprise and organization-level usage tracking via GitHub's public preview Copilot Usage Metrics API. Falls back to individual quota checking when enterprise metrics unavailable. - Add CopilotEnterpriseConfig to UsageConfig - Create providers/copilot/enterprise.ts for metrics API - Add loadCopilotEnterpriseConfig() with gh CLI token fallback - Update CopilotProvider to try enterprise path first Configuration via copilotEnterprise in usage-config.jsonc. Requires "Copilot usage metrics" policy enabled in enterprise settings.
GitHub-specific token reading logic now lives alongside other Copilot auth helpers instead of in general config layer.
Summary of ChangesHello @IgorWarzocha, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the application's ability to monitor GitHub Copilot usage by integrating with the new enterprise and organization-level metrics API. It provides a flexible and robust solution for large-scale Copilot deployments, ensuring that usage data can be retrieved efficiently, with intelligent fallback to individual quotas and simplified authentication. The changes aim to provide a clearer picture of Copilot consumption for administrators and users alike. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds significant new functionality for tracking GitHub Copilot usage for Enterprise and Organization accounts. The changes are well-structured, introducing a new provider for enterprise metrics and integrating it with a fallback to the existing individual quota checking. The automatic discovery of the GitHub CLI token is a nice touch for user experience.
My review focuses on improving code quality by reducing duplication, increasing robustness, and ensuring correctness in edge cases. Specifically, I've identified opportunities to:
- Refactor duplicated code in the new enterprise provider.
- Improve the robustness of parsing configuration files.
- Address a potential correctness issue in how metrics data is processed.
- Enhance type safety and maintainability by reusing defined types.
Overall, this is a great addition. Addressing these points will make the new feature more robust and easier to maintain.
| export async function readGitHubCliToken(): Promise<string | null> { | ||
| const ghConfigPath = join(homedir(), ".config", "gh", "hosts.yml") | ||
|
|
||
| try { | ||
| const file = Bun.file(ghConfigPath) | ||
| if (!(await file.exists())) { | ||
| return null | ||
| } | ||
|
|
||
| const content = await file.text() | ||
| const lines = content.split("\n") | ||
| let inGithubCom = false | ||
| for (const line of lines) { | ||
| const trimmed = line.trim() | ||
| if (trimmed === "github.com:" || trimmed === '"github.com":') { | ||
| inGithubCom = true | ||
| continue | ||
| } | ||
| if (inGithubCom && trimmed.startsWith("oauth_token:")) { | ||
| const match = trimmed.match(/oauth_token:\s*["']?([^"'\s]+)/) | ||
| if (match && match[1]) { | ||
| return match[1] | ||
| } | ||
| } | ||
| if (inGithubCom && trimmed.endsWith(":") && !trimmed.startsWith("oauth_token")) { | ||
| inGithubCom = false | ||
| } | ||
| } | ||
|
|
||
| return null | ||
| } catch { | ||
| return null | ||
| } | ||
| } |
There was a problem hiding this comment.
The readGitHubCliToken function manually parses the hosts.yml file by splitting it into lines and using a state machine. This approach is fragile and can easily break if the YAML file's format changes (e.g., different indentation, comments, or key ordering). A more robust solution would be to use a dedicated YAML parsing library (like js-yaml or yaml) to handle the file. This would make the code simpler and less prone to errors. For example: const config = yaml.parse(fileContent); const token = config?.['github.com']?.oauth_token;
src/providers/copilot/enterprise.ts
Outdated
| async function fetchEnterpriseMetrics( | ||
| enterprise: string, | ||
| authToken: string, | ||
| ): Promise<EnterpriseMetricsEntry[] | null> { | ||
| const url = `${GITHUB_API_BASE_URL}/enterprises/${enterprise}/copilot/metrics/reports/enterprise-28-day/latest` | ||
|
|
||
| try { | ||
| const response = await fetchWithTimeout( | ||
| url, | ||
| { | ||
| headers: { | ||
| Accept: "application/vnd.github+json", | ||
| Authorization: `Bearer ${authToken}`, | ||
| "X-GitHub-Api-Version": API_VERSION, | ||
| }, | ||
| }, | ||
| REQUEST_TIMEOUT_MS, | ||
| ) | ||
|
|
||
| if (!response.ok) { | ||
| return null | ||
| } | ||
|
|
||
| const data = (await response.json()) as EnterpriseMetricsResponse | ||
|
|
||
| // Fetch the first report link (NDJSON format) | ||
| if (!data.download_links || data.download_links.length === 0) { | ||
| return null | ||
| } | ||
|
|
||
| const reportUrl = data.download_links[0] | ||
| const reportResponse = await fetchWithTimeout(reportUrl, {}, REQUEST_TIMEOUT_MS) | ||
|
|
||
| if (!reportResponse.ok) { | ||
| return null | ||
| } | ||
|
|
||
| const reportText = await reportResponse.text() | ||
| const lines = reportText.trim().split("\n") | ||
|
|
||
| return lines | ||
| .filter((line) => line.trim().length > 0) | ||
| .map((line) => { | ||
| try { | ||
| return JSON.parse(line) as EnterpriseMetricsEntry | ||
| } catch { | ||
| return null | ||
| } | ||
| }) | ||
| .filter((entry): entry is EnterpriseMetricsEntry => entry !== null) | ||
| } catch { | ||
| return null | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Fetch and parse the latest 28-day organization metrics report. | ||
| * Aggregates data across all days to compute total usage. | ||
| */ | ||
| async function fetchOrganizationMetrics( | ||
| organization: string, | ||
| authToken: string, | ||
| ): Promise<OrganizationMetricsEntry[] | null> { | ||
| const url = `${GITHUB_API_BASE_URL}/orgs/${organization}/copilot/metrics/reports/organization-28-day/latest` | ||
|
|
||
| try { | ||
| const response = await fetchWithTimeout( | ||
| url, | ||
| { | ||
| headers: { | ||
| Accept: "application/vnd.github+json", | ||
| Authorization: `Bearer ${authToken}`, | ||
| "X-GitHub-Api-Version": API_VERSION, | ||
| }, | ||
| }, | ||
| REQUEST_TIMEOUT_MS, | ||
| ) | ||
|
|
||
| if (!response.ok) { | ||
| return null | ||
| } | ||
|
|
||
| const data = (await response.json()) as OrganizationMetricsResponse | ||
|
|
||
| // Fetch the first report link (NDJSON format) | ||
| if (!data.download_links || data.download_links.length === 0) { | ||
| return null | ||
| } | ||
|
|
||
| const reportUrl = data.download_links[0] | ||
| const reportResponse = await fetchWithTimeout(reportUrl, {}, REQUEST_TIMEOUT_MS) | ||
|
|
||
| if (!reportResponse.ok) { | ||
| return null | ||
| } | ||
|
|
||
| const reportText = await reportResponse.text() | ||
| const lines = reportText.trim().split("\n") | ||
|
|
||
| return lines | ||
| .filter((line) => line.trim().length > 0) | ||
| .map((line) => { | ||
| try { | ||
| return JSON.parse(line) as OrganizationMetricsEntry | ||
| } catch { | ||
| return null | ||
| } | ||
| }) | ||
| .filter((entry): entry is OrganizationMetricsEntry => entry !== null) | ||
| } catch { | ||
| return null | ||
| } | ||
| } |
There was a problem hiding this comment.
The functions fetchEnterpriseMetrics and fetchOrganizationMetrics are nearly identical. The logic for making the API call, fetching the report, and parsing the NDJSON response should be extracted into a single, generic helper function. The two current functions could then become thin wrappers that call this helper with the appropriate API endpoint URL. This would significantly reduce code duplication and make the code easier to maintain.
src/providers/copilot/enterprise.ts
Outdated
| }, 0) | ||
|
|
||
| // Get the most recent date for reset time calculation | ||
| const mostRecentEntry = entries[entries.length - 1] |
There was a problem hiding this comment.
This line assumes that the entries array is sorted by date when it selects the last element as the most recent. This is not a safe assumption as the API doesn't guarantee the order. To ensure you're using the correct date for the reset time calculation, you should explicitly sort the entries array by date before this line.
| const mostRecentEntry = entries[entries.length - 1] | |
| entries.sort((a, b) => new Date(a.date).getTime() - new Date(b.date).getTime()); | |
| const mostRecentEntry = entries[entries.length - 1]; |
src/providers/copilot/enterprise.ts
Outdated
| interface EnterpriseMetricsResponse { | ||
| download_links: string[] | ||
| report_start_day: string | ||
| report_end_day: string | ||
| } | ||
|
|
||
| interface OrganizationMetricsResponse { | ||
| download_links: string[] | ||
| report_start_day: string | ||
| report_end_day: string | ||
| } |
There was a problem hiding this comment.
The interfaces EnterpriseMetricsResponse and OrganizationMetricsResponse are identical. They can be merged into a single interface, for example MetricsReportResponse, to reduce code duplication. This new interface can then be used in both fetchEnterpriseMetrics and fetchOrganizationMetrics when parsing the JSON response.
src/providers/copilot/enterprise.ts
Outdated
| interface EnterpriseMetricsEntry { | ||
| /** Enterprise slug */ | ||
| enterprise_id: string | ||
| /** Date in YYYY-MM-DD format */ | ||
| date: string | ||
| /** Total number of active users */ | ||
| total_active_users: number | ||
| /** Total number of engaged users */ | ||
| total_engaged_users: number | ||
| /** Total lines of code suggested */ | ||
| total_lines_suggested: number | ||
| /** Total lines of code accepted */ | ||
| total_lines_accepted: number | ||
| /** Number of code suggestions */ | ||
| total_suggestions_count: number | ||
| /** Number of accepted suggestions */ | ||
| total_acceptances_count: number | ||
| /** Number of completions */ | ||
| completions_count: number | ||
| /** Number of chat conversations */ | ||
| chat_conversations_count: number | ||
| /** Number of chat acceptances */ | ||
| chat_acceptances_count: number | ||
| /** Number of premium interactions */ | ||
| premium_interactions_count?: number | ||
| /** Total number of premium requests */ | ||
| total_premium_requests?: number | ||
| } | ||
|
|
||
| interface OrganizationMetricsEntry { | ||
| /** Organization ID */ | ||
| organization_id: string | ||
| /** Date in YYYY-MM-DD format */ | ||
| date: string | ||
| /** Total number of active users */ | ||
| total_active_users: number | ||
| /** Total number of engaged users */ | ||
| total_engaged_users: number | ||
| /** Total lines of code suggested */ | ||
| total_lines_suggested: number | ||
| /** Total lines of code accepted */ | ||
| total_lines_accepted: number | ||
| /** Number of code suggestions */ | ||
| total_suggestions_count: number | ||
| /** Number of accepted suggestions */ | ||
| total_acceptances_count: number | ||
| /** Number of completions */ | ||
| completions_count: number | ||
| /** Number of chat conversations */ | ||
| chat_conversations_count: number | ||
| /** Number of chat acceptances */ | ||
| chat_acceptances_count: number | ||
| /** Number of premium interactions */ | ||
| premium_interactions_count?: number | ||
| /** Total number of premium requests */ | ||
| total_premium_requests?: number | ||
| } |
There was a problem hiding this comment.
The EnterpriseMetricsEntry and OrganizationMetricsEntry interfaces have a lot of overlapping fields. To improve maintainability and reduce duplication, consider creating a BaseMetricsEntry interface with all the common properties. The two specific interfaces can then extend this base interface, each adding its unique ID field (enterprise_id or organization_id).
| } catch { | ||
| } |
There was a problem hiding this comment.
This catch block is empty, which swallows any potential errors during the individual quota fetch. While failing silently might be the intended behavior if auth isn't configured, an empty block is a code smell and can hide unexpected issues. It would be better to add a comment explaining why the block is intentionally empty, to clarify the intent for future maintainers.
} catch {
// This is expected to fail if individual Copilot auth is not configured.
// The provider will return null, which is the desired fallback behavior.
}
src/usage/config.ts
Outdated
| export async function loadCopilotEnterpriseConfig(): Promise<{ | ||
| enterprise?: string | ||
| organization?: string | ||
| token: string | ||
| } | null> { |
There was a problem hiding this comment.
The return type of loadCopilotEnterpriseConfig is defined as an inline anonymous type. This type is identical to the CopilotEnterpriseAuth interface already defined in src/providers/copilot/enterprise.ts. To improve maintainability and type safety, you should import CopilotEnterpriseAuth and use it here instead of redefining it. You will need to add import type { CopilotEnterpriseAuth } from '../providers/copilot/enterprise.js'; to the top of the file.
export async function loadCopilotEnterpriseConfig(): Promise<CopilotEnterpriseAuth | null> {![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e459054236
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
src/providers/copilot/enterprise.ts
Outdated
| return { | ||
| used: totalPremiumRequests, | ||
| total: -1, // Enterprise quotas are managed at the org level | ||
| percentRemaining: 0, // Cannot determine without quota info |
There was a problem hiding this comment.
In toCopilotQuotaFromMetrics, used is set to the total premium requests, but the Copilot UI treats copilot.used as the remaining count (see formatCopilotSnapshot where it is assigned to chatRemaining in src/ui/status.ts lines 129–132) and the bar expects percentRemaining to match. With enterprise data this shows usage as if it were remaining (and a 0% bar), which is misleading when users have any activity. Consider mapping enterprise data to the same “remaining” semantics or updating the display logic to distinguish usage-only data.
Useful? React with 👍 / 👎.
src/providers/copilot/enterprise.ts
Outdated
| // Aggregate total premium requests across all days | ||
| const totalPremiumRequests = entries.reduce((sum, entry) => { | ||
| return sum + (entry.total_premium_requests || entry.premium_interactions_count || 0) | ||
| }, 0) |
There was a problem hiding this comment.
The aggregation uses entry.total_premium_requests || entry.premium_interactions_count || 0, which treats 0 as missing. If a report day legitimately has total_premium_requests: 0 but also includes premium_interactions_count, the fallback will overwrite the zero and inflate usage. Use nullish coalescing or an explicit undefined check so zero counts remain zero.
Useful? React with 👍 / 👎.
- Consolidated duplicate metrics interfaces and extracted common fetch helper - Added explicit sorting of metrics entries by date - Fixed nullish coalescing for zero totals in aggregation - Aligned enterprise quota semantics (tracking usage only for now) - Reused CopilotEnterpriseAuth type in config layer - Added explanatory comments for empty catch blocks and enterprise limits
|
@codex have a look at new version. Read all the previous comments. Not all of them had been actioned, but we got as close as we could - some of them did not make sense in the grand scheme of things. Surgically executed all PR feedback:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c898ccb90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| resetTime, | ||
| completionsUsed: sortedEntries.reduce((sum, e) => e.completions_count ?? 0, 0), | ||
| completionsTotal: -1, |
There was a problem hiding this comment.
The completionsUsed reducer ignores the accumulator and returns only the current entry’s completions_count, so the final value reflects just the last day in the report instead of the 28‑day total. This underreports usage whenever there is more than one day of data. If the intent is to show aggregate completions, this should sum with the accumulator (e.g., sum + (e.completions_count ?? 0)).
Useful? React with 👍 / 👎.
The reducer was ignoring the accumulator and only returning the last entry's count. Fixed to sum all counts over the 28-day period.
1 participant
Summary
Adds support for GitHub Copilot enterprise and organization-level usage tracking via the public preview Copilot Usage Metrics API.
What's New
copilotEnterprisetousage-config.jsoncConfiguration
{ "copilotEnterprise": { "enterprise": "your-enterprise-slug", // or "organization": "your-org-name" "token": "" // optional, defaults to gh CLI token } }Requirements
Testing Status
🧪 Needs testing - This feature cannot be tested without an enterprise account. Looking for enterprise users to validate:
Files Changed
src/providers/copilot/enterprise.ts- New enterprise metrics providersrc/providers/copilot/auth.ts- AddedreadGitHubCliToken()src/providers/copilot/index.ts- Try enterprise path first, fallback to individualsrc/types.ts- AddedCopilotEnterpriseConfigsrc/usage/config.ts- AddedloadCopilotEnterpriseConfig()README.md- Documentation for enterprise setupCode Quality