Skip to content

perf: fix BenchmarkCompileMemoryUsage regression (+81.9%) — faster YAML parser + benchmark warm-up #22464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 4 commits into from
Mar 23, 2026
Merged

perf: fix BenchmarkCompileMemoryUsage regression (+81.9%) — faster YAML parser + benchmark warm-up #22464

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0629011
Initial plan
Copilot Mar 23, 2026
f48649b
perf: fix BenchmarkCompileMemoryUsage regression - faster YAML parser…
Copilot Mar 23, 2026
5d6250a
Merge branch 'main' into copilot/fix-regression-in-compile-memory-usage
pelikhan Mar 23, 2026
d0e8d74
Merge branch 'main' into copilot/fix-regression-in-compile-memory-usage
pelikhan Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pkg/workflow/compiler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import (
"github.com/github/gh-aw/pkg/console"
"github.com/github/gh-aw/pkg/logger"
"github.com/github/gh-aw/pkg/stringutil"
"github.com/goccy/go-yaml"
"go.yaml.in/yaml/v3"
)

var log = logger.New("workflow:compiler")
Expand Down
25 changes: 25 additions & 0 deletions pkg/workflow/compiler_performance_benchmark_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@ Analyze the issue: ${{ steps.sanitized.outputs.text }}

compiler := NewCompiler()

// Warm up: run once before timing to prime one-time caches (schema compilation, etc.)
_ = compiler.CompileWorkflow(testFile)

b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
_ = compiler.CompileWorkflow(testFile)
Comment on lines +44 to 49
Copy link

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The warm-up compile call ignores the returned error. If the test input becomes invalid (or CompileWorkflow starts returning errors for a regression), the benchmark will silently measure an error path. Consider checking the warm-up error and failing the benchmark (e.g., b.Fatal/b.Fatalf) before resetting the timer.

This issue also appears in the following locations of the same file:

  • line 105
  • line 156
  • line 209
  • line 298
Suggested change
_ = compiler.CompileWorkflow(testFile)
b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
_ = compiler.CompileWorkflow(testFile)
if err := compiler.CompileWorkflow(testFile); err != nil {
b.Fatalf("warm-up compile failed: %v", err)
}
b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
if err := compiler.CompileWorkflow(testFile); err != nil {
b.Fatalf("benchmark compile failed: %v", err)
}

Copilot uses AI. Check for mistakes.
Expand Down Expand Up @@ -98,6 +102,10 @@ Review the pull request: ${{ github.event.pull_request.number }}

compiler := NewCompiler(WithNoEmit(true))

// Warm up: run once before timing to prime one-time caches (schema compilation, etc.)
_ = compiler.CompileWorkflow(testFile)

b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
_ = compiler.CompileWorkflow(testFile)
Expand Down Expand Up @@ -145,6 +153,10 @@ Review and test the pull request with multiple tools.

compiler := NewCompiler(WithNoEmit(true))

// Warm up: run once before timing to prime one-time caches (schema compilation, etc.)
_ = compiler.CompileWorkflow(testFile)

b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
_ = compiler.CompileWorkflow(testFile)
Expand Down Expand Up @@ -194,6 +206,11 @@ Standard workflow for memory profiling.

compiler := NewCompiler(WithNoEmit(true))

// Warm up: run once before timing to initialize one-time caches
// (schema compilation, regex caches) so they don't skew per-op metrics.
_ = compiler.CompileWorkflow(testFile)

b.ResetTimer()
b.ReportAllocs()

// Track memory allocations
Expand Down Expand Up @@ -232,6 +249,10 @@ Test parsing performance.

compiler := NewCompiler()

// Warm up: prime the schema compilation cache before timed measurement.
_, _ = compiler.ParseWorkflowFile(testFile)

b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
_, _ = compiler.ParseWorkflowFile(testFile)
Comment on lines +253 to 258
Copy link

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The warm-up ParseWorkflowFile call ignores its error. If parsing fails, the benchmark will still proceed and may measure the failure path. Consider asserting the warm-up succeeds (and perhaps also verify the file parses once) before calling b.ResetTimer().

Suggested change
_, _ = compiler.ParseWorkflowFile(testFile)
b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
_, _ = compiler.ParseWorkflowFile(testFile)
if _, err := compiler.ParseWorkflowFile(testFile); err != nil {
b.Fatalf("warm-up ParseWorkflowFile failed: %v", err)
}
b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
if _, err := compiler.ParseWorkflowFile(testFile); err != nil {
b.Fatalf("ParseWorkflowFile failed: %v", err)
}

Copilot uses AI. Check for mistakes.
Expand Down Expand Up @@ -274,6 +295,10 @@ Test validation performance.
compiler := NewCompiler(WithNoEmit(true))
compiler.SetStrictMode(true)

// Warm up: prime the schema compilation cache before timed measurement.
_ = compiler.CompileWorkflow(testFile)

b.ResetTimer()
b.ReportAllocs()
for b.Loop() {
_ = compiler.CompileWorkflow(testFile)
Expand Down
2 changes: 1 addition & 1 deletion pkg/workflow/schema_validation.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ import (
"strings"
"sync"

"github.com/goccy/go-yaml"
"github.com/santhosh-tekuri/jsonschema/v6"
"go.yaml.in/yaml/v3"
)

var schemaValidationLog = newValidationLogger("schema")
Expand Down
Loading