Skip to content

bugfix: decode prefermance issue fix#375

Merged
AlexStocks merged 5 commits intoapache:masterfrom
nuclearg:bugfix_20250331_decode_perf_issue_fix
Jul 9, 2025
Merged

bugfix: decode prefermance issue fix#375
AlexStocks merged 5 commits intoapache:masterfrom
nuclearg:bugfix_20250331_decode_perf_issue_fix

Conversation

@nuclearg
Copy link
Copy Markdown

What this PR does:
decode.go 里面的 Decode() 每次在解析map或list时都会触发一次对 d.refHolders 的轮询 notify(),在解析复杂对象时(我的场景需要处理约10M的数据),这个 Decode() 会被疯狂递归调用,无数次对这个 d.refHolders() 进行无用的 notify(),产生严重的性能问题

我的机器是 MacBook Pro M3Pro 36G,在我本机上,处理10M的数据时,rt将达到5秒。此性能不可接受。

本次改动对 Decode() 的递归次数进行计数,只在最外层的 Decode() 时进行 notify() 操作。

可以通过MR中的 decode_largedata_test.go 验证。此测试用例随机生成一个巨大的map/list相互嵌套的结构并执行序列化/反序列化操作。未引入我的修改时,反序列化的rt远高于序列化(秒级)。引入此修改后序列化与反序列化rt持平(毫秒级)

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@nuclearg nuclearg changed the title bugfix: decode prefermance fix bugfix: decode prefermance issue fix Mar 31, 2025
@AlexStocks AlexStocks requested review from Copilot and wongoo and removed request for wongoo April 1, 2025 01:20
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses a performance bug in the Decode() function where repeated unnecessary calls to notify() on d.refHolders are causing significant slowdowns when processing large, complex data structures. The changes introduce a recursive depth counter in Decode() so that d.refHolders.notify() is only called at the outermost level.

  • Introduced a decodeRecursiveDepth counter in the Decoder struct
  • Updated Decode() to increment/decrement this counter and conditionally call notify()
  • Added a large data test case (decode_largedata_test.go) to validate the performance improvement

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
decode_largedata_test.go Added a test to generate and verify performance with large data
decode.go Updated Decode() with a recursive depth counter to control notify() calls
Comments suppressed due to low confidence (2)

decode.go:243

  • [nitpick] The variable name 'decodeRecursiveDepth' is descriptive, but consider a shorter variant like 'recursiveDepth' if it would improve consistency and readability within the codebase.
d.decodeRecursiveDepth++

decode.go:242

  • If the Decoder type might be used concurrently, please ensure that proper thread-safety measures are in place for 'decodeRecursiveDepth' to avoid potential race conditions.
func (d *Decoder) Decode() (interface{}, error) {

@tiltwind
Copy link
Copy Markdown
Contributor

tiltwind commented Apr 9, 2025

@nuclearg 感谢

  1. 有一个错误要解决
Error: decode_benchmark_test.go:54:2: ineffectual assignment to `now` (ineffassign)
	now = time.Now()
  1. 测试和压测可以考虑改一下:
func TestMultipleLevelRecursiveDep(t *testing.T) {

	// TODO : 测试 encode 和 decode 后的对象和源对象结构是一致的,值是相等的

}

func BenchmarkMultipleLevelRecursiveDep(b *testing.B) {
	// TODO: 构造一个多层循环依赖的对象, 无需大对象, 压测情况大批量请求可以看出是否有性能改进
	for i := 0; i < b.N; i++ {
		// TODO : 循环 encode 和 decode
	}
}

@nuclearg
Copy link
Copy Markdown
Author

@tiltwind 基准测试用例已提交

Copy link
Copy Markdown
Contributor

@wongoo wongoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nuclearg nuclearg requested a review from AlexStocks April 25, 2025 07:14
@AlexStocks
Copy link
Copy Markdown
Contributor

@nuclearg please fix the ci failure

@AlexStocks
Copy link
Copy Markdown
Contributor

@nuclearg please fix the ci failure.

fmt.Printf("hessian2 deserialize %s\n", rt)

if rt > 1*time.Second {
t.Error("deserialize too slow")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use t.Log instead

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO. This is a perf fix, run slow is a failure.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the CI failed

=== RUN   TestMultipleLevelRecursiveDep2
hessian2 serialize 3.948985828s 10660KB
hessian2 deserialize 1.218173434s
    decode_benchmark_test.go:75: deserialize too slow
--- FAIL: TestMultipleLevelRecursiveDep2 (11.67s)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nuclearg any feedback?

@AlexStocks
Copy link
Copy Markdown
Contributor

please fix the ci failure

@AlexStocks AlexStocks merged commit d39162a into apache:master Jul 9, 2025
1 of 2 checks passed
wqyenjoy pushed a commit to wqyenjoy/dubbo-go-hessian2 that referenced this pull request Jul 13, 2025
wqyenjoy pushed a commit to wqyenjoy/dubbo-go-hessian2 that referenced this pull request Aug 23, 2025
- Fix GitHub Actions workflow compatibility issues
- Add proper test skipping for Java-dependent tests in CI environments
- Improve import organization and code formatting
- Enhance test reliability and documentation
- Convert flaky tests to benchmarks for better CI stability

Fixes issues identified in PR apache#375
wqyenjoy pushed a commit to wqyenjoy/dubbo-go-hessian2 that referenced this pull request Aug 23, 2025
- Fix GitHub Actions workflow compatibility issues
- Add proper test skipping for Java-dependent tests in CI environments
- Improve import organization and code formatting across all modules
- Enhance test reliability and documentation
- Convert flaky tests to benchmarks for better CI stability
- Clean up code structure and remove redundant imports

This commit addresses the CI/CD pipeline failures identified in PR apache#375
while improving overall code quality and maintainability.

Fixes issues identified in PR apache#375
wqyenjoy pushed a commit to wqyenjoy/dubbo-go-hessian2 that referenced this pull request Aug 23, 2025
- Fix GitHub Actions workflow compatibility issues
- Add proper test skipping for Java-dependent tests in CI environments
- Improve import organization and code formatting across all modules
- Enhance test reliability and documentation
- Convert flaky tests to benchmarks for better CI stability
- Clean up code structure and remove redundant imports

This commit addresses the CI/CD pipeline failures identified in PR apache#375
while improving overall code quality and maintainability.

Fixes issues identified in PR apache#375
wqyenjoy pushed a commit to wqyenjoy/dubbo-go-hessian2 that referenced this pull request Aug 23, 2025
- Fix GitHub Actions workflow compatibility issues
- Add proper test skipping for Java-dependent tests in CI environments
- Improve import organization and code formatting across all modules
- Enhance test reliability and documentation
- Convert flaky tests to benchmarks for better CI stability
- Clean up code structure and remove redundant imports

This commit addresses the CI/CD pipeline failures identified in PR apache#375
while improving overall code quality and maintainability.

Fixes issues identified in PR apache#375
@wqyenjoy wqyenjoy mentioned this pull request Aug 23, 2025
mochengqian added a commit to mochengqian/dubbo-go-hessian2 that referenced this pull request Mar 21, 2026
PR apache#375 introduced decode performance optimizations but left several CI-breaking
issues. PR apache#382 attempted to fix them but introduced additional problems
(undefined abs64 function, golangci-lint config downgrade, removal of
imports-formatter, unconditional t.Skip).

This patch fixes all CI blockers while preserving the valuable parts:

- Fix compile error: replace undefined abs64() with math.Abs()
- Fix CI config: align Go version (1.25), golangci-lint (v2.4.0),
  restore imports-formatter
- Fix go.mod: add go 1.25 directive, keep dependencies aligned with master
- Revert non-idiomatic code changes: restore original switch/time.Equal patterns
- Remove junk comments: clean up "No third-party/internal imports" from 13 files
- Remove unconditional t.Skip: restore Java interop tests
- Upgrade GitHub Actions: checkout v2→v4, setup-java v1→v4, setup-go v2→v5

Preserve from original PRs:
- Decode performance optimization (60% improvement)
- BenchmarkMultipleLevelRecursiveDepLarge with structured verification
- Goroutine-safety documentation on Decoder
- Explicit type declarations in test files

Test results:
- All pure Go tests: PASS
- Java interop tests: require JDK 1.8 (CI env), not JDK 25
mochengqian added a commit to mochengqian/dubbo-go-hessian2 that referenced this pull request Mar 21, 2026
- Fix compile error: replace undefined abs64() with math.Abs()
- Fix CI config: align Go 1.25, golangci-lint v2.4.0, restore imports-formatter
- Fix go.mod: add go 1.25 directive, keep deps aligned with master
- Replace panic(err) with b.Fatal(err) in benchmarks
- Remove junk comments and restore idiomatic code
- Upgrade GitHub Actions: checkout v4, setup-java v4, setup-go v5
- Add Decoder goroutine-safety documentation
- Add BenchmarkMultipleLevelRecursiveDepLarge with structured verification
mochengqian added a commit to mochengqian/dubbo-go-hessian2 that referenced this pull request Mar 21, 2026
- Fix compile error: replace undefined abs64() with math.Abs()
- Fix CI config: align Go 1.25, golangci-lint v2.4.0, restore imports-formatter
- Fix go.mod: add go 1.25 directive, keep deps aligned with master
- Replace panic(err) with b.Fatal(err) in benchmarks
- Remove junk comments and restore idiomatic code
- Upgrade GitHub Actions: checkout v4, setup-java v4, setup-go v5
- Add Decoder goroutine-safety documentation
- Add BenchmarkMultipleLevelRecursiveDepLarge with structured verification
mochengqian added a commit to mochengqian/dubbo-go-hessian2 that referenced this pull request Mar 21, 2026
- Fix compile error: replace undefined abs64() with math.Abs()
- Fix CI config: align Go 1.25, golangci-lint v2.4.0, restore imports-formatter
- Fix go.mod: add go 1.25 directive, keep deps aligned with master
- Replace panic(err) with b.Fatal(err) in benchmarks
- Remove junk comments and restore idiomatic code
- Upgrade GitHub Actions: checkout v4, setup-java v4, setup-go v5
- Add Decoder goroutine-safety documentation
- Add BenchmarkMultipleLevelRecursiveDepLarge with structured verification
AlexStocks added a commit that referenced this pull request Mar 28, 2026
fix: resolve CI failures from PR #375 and improve code quality
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants