Skip to content

feat: add parallel fetching for services#32

Merged
k1LoW merged 2 commits intomainfrom
parallel-get
Sep 22, 2025
Merged

feat: add parallel fetching for services#32
k1LoW merged 2 commits intomainfrom
parallel-get

Conversation

@k1LoW
Copy link
Copy Markdown
Contributor

@k1LoW k1LoW commented Sep 22, 2025

This pull request introduces significant performance improvements to the tailor/resource.go resource fetching logic by adding parallelism at multiple levels. The main changes refactor the code to use goroutines and errgroup for concurrent fetching of pipeline, TailorDB, and StateFlow resources, as well as their nested details. Mutexes are introduced to ensure thread-safe writes to shared data structures.

Key improvements and changes:

Parallelization and Concurrency:

  • Refactored the top-level Resources method to use errgroup for concurrent fetching of Pipeline, TailorDB, and StateFlow services, reducing overall resource fetch time. (tailor/resource.go)
  • Introduced parallel fetching for nested resources (e.g., pipeline resolvers, TailorDB types) using goroutines and errgroup, with mutexes to ensure thread safety when appending to shared slices. (tailor/resource.go) [1] [2] [3] [4]

Code Structure and Maintainability:

  • Extracted resource-fetching logic into dedicated helper methods (fetchPipelineServices, fetchPipelineResolvers, fetchPipelineResolverDetails, fetchTailorDBServices, fetchTailorDBTypes, fetchTailorDBTypeDetails, fetchStateFlowServices, fetchExecutionResults) for better readability and testability. (tailor/resource.go) [1] [2] [3] [4] [5]

Thread Safety:

  • Added a sync.Mutex field to the Resources struct and used local mutexes in helper functions to prevent race conditions when updating shared slices. (tailor/resource.go) [1] [2] [3] [4]

Dependency Updates:

  • Added golang.org/x/sync to go.mod to provide errgroup for managing goroutine lifecycles and error propagation. (go.mod)

These changes collectively make resource loading much faster and more robust, especially for projects with a large number of pipelines or database types.

@k1LoW k1LoW self-assigned this Sep 22, 2025
@k1LoW k1LoW added enhancement New feature or request minor labels Sep 22, 2025
@github-actions
Copy link
Copy Markdown
Contributor

Code Metrics Report

main (01c749f) #32 (adffa60) +/-
Coverage 30.7% 26.6% -4.1%
Code to Test Ratio 1:1.4 1:1.3 -0.2
Test Execution Time 6s 7s +1s
Details
  |                     | main (01c749f) | #32 (adffa60) |  +/-  |
  |---------------------|----------------|---------------|-------|
- | Coverage            |          30.7% |         26.6% | -4.1% |
  |   Files             |             12 |            12 |     0 |
  |   Lines             |            482 |           556 |   +74 |
  |   Covered           |            148 |           148 |     0 |
- | Code to Test Ratio  |          1:1.4 |         1:1.3 |  -0.2 |
  |   Code              |           1354 |          1462 |  +108 |
  |   Test              |           1924 |          1924 |     0 |
- | Test Execution Time |             6s |            7s |   +1s |

Code coverage of files in pull request scope (0.0% → 0.0%)

Files Coverage +/- Status
tailor/resource.go 0.0% 0.0% modified

Reported by octocov

@k1LoW k1LoW marked this pull request as ready for review September 22, 2025 05:47
@k1LoW k1LoW requested a review from a team as a code owner September 22, 2025 05:47
@k1LoW k1LoW enabled auto-merge September 22, 2025 05:49
Copy link
Copy Markdown

@Mistat Mistat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@k1LoW k1LoW disabled auto-merge September 22, 2025 06:58
@k1LoW
Copy link
Copy Markdown
Contributor Author

k1LoW commented Sep 22, 2025

Thank you!

@k1LoW k1LoW merged commit 07d9c08 into main Sep 22, 2025
1 check passed
@k1LoW k1LoW deleted the parallel-get branch September 22, 2025 06:58
@github-actions github-actions Bot mentioned this pull request Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request minor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants