Skip to content

Conversation

@mkysel
Copy link
Collaborator

@mkysel mkysel commented Jun 30, 2025

The DB goroutine lifecycle was tied to the lifecycle of the GRPC connection. There were no guarantees that the DB conn won't outlive the main routine when being rebuilt. This could lead to concurrent DB writing coroutines, out-of-order writes and other nastiness.

@mkysel mkysel requested a review from a team as a code owner June 30, 2025 20:38
@graphite-app
Copy link

graphite-app bot commented Jun 30, 2025

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

  • Queue - adds this PR to the back of the merge queue
  • Hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

recvChan := make(chan *message_api.SubscribeEnvelopesResponse)
errChan := make(chan error)
writeQueue := make(chan *envUtils.OriginatorEnvelope, 10)
errChan := make(chan error, 1)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

guarantees that the reader routine can exit if the main routine is doing something else. Otherwise the write to err might block forever.

@macroscopeapp
Copy link

macroscopeapp bot commented Jun 30, 2025

Separate database worker from sync by extracting envelope storage logic from originatorStream into new EnvelopeSink component

The envelope storage logic is extracted from the originatorStream struct into a new EnvelopeSink component that handles database operations through a queue-based worker pattern. The changes include:

  • Creates new envelopeSink.go with EnvelopeSink struct containing methods for storing envelopes, handling reserved topics, calculating fees, and getting payer IDs
  • Refactors originatorStream.go to remove database-related fields and methods, adding writeQueue field to pass envelopes to EnvelopeSink
  • Updates syncWorker.go to create write queue channels and coordinate between originatorStream and EnvelopeSink components
  • Modifies originatorStream_test.go with new test functions for EnvelopeSink and updated existing tests to work with separated components

📍Where to Start

Start with the subscribeToNode method in syncWorker.go to see how the write queue is created and how the EnvelopeSink and originatorStream components are coordinated.


Macroscope summarized 9804f6b.

@@ -0,0 +1,247 @@
package sync
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other than the struct, all content is copied from the original place without changes.

@mkysel mkysel mentioned this pull request Jun 30, 2025
Copy link
Collaborator

@fbac fbac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Love the separation of concerns.

@mkysel mkysel merged commit 1231f47 into main Jul 1, 2025
9 checks passed
@mkysel mkysel deleted the mkysel/separate-db-from-sync branch July 1, 2025 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants