Skip to content

Conversation

@fbac
Copy link
Collaborator

@fbac fbac commented Sep 19, 2025

Change SelectVectorClock to return gateway_time with latest originator_sequence_id per originator_node_id and initialize originator stream cursors to improve vector clock performance

This updates the vector clock query and stream initialization to use gateway timestamps instead of raw envelopes. - Replace SelectVectorClock SQL with a lateral join that returns originator_node_id, latest originator_sequence_id, and gateway_time, and regenerate query code to scan into GatewayTime time.Time in envelopes.sql.go and update the SQL in envelopes.sql. - Create indexes idx_gateway_envelopes_vector_clock and idx_gateway_envelopes_originator_node_id in 00016_vector_clock.up.sql and drop them in 00016_vector_clock.down.sql. - Introduce a lightweight cursor (sequenceID, timestampNS) and refactor originatorStream to track validation state using the cursor instead of the last envelope in originator_stream.go with matching test adjustments in originator_stream_test.go. - Initialize stream position from SelectVectorClock by constructing a cursor using OriginatorSequenceID and GatewayTime.UnixNano() in syncWorker.setupStream in sync_worker.go.

📍Where to Start

Start with syncWorker.setupStream in sync_worker.go, then review the SelectVectorClock changes in envelopes.sql and the regenerated code in envelopes.sql.go.


Macroscope summarized 8c414bd.

@fbac fbac requested a review from a team as a code owner September 19, 2025 16:44
@graphite-app
Copy link

graphite-app bot commented Sep 19, 2025

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

  • Queue - adds this PR to the back of the merge queue
  • Hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

@fbac fbac changed the title 09 19 performant vector clock Performant vector clock Sep 19, 2025
@fbac fbac changed the title Performant vector clock Enhance vector clock performance Sep 19, 2025
@fbac fbac merged commit c2fc25f into main Sep 19, 2025
11 checks passed
@fbac fbac deleted the 09-19-performant_vector_clock branch September 19, 2025 17:12
Copy link
Contributor

@neekolas neekolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there similar issues with SelectGatewayEnvelopes? That's the one that gets used the most

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants