Skip to content

feat(dev/logs/metrics): Add sentry.timestamp.sequence definition to the logs and metrics specs#16492

Merged
logaretm merged 13 commits intomasterfrom
awad/add-log-sequence-attr
Mar 4, 2026
Merged

feat(dev/logs/metrics): Add sentry.timestamp.sequence definition to the logs and metrics specs#16492
logaretm merged 13 commits intomasterfrom
awad/add-log-sequence-attr

Conversation

@logaretm
Copy link
Member

@logaretm logaretm commented Feb 20, 2026

This pull request updates the telemetry logs specification to introduce deterministic log ordering via a new attribute, sentry.timestamp.sequence. This change addresses issues in environments where multiple logs can share identical timestamps, ensuring logs can always be ordered correctly.

The attribute addresses an issue that came up in Cloudflare’s runtime which freezes all timers and clock functions, causing all timestamp calls to return the same value. This can lead to logs being displayed out of order if they don’t get sent in one request. Instead, they’ll appear in the order of arrival in ingest, which is subject to request latencies and out-of-order request completion.

This is a proposal as we have yet to see what product thinks about this.

@vercel
Copy link

vercel bot commented Feb 20, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
develop-docs Ready Ready Preview, Comment Mar 4, 2026 2:24pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
sentry-docs Ignored Ignored Preview Mar 4, 2026 2:24pm

Request Review

@logaretm logaretm changed the title feat: Add sentry.log.sequence definition to the logs' spec feat(dev/logs): Add sentry.log.sequence definition to the logs' spec Feb 20, 2026
Comment on lines +309 to +310
The sequence number **MUST NOT**:
- Be intentionally reset during the process lifetime.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You only need it as a discriminator per timestamp, so maybe that's enough as a constraint?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be a rare scenario, but If it does get reset during the lifetime of the SDK process then on environments like Cloudflare it will produce the same timestamps again since the time is frozen throughout their lifetime, so it won't be unique anymore. So, I wanted to close that door.

What do you think?

"sentry.message.parameter.1": { "value": 85, "type": "integer" },
"sentry.message.parameter.2": { "value": 60, "type": "integer" }
"sentry.message.parameter.2": { "value": 60, "type": "integer" },
"sentry.log.sequence": { "value": 0, "type": "integer" }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should be top-level, it feels heavily tied to the timestamp and a fundamental attribute of a log.

@k-fish wdyt?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I namespaced it initially to signal to the user that they should not mess with this (they shouldn't be able anyways).

But yea I agree, it is tied to the logs timestamps heavily and has little to no-use outside of that.


The sequence number **MUST NOT**:
- Be intentionally reset during the process lifetime.
- Be used to modify or synthesize sub-millisecond timestamp precision.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? Adding a separate attribute means to do log ordering naturally we'll now have to either:

A. Include yet another column on every request in order to have logs ordered properly when a users sorts by time (log.sequence), we already are using timestamp and precise timestamp when doing log ordering. Adding a column adds additional query bytes to every request.
B. NOT sort properly by this monotonic counter, and have it as a free floating attribute that a user will have to know about and then sort by independently, with a count from beginning of the lifetime being hard to intuit.

Is it possible we send it and have relay transform the precise timestamp into a synthesized version where we add an annotation (meta) to indicate the log sequence value it was modified by and why (because their sdk doesn't support precise timestamps). That way we can put a warning in the timestamp tooltip that explains it, and encode the sequence into the precise timestamp where we have an extra 32 bits being essentially unused at millisecond precision? Thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible we send it and have relay transform the precise timestamp into a synthesized version where we add an annotation (meta) to indicate the log sequence value it was modified by...

Nar suggested something similar about doing the synthesizing on the SDK level. I included that part to avoid lying to the user about the actual time. I think your suggestion is more transparent so we could do that, but it feels like it involves more changes downstream to the UI.

I have a few thoughts on this:

  1. We can't dynamically tell if the clock is frozen or not so we can't synthesize conditionally unless we send in another flag to tell relay to do that, but then again users cross-deploy a lot and they can have a node SDK running on Cloudflare, so the synthesizing would need to happen all the time. I initially wanted us to use this for the off chance some SDK got its clock frozen for whatever reason.
  2. We could skew the ms value but with 32 bit of space, it may not grow big enough to overflow into seconds range.

I would've liked for this to be completely transparent to the user, How considerable is the cost of adding an secondary sort integer to our queries?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked with Kevan offline and we figured a good compromise is to do a reset of the counter whenever we detect that the clock did advance, this would keep the sequence numbers small for most cases which means it won't grow larger enough to skew the numbers.

We also will monitor how large the sequence numbers grows to see if we should modify our approach.

Copy link
Member

@k-fish k-fish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General +1 that we need a monotonic counter if we can't get a more precise timestamp


<SpecSection id="log-ordering" status="candidate" since="1.16.0">

Some runtimes (notably Cloudflare Workers) freeze timer APIs (`Date.now()`, `performance.now()`) during synchronous execution, causing multiple logs to share identical timestamps. To guarantee deterministic ordering, SDKs **MUST** attach a `sentry.log.sequence` integer attribute to every log.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: sentry.log.sequence it actually is probably necessary for our other telemetry too, at least for metrics I know it could be a problem.

imo something like sentry.timestamp.sequence makes more sense to me, my $0.02

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the clockfreeze thing on cloudflare does affect all of our telemetry.

  • Traces and spans always display 0ms since startTime and endTime are always the same, the counter wouldn't help with that tho.
  • Metrics are probably affected, but I haven't been exposed to them yet. I can explore further there if we want to make this a general telemetry attribute.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked with Kevan offline and we will do the same thing for metrics, so the attribute name will change to sentry.timestamp.sequence

@logaretm logaretm force-pushed the awad/add-log-sequence-attr branch from 7a3e04f to fbfa09e Compare February 24, 2026 17:25
@logaretm logaretm changed the title feat(dev/logs): Add sentry.log.sequence definition to the logs' spec feat(dev/logs): Add sentry.timestamp.sequence definition to the logs and metrics specs Feb 24, 2026
@logaretm logaretm changed the title feat(dev/logs): Add sentry.timestamp.sequence definition to the logs and metrics specs feat(dev/logs/metrics): Add sentry.timestamp.sequence definition to the logs and metrics specs Feb 24, 2026
@logaretm logaretm requested review from Dav1dde and k-fish February 24, 2026 20:13
Copy link
Member

@k-fish k-fish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add one final piece about sequence not being sent if not required (the sdk has better precision than 1ms or non frozen time) then 👍 from me

@logaretm logaretm force-pushed the awad/add-log-sequence-attr branch from 60b43dc to 578eb4d Compare February 25, 2026 17:29
@logaretm logaretm marked this pull request as ready for review February 25, 2026 18:57
Copilot AI review requested due to automatic review settings February 25, 2026 18:57
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the SDK telemetry logs and metrics specifications to add a new attribute, sentry.timestamp.sequence, intended to enable deterministic ordering when multiple items share identical timestamps (e.g., in runtimes with frozen timers).

Changes:

  • Bumped spec versions and added changelog entries for introducing sentry.timestamp.sequence.
  • Added sentry.timestamp.sequence to the “default attributes” requirements and to envelope examples.
  • Introduced new “Log Ordering” / “Metric Ordering” sections describing how the sequence counter behaves.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.

File Description
develop-docs/sdk/telemetry/metrics.mdx Adds sentry.timestamp.sequence to default metric attributes, documents metric ordering rules, updates examples, and bumps spec version/changelog.
develop-docs/sdk/telemetry/logs.mdx Adds sentry.timestamp.sequence to default log attributes, documents log ordering rules, updates examples, and bumps spec version/changelog.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

version: ">=1.0.0"
spec_changelog:
- version: 2.6.0
date: 2026-02-06
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the changelog, version 2.6.0 is dated 2026-02-06 but the previous 2.5.0 entry is dated 2026-02-12. This makes the timeline inconsistent with the version ordering; update the 2.6.0 date (or the 2.5.0 date) so newer versions don’t appear to precede older ones.

Suggested change
date: 2026-02-06
date: 2026-02-19

Copilot uses AI. Check for mistakes.
Comment on lines 588 to +592
"endpoint": { "value": "/api/users", "type": "string" },
"method": { "value": "POST", "type": "string" },
"sentry.sdk.name": { "value": "sentry.javascript.browser", "type": "string" },
"sentry.sdk.version": { "value": "10.17.0", "type": "string" }
"sentry.sdk.version": { "value": "10.17.0", "type": "string" },
"sentry.timestamp.sequence": { "value": 1, "type": "integer" }
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this full envelope example, the first metric item includes sentry.environment and sentry.release, but this item omits them even though they’re listed as default attributes (when defined) and would normally be consistent across metrics from the same SDK instance. Consider adding them here too, or removing them from the earlier item, so the example matches the spec rules.

Copilot uses AI. Check for mistakes.
@logaretm
Copy link
Member Author

@Dav1dde and @cleptric, After going through it with Kevan, this is what we have right now. Any thoughts?

@Dav1dde
Copy link
Member

Dav1dde commented Feb 26, 2026

👍 from me

Copy link
Contributor

@coolguyzone coolguyzone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding!

logaretm and others added 2 commits February 27, 2026 17:00
Co-authored-by: Alex Krawiec <alex.krawiec@sentry.io>
Co-authored-by: Alex Krawiec <alex.krawiec@sentry.io>
@linear
Copy link

linear bot commented Mar 3, 2026

Copy link
Member

@cleptric cleptric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe worth addressing the clanker comments, too.

logaretm and others added 3 commits March 4, 2026 15:12
Co-authored-by: Michi Hoffmann <cleptric@users.noreply.github.com>
Co-authored-by: Michi Hoffmann <cleptric@users.noreply.github.com>
@logaretm logaretm merged commit 8c89c8e into master Mar 4, 2026
17 checks passed
@logaretm logaretm deleted the awad/add-log-sequence-attr branch March 4, 2026 14:53
@github-actions github-actions bot locked and limited conversation to collaborators Mar 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants