Fix performance regression related to delayed events processing by anoadragon453 · Pull Request #18926 · element-hq/synapse

anoadragon453 · 2025-09-17T12:41:32Z

Hopefully fixes #18925.

Carries out a number of optimisations to this bit of the codebase (note that on element.io we have about 7mil state deltas to process):

Replace a get_event DB query per state delta with a single, smaller query for all state deltas.
Bail out early if there are no delayed events in the DB to update.
~~Set the next "scheduled at" time to the minimum of all delayed events once, instead of doing so for every delta.~~ this ended up causing tests to fail, and has been reverted.

Highly recommended to review commit-by-commit.

MSC4140

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

This is preferable to pulling out entire events and only fetching the `sender` field.

We also only call it once for all events in the batch, instead of per state delta.

There's no need to do this for every event in the batch. Just find the min time for all delayed events.

There are zero delayed events on matrix.org's database. We can save a lot of time pulling out state deltas by first just checking if there are any delayed events to actually deal with.

anoadragon453 · 2025-09-17T13:15:54Z

Initial theory is that the delayed_state_events_are_sent_on_timeout Complement failure is related to 9f1c7e3.

MadLittleMods

I haven't look at whether this is a good solution holistically with the delayed events feature. But the change itself looks like an improvement to what was there before.

This reverts commit 9f1c7e3. This change caused delayed events to not be scheduled, thus in turn causing the `TestDelayedEvents/delayed_.*_events_are_sent_on_timeout` tests to fail. I don't fully understand why my change didn't work, but have spent enough time looking at the code. The optimisation was very slight anyhow, so I don't think it's worth persuing for longer.

Apply the same fix to a similar loop in the presence handling code.

If a room has been purged, then the associated event would be gone as well. Thus we wouldn't be able to pull out the sender. This is different from the `sender` field being null on an event.

anoadragon453 · 2025-09-22T13:29:46Z

+            if sender_str is sentinel:
+                # This can happen if a room is purged. State deltas related to
+                # the room are left behind, but the event/room no longer exist.
+                logger.warning(
+                    "Skipping state delta with event ID '%s' as it is an unknown event - the room may have been purged",
+                    delta.event_id,
+                )
+                continue


This was added after an exception was found when running an earlier draft of this PR on element.io.

anoadragon453 · 2025-09-23T08:46:34Z

CI failures are due to flakes.

anoadragon453 added 5 commits September 17, 2025 13:35

Add a storage function to pull out senders for event _ids

fadb7eb

This is preferable to pulling out entire events and only fetching the `sender` field.

Stop calling get_event and use new storage function

e3ad18c

We also only call it once for all events in the batch, instead of per state delta.

Only schedule next delayed event once

9f1c7e3

There's no need to do this for every event in the batch. Just find the min time for all delayed events.

Bail out early if there are no delayed events to send

76b885b

There are zero delayed events on matrix.org's database. We can save a lot of time pulling out state deltas by first just checking if there are any delayed events to actually deal with.

newsfile

a0faac9

anoadragon453 marked this pull request as ready for review September 17, 2025 12:42

anoadragon453 requested a review from a team as a code owner September 17, 2025 12:42

anoadragon453 changed the title ~~Add a storage function to pull out senders for event _ids~~ Fix performance regression related to delayed events processing Sep 17, 2025

MadLittleMods reviewed Sep 17, 2025

View reviewed changes

Comment thread synapse/handlers/delayed_events.py Outdated

Comment thread synapse/handlers/delayed_events.py

Comment thread synapse/handlers/delayed_events.py

Comment thread changelog.d/18926.bugfix Outdated

anoadragon453 added 3 commits September 18, 2025 11:46

Note 100 row limit in docstrings

8d97548

Note that order of fetching stream orderings is important

d1974a2

link MSC4140 in newsfile

4ff8184

anoadragon453 force-pushed the anoa/delayed_event_perf_regression branch from 13fd4bd to 4ff8184 Compare September 18, 2025 10:46

More error handling

68fb552

anoadragon453 requested a review from MadLittleMods September 18, 2025 14:00

anoadragon453 force-pushed the anoa/delayed_event_perf_regression branch from 1500d26 to 1eb25f1 Compare September 18, 2025 14:14

MadLittleMods reviewed Sep 18, 2025

View reviewed changes

Comment thread synapse/handlers/delayed_events.py Outdated

Comment thread synapse/handlers/delayed_events.py

MadLittleMods reviewed Sep 18, 2025

View reviewed changes

Comment thread synapse/handlers/delayed_events.py

anoadragon453 added 5 commits September 19, 2025 11:17

Prevent potential infinite loop if we bypass the max

ef38b2e

Apply the same fix to a similar loop in the presence handling code.

logger.error for missing sender

a67b036

state reset TODO

b4877a6

Add a test for get_senders_for_event_ids

6eaf31e

Handle the case where a room has been purged

3d44c99

If a room has been purged, then the associated event would be gone as well. Thus we wouldn't be able to pull out the sender. This is different from the `sender` field being null on an event.

anoadragon453 commented Sep 22, 2025

View reviewed changes

Update logging and TODO to discuss state resets

af7962d

MadLittleMods reviewed Sep 22, 2025

View reviewed changes

Comment thread synapse/handlers/delayed_events.py Outdated

MadLittleMods reviewed Sep 22, 2025

View reviewed changes

Comment thread synapse/handlers/delayed_events.py Outdated

MadLittleMods added the X-Release-Blocker label Sep 22, 2025

anoadragon453 added 3 commits September 22, 2025 18:31

Move Sentinel to util module

17d2845

Replace ad-hoc Sentinel usage with utility module

c732bba

Update logs and comments to better cover cases

8c427f2

MadLittleMods approved these changes Sep 22, 2025

View reviewed changes

anoadragon453 merged commit ddc7627 into develop Sep 23, 2025
42 of 44 checks passed

anoadragon453 deleted the anoa/delayed_event_perf_regression branch September 23, 2025 08:47

anoadragon453 mentioned this pull request Sep 23, 2025

Fix case where get_partial_current_state_deltas could return >100 rows #18960

Merged

3 tasks

anoadragon453 added a commit that referenced this pull request Sep 24, 2025

Fix performance regression related to delayed events processing (#18926)

0c8594c

MadLittleMods mentioned this pull request Sep 24, 2025

very slow sync with element-web 1.11.110 since upgrading synapse to v1.138.0 #18912

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix performance regression related to delayed events processing#18926

Fix performance regression related to delayed events processing#18926
anoadragon453 merged 19 commits into
developfrom
anoa/delayed_event_perf_regression

anoadragon453 commented Sep 17, 2025 •

edited by MadLittleMods

Loading

Uh oh!

anoadragon453 commented Sep 17, 2025

Uh oh!

MadLittleMods left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoadragon453 Sep 22, 2025

Uh oh!

Uh oh!

Uh oh!

anoadragon453 commented Sep 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

anoadragon453 commented Sep 17, 2025 • edited by MadLittleMods Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

anoadragon453 commented Sep 17, 2025

Uh oh!

MadLittleMods left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoadragon453 Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

anoadragon453 commented Sep 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

anoadragon453 commented Sep 17, 2025 •

edited by MadLittleMods

Loading

MadLittleMods left a comment •

edited

Loading