Add an Admin API endpoint for listing quarantined media by turt2live · Pull Request #19268 · element-hq/synapse

turt2live · 2025-12-02T20:51:35Z

Note: this was written using the copy/paste development technique, so may not adhere to great Python standards.

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

turt2live · 2025-12-02T21:44:32Z

looks like test flakes on not-my changes

anoadragon453 · 2025-12-03T16:57:58Z

The trial and Build release artifacts CI failures look related to your changes, I'm afraid (typically it's only the Complement or Sytest runs that will flake):

[FAIL]
Traceback (most recent call last):
  File "/tmp/tmp.mV4uD1eGTx/tests/rest/admin/test_media.py", line 833, in test_list_quarantined_media
    self.assertEqual(200, channel.code, msg=channel.json_body)
  File "/synapse/build/debian/matrix-synapse-py3/opt/venvs/matrix-synapse/lib/python3.13/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual
    super().assertEqual(first, second, msg)
  File "/usr/lib/python3.13/unittest/case.py", line 907, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.13/unittest/case.py", line 900, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 200 != 401 : {'errcode': 'M_MISSING_TOKEN', 'error': 'Missing access token'}

tests.rest.admin.test_media.ListQuarantinedMediaTestCase.test_list_quarantined_media

and looks due to this line not having an access token specified.

The test runs too fast to really quarantine in order

anoadragon453

Just some minor things. LGTM overall!

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

anoadragon453

LGTM, thanks!

anoadragon453 · 2025-12-12T13:28:19Z

@@ -0,0 +1 @@
+Add an admin API for retrieving a paginated list of quarantined media.


Suggested change

Add an admin API for retrieving a paginated list of quarantined media.

Add a ["Listing all quarantined media" Admin API](https://element-hq.github.io/synapse/latest/admin_api/media_admin_api.html#listing-all-quarantined-media) for retrieving a paginated list of quarantined media.

anoadragon453 · 2025-12-12T13:30:13Z

CI failures are flakes. Merging manually.

https://github.com/element-hq/synapse/pull/19268/changes#r2614217300 wasn't applied and the migration had copy/paste artifacts. PR: #19268

…)" This reverts commit 3f63638.

…)" (#19351) Fixes #19349 This reverts commit 3f63638 (#19268) as the DB migration was taking too long and blocking media access while it happened. See #19349 for further information. ### Pull Request Checklist  * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Travis Ralston <travpc@gmail.com>

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [element-hq/synapse](https://github.com/element-hq/synapse) | minor | `1.145.0` → `1.146.0` | --- > ⚠️ **Warning** > > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>element-hq/synapse (element-hq/synapse)</summary> ### [`v1.146.0`](https://github.com/element-hq/synapse/releases/tag/v1.146.0) [Compare Source](element-hq/synapse@v1.145.0...v1.146.0rc1) ### Synapse 1.146.0 (2026-01-27) No significant changes since 1.146.0rc1. #### Deprecations and Removals - [MSC2697](matrix-org/matrix-spec-proposals#2697) (Dehydrated devices) has been removed, as the MSC is closed. Developers should migrate to [MSC3814](matrix-org/matrix-spec-proposals#3814). ([#19346](element-hq/synapse#19346)) - Support for Ubuntu 25.04 (Plucky Puffin) has been dropped. Synapse no longer builds debian packages for Ubuntu 25.04. ### Synapse 1.146.0rc1 (2026-01-20) #### Features - Add a new config option [`enable_local_media_storage`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#enable_local_media_storage) which controls whether media is additionally stored locally when using configured `media_storage_providers`. Setting this to `false` allows off-site media storage without a local cache. Contributed by Patrice Brend'amour [@dr](https://github.com/dr).allgood. ([#19204](element-hq/synapse#19204)) - Stabilise support for [MSC4312](matrix-org/matrix-spec-proposals#4312 `m.oauth` User-Interactive Auth stage for resetting cross-signing identity with the OAuth 2.0 API. The old, unstable name (`org.matrix.cross_signing_reset`) is now deprecated and will be removed in a future release. ([#19273](element-hq/synapse#19273)) - Refactor Grafana dashboard to use `server_name` label (instead of `instance`). ([#19337](element-hq/synapse#19337)) #### Bugfixes - Fix joining a restricted v12 room locally when no local room creator is present but local users with sufficient power levels are. Contributed by [@nexy7574](https://github.com/nexy7574). ([#19321](element-hq/synapse#19321)) - Fixed parallel calls to `/_matrix/media/v1/create` being ratelimited for appservices even if `rate_limited: false` was set in the registration. Contributed by [@tulir](https://github.com/tulir) @ Beeper. ([#19335](element-hq/synapse#19335)) - Fix a bug introduced in 1.61.0 where a user's membership in a room was accidentally ignored when considering access to historical state events in rooms with the "shared" history visibility. Contributed by Lukas Tautz. ([#19353](element-hq/synapse#19353)) - [MSC4140](matrix-org/matrix-spec-proposals#4140): Store the JSON content of scheduled delayed events as text instead of a byte array. This fixes the inability to schedule a delayed event with non-ASCII characters in its content. ([#19360](element-hq/synapse#19360)) - Always rollback database transactions when retrying (avoid orphaned connections). ([#19372](element-hq/synapse#19372)) - Fix `InFlightGauge` typing to allow upgrading to `prometheus_client` 0.24. ([#19379](element-hq/synapse#19379)) #### Updates to the Docker image - Add [Prometheus HTTP service discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) endpoint for easy discovery of all workers when using the `docker/Dockerfile-workers` image (see the [*Metrics* section of our Docker testing docs](docker/README-testing.md#metrics)). ([#19336](element-hq/synapse#19336)) #### Improved Documentation - Remove docs on legacy metric names (no longer in the codebase since 2022-12-06). ([#19341](element-hq/synapse#19341)) - Clarify how the estimated value of room complexity is calculated internally. ([#19384](element-hq/synapse#19384)) #### Internal Changes - Add an internal `cancel_task` API to the task scheduler. ([#19310](element-hq/synapse#19310)) - Tweak docstrings and signatures of `auth_types_for_event` and `get_catchup_room_event_ids`. ([#19320](element-hq/synapse#19320)) - Replace usage of deprecated `assertEquals` with `assertEqual` in unit test code. ([#19345](element-hq/synapse#19345)) - Drop support for Ubuntu 25.04 'Plucky Puffin', add support for Ubuntu 25.10 'Questing Quokka'. ([#19348](element-hq/synapse#19348)) - Revert "Add an Admin API endpoint for listing quarantined media ([#19268](element-hq/synapse#19268))". ([#19351](element-hq/synapse#19351)) - Bump `mdbook` from 0.4.17 to 0.5.2 and remove our custom table-of-contents plugin in favour of the new default functionality. ([#19356](element-hq/synapse#19356)) - Replace deprecated usage of PyGitHub's `GitRelease.title` with `.name` in release script. ([#19358](element-hq/synapse#19358)) - Update the Element logo in Synapse's README to be an absolute URL, allowing it to render on other sites (such as PyPI). ([#19368](element-hq/synapse#19368)) - Apply minor tweaks to v1.145.0 changelog. ([#19376](element-hq/synapse#19376)) - Update Grafana dashboard syntax to use the latest from importing/exporting with Grafana 12.3.1. ([#19381](element-hq/synapse#19381)) - Warn about skipping reactor metrics when using unknown reactor type. ([#19383](element-hq/synapse#19383)) - Add support for reactor metrics with the `ProxiedReactor` used in worker Complement tests. ([#19385](element-hq/synapse#19385)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://gitea.alexlebens.dev/alexlebens/infrastructure/pulls/3533 Co-authored-by: Renovate Bot <renovate-bot@alexlebens.net> Co-committed-by: Renovate Bot <renovate-bot@alexlebens.net>

Fixes #19352 (See issue for history of this feature and previous PRs) > First, a [naive implementation](#19268) of the endpoint was introduced, but it quickly ran into [performance issues on query](#19312) and [long startup times](#19349), leading to its [removal](#19351). It also didn't actually work, and would fail to expose media when it was "unquarantined", so a [partial fix](#19308) was attempted, where the suggested direction is to use a [stream](https://element-hq.github.io/synapse/latest/development/synapse_architecture/streams.html#cheatsheet-for-creating-a-new-stream) instead of a timestamp column. This PR re-introduces the API building on the previous feedback: * Adds a stream which tracks when media becomes (un)quarantined. * Runs a background update to capture already-quarantined media. * Adds a new admin API to return rows from the stream table. We track both quarantine and unquarantine actions in the stream to allow downstream consumers to process the records appropriately. Namely, to allow our Synapse exchange in HMA to remove hashes for unquarantined media (use case further explained in the [issue](#19352)). **Note**: This knowingly does not capture all cases of media being quarantined. Other call sites are lower priority for T&S, and can be addressed in a future PR. ~~An issue will be created after this PR is merged to track those sites.~~ #19672 ### Pull Request Checklist  * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: turt2live <1190097+turt2live@users.noreply.github.com> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Eric Eastwood <erice@element.io>

turt2live and others added 9 commits December 2, 2025 13:50

Add an Admin API endpoint for listing quarantined media

0de3f91

changelog

2729ace

Attempt to fix linting

e319d41

Merge branch 'develop' into travis/hma-api

4fbba2d

Fix types of media IDs in tests

1dfe742

it helps to slice the right thing

b7bd546

Fix a bunch of type issues

d1a390c

remove import that wasn't supposed to be there

e1aaea4

register the servlet

60ade15

turt2live marked this pull request as ready for review December 2, 2025 21:44

turt2live requested a review from a team as a code owner December 2, 2025 21:44

turt2live and others added 5 commits December 3, 2025 10:21

access tokens are important

b456a8b

Attempt to fix linting

4ad03ab

Merge branch 'develop' into travis/hma-api

29ff87b

Just assert that there's media, rather than the specific media

631bb5d

The test runs too fast to really quarantine in order

Merge remote-tracking branch 'origin/travis/hma-api' into travis/hma-api

9a694fe

anoadragon453 requested changes Dec 11, 2025

View reviewed changes

turt2live and others added 3 commits December 11, 2025 13:27

Apply suggestions from code review

70ac6c0

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

Update docs

d8a85b6

Merge branch 'develop' into travis/hma-api

8ad5673

turt2live requested a review from anoadragon453 December 11, 2025 21:12

anoadragon453 approved these changes Dec 12, 2025

View reviewed changes

anoadragon453 merged commit 3f63638 into develop Dec 12, 2025
45 of 50 checks passed

anoadragon453 deleted the travis/hma-api branch December 12, 2025 13:30

turt2live added a commit that referenced this pull request Dec 16, 2025

Cleanup from previous PR

32ce7a3

https://github.com/element-hq/synapse/pull/19268/changes#r2614217300 wasn't applied and the migration had copy/paste artifacts. PR: #19268

This was referenced Dec 16, 2025

Fix "list quarantined media" API to have semi-stable ordering #19308

Closed

Add indexes for quarantined media lookups #19312

Closed

MadLittleMods added the A-Admin-API label Dec 23, 2025

turt2live mentioned this pull request Jan 6, 2026

DB Delta 04_add_quarantined_ts_to_media.sql is slow and blocks media access while updating #19349

Closed

turt2live added a commit that referenced this pull request Jan 6, 2026

Revert "Add an Admin API endpoint for listing quarantined media (#19268…

f6ed5a4

…)" This reverts commit 3f63638.

MadLittleMods mentioned this pull request Jan 6, 2026

Revert "Add an Admin API endpoint for listing quarantined media (#19268)" #19351

Merged

3 tasks

turt2live mentioned this pull request Jan 6, 2026

Admin API to list quarantined media #19352

Closed

MadLittleMods mentioned this pull request Mar 23, 2026

Add an API to list changes to quarantine state of media #19558

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an Admin API endpoint for listing quarantined media#19268

Add an Admin API endpoint for listing quarantined media#19268
anoadragon453 merged 17 commits into
developfrom
travis/hma-api

turt2live commented Dec 2, 2025 •

edited

Loading

Uh oh!

turt2live commented Dec 2, 2025

Uh oh!

anoadragon453 commented Dec 3, 2025

Uh oh!

anoadragon453 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoadragon453 left a comment

Uh oh!

anoadragon453 Dec 12, 2025

Uh oh!

anoadragon453 commented Dec 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1 @@
		Add an admin API for retrieving a paginated list of quarantined media. No newline at end of file

	Add an admin API for retrieving a paginated list of quarantined media.
	Add a ["Listing all quarantined media" Admin API](https://element-hq.github.io/synapse/latest/admin_api/media_admin_api.html#listing-all-quarantined-media) for retrieving a paginated list of quarantined media.

Conversation

turt2live commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

turt2live commented Dec 2, 2025

Uh oh!

anoadragon453 commented Dec 3, 2025

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

anoadragon453 Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

anoadragon453 commented Dec 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

turt2live commented Dec 2, 2025 •

edited

Loading