Releases

Synapse 1.39.0 is now available!

We've done quite a bit of work this release on Synapse's new interface for extension modules:

• Synapse's account_validity option has been extracted into a built-in extension module. If your configuration previously enabled account_validity, it will continue working as before, but it is now easier to override and customize.
• The third party event rules callbacks have also been experimentally ported to the new module interface.

We've also improved the Space Summary API to list all rooms which a user could join (#10298). This is especially important with the pending standardization of Room Version 8, which includes MSC3083: Restricting room membership based on membership in other rooms.

MadLittleMods landed several pull requests on the path to implementing MSC2716: Incrementally importing history into existing rooms (#10250, #10276).

More than anything else, this release contains an absolutely enormous load of internal cleanup. For example:

• ShadowJonathan ran pyupgrade on the entire codebase, bringing it up to modern Py3.6+ conventions, then separately used com2ann to move us from legacy # type: foo type hints to more modern inline annotations.
• We finished converting several integer columns in our database to bigint as a precautionary measure.
• We've made innumerable improvements to our CI configuration; including moving completely to GitHub Actions.
• Countless small improvements to performance, reliability, and error logging.

But that's not all! Synapse 1.39 also allows for setting credentials for HTTP proxy connections thanks to work by dklimpel; previously, Synapse was only able to provide authentication for HTTPS proxies.

Please see the Upgrade Notes and Release Notes for a complete list of changes in this rele ase.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including andir, dklimpel, ilmari, LukeWalsh, moritzdietz, ShadowJonathan, and xmunoz.

It's only been two weeks since Dendrite 0.4 landed, but there's already a significant new release with Dendrite 0.4.1 (it's amazing how much work we can do on Dendrite when not off chasing low-bandwidth and P2P Matrix!)

This release further improves memory performance and radically improves state resolution performance (rumour has it that it's a 10x speed-up). Meanwhile, SS API sytest coverage is up to 91%(!!) and CS API is now at 63%.

We're going to try to keep the pressure up over the coming weeks - and once sytest is at 100% coverage (and we're not missing any big features which sytest doesn't cover yet) we'll be declaring a 1.0 :)

If you're running Dendrite, please upgrade. If not, perhaps this would be a good version to give it a try? You can get it, as always from, https://github.com/matrix-org/dendrite/releases/tag/v0.4.1. The changelog follows:

Features

• Support for room version 7 has been added
• Key notary support is now more complete, allowing Dendrite to be used as a notary server for looking up signing keys
• State resolution v2 performance has been optimised further by caching the create event, power levels and join rules in memory instead of parsing them repeatedly
• The media API now handles cases where the maximum file size is configured to be less than 0 for unlimited size
• The initial_state in a /createRoom request is now respected when creating a room
• Code paths for checking if servers are joined to rooms have been optimised significantly

Fixes

• A bug resulting in cannot xref null state block with snapshot during the new state storage migration has been fixed
• Invites are now retired correctly when rejecting an invite from a remote server which is no longer reachable
• The DNS cache cache_lifetime option is now handled correctly (contributed by S7evinK)
• Invalid events in a room join response are now dropped correctly, rather than failing the entire join
• The prev_state of an event will no longer be populated incorrectly to the state of the current event
• Receiving an invite to an unsupported room version will now correctly return the M_UNSUPPORTED_ROOM_VERSION error code instead of M_BAD_JSON (contributed by meenal06)

-- Team Dendrite

Synapse 1.38.0 is out now!

NOTE: We released Synapse 1.38.1 on Thursday, July 22nd. It mitigates a client bug with Synapse 1.38.0's smaller sync responses which prevented new Element Android sessions from successfully participating in encrypted conversations. Server administrators are strongly encouraged to upgrade.

(Big) Integers

Synapse's database schema used integer columns in a few places where values could potentially overflow a maximum value of 2³¹. One such column is events.stream_ordering, which surpassed 2³¹ on matrix.org last week.

To prevent overflows, Synapse 1.38 will automatically convert several integer columns to bigint as a background update. While homeservers will function normally during this task, it could result in increased disk I/O for several hours or days. Note that homeservers may need several gigabytes of free space to successfully rebuild associated database indexes and complete the upgrade.

See the upgrade notes for more details.

Expiring Caches

Synapse has a new configuration option, caches.expiry_time, which can be set to enable evicting items from caches if they go too long without being accessed. This helps servers reclaim memory used by large yet infrequently used caches.

Smaller Sync Responses

The response to /sync now omits optional keys when they would otherwise be empty. This can significantly reduce the size of incremental syncs, as demonstrated in #6579. Thanks to deepbluev7 for initially submitting this in #9919, which made it into this release via #10214.

Everything Else

A few other items worth calling out:

• This release includes an experimental implementation of MSC2918: Refresh tokens, which adds initial support for complementary access / refresh tokens in line with OAuth best practices (#9450).
• Synapse now ships a script to review recently registered accounts, which can be useful in cleaning up servers in the wake of malicious, automated registrations like we witnessed during last month's spam attack.
• We've also fixed a few rough edges (#10263, #10303, #10336) in the spam mitigations from 1.37.1, and would encourage you to update.
• The Admin API for querying user information now includes information about a user's SSO identities in its response.

These are just the highlights; please see the Upgrade Notes and Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including deepbluev7, dklimpel, fkr, and sideshowbarker

After quite a significant gap between releases — version 0.3.11 was released at the beginning of March — we're happy to finally announce the release of Dendrite 0.4.0 today!

The full changelog for the release is available on GitHub, but we wanted to take the opportunity to talk a little about some of the changes that have gone into this release.

Recently our release cadence for Dendrite has slowed as we have spent more time within the team working on Pinecone and Low Bandwidth Matrix. These are major areas of research for us which we hope will unlock a number of new opportunities within the Matrix ecosystem, allowing us to build on Matrix anywhere and to reduce the protocol-level footprint. However, Dendrite has not been forgotten amidst the excitement and we will be spending more time working on Dendrite again in the coming months.

State storage

One of the major features in v0.4.0 is that we've introduced newly-refactored state storage in the roomserver database. The goal here is to make state storage significantly more efficient by ensuring that we deduplicate state blocks and snapshots wherever we can. By ensuring that all state blocks and snapshots are ordered strictly, and by enforcing uniqueness constraints on the hashes of the blocks/snapshots, we've been able to achieve this.

This was largely spurred on by watching dendrite.matrix.org consuming a rather alarming amount of disk space on a daily basis. In this particular instance, moving to the new state storage resulted in a 15x improvement on disk utilisation for state blocks and a further 2x improvement for state snapshot references immediately after the migration, and the growth rate of the database has slowed substantially since.

Ensuring that we don't waste disk space is one of the most important factors in ensuring that Dendrite operates well at any scale — future datacentre deployments supporting many users will find storage overheads decreased and small/embedded single-user deployments (such as P2P, on mobile devices or in the browser) will fit much more effectively onto resource-constrained targets.

After upgrading to v0.4.0, Dendrite will run an automatic migration to update your homeserver to the new state storage format. This might take a while on larger databases so please expect some downtime.

Optimisations

We've continued to squeeze further optimisations into the federation and state resolution code, aiming to reduce the amount of CPU burn and memory utilisation. Some of the feedback that we receive most often from those that have been experimenting with the Dendrite betas is around the sudden spikes in resource usage, especially when joined to large rooms.

The bulk of this resource usage comes either from attempting to reconcile missing events or running state resolution in rooms with lots of members, as potentially large state sets of events need to be brought into memory in order to do so. We've introduced some transaction-level caches for dealing with missing auth/prev events to reduce the memory pressure and we've also tweaked the caching around around /get_missing_events to ensure we don't duplicate any state events in memory.

Resource spikes aren't completely eliminated but this should smooth out CPU and memory utilisation significantly. In the case of dendrite.matrix.org, which is joined to some 6500 rooms at present, memory utilisation of the Dendrite process typically sits around 1.5GB at present.

State Resolution v2 has also seen further optimisations in the power-level checking, which should reduce CPU usage even more.

Bridges

Thanks to Half-Shot's perseverance and contributions, we've merged a couple PRs and worked on some further fixes for getting Application Services working correctly in Dendrite. Whilst not entirely feature-complete and with a number of features still to go, enough support is now present to support basic bridging functionality.

We've done quite a bit of preliminary testing with matrix-appservice-irc and have also heard a number of success stories from the community with mautrix-whatsapp and mautrix-telegram. Others may work too — let us know what you find!

Bug-hunting

A number of bugs in various places (including the roomserver, federation API and media API) which could cause Dendrite to crash have also been fixed. Some of these have been contributed by the community in pull requests, so we extend our thanks to anyone who has submitted a fix to the project.

A special mention also goes to Jakob Varmose Bentzen for reporting a security issue to us around the legacy /v1/register endpoint, where a flaw in the legacy shared secret registration allowed malicious users to create accounts. We've since removed this legacy endpoint and the vulnerability is now fixed.

What's next

There are still a number of missing user-facing features which we will be working on over the coming months, as well as some architectural issues that we will look to address.

A notable area of work involves attempting to remove the dependency on Kafka for polylith deployments. Kafka is very resource-heavy in operation and somewhat limits us to the types of interactions that we can perform between components. It's also very difficult to manage retention correctly, in the interests of not endlessly consuming disk space here either.

As usual, Dendrite is still considered beta so you may not want to rely on it for production systems, although it should be stable enough to experiment with. If you find any bugs or anything that doesn't look right, please let us know:

• #dendrite:matrix.org
• GitHub Issues

We're also open to contributions, so don't be afraid to open pull requests. Finally, thanks for your continued support!

— Team Dendrite

Hi all,

Over the last few days we've seen a distributed spam attack across the public Matrix network, where large numbers of spambots have been registered across servers with open registration and then used to flood abusive traffic into rooms such as Matrix HQ.

The spam itself has been handled by temporarily banning the abused servers. However, on Monday and Tuesday the volume of traffic triggered performance problems for the homeservers participating in targeted rooms (e.g. memory explosions, or very delayed federation). This was due to a combination of factors, but one of the most important ones was Synapse issue #9490: that one busy room could cause head-of-line blocking, starving your server from processing events in other rooms, causing all traffic to fall behind.

We're happy to say that Synapse 1.37.1 fixes this and we now process inbound federation traffic asynchronously, ensuring that one busy room won't impact others. First impressions are that this has significantly improved federation performance and end-to-end encryption stability — for instance, new E2EE keys from remote users for a given conversation should arrive immediately rather than being blocked behind other traffic.

Please upgrade to Synapse 1.37.1 as soon as possible, in order to increase resilience to any other traffic spikes.

Also, we highly recommend that you disable open registration or, if you keep it enabled, use SSO or require email validation to avoid abusive signups. Empirically adding a CAPTCHA is not enough. Otherwise you may find your server blocked all over the place if it is hosting spambots.

Finally, if your server has open registration, PLEASE check whether spambots have been registered on your server, and deactivate them. Once deactivated, you will need to contact [email protected] to request that blocks on your server are removed.

Your best bet for spotting and neutralising dormant spambots is to review signups on your homeserver over the past 3-5 days and deactivate suspicious users. We do not recommend relying solely on lists of suspicious IP addresses for this task, as the distributed nature of the attack means any such list is likely to be incomplete or include shared proxies which may also catch legitimate users.

To ease review, we're working on an auditing script in #10290; feedback on whether this is useful would be appreciated. Problematic accounts can then be dealt with using the Deactivate Account Admin API.

Meanwhile, over to Dan for the Synapse 1.37 release notes.

Synapse 1.37 Release Announcement

Synapse 1.37 is now available!

**Note: ** The legacy APIs for Spam Checker extension modules are now considered deprecated and targeted for removal in August. Please see the module docs for information on updating.

This release also removes Synapse's built-in support for the obsolete ACMEv1 protocol for automatically obtaining TLS certificates. Server administrators should place Synapse behind a reverse proxy for TLS termination, or switch to a standalone ACMEv2 client like certbot.

Knock, knock?

After nearly 18 months and 129 commits, Synapse now includes support for MSC2403: Add "knock" feature and Room Version 7! This feature allows users to directly request admittance to private rooms, without having to track down an invitation out-of-band. One caveat: Though the server-side foundation is there, knocking is not yet implemented in clients.

A Unified Interface for Extension Modules

Third party modules can customize Synapse's behavior, implementing things like bespoke media storage providers or user event filters. However, Synapse previously lacked a unified means of enumerating and configuring third-party modules. That changes with Synapse 1.37, which introduces a new, generic interface for extensions.

This new interface consolidates configuration into one place, allowing for more flexibility and granularity by explicitly registering callbacks with specific hooks. You can learn more about the new module API in the docs linked above, or in Matrix Live S6E29, due out this Friday, July 2nd.

Safer Reauthentication

User-interactive authentication ("UIA") is required for potentially dangerous actions like removing devices or uploading cross-signing keys. However, Synapse can optionally be configured to provide a brief grace period such that users are not prompted to re-authenticate on actions taken shortly after logging in or otherwise authenticating.

This improves user experience, but also creates risks for clients which rely on UIA as a guard against actions like account deactivation. Synapse 1.37 protects users by exempting especially risky actions from the grace period. See #10184 for details.

Smaller Improvements

We've landed a number of smaller improvements which, together, make Synapse more responsive and reliable. We now:

• More efficiently respond to key requests, preventing excessive load (#10221, #10144)
• Render docs for each vX.Y Synapse release, starting with v1.37 (#10198)
• Ensure that log entries from failures during early startup are not lost (#10191)
• Have a notion of database schema "compatibility versions", allowing for more graceful upgrades and downgrades of Synapse (docs)

We've also resolved two bugs which could cause sync requests to immediately return with empty payloads (#8518), producing a tight loop of repeated network requests.

Everything Else

Lastly, we've merged an experimental implementation of MSC2716: Incrementally importing history into existing rooms (#9247) as part of Element's work to fully integrate Gitter into Matrix.

These are just the highlights; please see the Upgrade Information and Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including aaronraimist, Bubu, dklimpel, jkanefendt, lukaslihotzki, mikure, and Sorunome,

Synapse 1.36.0 is out, and it's a big one!

Room Join Memory Improvements

We did it! Synapse no longer experiences a memory spike when joining large / complex rooms.

These improvements mainly arise from processing join responses incrementally, rather than trying to load everything into memory at once. However, realizing these gains involved a fair bit of rewriting, as the entire processing pipeline had to work incrementally, and with appropriately sized batches, to avoid downstream bottlenecks. You can hear more about our original plans for this work in last month's Matrix Live: S6E23 — Dan and Erik talk about Synapse.

Presence Improvements

Running presence on a single worker process is now expected to work correctly. This feature first debuted in Synapse 1.33, but a few bugs cropped up which could lead to presence state becoming outdated. With #10149 merged, we believe the last of these issues to be resolved.

We had also noticed a recent increase in presence load on federation workers; this was ultimately tracked to two bugs, both fixed in this release: We were processing local presence via federation workers (#10163) and we were occasionally sending duplicate presence updates (#10165).

With both issues fixed, outgoing federation load has returned to normal levels:

(Thank you to David Mehren for this graph from issue #10153)

Everything Else

Synapse now has two new Admin APIs for unprotecting and removing media from quarantine, thanks to contributions by dklimpel.

Synapse now implements the stable /_matrix/client/r0/rooms/{roomId}/aliases endpoint originally introduced by MSC2432, and, thanks to contributions by govynnus, makes the reason and score fields of event reports optional per MSC2414.

These are just the highlights; please see the Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including 14mRh4X0r, aaronraimist, bradtgmurray, crcastle, dklimpel, govynnus, and RhnSharma.

Synapse 1.35.0 is out! This release focused on improving internals as we drive toward better memory performance during room joins, but more on that below.

Update: Synapse 1.35.1 was published on Thursday, June 3rd. It resolves a bug (#10109) which mistakenly listed invite-only rooms in the Spaces summary.

We'd also like to call the attention of client developers to a deprecation: The unstable prefixes used during development of MSC2858: Multiple SSO Identity Providers will be removed from Synapse 1.38, due out in August. Please ensure your client supports the stable identifiers for this feature.

Spaces: On by Default

Following the successful release of Synapse 1.34, the experimental Spaces flag is now enabled by default. If you had manually enabled the experimental_features: { spaces_enabled: true } flag in your homeserver configuration, you may now remove it.

Bug Squashing

This release of Synapse fixes an issue which could cause federated room joins to fail when the join response exceeded a size limit which was too low (#10082). We've also improved what Synapse logs when it drops a connection in similar circumstances (#10091), which should aid diagnosis if a similar issue were to arise in the future.

GitHub user thermaq contributed a fix (#10014) for a bug which could cause user presence state to become stale.

Lastly our OpenTracing support now allows for profiling end-to-end performance on a per-user basis (#9978).

An Update on Room Joins

We've been hammering away at shrinking Synapse's memory footprint when joining large / complex rooms, and while we're not there yet, the end is in sight! In particular, this release includes many internal refactorings, including using ijson to parse the JSON response to /send_join (#9958), clearing the way for substantial improvements.

Memory usage still spikes because we're effectively doing the same work with a different library, but ijson's design allows for iterative parsing. This will pay dividends once we modify the code downstream of /send_join to take advantage of it.

Concretely, Erik Johnston has an experimental branch of Synapse which completely eliminates the memory spike:

The remaining work is centered on splitting that branch into self-contained, reviewable pull requests, like a rewrite of the Synapse Keyring class (#10035). After that's merged, we'll need to make one further change to properly batch up work, at which point we should attain the efficiency gains from Erik's experiment.

Everything Else

GitHub user savyajha contributed a security hardened systemd unit file which effectively sandboxes Synapse (#9803). While not enabled by default, we'd encourage security conscious users to review the example file and associated documentation.

Please see the Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including dklimpel, jerinjtitus, junquera, lonyeon, savyajha, and thermaq.

Synapse 1.34.0 is now available, and it's loaded with new features and performance improvements.

Note: This release deprecates and replaces the room_invite_state_types configuration option. If you've customized that for your homeserver, please review the Upgrade Notes.

We've also marked the v1 room deletion Admin API as deprecated. Instead of sending a POST to a path ending in /delete, administrators are encourage to instead send an HTTP DELETE to /_synapse/admin/v1/rooms/<room_id>. Thanks to ThibF for implementing this (#9889).

Spaces

The highlight of this release is support for Spaces, now that MSC1772: Matrix Spaces has merged into the Matrix spec!

Synapse also has support for MSC2946: Spaces Summary and MSC3083: Restricting room membership based on space membership, but these are off by default as they're still under development. To enable these experimental MSCs, set experimental_features: { spaces_enabled: true } in your homeserver configuration. These are enabled on the matrix.org homeserver, and we encourage you to experiment with Spaces there and let us know in the Spaces Feedback Room if you encounter any issues.

Memory and Caching

Memory consumption and caching have been a major focus of the Synapse team this quarter, and we've made significant strides:

• Synapse has a new gc_min_interval configuration option with reasonable defaults to prevent Python's garbage collector from running too frequently and thrashing when a large homeserver has its collection thresholds set too low.
• Synapse will report memory allocation stats to Prometheus when using jemalloc.
• Synapse will also measure Redis cache response times and report those to Prometheus.
• For debugging, Synapse can optionally track the memory use of each LruCache.

We have a few more tricks up our sleeves; to learn more about how we're planning to improve the memory cost of joining large rooms, check out last week's Matrix Live.

Other Fixes and Improvements

We've also landed significant improvements to:

• Sending events when Redis is available (#9905, #9950, #9951)
• Joining large rooms when presence is enabled (#9910, #9916)
• Backfilling history in large rooms (#9935)

...and fixed bugs that:

• Prevented cross-account m.room_key_request messages from being delivered (#9961, #9965)
• Incorrectly applied room creation / invitation rate limits to users and app services which should have been exempt (#9968)

The health check on our Docker images now responds more quickly upon successful startup thanks to improvements by maquis196 (#9913), and for especially privacy-conscious homeservers, device names can now be shielded over federation thanks to a contribution by aaronraimist (#9945).

New Access Token Format

Inspired by GitHub's new token format, we've restructured Synapse's access token format to follow the pattern:

syt_<UnpaddedBase64MXIDLocalPart>_<20RandomLetters>_<CRC32>

So a token for @example:matrix.org might look like:

syt_ZXhhbXBsZQ_KfJetOcLWEKCvYdKnQLV_0i3W80

Existing tokens remain valid; this is just for new tokens. We hope the new format reduces network overhead while also making it easier identify misplaced tokens in logs and repositories.

Everything Else

See the Upgrading Instructions and Release Notes for further information on this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including aaronraimist, maquis196, and ThibF.

Synapse 1.33.2 is now available.

This release fixes a denial of service issue (CVE-2021-29471) where evaluating specially crafted push rules could lead to excessive CPU load. Server administrators are encouraged to upgrade.

To learn more about Synapse 1.33, see last week's release announcement.

Synapse 1.33.0 is out! Three main items of note:

1. We plan to release 1.33.1 1.33.2 with a low severity security fix on Tuesday next week, and we're interested in your thoughts on decoupling routine security fixes from normal releases. Please weigh in on this discussion.

   Note: We shipped 1.33.1 with a small dependency fix when installing Synapse via pip. A security release is still planned for Tuesday, which will now be 1.33.2.

2. If you use Synapse's optional account revalidation feature (see account_validity in config.yaml), you'll want to review the upgrading instructions as we've made a few small changes to the email templates it uses.

3. Synapse now has very experimental support for moving presence off of the main process. This has not yet been extensively validated, so please proceed with caution. We expect to get this to a point where we can confidently recommend it in the coming weeks.

Otherwise, this is another release focused on internals. We're driving toward a goal of reducing excess memory consumption when joining large or complex rooms, and most of our effort (aside from the presence work) has been focused on measurement, instrumentation, and experimentation for that.

We did manage to slightly speed up room joins, improve the performance of the user directory, and refine our implementation of MSC3083. Additionally, thanks to work by ShadowJonathan, Synapse now passes all of flake8-bugbear's lints.

See the Upgrading Instructions and Release Notes for further information.

Thank You

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including rkfg, and ShadowJonathan.