All the threes, Synapse 0.33.3!
This release brings together a lot of bugfixes, and also some preparation for support for Lazy Loading and Room Versioning.
We also have, as a great contribution from @vojeroen, SNI extension support! With v0.33.3, Synapse now supports sending SNI over federation for vhosted servers, which resolves this long-standing request.
As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.3 or any of the sources mentioned at https://github.com/matrix-org/synapse.
Features
- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
- Add /_media/r0/config (#3184)
- speed up /members API and add
at and membership params as per MSC1227 (#3568)
- implement
summary block in /sync response as per MSC688 (#3574)
- Add lazy-loading support to /messages as per MSC1227 (#3589)
- Add ability to limit number of monthly active users on the server (#3633)
- Support more federation endpoints on workers (#3653)
- Basic support for room versioning (#3654)
- Ability to disable client/server Synapse via conf toggle (#3655)
- Ability to whitelist specific threepids against monthly active user limiting (#3662)
- Add some metrics for the appservice and federation event sending loops (#3664)
- Where server is disabled, block ability for locked out users to read new messages (#3670)
- set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. (#3694)
- For resource limit blocked users, prevent writing into rooms (#3708)
Bugfixes
- Fix occasional glitches in the synapse_event_persisted_position metric (#3658)
- Fix bug on deleting 3pid when using identity servers that don't support unbind API (#3661)
- Make the tests pass on Twisted < 18.7.0 (#3676)
- Don't ship recaptcha_ajax.js, use it directly from Google (#3677)
- Fixes test_reap_monthly_active_users so it passes under postgres (#3681)
- Fix mau blocking calculation bug on login (#3689)
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users (#3692)
- Improve HTTP request logging to include all requests (#3700, #3723)
- Avoid timing out requests while we are streaming back the response (#3701)
- Support more federation endpoints on workers (#3705, #3713)
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning (#3710)
- Fix bug where
state_cache cache factor ignored environment variables (#3719)
Deprecations and Removals
Internal Changes
- The test suite now can run under PostgreSQL. (#3423)
- Refactor HTTP replication endpoints to reduce code duplication (#3632)
- Tests now correctly execute on Python 3. (#3647)
- Sytests can now be run inside a Docker container. (#3660)
- Port over enough to Python 3 to allow the sytests to start. (#3668, #3732)
- Update docker base image from alpine 3.7 to 3.8. (#3669)
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. (#3678)
- Synapse's tests are now formatted with the black autoformatter. (#3679)
- Implemented a new testing base class to reduce test boilerplate. (#3684)
- Rename MAU prometheus metrics (#3690)
- add new error type ResourceLimit (#3707)
- Logcontexts for replication command handlers (#3709)
- Update admin register API documentation to reference a real user ID. (#3712)
Folks, it's release time, Synapse 0.33.2 has landed.
The release focuses on performance, notable highlights include reducing CPU consumption through speeding up state delta calculations (#3592) and reducing I/O through lazily loading state on the master process (#3579, #3581, #3582, #3584)
Separately work continues on our python 3 port and we hope to have something concrete to trial very soon - we're really excited about this and expect step change improvements in CPU and memory use.
Finally we have some ground work for upcoming room membership lazy loading, there is nothing to see here as yet, but rest assured we will make a lot of noise as soon as it is ready. Stay tuned.
or any of the sources mentioned at https://github.com/matrix-org/synapse.
Synapse 0.33.2 (2018-08-09)
No significant changes.
Synapse 0.33.2rc1 (2018-08-07)
Features
- add support for the lazy_loaded_members filter as per MSC1227 (#2970)
- add support for the include_redundant_members filter param as per MSC1227 (#3331)
- Add metrics to track resource usage by background processes (#3553, #3556, #3604, #3610)
- Add
code label to synapse_http_server_response_time_seconds prometheus metric (#3554)
- Add support for client_reader to handle more APIs (#3555, #3597)
- make the /context API filter & lazy-load aware as per MSC1227 (#3567)
- Add ability to limit number of monthly active users on the server (#3630)
- When we fail to join a room over federation, pass the error code back to the client. (#3639)
- Add a new /admin/register API for non-interactively creating users. (#3415)
Bugfixes
- Make /directory/list API return 404 for room not found instead of 400 (#2952)
- Default inviter_display_name to mxid for email invites (#3391)
- Don't generate TURN credentials if no TURN config options are set (#3514)
- Correctly announce deleted devices over federation (#3520)
- Catch failures saving metrics captured by Measure, and instead log the faulty metrics information for further analysis. (#3548)
- Unicode passwords are now normalised before hashing, preventing the instance where two different devices or browsers might send a different UTF-8 sequence for the password. (#3569)
- Fix potential stack overflow and deadlock under heavy load (#3570)
- Respond with M_NOT_FOUND when profiles are not found locally or over federation. Fixes #3585 (#3585)
- Fix failure to persist events over federation under load (#3601)
- Fix updating of cached remote profiles (#3605)
- Fix 'tuple index out of range' error (#3607)
- Only import secrets when available (fix for py < 3.6) (#3626)
Internal Changes
- Remove redundant checks on who_forgot_in_room (#3350)
- Remove unnecessary event re-signing hacks (#3367)
- Rewrite cache list decorator (#3384)
- Move v1-only REST APIs into their own module. (#3460)
- Replace more instances of Python 2-only iteritems and itervalues uses. (#3562)
- Refactor EventContext to accept state during init (#3577)
- Improve Dockerfile and docker-compose instructions (#3543)
- Release notes are now in the Markdown format. (#3552)
- add config for pep8 (#3559)
- Merge Linearizer and Limiter (#3571, #3572)
- Lazily load state on master process when using workers to reduce DB consumption (#3579, #3581, #3582, #3584)
- Fixes and optimisations for resolve_state_groups (#3586)
- Improve logging for exceptions when handling PDUs (#3587)
- Add some measure blocks to persist_events (#3590)
- Fix some random logcontext leaks. (#3591, #3606)
- Speed up calculating state deltas in persist_event loop (#3592)
- Attempt to reduce amount of state pulled out of DB during persist_events (#3595)
- Fix a documentation typo in on_make_leave_request (#3609)
- Make EventStore inherit from EventFederationStore (#3612)
- Remove some redundant joins on event_edges.room_id (#3613)
- Stop populating events.content (#3614)
- Update the /send_leave path registration to use event_id rather than a transaction ID. (#3616)
- Refactor FederationHandler to move DB writes into separate functions (#3621)
- Remove unused field "pdu_failures" from transactions. (#3628)
- rename replication_layer to federation_client (#3634)
- Factor out exception handling in federation_client (#3638)
- Refactor location of docker build script. (#3644)
- Update CONTRIBUTING to mention newsfragments. (#3645)
Hi All,
We have patched two securities vulnerabilities (details follow), we do not believe either have been exploited in the wild, but recommend upgrading asap.
As always you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.1 or from any of the sources mentioned at https://github.com/matrix-org/synapse/
Thanks
Changes in Synapse v0.33.1 (2018-08-2)
- Fix a potential issue where servers could request events for rooms they have not joined. (#3641)
- Fix a potential issue where users could see events in private rooms before they joined. (#3642)
Folks, Synapse 0.32.0 is an important security update: please upgrade as soon as you can.
The release focuses on security; fixing several federation bugs and adding new features for countering abuse. Notably it includes the ability to blacklist & whitelist servers allowed to send events to a room on a per-room basis via the new
m.room.server_acl
state event: see MSC1383 for details. This also closes out https://github.com/matrix-org/matrix-doc/issues/709 - one of our oldest feature requests from users who wish to be able to limit the servers allowed to participate in a given room.
It's important to understand that server ACLs only work if all the servers participating in the room honour them. In future this will be handled better (as part of ongoing work in making it easier to incrementally version and upgrade the federation protocol). This means that for the ACLs to work, any servers which don't yet implement ACLs (e.g. older Synapses) have to be ACL'd from the room for the access control to work.
Therefore please upgrade as soon as possible to avoid this problem.
This ongoing flurry of security work is in general all part of moving towards the long-awaited stable release of the Server-Server API. In parallel we've been working on the other main outstanding point: State Resets (i.e. scenarios where you get unexpected results when resolving conflicts between different servers' copies of a room). There will be a few more major changes and upgrades on the horizon as we fix these, but then we'll finally be able to cut an r0 release of the Server-Server API and Matrix will be one massive step closer to being out of beta!
As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.32.1
or any of the sources mentioned at https://github.com/matrix-org/synapse.
Changes in synapse v0.32.0 (2018-07-06)
No changes since 0.32.0rc1
Synapse 0.32.0rc1 (2018-07-05)
Features
- Add blacklist & whitelist of servers allowed to send events to a
room via
m.room.server_acl event. (merge)
- Cache factor override system for specific caches (#3334)
- Add metrics to track appservice transactions (#3344)
- Try to log more helpful info when a sig verification fails
(#3372)
- Synapse now uses the best performing JSON encoder/decoder according
to your runtime (simplejson on CPython, stdlib json on PyPy).
(#3462)
- Add optional ip_range_whitelist param to AS registration files to
lock AS IP access (#3465)
- Reject invalid server names in federation requests (#3480)
- Reject invalid server names in homeserver.yaml (#3483)
Bugfixes
- Strip access_token from outgoing requests (#3327)
- Redact AS tokens in logs (#3349)
- Fix federation backfill from SQLite servers (#3355)
- Fix event-purge-by-ts admin API (#3363)
- Fix event filtering in get_missing_events handler (#3371)
- Synapse is now stricter regarding accepting events which it cannot
retrieve the prev_events for. (#3456)
- Fix bug where synapse would explode when receiving unicode in HTTP
User-Agent header (#3470)
- Invalidate cache on correct thread to avoid race (#3473)
Improved Documentation
Deprecations and Removals
- Remove was_forgotten_at (#3324)
Misc
Folks,
v0.31.1 fixes a security bug in the get_missing_events federation API where event visibility rules were not applied correctly.
We are not aware of it being actively exploited but please upgrade asap.
Sorry for the inconvenience, Synapse and the Matrix spec are still in beta and we still ironing out gaps such as this one.
You can get the release here.
Changes in synapse v0.31.1 (2018-06-08)
v0.31.1 fixes a security bug in the get_missing_events federation API
where event visibility rules were not applied correctly.
We are not aware of it being actively exploited but please upgrade asap.
Bug Fixes:
- Fix event filtering in get_missing_events handler (PR #3371)
Good people, it's release time.
With the core team focusing on upcoming performance work and GDPR management tooling, v0.31.0 is most notable for improvements to system stats. Additionally, work continues on our py3 port and a host of small bug fixes and perf improvements.
Get it now from https://github.com/matrix-org/synapse/releases/tag/v0.31.0
Changes in synapse v0.31.0 (2018-06-06)
Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, seedocs/metrics-howto.rst
Bug Fixes:
- Fix metric documentation tables (PR #3341)
- Fix LaterGauge error handling (694968f)
- Fix replication metrics (b7e7fd2)
Changes in synapse v0.31.0-rc1 (2018-06-04)
Features:
- Switch to the Python Prometheus library (PR #3256, #3274)
- Let users leave the server notice room after joining (PR #3287)
Changes:
- daily user type phone home stats (PR #3264)
- Use iter* methods for _filter_events_for_server (PR #3267)
- Docs on consent bits (PR #3268)
- Remove users from user directory on deactivate (PR #3277)
- Avoid sending consent notice to guest users (PR #3288)
- disable CPUMetrics if no /proc/self/stat (PR #3299)
- Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat!
- Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307)
- Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat!
- Reduce stuck read-receipts: ignore depth when updating (PR #3318)
- Put python's logs into Trial when running unit tests (PR #3319)
Changes, python 3 migration:
Bugs:
- Fix federation backfill bugs (PR #3261)
- federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!
It's release o'clock - GDPR time!!!!
v0.30.0 sees the introduction of Server Notices, which provides a channel whereby server administrators can send messages to users on the server, as well as Consent Management for tracking whether users have agreed to the terms and conditions set by the administrator of a server - and blocking access to the server until they have.
In conjunction these features support GDPR compliance in the form of providing a client agnostic means to contact users and ask for consent/agreement to a Privacy Notice.
For more information about our approach to GDPR compliance take a look here (although be aware that our position has evolved a bit; see the upcoming new privacy policy for the Matrix.org homeserver for details).
Additionally there are a host of bug fixes and refactors as well as an enhancement to our Dockerfile.
Get it now from https://github.com/matrix-org/synapse/releases/tag/v0.30.0
Changes in synapse v0.30.0 (2018-05-24)
'Server Notices' are a new feature introduced in Synapse 0.30. They provide a
channel whereby server administrators can send messages to users on the server.
They are used as part of communication of the server policies (see Consent Tracking),
however the intention is that they may also find a use for features such
as "Message of the day".
This feature is specific to Synapse, but uses standard Matrix communication mechanisms,
so should work with any Matrix client. For more details see here.
Further Server Notices/Consent Tracking Support:
- Allow overriding the server_notices user's avatar (PR #3273)
- Use the localpart in the consent uri (PR #3272)
- Support for putting %(consent_uri)s in messages (PR #3271)
- Block attempts to send server notices to remote users (PR #3270)
- Docs on consent bits (PR #3268)
Changes in synapse v0.30.0-rc1 (2018-05-23)
GDPR Support:
- ConsentResource to gather policy consent from users (PR #3213)
- Move RoomCreationHandler out of synapse.handlers.Handlers (PR #3225)
- Infrastructure for a server notices room (PR #3232)
- Send users a server notice about consent (PR #3236)
- Reject attempts to send event before privacy consent is given (PR #3257)
- Add a 'has_consented' template var to consent forms (PR #3262)
- Fix dependency on jinja2 (PR #3263)
Features:
Changes:
- Remove unused
update_external_syncs (PR #3233)
- Use stream rather depth ordering for push actions (PR #3212)
- Make purge_history operate on tokens (PR #3221)
- Don't support limitless pagination (PR #3265)
Bug Fixes:
- Fix logcontext resource usage tracking (PR #3258)
- Fix error in handling receipts (PR #3235)
- Stop the transaction cache caching failures (PR #3255)
It's release time people, not to be outdone by our friends on the Riot web team, Synapse v0.29.1 lands today.
v0.29.1 contains an officially supported docker image (many thanks to the contribution from @kaiyou), continued progress towards Python 3 (thanks to @NotAFile) - as well as a heap of refactorings and bug fixes.
Something worth noting is a potentially breaking change in the error code that /login returns in the Client Server API. Details follow, but the change closes a gap between Synapse behaviour and the spec.
We'd like to give huge thanks to Silvio Fricke and Andreas Peters for writing and maintaining Synapse's first Dockerfile, as well as allmende, jcgruenhage, ptman, and ilianaw for theirs! The new Dockerfile from kaiyou has ended up being merged into the main synapse tree and we're going to try to maintain it going forwards, but folks should use whichever one they prefer.
You can pick it up from https://github.com/matrix-org/synapse/releases/tag/v0.29.1 and thanks to everyone who tested the release candidate.
Changes in synapse v0.29.1 (2018-05-17)
Changes:
- Update docker documentation (PR #3222)
Changes in synapse v0.29.0 (2018-05-16)
No changes since v0.29.0-rc1
Changes in synapse v0.29.0-rc1 (2018-05-14)
Potentially breaking change:
- Make Client-Server API return 401 for invalid token (PR #3161).This changes the Client-server spec to return a 401 error code instead of 403 when the access token is unrecognised. This is the behaviour required by the specification, but some clients may be relying on the old, incorrect behaviour.Thanks to @NotAFile for fixing this.
Features:
Changes - General:
- nuke-room-from-db.sh: added postgresql option and help (PR #2337) Thanks to @rubo77!
- Part user from rooms on account deactivate (PR #3201)
- Make 'unexpected logging context' into warnings (PR #3007)
- Set Server header in SynapseRequest (PR #3208)
- remove duplicates from groups tables (PR #3129)
- Improve exception handling for background processes (PR #3138)
- Add missing consumeErrors to improve exception handling (PR #3139)
- reraise exceptions more carefully (PR #3142)
- Remove redundant call to preserve_fn (PR #3143)
- Trap exceptions thrown within run_in_background (PR #3144)
Changes - Refactors:
- Refactor /context to reuse pagination storage functions (PR #3193)
- Refactor recent events func to use pagination func (PR #3195)
- Refactor pagination DB API to return concrete type (PR #3196)
- Refactor get_recent_events_for_room return type (PR #3198)
- Refactor sync APIs to reuse pagination API (PR #3199)
- Remove unused code path from member change DB func (PR #3200)
- Refactor request handling wrappers (PR #3203)
- transaction_id, destination defined twice (PR #3209) Thanks to @damir-manapov!
- Refactor event storage to prepare for changes in state calculations (PR #3141)
- Set Server header in SynapseRequest (PR #3208)
- Use deferred.addTimeout instead of time_bound_deferred (PR #3127, #3178)
- Use run_in_background in preference to preserve_fn (PR #3140)
Changes - Python 3 migration:
Bug Fixes:
- synapse fails to start under Twisted >= 18.4 (PR #3157) Thanks to @Half-Shot!
- Fix a class of logcontext leaks (PR #3170)
- Fix a couple of logcontext leaks in unit tests (PR #3172)
- Fix logcontext leak in media repo (PR #3174)
- Escape label values in prometheus metrics (PR #3175, #3186)
- Fix 'Unhandled Error' logs with Twisted 18.4 (PR #3182) Thanks to @Half-Shot!
- Fix logcontext leaks in rate limiter (PR #3183)
- notifications: Convert next_token to string according to the spec (PR #3190) Thanks to @mujx!
- nuke-room-from-db.sh: fix deletion from search table (PR #3194) Thanks to @rubo77!
- add guard for None on purge_history api (PR #3160) Thanks to @krombel!
Well now, today sees the release of Synapse 0.28.0!
This release is particularly exciting as it's a major bump mainly thanks to lots and lots of contributions from the wider community - including support for running Synapse on PyPy (thanks Valodim) and lots of progress towards official Python3 support (thanks notafile)!! However, almost all the changes are under the hood (and some are quite major), so this is more a performance, bugfix and synapse internals release rather than adding many new APIs or features
As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.28.0 and thanks to everyone who tested the release candidates.
Changes in synapse v0.28.0 (2018-04-26)
Bug Fixes:
- Fix quarantine media admin API and search reindex (PR #3130)
- Fix media admin APIs (PR #3134)
Changes in synapse v0.28.0-rc1 (2018-04-24)
Minor performance improvement to federation sending and bug fixes.
(Note: This release does not include state resolutions discussed in matrix live)
Features:
- Add metrics for event processing lag (PR #3090)
- Add metrics for ResponseCache (PR #3092)
Changes:
Bug Fixes:
- Return 401 for invalid access_token on logout (PR #2938) Thanks to @dklug!
- Return a 404 rather than a 500 on rejoining empty rooms (PR #3080)
- fix federation_domain_whitelist (PR #3099)
- Avoid creating events with huge numbers of prev_events (PR #3113)
- Reject events which have lots of prev_events (PR #3118)
Today we release Synapse 0.27.3!
Hot on the heels of 0.27.2, notable changes include API support for joining groups/communities as well as a major bug fix (#3082), that is particularly important for those upgrading for the first time in a while. Also new metrics and phone home stats. Phone home stats include better visibility of system usage so we can have a better idea of how synapse's RAM and CPU is behaving in the wild. Also, recording the number of users on the server who have been using it for at least 30 days.
Phone home stats are entirely optional and can be enabled/disabled by setting "report_
stat
s" in homeserver.yaml.
Please consider enabling phone home stats if you currently have not done so - this data is really important to us in improving Matrix as a whole (and justifying future funding for Matrix.org).
As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.27.3 and thanks to everyone who tested the release candidates.
Changes in synapse v0.27.3 (2018-04-11)
Bug fixes:
- URL quote path segments over federation (#3082)
Changes in synapse v0.27.3-rc2 (2018-04-09)
v0.27.3-rc1 used a stale version of the develop branch so the changelog overstates
the functionality. v0.27.3-rc2 is up to date, rc1 should be ignored.
Changes in synapse v0.27.3-rc1 (2018-04-09)
Notable changes include API support for joinability of groups. Also new metrics
and phone home stats. Phone home stats include better visibility of system usage
so we can tweak synpase to work better for all users rather than our own experience
with matrix.org. Also, recording 'r30' stat which is the measure we use to track
overall growth of the Matrix ecosystem. It is defined as:-
Counts the number of native 30 day retained users, defined as:-
- Users who have created their accounts more than 30 days
- Where last seen at most 30 days ago
- Where account creation and last_seen are > 30 days"
Features:
- Add joinability for groups (PR #3045)
- Implement group join API (PR #3046)
- Add counter metrics for calculating state delta (PR #3033)
- R30 stats (PR #3041)
- Measure time it takes to calculate state group ID (PR #3043)
- Add basic performance statistics to phone home (PR #3044)
- Add response size metrics (PR #3071)
- phone home cache size configurations (PR #3063)
Changes:
- Add a blurb explaining the main synapse worker (PR #2886) Thanks to @turt2live!
- Replace old style error catching with 'as' keyword (PR #3000) Thanks to @NotAFile!
- Use .iter* to avoid copies in StateHandler (PR #3006)
- Linearize calls to _generate_user_id (PR #3029)
- Remove last usage of ujson (PR #3030)
- Use simplejson throughout (PR #3048)
- Use static JSONEncoders (PR #3049)
- Remove uses of events.content (PR #3060)
- Improve database cache performance (PR #3068)
Bug fixes:
- Add room_id to the response of
rooms/{'{'}roomId{'}'}/join (PR #2986) Thanks to @jplatte!
- Fix replication after switch to simplejson (PR #3015)
- Fix replication after switch to simplejson (PR #3015)
- 404 correctly on missing paths via NoResource (PR #3022)
- Fix error when claiming e2e keys from offline servers (PR #3034)
- fix tests/storage/test_user_directory.py (PR #3042)
- use PUT instead of POST for federating groups/m.join_policy (PR #3070) Thanks to @krombel!
- postgres port script: fix state_groups_pkey error (PR #3072)
