Well here it is: Synapse 1.0.

Synapse 1.0 is the reference implementation of the Matrix 1.0 spec. The goal of the release overall has been to focus on security and stability, such that we can officially declare Synapse (and Matrix) out of beta and recommended for production use. This means changing the default room protocol version used for new rooms to be v4, which includes the new state resolution algorithm, as well as collision-resistant event IDs, which are now formatted to be URL safe.

Synapse 1.0 also ships with support for the upcoming v5 room protocol (which enforces honouring server key validity periods), but this will not be used as the default for new rooms until a sufficient number of servers support it.

Please note that Synapse 1.0 does not include significant performance work or new features - our focus has been almost exclusively on providing a reference implementation of the Matrix 1.0 protocol. But having cleared our backlog on security/stability issues we will finally be now unblocked to pursue work around reducing RAM footprint, eliminating forward-extremity build up, and shipping new features like Edits, Reactions & E2E cross-signing support.

As part of the security work, Synapse 1.0 contains a breaking change that requires a valid TLS certificate on the federation API endpoint. Servers that do not configure their certificate will no longer be able to federate post 1.0.

It is also worth noting that Synapse 1.0.0 is the last release that will support Python 2.x and Postgres 9.4. For more information see here but the TL;DR is that you should upgrade asap.

This release has been a long time coming. Many thanks indeed to everyone who helped test the release candidates and provided feedback along the way.

Synapse 1.0 is just one component of a larger Matrix 1.0 release, which you can read all about here.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

The changelog since 0.99.5 follows:

Synapse 1.0.0 (2019-06-11)

Bugfixes

• Fix bug where attempting to send transactions with large number of EDUs can fail. (#5418)

Improved Documentation

• Expand the federation guide to include relevant content from the MSC1711 FAQ (#5419)

Internal Changes

• Move password reset links to /_matrix/client/unstable namespace. (#5424)

Synapse 1.0.0rc3 (2019-06-10)

Security: Fix authentication bug introduced in 1.0.0rc1. Please upgrade to rc3 immediately

Synapse 1.0.0rc2 (2019-06-10)

Bugfixes

• Remove redundant warning about key server response validation. (#5392)
• Fix bug where old keys stored in the database with a null valid until timestamp caused all verification requests for that key to fail. (#5415)
• Fix excessive memory using with default federation_verify_certificates: true configuration. (#5417)

Synapse 1.0.0rc1 (2019-06-07)

Features

• Synapse now more efficiently collates room statistics. (#4338, #5260, #5324)
• Add experimental support for relations (aka reactions and edits). (#5220)
• Ability to configure default room version. (#5223, #5249)
• Allow configuring a range for the account validity startup job. (#5276)
• CAS login will now hit the r0 API, not the deprecated v1 one. (#5286)
• Validate federation server TLS certificates by default (implements MSC1711). (#5359)
• Update /_matrix/client/versions to reference support for r0.5.0. (#5360)
• Add a script to generate new signing-key files. (#5361)
• Update upgrade and installation guides ahead of 1.0. (#5371)
• Replace the perspectives configuration section with trusted_key_servers, and make validating the signatures on responses optional (since TLS will do this job for us). (#5374)
• Add ability to perform password reset via email without trusting the identity server. (#5377)
• Set default room version to v4. (#5379)

Bugfixes

• Fixes client-server API not sending "m.heroes" to lazy-load /sync requests when a rooms name or its canonical alias are empty. Thanks to @dnaf for this work! (#5089)
• Prevent federation device list updates breaking when processing multiple updates at once. (#5156)
• Fix worker registration bug caused by ClientReaderSlavedStore being unable to see get_profileinfo. (#5200)
• Fix race when backfilling in rooms with worker mode. (#5221)
• Fix appservice timestamp massaging. (#5233)
• Ensure that server_keys fetched via a notary server are correctly signed. (#5251)
• Show the correct error when logging out and access token is missing. (#5256)
• Fix error code when there is an invalid parameter on /_matrix/client/r0/publicRooms (#5257)
• Fix error when downloading thumbnail with missing width/height parameter. (#5258)
• Fix schema update for account validity. (#5268)
• Fix bug where we leaked extremities when we soft failed events, leading to performance degradation. (#5274, #5278, #5291)
• Fix "db txn 'update_presence' from sentinel context" log messages. (#5275)
• Fix dropped logcontexts during high outbound traffic. (#5277)
• Fix a bug where it is not possible to get events in the federation format with the request GET /_matrix/client/r0/rooms/{roomId}/messages. (#5293)
• Fix performance problems with the rooms stats background update. (#5294)
• Fix noisy 'no key for server' logs. (#5300)
• Fix bug where a notary server would sometimes forget old keys. (#5307)
• Prevent users from setting huge displaynames and avatar URLs. (#5309)
• Fix handling of failures when processing incoming events where calling /event_auth on remote server fails. (#5317)
• Ensure that we have an up-to-date copy of the signing key when validating incoming federation requests. (#5321)
• Fix various problems which made the signing-key notary server time out for some requests. (#5333)
• Fix bug which would make certain operations (such as room joins) block for 20 minutes while attemoting to fetch verification keys. (#5334)
• Fix a bug where we could rapidly mark a server as unreachable even though it was only down for a few minutes. (#5335, #5340)
• Fix a bug where account validity renewal emails could only be sent when email notifs were enabled. (#5341)
• Fix failure when fetching batches of events during backfill, etc. (#5342)
• Add a new room version where the timestamps on events are checked against the validity periods on signing keys. (#5348, #5354)
• Fix room stats and presence background updates to correctly handle missing events. (#5352)
• Include left members in room summaries' heroes. (#5355)
• Fix federation_custom_ca_list configuration option. (#5362)
• Fix missing logcontext warnings on shutdown. (#5369)

Improved Documentation

• Fix docs on resetting the user directory. (#5282)
• Fix notes about ACME in the MSC1711 faq. (#5357)

Internal Changes

• Synapse will now serve the experimental "room complexity" API endpoint. (#5216)
• The base classes for the v1 and v2_alpha REST APIs have been unified. (#5226, #5328)
• Simplifications and comments in do_auth. (#5227)
• Remove urllib3 pin as requests 2.22.0 has been released supporting urllib3 1.25.2. (#5230)
• Preparatory work for key-validity features. (#5232, #5234, #5235, #5236, #5237, #5244, #5250, #5296, #5299, #5343, #5347, #5356)
• Specify the type of reCAPTCHA key to use. (#5283)
• Improve sample config for monthly active user blocking. (#5284)
• Remove spurious debug from MatrixFederationHttpClient.get_json. (#5287)
• Improve logging for logcontext leaks. (#5288)
• Clarify that the admin change password API logs the user out. (#5303)
• New installs will now use the v54 full schema, rather than the full schema v14 and applying incremental updates to v54. (#5320)
• Improve docstrings on MatrixFederationClient. (#5332)
• Clean up FederationClient.get_events for clarity. (#5344)
• Various improvements to debug logging. (#5353)
• Don't run CI build checks until sample config check has passed. (#5370)
• Automatically retry buildkite builds (max twice) when an agent is lost. (#5380)

0.99.5.2 contains a critical performance fix following a regression that was introduced in 0.99.5. Affected servers will have experienced increased CPU and RAM usage with a knock on effect of generally sluggish performance.

Separately, we are also looking into reports relating to further performance degradations that may have been introduced as part of 0.99.5, though consider the 0.99.5.2 fix to be a significant improvement on previous 0.99.5.x releases.

Please upgrade asap.

You can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page.

Synapse v0.99.5.2 Changelog

Bugfixes

• Fix bug where we leaked extremities when we soft failed events, leading to performance degradation. (#5274, #5278, #5291)

Okay folks, this is an important one. v0.99.5.1 will be the last release before we ship Synapse v1.0. It is really important that you upgrade to v0.99.5.1 because it implements rooms version 4 - which is the room version that Synapse 1.0 will default to.

This means that Synapse 1.0 servers will create new rooms as version 4 by default and servers that have not upgraded to at least v0.99.5.1 will not be able to join those rooms.

Over the coming days we will announce a release day for Synapse v1.0, the idea is to give admins 2 weeks notice so that anyone yet to configure their federation SSL certificate has time to do so. This is important, failure to configure your certs will mean not being able to federate with v1.0 servers. If you are not sure if you certs are valid, you can test here and read here for more info on what to do.

Aside from room v4, this release also includes the ability to blacklist specific IPs from federating as well as experimental support for edits and reactions. We are not quite ready to mark the feature 'done done', but it is very close. Watch out for news as the feature lands properly.

We're really close to v1.0 now, give us a few more days and we'll announce an official release date.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

Synapse v0.99.5.1 Changelog (since v0.99.4)

Features

• Add ability to blacklist IP ranges for the federation client. (#5043)
• Ratelimiting configuration for clients sending messages and the federation server has been altered to match login ratelimiting. The old configuration names will continue working. Check the sample config for details of the new names. (#5181)
• Drop support for the undocumented /_matrix/client/v2_alpha API prefix. (#5190)
• Add an option to disable per-room profiles. (#5196)
• Stick an expiration date to any registered user missing one at startup if account validity is enabled. (#5204)
• Add experimental support for relations (aka reactions and edits). (#5209, #5211, #5203, #5212)
• Add a room version 4 which uses a new event ID format, as per MSC2002. (#5210, #5217)

Bugfixes

• Fix image orientation when generating thumbnails (needs pillow>=4.3.0). Contributed by Pau Rodriguez-Estivill. (#5039)
• Exclude soft-failed events from forward-extremity candidates: fixes "No forward extremities left!" error. (#5146)
• Re-order stages in registration flows such that msisdn and email verification are done last. (#5174)
• Fix 3pid guest invites. (#5177)
• Fix a bug where the register endpoint would fail with M_THREEPID_IN_USE instead of returning an account previously registered in the same session. (#5187)
• Prevent registration for user ids that are too long to fit into a state key. Contributed by Reid Anderson. (#5198)
• Fix incompatibility between ACME support and Python 3.5.2. (#5218)
• Fix error handling for rooms whose versions are unknown. (#5219)

Internal Changes

• Make /sync attempt to return device updates for both joined and invited users. Note that this doesn't currently work correctly due to other bugs. (#3484)
• Update tests to consistently be configured via the same code that is used when loading from configuration files. (#5171, #5185)
• Allow client event serialization to be async. (#5183)
• Expose DataStore._get_events as get_events_as_list. (#5184)
• Make generating SQL bounds for pagination generic. (#5191)
• Stop telling people to install the optional dependencies by default. (#5197)

Hey ho Synapse release day.

0.99.4 is a maintenance release collecting together all of the bug fixes and performance improvements over the past few weeks, additionally there is further support for the upcoming 1.0 release (more info coming soon). One thing worth calling out is how many community contributions have made their way into 0.99.4, take a look at the change log for details, but many thanks to everyone submitting PRs, keep them coming!

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

Synapse 0.99.4 Changelog

Features

• Add systemd-python to the optional dependencies to enable logging to the systemd journal. Install with pip install matrix-synapse[systemd]. (#4339)
• Add a default .m.rule.tombstone push rule. (#4867)
• Add ability for password provider modules to bind email addresses to users upon registration. (#4947)
• Implementation of MSC1711 including config options for requiring valid TLS certificates for federation traffic, the ability to disable TLS validation for specific domains, and the ability to specify your own list of CA certificates. (#4967)
• Remove presence list support as per MSC 1819. (#4989)
• Reduce CPU usage starting pushers during start up. (#4991)
• Add a delete group admin API. (#5002)
• Add config option to block users from looking up 3PIDs. (#5010)
• Add context to phonehome stats. (#5020)
• Configure the example systemd units to have a log identifier of matrix-synapse instead of the executable name, python. Contributed by Christoph Müller. (#5023)
• Add time-based account expiration. (#5027, #5047, #5073, #5116)
• Add support for handling /versions, /voip and /push_rules client endpoints to client_reader worker. (#5063, #5065, #5070)
• Add an configuration option to require authentication on /publicRooms and /profile endpoints. (#5083)
• Move admin APIs to /_synapse/admin/v1. (The old paths are retained for backwards-compatibility, for now). (#5119)
• Implement an admin API for sending server notices. Many thanks to @krombel who provided a foundation for this work. (#5121, #5142)

Bugfixes

• Avoid redundant URL encoding of redirect URL for SSO login in the fallback login page. Fixes a regression introduced in #4220. Contributed by Marcel Fabian Krüger ("zaugin"). (#4555)
• Fix bug where presence updates were sent to all servers in a room when a new server joined, rather than to just the new server. (#4942, #5103)
• Fix sync bug which made accepting invites unreliable in worker-mode synapses. (#4955, #4956)
• start.sh: Fix the --no-rate-limit option for messages and make it bypass rate limit on registration and login too. (#4981)
• Transfer related groups on room upgrade. (#4990)
• Prevent the ability to kick users from a room they aren't in. (#4999)
• Fix issue #4596 so synapse_port_db script works with --curses option on Python 3. Contributed by Anders Jensen-Waud [email protected]. (#5003)
• Clients timing out/disappearing while downloading from the media repository will now no longer log a spurious "Producer was not unregistered" message. (#5009)
• Fix "cannot import name execute_batch" error with postgres. (#5032)
• Fix disappearing exceptions in manhole. (#5035)
• Workaround bug in twisted where attempting too many concurrent DNS requests could cause it to hang due to running out of file descriptors. (#5037)
• Make sure we're not registering the same 3pid twice on registration. (#5071)
• Don't crash on lack of expiry templates. (#5077)
• Fix the ratelimting on third party invites. (#5104)
• Add some missing limitations to room alias creation. (#5124, #5128)
• Limit the number of EDUs in transactions to 100 as expected by synapse. Thanks to @superboum for this work! (#5138)
• Fix bogus imports in unit tests. (#5154)

Internal Changes

• Add test to verify threepid auth check added in #4435. (#4474)
• Fix/improve some docstrings in the replication code. (#4949)
• Split synapse.replication.tcp.streams into smaller files. (#4953)
• Refactor replication row generation/parsing. (#4954)
• Run black to clean up formatting on synapse/storage/roommember.py and synapse/storage/events.py. (#4959)
• Remove log line for password via the admin API. (#4965)
• Fix typo in TLS filenames in docker/README.md. Also add the '-p' commandline option to the 'docker run' example. Contributed by Jurrie Overgoor. (#4968)
• Refactor room version definitions. (#4969)
• Reduce log level of .well-known/matrix/client responses. (#4972)
• Add config.signing_key_path that can be read by synapse.config utility. (#4974)
• Track which identity server is used when binding a threepid and use that for unbinding, as per MSC1915. (#4982)
• Rewrite KeyringTestCase as a HomeserverTestCase. (#4985)
• README updates: Corrected the default POSTGRES_USER. Added port forwarding hint in TLS section. (#4987)
• Remove a number of unused tables from the database schema. (#4992, #5028, #5033)
• Run black on the remainder of synapse/storage/. (#4996)
• Fix grammar in get_current_users_in_room and give it a docstring. (#4998)
• Clean up some code in the server-key Keyring. (#5001)
• Convert SYNAPSE_NO_TLS Docker variable to boolean for user friendliness. Contributed by Gabriel Eckerson. (#5005)
• Refactor synapse.storage._base._simple_select_list_paginate. (#5007)
• Store the notary server name correctly in server_keys_json. (#5024)
• Rewrite Datastore.get_server_verify_keys to reduce the number of database transactions. (#5030)
• Remove extraneous period from copyright headers. (#5046)
• Update documentation for where to get Synapse packages. (#5067)
• Add workarounds for pep-517 install errors. (#5098)
• Improve logging when event-signature checks fail. (#5100)
• Factor out an "assert_requester_is_admin" function. (#5120)
• Remove the requirement to authenticate for /admin/server_version. (#5122)
• Prevent an exception from being raised in a IResolutionReceiver and use a more generic error message for blacklisted URL previews. (#5155)
• Run black on the tests directory. (#5170)
• Fix CI after new release of isort. (#5179)

TL;DR DON'T PANIC - Synapse 1.0 will support Postgres 9.4 and Python 2.7

Folks, this is an update to explain that we will be shortly deprecating Synapse support for Postgres 9.4 and Python 2.x.

What are we doing?

From the dates described below, we will no longer guarantee support for deprecated versions. This means that Synapse may continue to work with these versions but we will not make any attempt to ensure compatibility and will remove old library versions from our CI.

When is this happening?

Synapse 1.0 will continue to support both technologies, but subsequent releases may not:-

For Python, we shared that we would discontinue to Python 2.x support from April 1st 2019, so for the first release that follows 1.0 we do not guarantee Python 2.x support.

For Postgres, will give server admins 6 weeks to upgrade to a newer version, and will guarantee support up until 20th May 2019.

Why would you do this to us?

We have multiple reasons, but broadly:-

• We want to make use of new language features not supported in old versions. This will enable us to continue to improve the performance and maintainability of Synapse.
• Python 2.x overall will be end of life'd at the end of the year. Postgres 9.4's final release will follow 2 months later on 13th February 2020.
• Since very few server admins still use these technologies on the wild, providing support is costly and we want to reduce our overall maintenance load.

La la la I am ignoring you - what will happen?

You will be able to upgrade to Synapse 1.0, but will likely experience incompatibilities that prevent you upgrading further. Seriously, you really need to upgrade.

Okay, but I have questions, where should I go?

Come and say Hi in #synapse:matrix.org and we'll do our best to help you.

Hey hey, a Synapse release for you today.

The big news in 0.99.3 is that the user directory has been completely re-written and should now be much more performant - this will benefit all installations, but especially those housing larger servers.

Aside from that we continue our 1.0 preparations and relatedly we've improved our docs, in particular to explain how .well-known works. On the perf side we've added rate limiting to login and register endpoints as well as now batching up read receipts to send over federation.

I've said it before, and I'll say it again:-

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate - self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.3 changelog

Features

• The user directory has been rewritten to make it faster, with less chance of falling behind on a large server. (#4537, #4846, #4864, #4887, #4900, #4944)
• Add configurable rate limiting to the /register endpoint. (#4735, #4804)
• Move server key queries to federation reader. (#4757)
• Add support for /account/3pid REST endpoint to client_reader worker. (#4759)
• Add an endpoint to the admin API for querying the server version. Contributed by Joseph Weston. (#4772)
• Include a default configuration file in the 'docs' directory. (#4791, #4801)
• Synapse is now permissive about trailing slashes on some of its federation endpoints, allowing zero or more to be present. (#4793)
• Add support for /keys/query and /keys/changes REST endpoints to client_reader worker. (#4796)
• Add checks to incoming events over federation for events evading auth (aka "soft fail"). (#4814)
• Add configurable rate limiting to the /login endpoint. (#4821, #4865)
• Remove trailing slashes from certain outbound federation requests. Retry if receiving a 404. Context: #3622. (#4840)
• Allow passing --daemonize flags to workers in the same way as with master. (#4853)
• Batch up outgoing read-receipts to reduce federation traffic. (#4890, #4927)
• Add option to disable searching the user directory. (#4895)
• Add option to disable searching of local and remote public room lists. (#4896)
• Add ability for password providers to login/register a user via 3PID (email, phone). (#4931)

Bugfixes

• Fix a bug where media with spaces in the name would get a corrupted name. (#2090)
• Fix attempting to paginate in rooms where server cannot see any events, to avoid unnecessarily pulling in lots of redacted events. (#4699)
• 'event_id' is now a required parameter in federated state requests, as per the matrix spec. (#4740)
• Fix tightloop over connecting to replication server. (#4749)
• Fix parsing of Content-Disposition headers on remote media requests and URL previews. (#4763)
• Fix incorrect log about not persisting duplicate state event. (#4776)
• Fix v4v6 option in HAProxy example config. Contributed by Flakebi. (#4790)
• Handle batch updates in worker replication protocol. (#4792)
• Fix bug where we didn't correctly throttle sending of USER_IP commands over replication. (#4818)
• Fix potential race in handling missing updates in device list updates. (#4829)
• Fix bug where synapse expected an un-specced prev_state field on state events. (#4837)
• Transfer a user's notification settings (push rules) on room upgrade. (#4838)
• fix test_auto_create_auto_join_where_no_consent. (#4886)
• Fix a bug where hs_disabled_message was sometimes not correctly enforced. (#4888)
• Fix bug in shutdown room admin API where it would fail if a user in the room hadn't consented to the privacy policy. (#4904)
• Fix bug where blocked world-readable rooms were still peekable. (#4908)

Internal Changes

• Add a systemd setup that supports synapse workers. Contributed by Luca Corbatto. (#4662)
• Change from TravisCI to Buildkite for CI. (#4752)
• When presence is disabled don't send over replication. (#4757)
• Minor docstring fixes for MatrixFederationAgent. (#4765)
• Optimise EDU transmission for the federation_sender worker. (#4770)
• Update test_typing to use HomeserverTestCase. (#4771)
• Update URLs for riot.im icons and logos in the default notification templates. (#4779)
• Removed unnecessary $ from some federation endpoint path regexes. (#4794)
• Remove link to deleted title in README. (#4795)
• Clean up read-receipt handling. (#4797)
• Add some debug about processing read receipts. (#4798)
• Clean up some replication code. (#4799)
• Add some docstrings. (#4815)
• Add debug logger to try and track down #4422. (#4816)
• Make shutdown API send explanation message to room after users have been forced joined. (#4817)
• Update example_log_config.yaml. (#4820)
• Document the generate option for the docker image. (#4824)
• Fix check-newsfragment for debian-only changes. (#4825)
• Add some debug logging for device list updates to help with #4828. (#4828)
• Improve federation documentation, specifically .well-known support. Many thanks to @vaab. (#4832)
• Disable captcha registration by default in unit tests. (#4839)
• Add stuff back to the .gitignore. (#4843)
• Clarify what registration_shared_secret allows for. (#4844)
• Correctly log expected errors when fetching server keys. (#4847)
• Update install docs to explicitly state a full-chain (not just the top-level) TLS certificate must be provided to Synapse. This caused some people's Synapse ports to appear correct in a browser but still (rightfully so) upset the federation tester. (#4849)
• Move client read-receipt processing to federation sender worker. (#4852)
• Refactor federation TransactionQueue. (#4855)
• Comment out most options in the generated config. (#4863)
• Fix yaml library warnings by using safe_load. (#4869)
• Update Apache setup to remove location syntax. Thanks to @cwmke! (#4870)
• Reinstate test case that runs unit tests against oldest supported dependencies. (#4879)
• Update link to federation docs. (#4881)
• fix test_auto_create_auto_join_where_no_consent. (#4886)
• Use a regular HomeServerConfig object for unit tests rater than a Mock. (#4889)
• Add some notes about tuning postgres for larger deployments. (#4895)
• Add a config option for torture-testing worker replication. (#4902)
• Log requests which are simulated by the unit tests. (#4905)
• Allow newsfragments to end with exclamation marks. Exciting! (#4912)
• Refactor some more tests to use HomeserverTestCase. (#4913)
• Refactor out the state deltas portion of the user directory store and handler. (#4917)
• Fix nginx example in ACME doc. (#4923)
• Use an explicit dbname for postgres connections in the tests. (#4928)
• Fix ClientReplicationStreamProtocol.__str__(). (#4929)

TL;DR

• If you run a Synapse ensure that your federation certificates are valid here.
• If they are not valid check out the FAQ.
• Follow along with progress at https://arewereadyyet.com
• Tell all your admin friends.

Folks, as you know we are now very close to achieving Matrix 1.0 and finally being in a position to shed our ‘beta' tag. It has been a long time coming and speaks to the huge effort from hundreds of people over the past 5 years.

A critical step towards this goal is the release of Synapse 1.0. We want to ship Synapse 1.0 as soon as possible but can't do so without your help!

We'd like to introduce AreWeReadyYet.com - a quick and easy way for everyone to track the progress and check if their federation is ready for Matrix 1.0!!

Synapse 1.0 is good news for anyone running a Synapse installation - it contains critical bug fixes, security patches, a new room algorithm version and dramatically improved user and room search. However, as part of the security work it also contains a breaking change from previous Synapse versions. From 1.0 onwards it will necessary to ensure a valid TLS certificate on the federation API. Self signed certificates will no longer be accepted. Why would we do such a thing?

In anticipation for this, everyone currently running a homeserver must ensure that they have checked their federation certificate (check yours here). Failure to do so will mean being unable to federate with any Matrix 1.0 compliant server. If your server fails the check, our FAQ has all the details on what you need to do.

This post is a call to arms to try and get as many admins to upgrade their certificates as possible. We are tracking adoption at https://arewereadyyet.com - currently this sits at about 55% - we need this figure to be higher before we can pull the lever.  

So what are you waiting for? Check that your server has valid certs - then tell all your admin pals to do the same. Friends don't let friends miss out on Synapse 1.0, send them to arewereadyyet.com (or tweet here to remind them!) We really need the community to help us here because at some point soon, we will need to pull the lever and release.

Once we make more progress on adoption, we will announce an official release date and finally get Synapse out of beta!

Well now, what have we here? Synapse 0.99.2 is the latest in the 0.99.x series as we step ever closer to 1.0.

0.99.2 is an incremental release including a bunch of performance improvements, enhancements to room upgrades and generally a plethora of bug fixes.

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate - self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.2 changelog

Features

• Added an HAProxy example in the reverse proxy documentation. Contributed by Benoît S. (“Benpro”). (#4541)
• Add basic optional sentry integration. (#4632, #4694)
• Transfer bans on room upgrade. (#4642)
• Add configurable room list publishing rules. (#4647)
• Support .well-known delegation when issuing certificates through ACME. (#4652)
• Allow registration and login to be handled by a worker instance. (#4666, #4670, #4682)
• Reduce the overhead of creating outbound federation connections over TLS by caching the TLS client options. (#4674)
• Add prometheus metrics for number of outgoing EDUs, by type. (#4695)
• Return correct error code when inviting a remote user to a room whose homeserver does not support the room version. (#4721)
• Prevent showing rooms to other servers that were set to not federate. (#4746)

Bugfixes

• Fix possible exception when paginating. (#4263)
• The dependency checker now correctly reports a version mismatch for optional dependencies, instead of reporting the dependency missing. (#4450)
• Set CORS headers on .well-known requests. (#4651)
• Fix kicking guest users on guest access revocation in worker mode. (#4667)
• Fix an issue in the database migration script where the e2e_room_keys.is_verified column wasn't considered as a boolean. (#4680)
• Fix TaskStopped exceptions in logs when outbound requests time out. (#4690)
• Fix ACME config for python 2. (#4717)
• Fix paginating over federation persisting incorrect state. (#4718)

Internal Changes

• Run black to reformat user directory code. (#4635)
• Reduce number of exceptions we log. (#4643, #4668)
• Introduce upsert batching functionality in the database layer. (#4644)
• Fix various spelling mistakes. (#4657)
• Cleanup request exception logging. (#4669, #4737, #4738)
• Improve replication performance by reducing cache invalidation traffic. (#4671, #4715, #4748)
• Test against Postgres 9.5 as well as 9.4. (#4676)
• Run unit tests against python 3.7. (#4677)
• Attempt to clarify installation instructions/config. (#4681)
• Clean up gitignores. (#4688)
• Minor tweaks to acme docs. (#4689)
• Improve the logging in the pusher process. (#4691)
• Better checks on newsfragments. (#4698, #4750)
• Avoid some redundant work when processing read receipts. (#4706)
• Run push_receipts_to_remotes as background job. (#4707)
• Add prometheus metrics for number of badge update pushes. (#4709)
• Reduce pusher logging on startup (#4716)
• Don't log exceptions when failing to fetch remote server keys. (#4722)
• Correctly proxy exception in frontend_proxy worker. (#4723)
• Add database version to phonehome stats. (#4753)

No Ben this week, apparently he is allowed to go on holiday from time to time. Also no Matrix Live because we are terrible people - bring back Ben, that's what I say.

Quaternion

kitsune:

Quaternion (master branch, and upcoming v0.0.9.4) can now open rooms by their aliases or ids upon pressing Ctrl+O, as long as those rooms are already in your room list (opening arbitrary public rooms will come in later versions). You can even paste matrix.to URIs for users (will open direct chat) and rooms in the same dialog. Navigation to known rooms inside Quaternion also works.

libQMatrixClient 0.5 is out - with members lazy-loading, room versions and upgrades and much more: https://github.com/QMatrixClient/libqmatrixclient/releases/tag/0.5.0

Which is, err, pretty amazing actually. As far as I know this is the only independent SDK supporting lazy loading and room upgrades. Congratulations.

weechat-matrix

Poljar has been hard at work on weechat-matrix

Matthew reports:-

weechat-matrix's e2e support is really impressive (via matrix-nio and python-olm). It can only read rather than send right now, but otherwise looks to be massively on the right track. It even does fingerprint-based verification!

Synapse

We shipped 0.99.2 this week, it's a point release containing all the usual bug fixes and perf improvements. We have also been taking a look at our docs and trying to improve where we can.

Hawkowl has spent some time improving CI so that we don't get queued up for hours waiting for builds (woo).

Admins - your weekly reminder that if you've not already done so, you must ensure the TLS certificate on your federation endpoints is no longer self signed - see our handy guide for all the details.

matrix-puppet-bridge

tom:

matrix-puppet-bridge v1.17.0 has been released! This release:

• Handles m.notice messages sent from Matrix as text, instead of ignoring them (thanks go to https://matrix.to/#/@cdknight1:matrix.org !)
• Sets rooms invite-only when they're touched, instead of relying on others not knowing the room ID (thanks to https://matrix.to/#/@AndrewJDR:matrix.org , from all of us who federate on the homeservers we use to bridge!)

Matrix rooms representing remote rooms being joinable by anyone who knows the room ID (which is generated, at least in part, from the remote room ID in all the matrix-puppet-bridge applications) was a big deficiency, and it's finally resolved. It wasn't known whether or not we could do this, and have our ghost users still be able to join the rooms (they need to be invited instead of just joining themselves), until it was attempted and tested in a few of the bridge applications.
Because it is a big deal, new minor versions of matrix-puppet-slack, matrix-puppet-facebook, matrix-puppet-hangouts, matrix-puppet-signal, matrix-puppet-imessage and matrix-puppet-groupme, bumping the matrix-puppet-bridge version to 1.17.0, have been released.

Riot iOS

• Support .well-known data from login response
• Create right permalinks for room v3
• Fix builder to Xcode 10.1 in order to still be able to release
• Update swift version. Add swiftLint

matrix-media-repo

TravisR reports that:

matrix-media-repo has alpha-quality support for s3 (and s3-like services) on the travis/s3 branch. Intrepid testers are encouraged to give it a shot, and report bugs. Caution: may upload your cat.

Scylla

It's been a little while since we saw any news about Scylla, a web-based client written in Elm. VaNilLa said:

Scylla, the Elm-based Matrix client, has received a few improvements. Included in this update:

• Files that are uploaded now keep their original name (as opposed to being called File)
• Sent messages are displayed right away, instead of when they are delivered to the server
• Usernames are only shown once per several consecutive messages, to avoid clutter Some CSS changes have been made to keep the interface looking clean

Almost forgot! You can find Scylla's source code here and a live instance here!

Riot Android

• Implementation of .well-known support (SDK and Riot)
• Minor change on some colors of the themes (link, home badges)
• Many issue will be fixed regarding linkification
• KeysBackup: improvement on recovery process: importing keys step is 8 times faster, and user get more feedback during the process which can take several seconds
• We will prepare a new release for the beginning of next week.
• PlayStore new descriptions have been updated for the following languages: Bulgarian, German, English (US), French, Hungarian, Russian and Chinese (Taiwan).

RiotX (Android)

• Timeline:
  • HTML formatting
  • Implementation of pills (need optimization)
• Many Github issues have been created to track parity with the Riot Android

Riotic: new fork with updates

Aaron Raimist has made some updates to Riotic, which was a good chance for me to revisit it. It works nicely and is a great alternative to the Electron version of Riot. I also like being able to use https://riot.im/develop as an app.

I've slightly tweaked Joakim Ahlen's Riot wrapper for macOS, riotic, which uses the native WKWebView instead of Electron. I updated the app to be sandboxed so it has very limited access to your system. I also updated the interface to follow macOS conventions and updated it to use the latest version of Swift.

riotic does have some limitations though. Riot doesn't support VoIP on Safari so riotic can't support VoIP either and WKWebView doesn't support notifications as far as I can tell. Right now it uses a really old Riot icon, maybe I'll ask about using one of these community made icons https://github.com/vector-im/riot-web/pull/4474.

It does have some advantages over the official Riot Electron app though. The app is only ~12 MB compared to Riot which is ~180 MB, it also uses significantly less RAM. riotic also allows you to pick what Riot URL to use so you can run /develop as a desktop app.

You can download riotic from https://github.com/aaronraimist/riotic/releases/tag/v1.1.

 

synapse-netcore-workers

Half-Shot has been working on synapse-netcore-workers:

The synapse-netcore-worker project has continued to evolve. You can now federate with other servers using the federation sender implementation. It supports everything except device lists at the moment, so it supports PDUs/EDUs and can just be connected up to one of your existing synapse instances. It's not been battletested enough yet to be put in production (hence no dockerfile), but it's very fast.
Oh and for those of you who don't know, "synapse-netcore-worker" is Travis's .NET implementation of synapse workers, the room can be found at #synapse-netcore-workers:t2bot.io.

If you're as uninitiated as I was three days ago, this project is a replaceable worker component for Synapse, which just happens to be written in .NET.

But wait, there's more! Half-Shot continues with

To give an update to the earlier exploits of synapse-netcore-worker's federation sender, we've still got a few more things to iron out before we can suggest people use it actively. The hit list of remaining things to fix is in https://github.com/turt2live/synapse-netcore-workers/pull/5

Mautrix Whatsapp

That Ben guy eh? When he's not swanning off on holiday and having fun, he's writing super groovy guides to getting whatsapp bridging up and running. You don't even need a real device. Check it out.

Ruby SDK

Ananace reports that:

Just merged the protocol split branch I've been working on for the Ruby SDK, including a first PoC for an application service base. Not tested in any actual use as of yet, but expect Things TM in the next release.

Koma

yuforia reports that:

in project koma, a new bot picsay is created. It like the classic easter-egg program cowsay, but it uses actual photos instead of ASCII art. It configured to use any image just by editing a json file. So you can run your own version for fun.

TChap

Willem reports that:

I took some time to hack on Tchap again. This time I disabled virus scanning of thumbnails and downloads in Tchap to be able to see avatars. I also wrote a non-scanning virus scanner API implementation to be able to see files, pictures and videos that are sent. The updated Tchap can be found at https://github.com/14mRh4X0r/tchap-android, the virus scanner API implementation at https://git.snt.utwente.nl/14mRh4X0r/tchap-media-scanner.

Debian

andrewsh reports that:

Unless things get seriously on fire in the coming months, Debian buster will have Synapse 0.99.2 plus something newer in backports

Matrix WUG

Dandellion's Wug now supports Inuktitut Syllabics and Iñupiatun Orthography. In the bot's own words.

Hi I can help you translate X-SAMPA, Z-SAMPA to IPA, and transcribe into proto-indo european notation! Use (x/z/p) together with either / or [] as delimiters x/"hEloU/ z[or` 5aIk DIz] p/mreghnom/

I also can transcribe to Inuktitut Syllabics like this: i[tusaumaqattautijjutinik aulattijiit]. Find my source at https://github.com/Dali99/matrix-wug

New Rooms

Aaron Raimist has created a new room for Formula 1 fans:

Now that the Formula 1 season is getting underway it's probably a good time to announce @CIA:matrix.org's new(ish) room: #f1:matrix.org When this blog post is released there will be 16 days left before the first race of the 2019 season
There was previously a Formula 1 room but it was merged with Snoonet's IRC channel which tends to be extremely busy. This is a matrix only room.

That's it folks, your normal Ben orientated programming will continue next week. Bring back Ben, bring back Ben.

Good people,

2019 is a big year for Matrix, in the next month we will have shipped:

• Matrix spec 1.0 (including the first stable release of the Server to Server Spec)
• Synapse 1.0
• Riot 1.0

This is huge in itself, but is really only the beginning, and now we want to grow the ecosystem as quickly as possible. This means landing a mix of new features, enhancing existing ones, some big performance improvements as well as generally making life easier for our regular users, homeserver admins and community developers.

Today we are sharing the Matrix core team's backend roadmap. The idea is that this will make it easier for anyone to understand where the project is going, what we consider to be important, and why.

To see the roadmap in its full glory, take a look here.

What is a roadmap and why is it valuable?

A roadmap is a set of high level projects that the team intend to work on and a rough sense of the relative priority. It is essential to focus on specific goals, which inevitably means consciously not working on other initiatives.

Our roadmap is not a delivery plan - there are explicitly no dates. The reason for this is that we know that other projects will emerge, developers will be needed to support other urgent initiatives, matrix.org use continues to grow exponentially and will require performance tweaking.

So simply, based on what we know now, this is the order we will work on our projects.

Why are we sharing it?

We already share our day to day todo list, and of course our commit history, but it can be difficult for a casual observer to see the bigger picture from such granular data. The purpose of sharing is that we want anyone from the community to understand where our priorities lie.

We are often asked ‘Why are you not working on X, it is really important' where the answer is often ‘We agree that X is really important, but A, B and C are more important and must come first'.

The point of sharing the roadmap is to make that priority trade off more transparent and consumable.

How did we build it?

The core contributors to Synapse and Dendrite are 6 people, of 5 nationalities spread across 3 locations. After shipping the r0 release of the Server to Server spec last month we took some time to step back and have a think about what to do after Synapse 1.0 lands. This meant getting everyone in one place to talk it through.

We also had Ben (benpa) contribute from a community perspective and took input from speaking to so many of you at FOSDEM.

In the end we filled a wall with post-its, each post-it representing a sizeable project. The position of the post-it was significant in that the vertical axis being a sense of how valuable we thought the task would be, and the horizontal axis being a rough guess on how complex we considered it to be.

We found this sort of grid approach to be really helpful in determining relative priority.

After many hours and plenty of blood, sweat and tears we ended up with something we could live with and wrote it up in the shared board.

And this is written in blood right?

Not at all (it's written in board marker). This is simply a way to express our plan of action and we are likely to make changes to it dynamically. However, this means that at any given moment, if someone wants to know what we are working on then the roadmap is the place to go.

But wait I want to know more!

Here is a video of myself and Matthew to talk you through the projects

Interesting, but I have questions ...

Any feedback gratefully received, come and ask questions in #synapse or #dendrite or feel free to ping me direct at @neilj:matrix.org