On Monday, 13th December we plan to publish a security release of libolm at 15:00 UTC to address a single high severity issue. To the best of our knowledge, only matrix-js-sdk and clients relying on it for E2EE are affected by this issue. This includes Element Web/Desktop and their forks (like SchildiChat). The release of libolm will be immediately followed by a security release of matrix-js-sdk and the affected clients. Users of these clients are encouraged to upgrade as soon as the patched versions are released.
We will be reaching out to downstream packagers to ensure they can prepare patched versions of the affected packages at the time of the release. The details of the vulnerability will be disclosed in a blog post on the day of the release. There is so far no evidence of the vulnerability being exploited in the wild.
The patched version numbers will be as follows:
libolm 3.2.8
matrix-js-sdk 15.2.1
Element Web/Desktop 1.9.7
Thank you for your patience while we work to resolve this issue.
One thing you might not know is that TWIM bot is a space traveler, sent by the Matrix scientists to explore that zone called "The Possibilities". The #twim:matrix.org room is a portal to its energy tank, and we had received a distress signal!
To help the TWIM explorer fulfil its mission, we asked the Matrix community to fuel it with news before it crashed into space debris made of aggregated ignorance!
This week again, the community has been very active and explored many possibilities of the Matrix universe!
Matrix Live 🎙
For this week's Matrix Live my guest is Amandine and we're discussing how Element and 50 other organisations are trying to shape the future of EU's law for more interoperability. Bonus point: we have a double bridge demo with Matrix, Slack and Telegram!
Dept of Status of Matrix 🌡️
FOSDEM!
This year, the Matrix.org Foundation is excited to host the first ever Matrix.org Foundation and Community devroom at FOSDEM. A full day of talks, demos and workshops around Matrix itself and projects built on top of Matrix. Read (and answer to) our Call for Partipactions!
A group of Finnish Matrix admins have set up a free homeserver for Finnish public called pikaviestin.fi (literally instant messenger dot fi). It offers a bunch of bridges and registration requires an e-mail address in one of Finnish e-mail providers or organizations. We welcome all Finns to register there and help decentralize Matrix. Support room can be found at #aula:pikaviestin.fi
That's a fantastic initiative! Kudos to all the sysadmins involved!
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
The end of the year is drawing to a close. Thus many of the Spec Core Team members are focusing on implementation in order to meet deadlines. Review is still occurring though! As above, we have MSC3419 (allow guests to send more event types). This was born out of next-generation VoIP work, but it should have a positive impact on improving the guest experience in Matrix on the whole.
Otherwise work is still ongoing by Bruno and others on untangling the aggregations MSCs, specifically MSC2675 and MSC2676.
And finally, Alexandre Franke has PR'd some work to allow for matrix.org's OpenAPI spec to be widely available, meaning anyone with a Swagger (or other OpenAPI viewer) client can easily pull it and start sending requests against a Matrix homeserver. Fun times!
I want to start by drawing attention to a blog post which we published today: Type coverage for Sydent: motivation. This the first in a series of three articles discussing what we've learned from making Sydent pass the mypy type checker in strict mode. Improving type coverage across Synapse, Sygnal, and Sydent has been a major focus of the backend team at Element for the past few months, and we think we've learned a few useful things in the process.
This week we also released Synapse 1.48 with loads of internal improvements, new Admin APIs, better alignment with the Matrix 1.1 spec, and more. We're planning one more release for the year, 1.49 on December 14th, and then we're taking a break until Synapse 1.50 on January 11th.
Importantly: Synapse 1.49 will be the last release to support Python 3.6, PostrgreSQL 9.6, and Ubuntu 18.04 LTS (Bionic) — if you're reliant on any of these platforms, please ensure you have plans to upgrade.
Let us know what you think of the article (and the Synapse release!), and we'll see you next week!
Sydent is the reference Matrix Identity server. It provides a lookup service, so that you can find a Matrix user via their email address or phone number (if they have chosen to share it).
I've just published a blog post (part one of three) about our efforts to improve Sydent's type coverage. It should hopefully be of interest to anyone who works with Python or is interested in static analysis more generally.
In the vein of Gitter feature parity on Matrix, we've made the first steps towards a better public static archive. We merged an experimental implementation of MSC3030 into Synapse which lets you use the unstable /timestamp_to_event client API endpoint go from a given timestamp to the closest event ID. This will allow us to implement a calendar jump to date interface to be able to navigate to any day in the rooms history. Our first target to add the jump to date UI in is Hydrogen since we plan to server-side render Hydrogen for the actual public static archive as well.
To enable the MSC3030 unstable API endpoints in Synapse, add experimental_features -> msc3030_enabled: true to your homeserver.yaml:
GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>
{
"event_id": ...
"origin_server_ts": ...
}
Also as part of MSC3030, when you use the client API endpoint, if your homeserver sees that the closest event it has locally in the database is next to a gap in the history, it will go out and ask other federated homeservers what they have as the closest event instead.
GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>
{
"event_id": ...
"origin_server_ts": ...
}
*--
MSC2716 to import batches of historical messages is still marching along getting some polishing passes and strengthening the assertions in the Complement tests to make sure things are going absolutely correctly. It's also good to see Beeper utilizing it and catching a fewbugs along the way 💪.
Dept of Bridges 🌉
Hookshot
Half-Shot reports very late, to the great despair of TWIM's editor:
Hookshot gets provisioning!
Stop the press. This is a last minute TWIM. We've been beavering away on matrix-hookshot. It's gained many features in the last week, but the big thing is that hookshot has gained the ability to provision connections over a provisioning API, which means it should hook nicely into Dimension (and other integration managers, in the future)!
Other notable features are:
Support for multiple webhooks per room
Support for the username/text fields on an incoming webhook (slack style)
Named webhooks, so each hook now has a sensible displayname
The ability to spawn GitHub actions from rooms using the !gh workflow run command
Lots of new supported events from GitLab, such as reviews and tag pushes
Hosted documentation (so all of the above is easy to setup), it's a bit in progress atm.
We're aiming for a release very very soon, hopefully in the next week or so!
SchildiChat is a fork of Element that focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.
In a new release being published just now we added the possibility to mark rooms as unread also on Web/Desktop (using MSC2867, huge thanks to @alangecker for his PR on Element Web!).
This has already been implemented in SchildiChat-Android and is now enabled on both by default.
We finally figured out what caused the issues with the flatpak on GNOME, especially on Arm. It should now work properly, if you use Flathub. On the Pinephone (and other systems, that don't set a locale/use the C locale), timestamps should now not be needlessly long anymore. Redactions got a face-lift to distinguish them better from normal messages. We added a workaround for Synapse not allowing you to leave a banned room. We now delete the room from the room list permanently if Synapse returns "unknown room" when trying to leave it. Spaces can now show the entire hierarchy in the sidebar (if you pull it out) and you can navigate to subspaces by clicking on them in the roomlist, even if you collapsed the space hierarchy in the sidebar.
Today we have released FluffyChat 1.0.0 with a whole new design, a lot of bug fixes and huge performance improvements.
New design
The new design has bigger message bubbles with fancy shadows and bigger fonts. The contrast has been improved and some elements, like the time on every single message bubble, are now hidden by default. But they are not gone! Detailed message information are now accessible in the new message info page, where we not also can see the message type and the timestamp, but also the whole JSON source code of each timeline event.
Spaces
Spaces have got a lot improvements and bug fixes. They have moved to the bottom bar of the chat list (while this bottom bar is still hidden if you have not joined any space yet). The multi account switcher have instead been moved to a top left drop-down menu. So we finally got rid of the drawer, which seems to be a deprecated material design feature anyway. This new UX makes spaces much easier to use. You can long press on them to go to the space settings and long press on any chat in the chat list, to add or remove a chat to (or from) a space.
We still have no support for the spaces summary API though so we don't have yet the ability to discover new rooms inside of a space but this feature might land soon in the Matrix Dart SDK.
Multi Account
FluffyChats multi account is still in beta but got a lot of bug fixes as well. You are now able to sort your accounts in "bundles" which can be very handy. The new account switcher button gives you a much better overview over your connected Matrix accounts now.
Performance
We did a lot refactoring under the hood in our Matrix Dart SDK and have improved our in-app database a lot. On the web it now uses IndexedDB natively while it tunes all database transactions on all platforms. This leads to the fastest FluffyChat experience we ever had and makes the app finally kinda usable with bigger accounts on all platforms. The room list is now lazy loaded which speeds up the app start (especially with multi account enabled) a lot.
Choose your own primary color
This was a long requested feature. You can now choose your favorite color to style your FluffyChat for your needs:
What will you choose? Let me know in the comments. I mostly like blue on my Ubuntu desktop.
New major version?
Ahhh by the way... What does it mean that we now have FluffyChat 1.0.0? It does NOT mean that the previous versions were not yet stable or ready for daily use. It just means that we make so many changes at once that we thought, bumping the first digit of our pseudo-semver version string might make sense. We totally messed up our versioning and are now going to do it better. Promised!!
What's next?
We are often asked: What is the roadmap of FluffyChat?
Well... we still don't have a clear roadmap and might never have. FluffyChat is completely driven by volunteers. But what I can say that we would like to do in the next months is:
Better QA -> We would like to write some integration tests, push release candidates before new releases and involve everyone in testing them to offer the best stability possible.
Native video calls -> Yes! There will soon land support for native video calls in the Matrix Dart SDK and we are going to implement this in FluffyChat.
Stories -> Like you might know from SnapChat, WhatsApp or Instagram, stories are little messages you can send to all of your contacts and which will disappear after 24 hours. I would really like to implement this in FluffyChat!
Better notifications for iOS
Deeper support for spaces
Knocking feature
Drag&Drop for web
But as I said this is what we would like to do. We can't give any warranties on anything. We can only do our best. But you can help us if you like (You don't have to).
Join the FluffyChat community: https://matrix.to/#/#fluffychat:matrix.org
Report bugs at our issue tracker: https://gitlab.com/famedly/fluffychat/-/issues
Help with the translations and join our translators team: https://matrix.to/#/#fluffychat-translation:matrix.org
Help with development directly in GitLab <3
... or support us on Liberapay so we can organize more FluffyChat developer meetings: https://matrix.to/#/#fluffychat-translation:matrix.org
The complete changelog for FluffyChat 1.0.0:
design: Chat backup dialog as a banner
design: Encrypted by design, all users valid is normal not green
design: Move video call button to menu
design: Display edit marker in new bubbles
design: Floating input bar
design: Minor color changes
design: Move device ID to menu
design: Place share button under qr code
design: Redesign and simplify bootstrap
design: Remove cupertino icons
feat: Display typing indicators with gif
feat: Fancy chat list loading animation
feat: New database backend with FluffyBox
feat: Make the main color editable for users
feat: Move styles one settings level up
feat: Multiple mute, pin and mark unread
feat: New chat design
feat: New chat details design
feat: New Public room bottom sheet
feat: New settings design
feat: Nicer images, stickers and videos
feat: nicer loading bar
feat: Open im.fluffychat uris
feat: Redesign multiaccounts and spaces
feat: Redesign start page
feat: Send reactions to multiple events
feat: Speed up app start
feat: Use SalomonBottomBar
feat: Drag&Drop to send multiple files on desktop and web
fix: Adjust color
fix: Automatic key requests
fix: Bootstrap loop
fix: Chat background
fix: Chat list flickering
fix: Contrast in dark mode
fix: Crash when there is no prev message
fix: Do display error image widget
fix: Do not display bottombar in selectmode
fix: Dont enable encryption with bots
fix: Dont loose selected events
fix: Dont rerun server checks
fix: download path for saving files
fix: Hide FAB in new chat page if textfield has focus
fix: Let bottom space bar scroll
fix: Load spaces on app start
fix: Only mark unread if actually marked
fix: Public room design
fix: Remove avatar from room
fix: Remove broken docker job
fix: Report sync status error
fix: Self sign while bootstrap
fix: Sender name prefix in DM rooms
fix: Set room avatar
fix: Various multiaccount fixes
fix: Wrong version in snap packages
What a massive update! Little birds told me we will hear about FluffyChat very soon!
On Web, work continues on notifications and integration with homeserver APIs to improve user experience.
On Mobile, link sharing has been added and work is about to start on notifications.
Polls
Polls are nearly ready! If you enable this feature in labs, you can create a poll with several options, and people can vote on it.
We’re working on the finishing touches, and the first version of polls will be available in a release (on Element Desktop, Web, Android and iOS) within a few weeks.
Community testing
We closed 34 encryption bugs which had been resolved by improvements to the workflows and user interfaces.
Due to the overwhelming success with bug squash sessions in the last few weeks, we are making these a regular feature. Our next session will be on Thursday 9 December at 17:00 UTC.
Trixnity, a multiplatform Matrix SDK written in Kotlin, has grown up since the last release 6 month ago! It has it first release candidate for v1.0.0!
If you don't heard about Trixnity: Trixnity aims to be strongly typed, customizable and easy to use. You can register custom events and Trixnity will take care, that you can send and receive that type.
The most exciting thing is the new trixnity-client module. It provides a high level client implementation and allows you to easily implement clients for Desktop, Mobile and Web. You just need to render data from and passing user interactions to Trixnity. The key features are:
exchangeable database
fast cache on top of the database
E2E (olm, megolm)
verification
room list
timelines
user and room display name calculation
asynchronous message sending without caring about E2E stuff or online status
media support (thumbnail generation, offline "upload", etc.)
redactions
At the moment, Trixnity only supports JVM in all modules, but JS and Native will follow soon (to be exact: when Kotlin 1.6.10 and ktor 2.0.0 is released). I also implemented the module trixnity-olm, which implements the wrappers of libolm for Kotlin JVM/JS/Native.
Cross signing is one of the next big features, I want to implement.
I'm exploring the matrix-rust-sdk on my live stream every week. I'm working on a simple Rust bot for Matrix. Come watch me struggle with the compiler on PeerTube or Twitch every wednesday at 14:00 UTC!
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
matrix.awesomesheep48.me
1042
2
0x1a8510f2.space
1076
3
dendrite.s3cr3t.me
4405
The Adventures of TWIM bot continued
Following the late reports of the spec and hookshot updates, TWIM bot's ship went into hyperspeed. Our dear bot lost control of the ship and we lost its signal. We're doing our best to contact it and hope it's safe!
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
This is the first of three posts on improving type coverage in Sydent. Join us next Friday for the second part!
Sydent is the reference Matrix Identity server. It provides a lookup service, so that you can find a Matrix user via their email address or phone number (if they've chosen to share it).
We recently worked on improving Sydent's type coverage: the proportion of its source code with explicit annotations denoting the kind of data that each variable, expression and return value is expected to hold. These annotations are optional, but if present, they allow tools like mypy to analyze your programs and spot entire classes of bugs before you ship them! In this instance, we aimed to make Sydent pass mypy --strict, which it now does. Here's what the process looked like:
Two lines show two different measures of how well-typed the project is. The grey region covers our two-week sprint towards improving coverage; the earliest data point is from just before previous efforts to improve typing earlier in the year.
In a series of posts, I'd like to reflect on this sprint and share what we've learned. In particular, I aim to:
explain why we wanted to improve type coverage now;
work through examples to see how (if?) mypy could have spotted bugs;
describe the annotation process;
illustrate common patterns I learned along the way;
discuss the checks that mypy provides; and finally
reflect on the state of Python's typing ecosystem.
In this first part, we'll concentrate on the first two topics; the second covers the middle two; and the third the last two.
Why do this now?
It took us a long time (too long) to notice that the Sydent instance serving matrix.orgwas failing to send SMS messages for verification. We suspected that something was going wrong with our API call to OpenMarket. Our first step was to improve logging, so we could start to deduce what was going wrong and why. Whilst trawling through logs, we spotted
one problem which meant we weren't actually sending off the API request in the first place. Further investigation revealed a strings-versus-bytes confusion which meant that we would always (incorrectly) interpret the API response as having failed.
All in all, phone number verification was unknowingly broken in the 2.4.0 release, to be fixed in 2.4.6 a month later. How could we do better? Better test coverage is (as ever) one answer. But it struck me that the two bugs we'd encountered might be ripe for automatic detection:
we created an Awaitable but didn't await it or use it in any way, and
we tried to look up a str key in a dictionary which mapped bytes to bytes.
Could a static analysis tool like mypy detect these? How much work would it take to do so? Are there other bugs and problems we could spot with it? I was curious to answer these questions and learn more about the tools that Python's typing ecosystem provides.
Could typing have spotted these problems?
Let's start with the first of question: what can mypy detect?
The missing await
Instead of writing x = await foo(), we simply had x = foo() and didn't then go on to await x. Mypy doesn't have means to detect this at present. There was interest in this issue on such a feature, with related threads here and here.
Are there other opportunities to spot the error? Here's the relevant bit of source code from before the fix.
The call to requestToken produces a value of type Awaitable[int]. If we tried to assign that to an expression of type int we'd get an error that mypy can spot.
$ cat example.py
async def foo() -> int:
return 1
async def bar():
x = foo() # no error
y: int = foo() # error: rhs is Awaitable[int], but lhs expects int
$ mypy --check-untyped-defs example.py
example.py:6: error: Incompatible types in assignment (expression has type "Coroutine[Any, Any, int]", variable has type "int")
Found 1 error in 1 file (checked 1 source file)
Note that we have to specifically ask mypy to typecheck the body of bar by passing --check-untyped-defs; by default, mypy will only typecheck annotated code.
We might also have been able to detect the error by looking at how we used sid. Unfortunately, the only use of was in a conversion str(sid), which is a perfectly type-safe call for both sid: int and sid: Awaitable[int]. But let's put that aside for a second—suppose we added "sid": sid directly into the resp dictionary. Could mypy have spotted there was a problem with that?
Unfortunately not. Because resp has no annotation, we have to rely on how it's used to spot any type inconsistencies. There's only one use of resp: as the return value from its enclosing function, render_POST. Mypy's only chance to spot a type problem would be to compare the mypy's inferred type for resp to the return type of render_POST. What are those types? We can use reveal_type to see the former is Dict[str, object]. For the latter:
The return type is JsonDict, which is an alias for Dict[str, Any]. This is bad news, because Dict[str, object] and Dict[str, Any] are compatible. Digging a level deeper, this is because sid: Any holds true for both sid: int and sid: Awaitable[int]—so there's no error to spot here. The Any type is compatible with any other type whatsoever! Mypy uses Any as a way to defer all type checking to runtime; mypy won't (and can't!) statically analyse the usage of an expression of type Any. Indeed, mypy's reports will tell you how many Anys you're working with, and offer a variety of options to warn or error on usages of Any.
If we were inserting sid directly into a dictionary, we could do better by annotating the dictionary (or the function's return type) as a TypedDict. This is a way to specify a dictionary with a fixed set of keys, each with a fixed type. It comes in really handy for Sydent, Sygnal and Synapse—all of the Matrix APIs exchange JSON dictionaries, so anything we can do to teach mypy about their shape and types is gold dust.
In short, there were options for detecting this with some code changes, but no magic wand that would have spotted the error in the code as written.
headers = dict(resp.headers.getAllRawHeaders())
request_id = None
if "X-Request-Id" in headers:
request_id = headers["X-Request-Id"][0]
In this sample, resp.headers is a twisted.web.http_headers.Headers instance. getAllRawHeaders is documented as returning an iterable of (bytes, Sequence[bytes]) pairs. Even better, mypy can see this because getAllRawHeaders is annotated (many thanks to the twisted authors for this). Mypy should be able to deduce that we build a dictionary headers: Dict[bytes, Sequence[bytes]. We can check this using reveal_type:
$ mypy
sydent/sms/openmarket.py:110: note: Revealed type is "builtins.dict[builtins.bytes*, typing.Sequence*[builtins.bytes]]"
(The * in builtins.bytes* here means mypy has inferred that the dictionary's keys are bytes, rather than being told explicitly that they must be bytes.)
That's all fine and dandy. But why didn't we spot this before if the annotations were all in place in twisted? Let's put aside the fact that, erm, we weren't running mypy in Sydent's CI until the recent sprint, unlike our otherprojects. Checking out the problematic version, we can run mypy on the file we know to contain the bug.
$ mypy sydent/sms/openmarket.py
sydent/sms/openmarket.py:82: error: Dict entry 0 has incompatible type "str": "int"; expected "str": "str" [dict-item]
sydent/sms/openmarket.py:102: note: Revealed type is "twisted.web.iweb.IResponse*"
sydent/sms/openmarket.py:104: note: Revealed type is "Any"
sydent/sms/openmarket.py:105: note: Revealed type is "Any"
sydent/sms/openmarket.py:106: note: Revealed type is "builtins.dict[Any, Any]"
Found 1 error in 1 file (checked 1 source file)
Ahh, the Any type. As mentioned above, this represents a value whose type can't be statically determined. We're left to runtime checks to detect the problem. But we won't detect it at runtime, because dictionaries don't enforce any kind of type requirements on their keys and values.
The problem here is that mypy can't see that resp.headers is a twisted Headers object. If we could inform it of this, mypy would spot our bug:
$ mypy sydent/sms/openmarket.py
sydent/sms/openmarket.py:82: error: Dict entry 0 has incompatible type "str": "int"; expected "str": "str" [dict-item]
sydent/sms/openmarket.py:104: note: Revealed type is "twisted.web.iweb.IResponse*"
sydent/sms/openmarket.py:106: note: Revealed type is "twisted.web.http_headers.Headers"
sydent/sms/openmarket.py:107: note: Revealed type is "typing.Iterator[Tuple[builtins.bytes, typing.Sequence[builtins.bytes]]]"
sydent/sms/openmarket.py:108: note: Revealed type is "builtins.dict[builtins.bytes*, typing.Sequence*[builtins.bytes]]"
sydent/sms/openmarket.py:114: error: Invalid index type "str" for "Dict[bytes, Sequence[bytes]]"; expected type "bytes" [index]
sydent/sms/openmarket.py:114: error: Argument 1 to "split" of "bytes" has incompatible type "str"; expected "Optional[bytes]" [arg-type]
Found 3 errors in 1 file (checked 1 source file)
There it is, on line 114: Invalid index type "str" for "Dict[bytes, Sequence[bytes]]"; expected type "bytes".
Unfortunately it'd be a pain to annotate our application code to mark every use of IResponse.headers as a Headers object. We'll see a better way to do things in this the next post, which discusses the nitty-gritty details of adding annotations file-by-file.
Many thanks for reading! If you've got any corrections, comments or queries, I'm available on Matrix at @dmrobertson:matrix.org.
This year, the Matrix.org Foundation is excited to host the first ever Matrix.org Foundation and Community devroom at FOSDEM. A full day of talks, demos and workshops around Matrix itself and projects built on top of Matrix.
Matrix is the open source project that publishes the Matrix open standard for secure, decentralised, real-time communication, and its Apache licensed reference implementations.
We encourage people working on the Matrix protocol or building on it in an open source project to submit a proposal! Note that companies are welcome to talk about the Matrix details of their open source projects, but marketing talks are not welcome.
We want this devroom to be a space where the Matrix community can show its work, where developers can talk about the challenges they faced and how they overcame them, and where people can get a glimpse of the future of the Matrix protocol and ecosystem.
Talk Details
The talks will be pre-recorded in January. They will be played during FOSDEM, followed by a session of live Q&A depending on the format. During the playback of the talk, people will be able to comment and ask questions in the chat (via Matrix!).
The talks can follow one of three formats:
5 min lightning talk, ideal to showcase your project and make people want to have a look at it
20 min talk + 10 min Q&A, for topics that can be covered briefly
50 min talk + 10 min Q&A for more complex subjects which need more focus
We strongly encourage you to prepare a demo when it makes sense, so people can actually see what your work looks like in practice!
Of course, the proposal must respect the FOSDEM terms as well:
The conference language is English. All content must relate to Free and Open Source Software. By participating in the event you agree to the publication of your recordings, slides and other content provided under the same licence as all FOSDEM content (CC-BY).
We expect to receive more requests than we have slots available. The devroom organisers (two community members and one core team rep) will be reviewing the proposals and accepting them based on the potential positive impact the project has on Matrix (as defined in by the Mission section of https://matrix.org/foundation).
If a project proposal has been turned down, it doesn't mean we don't believe it has good potential. Maintainers are invited to join the #twim:matrix.org Matrix room to give it some visibility.
NOTE: Synapse 1.49, due out on December 14th, will be the last release of Synapse to support Python 3.6 or PostgreSQL 9.6 per our platform dependency deprecation policy. Accordingly, we will remove support for Ubuntu 18.04 LTS (Bionic) at the same date, as it ships with Python 3.6.
Password resets and identity servers
This release removes the long-deprecated trust_identity_server_for_password_resets configuration option. This option was initially deprecated in Synapse 1.4.0 back in October 2019.
Admins of servers still using this configuration option will need to update their Synapse configuration to send password resets through an SMTP server directly rather than relying on identity servers to send them on their behalf.
New admin APIs and improved alignment with Matrix 1.1
This release also introduces a handful of new admin APIs, allowing administrators to un-shadow-ban users, block a room, and run specific background updates (but we'll talk about this last one a bit later on). The delete room API has also been updated to be able to run in the background or to block a room pre-emptively, even if the server doesn't know about it yet.
This release also brings Synapse into greater alignment with version 1.1 of the Matrix specification by adding support for API paths beginning /_matrix/client/v3 and /_matrix/media/v3.
Background updates
When Synapse updates from one version to another, it might need to run large scale updates on its database. In order to avoid blocking startup for too long while waiting for these updates to run, Synapse runs them in the background after starting.
Lately the Synapse team has been doing some work to improve the performance of these background updates. More specifically, this release includes a performance fix for a background update introduced in Synapse 1.47.0, as well as a new admin API to let admins rerun specific updates.
This release also includes some improved support of MSC3440 to help threading. It also adds support for the stable identifiers from MSC2778, bringing Synapse closer to supporting end-to-end (or end-to-bridge) encryption support for application services.
We also now publish a Docker image, matrixdotorg/synapse:develop, which tracks the development head of Synapse.
Please see the Synapse Release Notes for a complete list of changes in this release.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including Dirk Klimpel, Stanislav Motylkov, Tulir Asokan and Neeeflix.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
MSC2675 (serverside aggregations) is getting lots of updates from Bruno in order to align the proposal with what is currently implemented in the wild (as it's easier to iterate on incremental improvements from a starting point grounded in reality). This MSC is a bit of a special case though, as it was implemented with stable prefixes before the MSC landed (in the before times...).
Regardless, thank you very much to Bruno for going through and finally untangling and help land aggregations in the spec! This MSC is one of four which describe how aggregations should work in Matrix, and it's great to see them finally being properly spec'd, especially as further features start to be built on top of them (such as threading!).
This proposal aims to allow appservices to get ever greater visibility into what is happening on the homeserver, while still maintaining full process separation.
A lot of the time solutions to complex problems require knowing when a user has registered or logged in, with what name/3pids etc, when users change their emails, etc. Hooking into these non-room-based actions can help with developing useful features. Maybe you want an appservice that plays a sound effect whenever a user signs up to your homeserver!
These days, that is often achieved by homeserver implementation-specific solutions, such as modules in Synapse. Being able to notify of these events using a standard API shape would be hugely beneficial to generalisation of projects.
So give the proposal a look over and review if that interests you!
Some exciting MSCs this week, I've been waiting for server-side aggregations for long!
As predicted last week, we released Synapse 1.47.1 on Tuesday. This is a security release which fixes an issue with Synapse's built-in media repository. Admins are strongly encouraged to upgrade.
Otherwise, relatively slow week: a handful of of the team have been away (Happy Thanksgiving, Americans!), but we did release Synapse 1.48.0rc1. Most importantly, this release candidate includes changes to improve the efficiency of large background updates from past releases, which should significantly reduce database load when upgrading. It also adds support for the /v3 APIs defined in version 1.1 of the Matrix specification.
We'll talk more about 1.48 when it's formally released next week, but as always, we appreciate folks trying out the release candidates and letting us know how they behave.
Administrators, keep your users safe: update as soon as you can!
A bit of an earlier update this week, but I wanted to make sure to note that my Hem Charts have been updated to matrix-synapse 1.47.1 for the security fix - and element-web has also been bumped to 1.9.5
mautrix-googlechat has seen lots of improvements over the past few days. New features include:
Bridging edits, deletions, reactions, formatting and read receipts in both directions (even /rainbow somewhat works from Matrix)
Bridging typing notifications and any types of files from Matrix to Google Chat
Bridging Google Meet links from Google Chat to Matrix
Syncing group members from Google Chat
(edits and deletions are only available on Google Workspace accounts, not normal accounts. I have no idea why they did that, but that's just how Google Chat works 🤷)
There's still a bug where it sometimes silently stops receiving messages, which I'm currently trying to solve (or work around). After that I'll make a v0.3.0 release. Backfilling history may also happen in the near future
An add-on for the matrix-appservice-webhooks bridge. Webhooks are essentially web interfaces for applications to "push" data to. The bridge can receive messages in a certain format, which is nice if the notifying app can be configured. Often it cannot.
Do you like to receive notifications in matrix?
Matrix Webhook Receiver (MWR) is an add-on for the matrix-appservice-webhooks bridge. Webhooks are essentially web interfaces for applications to "push" data to.
The bridge can receive messages in a certain format, which is nice if the notifying app can be configured. Often it cannot.
This is where MWR comes in:
It can receive any (JSON) content, optionally reformat it nicely (customizable!), and forward it to the webhooks bridge which will post it to a room for you. If you are running any software service, there is a good chance it can notify you via webhooks!
Right now, several example configurations exist, ready for you to use:
Hey folks! Some exciting new news on the bridge front: I've renamed matrix-github to matrix-hookshot to better reflect it's not-just-GitHub-ness. That's not all though, as there are new features too:
The bridge now supports Rust as a companion language (we're aiming to rewrite critical sections in rust). Some parts of the formatting code have already been rewritten.
The bridge now supports JIRA (full puppeting!)
The bridge now supports generic webhooks too, with the ability to write custom handling code inside the state event to process these hooks into pretty messages.
Basic support for GitHub discussions.
In the works:
A provisioning API to hook into integration managers
More GitLab support
Better GitHub discussions support
We're not quite ready for a 0.2.0 release, but please check us out at https://github.com/Half-Shot/matrix-hookshot.
An interesting update, and Half-Shot even demoes it in today's Matrix Live!
After the 0.9.0 release last week, we have of course been busy fixing all the bugs different people reported. Messing around with the sticker pack editor and then leaving room should not make Nheko crash anymore. The problems where the flatpak has issues starting on Gnome systems are still under investigation. We thought we had a solution, but that seems to have broken other stuff! 💥
Apart from that we have been doing some after release party cleanup. Apart from some refactorings, you can now filter your rooms on whether they are a direct chat or not in the sidebar. This is in addition to the filters we already had for favourites, spaces and your other personal tags. User colors should also now be much less biased towards blue and jdenticons should have more variance. Expect the next release to be a much more colorful experience!
Speaking of colors, Twily made this awesome ZX Spectrum inspired logo after we changed our Gitlab bot to be more colorful! Check it out:
We're still distracted with SDK work and other things less visible for users, but this week we've also released 0.2.22 that fixes login on Element One (and other servers using SSO login and not yet supporting the experimental dehydrated devices).
simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio. Version 2.4.0 provides several new features and a fix.
New Features:
Newlines are now supported when sending markdown messages.
The msgtype of text and markdown messages can now be specified. Text and markdown messages can now optionally be sent as "m.notice" to avoid alerting everybody of the new message. The default msgtype will continue to be "m.text".
New Fixes:
Fixed issue where the homeserver was hardcoded in an http request.
Example usage is shown below:
import simplematrixbotlib as botlib
creds = botlib.Creds("https://home.server", "user", "pass")
bot = botlib.Bot(creds)
PREFIX = '!'
@bot.listener.on_message_event
async def echo(room, message):
match = botlib.MessageMatch(room, message, bot, PREFIX)
if match.is_not_from_this_bot() and match.prefix() and match.command(
"echo"):
response = " ".join(arg for arg in match.args())
await bot.api.send_text_message(room.room_id, response, "m.notice") ## Uses the msgtype of m.notice instead of m.text
bot.run()
A thank you to HarHarLinks for their contributions to version 2.4.0!
This week saw three releases of jOlm which fix a native memory management issue, an Olm API (buffer) issue and add a few other improvements. Everyone is strongly encouraged to update to the latest release.
maubot v0.2.0 was released last weekend. Highlights:
Enabling encryption should be much easier: the device ID can be entered in the web UI or you can just do mbc auth --update-client to automatically log in and store the access token and device ID in maubot.
mbc auth can now log in with SSO.
The standalone mode for running a single plugin with a static config is now mostly functional and somewhat documented.
Also, I finally took a day to figure out Sphinx/autodoc and made some decent-looking autogenerated docs for mautrix-python. I'll probably extend that to generate maubot-specific API references too eventually.
Federated sign-in component for your web app (using Matrix)
This week's update:
Gained 180 stars on GitHub since release (thanks!)
Added login states, accessible from the API
Sign out
Added CSS styling via variables
Updated demo
more on https://github.com/mishushakov/signin-with-matrix
As last week, a note to keep in mind that this is a community project and that there is a MSC to make Matrix more OAuth2 friendly. More on that very soon!
Today is a good day for those calling me a spy, someone not wanting to care about privacy and for those who did publicly harass me for server_stats.
People using the API likely already noticed it wasn't reachable for a while. Effective immediately I am currently leaving all rooms the bot is part of. This will take days or even months considering this are 6397 rooms at the time of writing. I am not going into the motives of why I am shutting it down. It comes down to personal reasons.
There won't be any dump of the data. The source will be kept public. Note though if anyone ever tries to run it that you need about 600GB of space for synapse, a lot of CPU, a lot of RAM and plenty of workers as this can easily crash synapse.
Server_Stats was an incredibly useful project. It pains me a lot to see it go, but it pains me even further that its author got harassed. This is not an acceptable behaviour, and we are better than that as a community. Thanks for this incredible project MTRNord, it's been both exciting and useful.
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.
Today we are releasing Synapse 1.47.1, a security update based on last week's release of Synapse 1.47.0. This release patches one high severity issue affecting Synapse installations 1.47.0 and earlier using the media repository. An attacker could cause these Synapses to download a remote file and store it in a directory outside the media repository.
Attackers cannot control the exact name or destination of the stored file.
To quote from the advisory:
GHSA-3hfw-x7gx-437c / CVE-2021-41281: Path traversal when downloading remote media.
Impact
Synapse instances with the media repository enabled can be tricked into downloading a file from a remote server into an arbitrary directory, potentially outside the media store directory.
The last two directories and file name of the path are chosen randomly by Synapse and cannot be controlled by an attacker, which limits the impact.
Homeservers with the media repository disabled are unaffected. Homeservers configured with a federation whitelist are also unaffected.
The advisory has full details, including workarounds.
This issue was discovered and fixed by our internal security team.
Friday already? Did this week already happen? It looked like the spacetime continuum was broken and we didn't know who did it. We needed witnesses to solve the case: did things really happen this week? And the witnesses showed up! A huge thanks to everyone in the Matrix community who reported their progress, and to everyone currently working on making awesome projects around Matrix!
The case is closed: the week has not been stolen from us. Time appears to have wings, and flies faster than we had anticipated.
Matrix Live 🎙
A very very dense and exciting wrap up of what's happening these days in the Matrix space by Matrix Foundation co-founders Matthew & Amandine! Matrix is stepping up a gear with blazing fast Sync v3, Threading Support, VoIP, VR, a new release of the Spec, always more monthly active users, a full security audit and progress on P2P.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
This week we finally, finally had one of the aggregation-related MSCs, MSC2674 (event relationships) enter final comment period! This MSC, along with several others, document the stuff that powers message edit, reactions, the upcoming threading and polls MSCs, and much more! So it's really great to see the MSCs start to actually land.
Speaking of threading MSC3440 has had a good amount of review from the Spec Core Team last week. Threading in Matrix has been a long awaited feature for chat applications - as well as helping extend the flexibility of Matrix as a data structure even further. The MSC relies on both MSC2674 and MSC2675 (or a modified version of it), so the latter will be an area of focus for review for next week.
This is definitely a feature that I would love to have for chat. Note that this MSC proposes marking a room as unread, rather than a specific point in the room's timeline. This is intentional as noted in the document, as the latter is more complicated, as it intersects with sending out read receipts to other users.
When we release the fix, the changes will be publicly known and bad actors will have the ability to deduce the vulnerability. Most servers don't run release candidates, so releasing the fix in an RC will mean there's a larger window for an attacker to exploit the problem. Internally, there will be an RC deployed to test homeservers and eventually matrix.org. This means we'll be able to confidently recommend the upgrade to server administrators.
And that's the very reason we have dedicated security releases, instead of just rolling the security fixes into a feature release. The security release doesn't contain anything new apart from the security fix so it minimizes the chance of things going wrong.
Keep your servers up to date, and your users safe, administrators!
Implement "best effort" basic IRC moderation in plumbed rooms if bot has ops on IRC
Allow configuring topic sync for plumbs (IRC<->Matrix or one way)
Allow using forward slash (/) as MXID separator for IRC ghosts
Bump max mautrix version to <0.12
Small fixes
Plumb moderation! If the bridge bot has ops on IRC it will do its best to map kicks and bans (regarding IRC users) from Matrix. This definitely isn't perfect and is meant as a convenience.
Topic synchronization is now configurable for plumbs as well to make it possible to share the same topic between an IRC channel and a plumbed Matrix room. Default is still off and it requires the bridge bot to have enough PL to work.
The separator for IRC ghosts can now be changed to forward slash (/) from the default underscore (_). This happens by modifying the regex in the registration file. Only do this for new installations and it will cause all IRC users to duplicate in rooms who you can't remove and probably other bad side effects as well. The default may be changed in the future.
There were lots of refactoring issues so I hope I fixed all of them 🙈.
So, Nheko has a small little release this morning! Okay, that's a lie, it was actually pretty big! You can find the full changelog and some of our binaries here: https://github.com/Nheko-Reborn/nheko/releases/tag/v0.9.0
As always, thank you everyone, who contributed. There were over 30 authors this release! If you haven't tried Nheko in a while, give it a whirl. Lots of stuff changed, some things might not even have been mentioned in TWIM! I put the first few lines of the changelog below for your convenience:
Highlights
Somewhat stable end to end encryption 🔐
Show the room verification status
Configure Nheko to only send to verified users
Store the encryption keys securely in the OS-provided secrets service.
Support online keybackup as well as sharing historical session keys.
Crosssigning bootstrapping 🔄
Crosssigning is used to simplify the verification process. In this release
Nheko can setup crosssigning on a new account without having to use a
different client.
Nheko now also prompts you, if there are any unverified devices and asks you to verify them.
Room directory (Manu) 📂
Search for rooms on your server and other servers. (Prezu)
If their topic interests you and it has the right amount of members, join
the room and the discussion!
Custom sticker packs 🐈⬛
Add a custom sticker picker, that allows you to send stickers from MSC2545.
Support creating new sticker (and emote) packs.
You can share packs in a room and enable them globally or just for that
room.
Token authenticated registration (Callum) 🎫
Sign up with a token to servers, that have otherwise disabled registration.
This was done as part of GSoC and makes it easier to run private servers for
your family and friends!
Features
Support email in registration (required on matrix.org for example)
Warn, if an @room would mention the whole room, because some people don't like that.
Support device removal as well as renaming. (Thulinma)
Show your devices without encryption support, when showing your profile.
(Thulinma)
Move to the next room with unread messages by pressing Alt-A. (Symphorien)
Support jdenticons as a placeholder for rooms or users without avatars.
(LorenDB)
You will need to install https://github.com/Nheko-Reborn/qt-jdenticon
Properly sign macOS builds.
Support animated images like GIF and WebP.
Optionally just play them on hover.
Support accepting knocks in the timeline.
Close a room when clicking it again. (LorenDB)
Close image overlay with escape.
Support .well-known discovery during registration.
Limited spaces support.
No nice display of nested spaces.
No previews of unjoined rooms.
No way to edit a space.
Render room avatar changes in the timeline. (BShipman)
Support pulling out the sidebar to make it wider.
Allow editing pending messages instead of blocking until they are sent.
(balsoft)
Support mnemonics in the context menus. (AppAraat)
Support TOFU for encryption. (Trust on first use)
Right click -> copy address location.
Forward messages. (Jedi18)
Alt-F to forward messages.
A new video and audio player, that should look a bit nicer.
As always, come check us out and chat about Nheko in #nheko:nheko.im ♥
That's one massive update for Nheko! Thanks Nheko contributors!
We fixed regressions reported on our previous release candidates. Sorry for the delay but the current release candidate 1.6.8 should be available on the App Store on Monday
The work to replace Matomo by PostHog has been resumed
We are still working on making the MatrixKit obsolete
Space creation / invites: will start design and code review starting next week
Start space management integration in rooms
Element Android
Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!
Voice message draft is currently under active development. We want the feature to work well before we release it, and it was an opportunity to rework the whole feature, to improve its architecture.
after the Miounne update posted a minute ago here is another one, and it's about time: Time To Matrix (ttm) got v1.4.0 release!
Time To Matrix is a time-like command that will send end of an arbitrary command output and some other info (like exit status) to matrix room.
With new release, following things were added:
arch linux AUR package
automatic room alias resolving, so you can use #ttm:etke.cc instead of !XODRhTLplrymaFicdK:etke.cc
help message and human-readable errors
option to change message type (m.text or m.notice)
option to omit plaintext and send only html-formatted message (to get some more space for log)
option to override message type to m.notice if the command exits with non-zero exit code (by default m.text is sent, so you will get m.notice on failure)
simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio. Version 2.3.0 adds support for additional configuration via config files and other methods. Currently, there is only one setting that can be changed, however many existing and future features will be able to be enabled or disabled via this config.
Example usage is shown below:
"""
random_user
!echo something
echo_bot
something
"""
import simplematrixbotlib as botlib
creds = botlib.Creds("https://home.server", "user", "pass")
config = botlib.Config()
config.load_toml("config.toml")
bot = botlib.Bot(creds, config)
PREFIX = '!'
@bot.listener.on_message_event
async def echo(room, message):
match = botlib.MessageMatch(room, message, bot, PREFIX)
if match.is_not_from_this_bot() and match.prefix() and match.command("echo"):
await bot.api.send_text_message(room.room_id,
" ".join(arg for arg in match.args()))
bot.run()
New feature: if a user on your homeserver reports abuse, Mjölnir may now show the abuse report in your moderation room and offer you two-click moderation options. This feature is considered a preview for the time being.
Performance improvements for protections that need to lock back in the history of a room, decreasing the number of cases in which we could end up timing out.
Many improvements to testing.
Note: Any rumor of a v1.2.0 Docker image borked by yours truly is sadly true. There should be no risk in 1.2.0 but, to be on the safe side, if you have updated to 1.2.0, please update to 1.2.1.
That's one feature I've wanted for a while, and it's going to make moderation a lot easier! Thanks Mjolnir teams for keeping us safe!
Today I want to showcase you MinesTRIX. MinesTRIX is a decentralized social media based on matrix.
The goal is to create a privacy respectful social media using the power of matrix while trying to be as simple as possible.
Two Objectives
Showing that matrix could be used to build such a system.
There was the "Sign in with Matrix" project recently
I tried to do something similar with https://matrix-login.lyc.fi / https://gitlab.com/ptman/matrix-login
An important note on the interesting projects using Matrix for the login: those are community projects, and there are MSCs in the works to "do it right" at the Spec level!
The direction we're headed in the Matrix spec core team is instead towards replacing Matrix's current auth mechanisms with normal Open ID Connect (rather than wrapping our own OIDC-like thing, as we do today) - as per https://github.com/sandhose/matrix-doc/blob/msc/sandhose/oauth2-profile/proposals/2964-oauth2-profile.md The common login flow would then be for users to be authed by their server using a trusted OIDC identity provider, rather than ever trusting arbitrary clients with their credentials.
I have compiled a list of public homeservers available for registration, since previous such efforts to make these homeservers more discoverable fell through. This list serves as a sanitized version of the asra.gr list, with only homeservers intended for public consumption included. It is a static list and does not include pings, but rather than focusing on the technical aspect, my list has an emphasis on the written rules of a homeserver, which I believe to play a larger role in the Matrix experience. Hope this can spark other efforts in maintaining a better list!
That's one very useful list of hand curated servers! The transparency about the inclusion criteria is very much appreciated. Good job!
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
On Tuesday, 23rd November we plan to release Synapse 1.47.1 at 12:00 UTC to address a single high severity issue. This vulnerability was discovered internally by our security team. Synapse is a Matrix homeserver implementation developed by the matrix.org team and the wider Matrix community.
If you're a server administrator running Synapse, please be prepared to upgrade as soon as the patched version is released.
We will be reaching out to downstream packagers to ensure they can prepare patched versions of affected packages at the time of the release. The details of the vulnerability will be disclosed in a blog post on the day of the release. There is so far no evidence of the vulnerability being exploited in the wild.
Thank you for your patience while we work to resolve this issue.
Edit, 2021-11-19: The opening paragraph was amended to note that the vulnerability was discovered internally.
Edit, 2021-11-22: The opening paragraph was amended to include a release time of 12:00 UTC.
NOTE: We anticipate publishing a security release, Synapse 1.47.1, on the coming Tuesday, the 23rd of November.
Synapse 1.47.1 will contain a fix for a high severity issue.
Synapse 1.47.0 features additions to the admin and module APIs, a plethora of fixes for long-standing bugs, and a raft of internal improvements. Server administrators should note that this release removes a deprecated API for deleting a room and deprecates a module callback. Consult the upgrade notes for more details.
We fixed a variety of different bugs in this release, many of which were long-standing. It's good to see them dealt with! Worth mentioning in particular:
Improvements to thehandling of the device_inbox table, which trim redundant data to reduce database bloat.
Fixes related to restartingworkers, to ensure a more reliable upgrade process.
Additionally, work continues on improving type-checking coverage, both inSynapse and inSygnal.
Sydent 2.5.1
This week also saw the release of Sydent 2.5.1, the reference implementation of a Matrix Identity Server. This is a minor release which mainly tidies up error handling to reduce the amount of noise in logs. It should also make it easier for us to diagnose some outstanding bugs which remain to be squashed.
Everything Else
In the background, we're still working away at implementing MSC3440 to facilitate threading. Early tests are promising. We're also exploring MSC2775 as a means to speed up room joins. Both will be long term projects that should hopefully reap major rewards for the Matrix ecosystem. Lastly, there's support for MSC3228 to allow identity servers to provide bespoke invites to spaces. We mentioned this last time in Sydent release notes; now we've got support for it on the Synapse side.
Please see the Synapse Release Notes for a complete list of changes in this release.
