matrix-spec-proposals/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md

MSC4028: Push all encrypted events except for muted rooms

About notifications handling, it is currently not possible for mobile clients which get push notifications to properly implement a mentions-and-keywords-only room when encryption is enabled.

Like it was already described in MSC3996: Encrypted mentions-only rooms:

Currently, every new event in an encrypted room might be pushed to mobile clients due to the default underride rule .m.rule.encrypted.

However, a room can be configured to be mentions-and-keywords-only by creating a room-specific push rule with the id property set to the room ID & actions set to do nothing.¹ Since this is evaluated before the .m.rule.encrypted rule (almost) no events get pushed for a mentions-and-keywords-only room.

Additionally, a room can be configured to be muted by creating the override push rule which matches the room ID & has actions set to do nothing, e.g.:

{
    "rule_id": "!abcdef:example.com",
    "conditions": [
      {
         "key": "room_id",
         "kind": "event_match",
         "pattern": "!abcdef:example.com"
      }
    ],
    "default": false,
    "enabled": true,
    "actions": []
}

Proposal

A new predefined override push rule (.m.rule.encrypted_event) is added just after the .m.rule.master push rule to be able to push all encrypted events:

{
    "rule_id": ".m.rule.encrypted_event",
    "default": true,
    "enabled": true,
    "conditions": [
      {
         "kind": "event_property_is",
         "key": "type",
         "value": "m.room.encrypted"
      }
    ],
    "actions": [
      "notify",
      {
         "set_tweak": "org.matrix.msc4062.dont_email"
      }
   ]
}

Note: The “user-defined rules” are evaluated with a higher priority than “server-default rules”, except for the .m.rule.master rule which has always a higher priority than any other rule (see here). So all the override push rules created to mute rooms will be evaluated before this new one. The new rule will be evaluated before all the other “server-default rules”. When it is enabled, this new rule makes useless the 2 existing underride rules: .m.rule.encrypted_room_one_to_one and .m.rule.encrypted (predefined here).

Receiving notifications

When this push rule matches then homeserver would push the event to all registered clients, similar to how other rules work. Clients would decrypt the event and re-run push rules as normal. If the decrypted event results in no notification, the push is discarded ². If the decryption failed, the new .m.rule.encrypted_event rule will match on client side, but there is no reason to notify the end user, so the push is discarded ² too. Globally the client must discard the push when the event stays encrypted locally because this use case should not happen except if the event was not encrypted for the current session, or if we are in front of an unexpected "Unable To Decrypt" (UTD) issue.

The overall tradeoff is that clients will receive extra pushes some of the time.

Unread notification counts

Since the encryption has been introduced, the actual unread notification counts have to be updated/computed at the client side for the encrypted rooms. Nothing has been specified for that, so each client implemented its own mechanism. We want to harmonize this here.

As soon as the rule .m.rule.encrypted_event is present and enabled in the account push rules set, all the clients with or without push services should compute the unread notification count for each encrypted room during the client-server sync, by implementing the algorithm described below. Its complexity is similar to the existing implementations.

This algorithm depends on the unread notification count unread_notifications received from the server for each joined room in the sync response. It has to be adapted to the potential unread thread notification counts unread_thread_notifications that we will ignore here to ease the description.

Let's consider we receive the following data for a joined encrypted rooms in the sync response:

• number of new encrypted events in the timeline: n
• unread notification count: k

1. if k = 0 -> reset the local notification counts for this room (the unread notification count and the highlight count).
2. else if k <= n -> reset the local notification counts, then decrypt and run push rules on the k most recent events:
   • if the decrypted event resulted in a highlight notification -> increment the local highlight notifications count and the local unread notification count.
   • if the decrypted event results in a notification -> increment only the local unread notifications count.
   • if the decrypted event results in no notification -> no change is required on the local notification counts.
   • if the decryption failed -> increment the local unread notifications count.
3. else if k > n and no gap is mentioned in the sync response for this room -> update the local notification counts by taking into account the potential change of the read receipt position of the current user, then decrypt and run push rules on the n received events by applying the same rules as above.
4. else if k > n and a gap is mentioned in the sync response -> there is 2 options:

• flush the local room storage, this will reset the local notification counts, then decrypt and run push rules on the n received events by applying the same rules as above. In order to compute locally accurate notification counts, back paginate to retrieve (k-n) more encrypted events.

or

• update the local notification counts by taking into account the potential change of the read receipt position of the current user, then decrypt and run push rules on the n received events by applying the same rules as above. In order to compute locally accurate notification counts, back paginate to close the gap or at least retrieve (k-n) more encrypted events.

About the case 4, the back pagination should be mandatory for mentions-and-keywords-only rooms to detect if there are some mentions in the gap. The client may decide to not trigger back pagination for rooms in "all messages" mode, but in that case the local unread notification count is unknown. No badge can be displayed at the application side, only an unread marker.

Note: some unencrypted events (for example m.room.redaction) may be taken into account or not in this unreads count k. This depends on the push rules set. If you force "all messages" mode on a room by creating a room-specific push rule, some unencrypted events (like redaction events) will be included in this count. This is not the case if "all messages" mode is obtained by the combination of server predefined rules. The client have locally all the information (= the push rules) to handle this edge case during the implementation of the algorithm.

Listing notifications

This new push rule will impact the existing endpoint GET /_matrix/client/v3/notifications.

When this rule will be present and enabled in the account push rules set, the clients which retrieve the list of events that it has been notified about with this endpoint, will receive most of the encrypted events (except for muted rooms). They will be able to decrypt and re-run push rules locally. This should fix the notifications panel observed in some web clients where currently the notifications of encrypted events are missing.

Email notifications

A user may set up a pusher to receive emails with unread notifications (see the spec here with kind = "email"). Note that in the current Synapse implementation this pusher sends emails on a delay of ~10 minutes to give people time to see the notification and mark it as read. It looks like some other server implementations don't support email pushers though (see details in this comment).

Currently the existing email notifications are not really relevant in case of encrypted rooms. The users may receive emails with a long bunch of encrypted messages when the room is configured in "all messages" notification mode. This feature should be reviewed again but this is not the topic of this MSC.

In order to not disturb the existing email notifications mechanism, the server implementations which support it should ignore in the email notifications the events pushed because of the new push rule. Otherwise the number of emails will increase by including the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. These rooms are not handled for the moment in the email notifications.

As a first option, we suggested to ignore these pushed events by using the push rule Id: org.matrix.msc4028.encrypted_event. But this would require significant rework to the way Synapse handles push notifications (in particular the database will need to be modified -- this is a massive table so modifying the database is non-trivial). To avoid database modifications, we wrote MSC4062. The new tweak org.matrix.msc4062.dont_email would let us disable the email notification in the push rule definition.

Potential issues

Client side

Unexpected unread counts

Until a client is updated to support this MSC, it may display a high number of unread notifications for encrypted rooms. The first workaround would be to not display the count for the encrypted rooms when the rule .m.rule.encrypted_event is present and enabled in the account push rules set, until the suggested algorithm is implemented, at least for the mentions-and-keywords-only encrypted rooms.

Battery life

The mobile client will wake up more frequently than currently. We will have to control any potential issue with battery life when we will implement this MSC. The impact on the battery life will be definitely lower than the one observed on clients without push service which poll for events in the background. The end user will be able to reduce the impact by muting some encrypted rooms.

Server side

Event fetching

The clients with push services will request more frequently the server to retrieve the event contents. I'm mainly thinking about an encrypted room with a high number of members, each mobile client of these members will try to retrieve simultaneously the event content of each new encrypted event (pushed by the server). This scenario may happen today only when most of the members configure the room in "All messages" mode.

This may be a potential performance issue if each client fetches separately each pushed event. But this should not be the case, because the clients have to run a sync if they want to be able to receive potential new encryption keys through to_devices. So they will retrieve more than one event at the time.

BTW MSC3013: Encrypted Push could potentially mitigate this.

Push sending

Push is a huge bottleneck on the sending side. So calculating what to push and the unread counts (its the same process) is a big performance bottleneck currently when sending in large rooms. A bunch of optimisation has taken place to make that better, but its still quite expensive. We should plan an effort to limit potential issue server side here.

Alternatives

MSC3996: Encrypted mentions-only rooms: the suggested top-level property m.has_mentions may be an option to reduce the volume of pushes. But we would not be able to support notifications on keywords then.

Another alternative would be to define a new action: push_without_notify or silently_notify in order to push all encrypted events without incrementing the notifications count. The client implementation to handle these counts would be then less complex. But this would need to implement more changes in Synapse. The current proposal is just to add a new push rules without changing the rules handling server side. This alternative should be discussed as soon as possible.

Security considerations

None

Future extensions

None

Unstable prefix

During development the new push rule shall use org.matrix.msc4028.encrypted_event instead of .m.rule.encrypted_event.

Caution: this unstable push rule will be enabled by default like the stable push rule, the server owners should wait for the clients to support a minimum this MSC before enabling the MSC server side.

To ensure the server supports the functionality before a spec release, clients should interpret an unstable feature org.matrix.msc4028 with a value of true in the response of the /_matrix/clients/versions endpoint as the feature being supported by the homeserver. Once released in the specification, clients should be checking for server support through advertised spec versions instead.

Once this MSC has successfully passed a merge FCP, clients should use .m.rule.encrypted_event as the right push rule. The unstable org.matrix.msc4028.encrypted_event can be used only as a fallback if the right one is missing (backwards compatibility with slightly old servers).

Dependencies

None

---

1. Via either an explicit "dont_notify" action or an empty array. These are equivalent, see issue #643. ↩︎

2. In the past it was not possible to discard notifications on iOS: if a push notification was received it had to be displayed. This is no longer the case. ↩︎