From ec37fe44270b6dfaa6ad6e143a26bf428d99f3d6 Mon Sep 17 00:00:00 2001 From: Luca Becker Date: Thu, 1 Aug 2019 15:13:04 +0200 Subject: [PATCH 001/162] first draft of MSC2209 --- ...rules-other-keys-in-m.room.power.levels.md | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md diff --git a/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md b/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md new file mode 100644 index 00000000..93d9158a --- /dev/null +++ b/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md @@ -0,0 +1,45 @@ +# Update auth rules to check notifications key in m.room.power_levels + +## Introduction + +The key `notifications` was added to the `m.room.power_levels` event after the finalisation of the auth rules specified in room version 1. This leads to the fact, that this dictionary is not subject to the same validation as other dictionaries in the event, such as `users` or `events`. This especially means that Alice is able to alter any entry within the dictionary including ones, that are above her own power level, which is inconsistent with the behaviour for the other two dictionaries. + +[m.room.power_levels](https://matrix.org/docs/spec/client_server/r0.5.0#m-room-power-levels) +[room version 1](https://matrix.org/docs/spec/rooms/v1) + +## Proposal + +The auth rules regarding `m.room.power_levels` have been established in room version 1. The general idea here was that creators of a new `m.room.power_levels` event are only able to make changes to something that is equal to or lower than their own power level. +So, assuming a room with Alice (50), Bob(100), `m.room.power_levels` at 50, `m.room.name` at 75 and `m.room.topic` at 25 would mean the following: + +* Alice CAN alter `m.room.topic` to any power level up to her own, in this case 50 +* Alice is NOT able to alter `m.room.name` since the current value is higher than her own (75 > 50) +* Alice is NOT able to alter the power level of Bob, since his current value is higher than her own (100 > 50) +* Alice is free to set the level for anything that has not been defined such as `org.alice.message` up to a maximum of 50 + +Later on the key `notifications` was added to the `m.room.power_levels` event. It contains a mapping of notification keys to a power level, that is required for a user to trigger the specific notification. The most famous notification type is the `@room` notification. + +Going back to the original example because this key was added later on, the auth rules make no mention of it, which enables the following behaviour. *It is assumed that `@room` is at 75* + +* Alice can add any key to the dictionary and set the value to any value she wants, including ones higher than her own. +* Alice can alter the value for `@room` to any value she wants, including ones that are higher than her own, even though her own value is lower. + +The proposed solution is to alter the auth rules and add the `notification` dictionary to the same rules that handle `events` and `users`. + +So the rule [10.d](https://matrix.org/docs/spec/rooms/v1.html) of the auth rules in room version 1 would be altered to: + + +For each entry being added, changed or removed in events, users __and notifications__ keys: + + +## Tradeoffs + +The proposed solution would be a breaking change with current room versions and the alternative would be to leave the `notifications` key without any checks. + +## Security considerations + +This is likely to improve security because it prevents malicious users that were only given the right to emit `m.room.power_levels` so that they could alter a very specific key, such as, `invite` from altering the rules established for triggering notifications. + +## Conclusion + +The current spec conform behaviour regarding `notifications` is inconsistent with behaviour shown by the other dictionaries and room administrators are very likely expecting the `notifications` to work the same as them. The required change is minimal is and also in line with the general spirit of the auth rules regarding the `m.room.power_levels` event. From 42dbeee7d356e96f22c4b64bda4a3c0594dfd76f Mon Sep 17 00:00:00 2001 From: Luca Becker Date: Thu, 1 Aug 2019 18:32:11 +0200 Subject: [PATCH 002/162] MSC2209: 2nd draft @uhoregs changes --- ...rules-other-keys-in-m.room.power.levels.md | 64 ++++++++++++++----- 1 file changed, 48 insertions(+), 16 deletions(-) diff --git a/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md b/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md index 93d9158a..db4d9bee 100644 --- a/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md +++ b/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md @@ -2,44 +2,76 @@ ## Introduction -The key `notifications` was added to the `m.room.power_levels` event after the finalisation of the auth rules specified in room version 1. This leads to the fact, that this dictionary is not subject to the same validation as other dictionaries in the event, such as `users` or `events`. This especially means that Alice is able to alter any entry within the dictionary including ones, that are above her own power level, which is inconsistent with the behaviour for the other two dictionaries. +The key `notifications` was added to the `m.room.power_levels` event after the +finalisation of the auth rules specified in room version 1. This leads to the +fact, that this dictionary is not subject to the same validation as other +dictionaries in the event, such as `users` or `events`. This especially means +that Alice is able to alter any entry within the dictionary including ones, +that are above her own power level, which is inconsistent with the behaviour +for the other two dictionaries. [m.room.power_levels](https://matrix.org/docs/spec/client_server/r0.5.0#m-room-power-levels) [room version 1](https://matrix.org/docs/spec/rooms/v1) ## Proposal -The auth rules regarding `m.room.power_levels` have been established in room version 1. The general idea here was that creators of a new `m.room.power_levels` event are only able to make changes to something that is equal to or lower than their own power level. -So, assuming a room with Alice (50), Bob(100), `m.room.power_levels` at 50, `m.room.name` at 75 and `m.room.topic` at 25 would mean the following: +The auth rules regarding `m.room.power_levels` have been established in room +version 1. The general idea here was that creators of a new `m.room.power_levels` +event are only able to make changes to something that is equal to or lower than +their own power level. +So, assuming a room with Alice (50), Bob(100), `m.room.power_levels` at 50, +`m.room.name` at 75 and `m.room.topic` at 25 would mean the following: * Alice CAN alter `m.room.topic` to any power level up to her own, in this case 50 -* Alice is NOT able to alter `m.room.name` since the current value is higher than her own (75 > 50) -* Alice is NOT able to alter the power level of Bob, since his current value is higher than her own (100 > 50) -* Alice is free to set the level for anything that has not been defined such as `org.alice.message` up to a maximum of 50 +* Alice is NOT able to alter `m.room.name` since the current value is higher than +her own (75 > 50) +* Alice is NOT able to alter the power level of Bob, since his current value is +higher than her own (100 > 50) +* Alice is free to set the level for anything that has not been defined such as +`org.alice.message` up to a maximum of 50 -Later on the key `notifications` was added to the `m.room.power_levels` event. It contains a mapping of notification keys to a power level, that is required for a user to trigger the specific notification. The most famous notification type is the `@room` notification. +Later on the key `notifications` was added to the `m.room.power_levels` event. +It contains a mapping of notification keys to a power level, that is required +for a user to trigger the specific notification. The most famous notification +type is the `@room` notification. -Going back to the original example because this key was added later on, the auth rules make no mention of it, which enables the following behaviour. *It is assumed that `@room` is at 75* +Going back to the original example because this key was added later on, the auth +rules make no mention of it, which enables the following behaviour. *It is assumed +that `@room` is at 75* -* Alice can add any key to the dictionary and set the value to any value she wants, including ones higher than her own. -* Alice can alter the value for `@room` to any value she wants, including ones that are higher than her own, even though her own value is lower. +* Alice can add any key to the dictionary and set the value to any value she wants, +including ones higher than her own. +* Alice can alter the value for `@room` to any value she wants, including ones that +are higher than her own, even though her own value is lower. -The proposed solution is to alter the auth rules and add the `notification` dictionary to the same rules that handle `events` and `users`. +The proposed solution is to alter the auth rules and add the `notifications` dictionary +to the same rules that handle `events` and `users`. -So the rule [10.d](https://matrix.org/docs/spec/rooms/v1.html) of the auth rules in room version 1 would be altered to: +So the rule [10.d](https://matrix.org/docs/spec/rooms/v1.html) of the auth rules in +room version 1 would be updated in a new room version to: -For each entry being added, changed or removed in events, users __and notifications__ keys: +For each entry being added, changed or removed in events, users __and notifications__ +keys: ## Tradeoffs -The proposed solution would be a breaking change with current room versions and the alternative would be to leave the `notifications` key without any checks. +The proposed solution would be a breaking change with current room versions and +the alternative would be to leave the `notifications` key without any checks. ## Security considerations -This is likely to improve security because it prevents malicious users that were only given the right to emit `m.room.power_levels` so that they could alter a very specific key, such as, `invite` from altering the rules established for triggering notifications. +This is likely to improve security because it prevents malicious users that were +only given the right to emit `m.room.power_levels` so that they could alter a very +specific key, such as `invite`, from altering the rules established for triggering +notifications. ## Conclusion -The current spec conform behaviour regarding `notifications` is inconsistent with behaviour shown by the other dictionaries and room administrators are very likely expecting the `notifications` to work the same as them. The required change is minimal is and also in line with the general spirit of the auth rules regarding the `m.room.power_levels` event. +The current spec conform behaviour regarding `notifications` is inconsistent with +behaviour shown by the other dictionaries and room administrators are very likely +expecting the `notifications` to work the same as them. The required change is minimal +is and also in line with the general spirit of the auth rules regarding the +`m.room.power_levels` event. A new room version is, however, required. This can be +done with other pending changes. From 25e3602bbe0f76285f75abd4277959ee8add4948 Mon Sep 17 00:00:00 2001 From: Luca Becker Date: Sat, 3 Aug 2019 09:15:32 +0200 Subject: [PATCH 003/162] 3rd draft of MSC2209 --- .../2209-auth-rules-other-keys-in-m.room.power.levels.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md b/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md index db4d9bee..594c0b99 100644 --- a/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md +++ b/proposals/2209-auth-rules-other-keys-in-m.room.power.levels.md @@ -19,7 +19,7 @@ The auth rules regarding `m.room.power_levels` have been established in room version 1. The general idea here was that creators of a new `m.room.power_levels` event are only able to make changes to something that is equal to or lower than their own power level. -So, assuming a room with Alice (50), Bob(100), `m.room.power_levels` at 50, +So, assuming a room with Alice (50), Bob (100), `m.room.power_levels` at 50, `m.room.name` at 75 and `m.room.topic` at 25 would mean the following: * Alice CAN alter `m.room.topic` to any power level up to her own, in this case 50 @@ -51,8 +51,8 @@ So the rule [10.d](https://matrix.org/docs/spec/rooms/v1.html) of the auth rules room version 1 would be updated in a new room version to: -For each entry being added, changed or removed in events, users __and notifications__ -keys: +> For each entry being added, changed or removed in events, users __and notifications__ +>keys: ## Tradeoffs From b0f873785d338922876990d30e542079888b11d8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 21 Aug 2019 18:03:17 -0600 Subject: [PATCH 004/162] Proposal for room version 6 --- proposals/2240-rooms-v6.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 proposals/2240-rooms-v6.md diff --git a/proposals/2240-rooms-v6.md b/proposals/2240-rooms-v6.md new file mode 100644 index 00000000..a67a1ad7 --- /dev/null +++ b/proposals/2240-rooms-v6.md @@ -0,0 +1,21 @@ +# MSC2240: Room Version 6 + +A new room version, `6`, is proposed using [room version 5](https://matrix.org/docs/spec/rooms/v5.html) +as a starting point and incorporating the following MSCs: + +* [MSC2174](https://github.com/matrix-org/matrix-doc/pull/2174) - Moving the `redacts` key. +* [MSC2175](https://github.com/matrix-org/matrix-doc/pull/2175) - Removing the `creator` from create events. +* [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) - Modern redaction rules. +* [MSC2209](https://github.com/matrix-org/matrix-doc/pull/2209) - Including notifications in power level auth rules. +* [MSC2212](https://github.com/matrix-org/matrix-doc/pull/2212) - Third party user power levels. +* [MSC2213](https://github.com/matrix-org/matrix-doc/pull/2213) - Rejoinability of invite-only rooms. +* [MSC1849](https://github.com/matrix-org/matrix-doc/pull/1849) - Event aggregations/relationships. + +MSCs which require a new room version and are excluded from v6 are: +* [MSC2214](https://github.com/matrix-org/matrix-doc/pull/2214) - Joining upgraded private rooms. + This MSC is excluded due to the solution not being fully worked out. As this v6 proposal progresses, + it is possible for it to be included in the future. + + +Room version 6 upon being added to the specification shall be considered stable. No other room versions +are affected by this MSC. From d1efd488b7c3d62472db063a86b7084403cc4ab8 Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Fri, 30 Aug 2019 14:43:17 +0100 Subject: [PATCH 005/162] Proposal for mandating lowercasing when processing e-mail address localparts --- proposals/xxxx-email-lowercase.md | 65 +++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 proposals/xxxx-email-lowercase.md diff --git a/proposals/xxxx-email-lowercase.md b/proposals/xxxx-email-lowercase.md new file mode 100644 index 00000000..cef94fa8 --- /dev/null +++ b/proposals/xxxx-email-lowercase.md @@ -0,0 +1,65 @@ +# Proposal for mandating lowercasing when processing e-mail address localparts + +[RFC822](https://tools.ietf.org/html/rfc822#section-3.4.7) mandates that +localparts in e-mail addresses must be processed with the original case +preserved. [The Matrix spec](https://matrix.org/docs/spec/appendices#pid-types) +doesn't mandate anything about processing e-mail addresses, other than the fact +that the domain part must be converted to lowercase, as domain names are case +insensitive. + +On the other hand, most major e-mail providers nowadays process the localparts +of e-mail addresses as case insensitive. Therefore, most users expect localparts +to be treated case insensitively, and get confused when it's not. Some users, +for example, get confused over the fact that registering a 3PID association for +`john.doe@example.com` doesn't mean that the association is valid for +`John.Doe@example.com`, and don't expect to be expected to remember the exact +case they used to initially register the association (and sometimes get locked +out of their account because of that). So far we've seen that confusion occur +and lead to troubles of various degrees over several deployments of Synapse and +Sydent. + +## Proposal + +This proposal suggests changing the specification of the e-mail 3PID type in +[the Matrix spec appendices](https://matrix.org/docs/spec/appendices#pid-types) +to mandate that any e-mail address must be entirely converted to lowercase +before any processing, instead of only its domain. + +## Other considered solutions + +A first look at this issue concluded that there was no need to add such a +mention to the spec, and that it can be considered as an implementation detail. +However, [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) changes +this: because hashing functions are case sensitive, we need both clients and +identity servers to follow the same policy regarding case sensitivity. + +## Tradeoffs + +Implementing this MSC in identity servers would require the databases of +existing identity servers to be updated in a large part to convert the email +addresses of existing associations to lowercase, in order to avoid conflicts. +However, most of this update can usually be done by a single database query (or +a background job running at startup), so the UX improvement outweights this +trouble. + +## Potential issues + +Some users might already have two different accounts associated with the same +e-mail address but with different cases. This appears to happen in a small +number of cases, however, and can be dealt by the identity server's maintainer. + +For example, with Sydent, the process of dealing with such cases could look +like: + +1. list all MXIDs associated with a variant of the email address, and the + timestamp of that association +2. delete all associations except for the most recent one [0] +3. inform the user of the deletion by sending them an email notice to the email + address + +## Footnotes + +[0]: This is specific to Sydent because of a bug it has where v1 lookups are +already processed case insensitively, which means it will return the most recent +association for any case of the given email address, therefore keeping only this +association won't change the result of v1 lookups. From 60354f8cf90753667d797dd97a348550b5c121db Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Fri, 30 Aug 2019 14:47:02 +0100 Subject: [PATCH 006/162] MSC number --- proposals/{xxxx-email-lowercase.md => 2265-email-lowercase.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{xxxx-email-lowercase.md => 2265-email-lowercase.md} (100%) diff --git a/proposals/xxxx-email-lowercase.md b/proposals/2265-email-lowercase.md similarity index 100% rename from proposals/xxxx-email-lowercase.md rename to proposals/2265-email-lowercase.md From 524ec52f73fded7b43a0ae44b1c9b2e4175319c2 Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Mon, 2 Sep 2019 13:41:07 +0100 Subject: [PATCH 007/162] Wording Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- proposals/2265-email-lowercase.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2265-email-lowercase.md b/proposals/2265-email-lowercase.md index cef94fa8..0e862272 100644 --- a/proposals/2265-email-lowercase.md +++ b/proposals/2265-email-lowercase.md @@ -12,7 +12,7 @@ of e-mail addresses as case insensitive. Therefore, most users expect localparts to be treated case insensitively, and get confused when it's not. Some users, for example, get confused over the fact that registering a 3PID association for `john.doe@example.com` doesn't mean that the association is valid for -`John.Doe@example.com`, and don't expect to be expected to remember the exact +`John.Doe@example.com`, and don't expect to have to remember the exact case they used to initially register the association (and sometimes get locked out of their account because of that). So far we've seen that confusion occur and lead to troubles of various degrees over several deployments of Synapse and From 552f71a9f9cb088f501c0b899704908e71ba0dd9 Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Mon, 2 Sep 2019 13:41:23 +0100 Subject: [PATCH 008/162] Wording Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- proposals/2265-email-lowercase.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2265-email-lowercase.md b/proposals/2265-email-lowercase.md index 0e862272..50bad406 100644 --- a/proposals/2265-email-lowercase.md +++ b/proposals/2265-email-lowercase.md @@ -28,7 +28,7 @@ before any processing, instead of only its domain. ## Other considered solutions A first look at this issue concluded that there was no need to add such a -mention to the spec, and that it can be considered as an implementation detail. +mention to the spec, and that it can be considered an implementation detail. However, [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) changes this: because hashing functions are case sensitive, we need both clients and identity servers to follow the same policy regarding case sensitivity. From bddadfeb184b6546a2e74782d2786bfc168075ee Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Mon, 2 Sep 2019 13:41:33 +0100 Subject: [PATCH 009/162] Typo Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- proposals/2265-email-lowercase.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2265-email-lowercase.md b/proposals/2265-email-lowercase.md index 50bad406..166de458 100644 --- a/proposals/2265-email-lowercase.md +++ b/proposals/2265-email-lowercase.md @@ -39,7 +39,7 @@ Implementing this MSC in identity servers would require the databases of existing identity servers to be updated in a large part to convert the email addresses of existing associations to lowercase, in order to avoid conflicts. However, most of this update can usually be done by a single database query (or -a background job running at startup), so the UX improvement outweights this +a background job running at startup), so the UX improvement outweighs this trouble. ## Potential issues From 997360995ccc5d3d46b2d8983601afc10a0d47ac Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Mon, 2 Sep 2019 13:41:49 +0100 Subject: [PATCH 010/162] Wording Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- proposals/2265-email-lowercase.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2265-email-lowercase.md b/proposals/2265-email-lowercase.md index 166de458..3bc3c445 100644 --- a/proposals/2265-email-lowercase.md +++ b/proposals/2265-email-lowercase.md @@ -46,7 +46,7 @@ trouble. Some users might already have two different accounts associated with the same e-mail address but with different cases. This appears to happen in a small -number of cases, however, and can be dealt by the identity server's maintainer. +number of cases, however, and can be dealt with by the identity server's maintainer. For example, with Sydent, the process of dealing with such cases could look like: From 520c76a1cb8e8669ac624be3c36cc606aeb4cba6 Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Mon, 2 Sep 2019 13:56:01 +0100 Subject: [PATCH 011/162] Spell out that the proposal also concerns homeservers --- proposals/2265-email-lowercase.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/proposals/2265-email-lowercase.md b/proposals/2265-email-lowercase.md index 3bc3c445..5698a8c2 100644 --- a/proposals/2265-email-lowercase.md +++ b/proposals/2265-email-lowercase.md @@ -35,18 +35,19 @@ identity servers to follow the same policy regarding case sensitivity. ## Tradeoffs -Implementing this MSC in identity servers would require the databases of -existing identity servers to be updated in a large part to convert the email -addresses of existing associations to lowercase, in order to avoid conflicts. -However, most of this update can usually be done by a single database query (or -a background job running at startup), so the UX improvement outweighs this -trouble. +Implementing this MSC in identity servers and homeservers might require the +databases of existing instances to be updated in a large part to convert the +email addresses of existing associations to lowercase, in order to avoid +conflicts. However, most of this update can usually be done by a single database +query (or a background job running at startup), so the UX improvement outweighs +this trouble. ## Potential issues Some users might already have two different accounts associated with the same e-mail address but with different cases. This appears to happen in a small -number of cases, however, and can be dealt with by the identity server's maintainer. +number of cases, however, and can be dealt with by the identity server's or the +homeserver's maintainer. For example, with Sydent, the process of dealing with such cases could look like: From 6b0a8505ec4ef375d7b8dad0baeb01d12061bd09 Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Thu, 19 Sep 2019 17:34:25 +0100 Subject: [PATCH 012/162] Propose case folding instead of lowercasing --- proposals/2265-email-lowercase.md | 51 ++++++++++++++++++++++++++----- 1 file changed, 43 insertions(+), 8 deletions(-) diff --git a/proposals/2265-email-lowercase.md b/proposals/2265-email-lowercase.md index 5698a8c2..935e6f2c 100644 --- a/proposals/2265-email-lowercase.md +++ b/proposals/2265-email-lowercase.md @@ -1,4 +1,4 @@ -# Proposal for mandating lowercasing when processing e-mail address localparts +# Proposal for mandating case folding when processing e-mail address localparts [RFC822](https://tools.ietf.org/html/rfc822#section-3.4.7) mandates that localparts in e-mail addresses must be processed with the original case @@ -22,8 +22,13 @@ Sydent. This proposal suggests changing the specification of the e-mail 3PID type in [the Matrix spec appendices](https://matrix.org/docs/spec/appendices#pid-types) -to mandate that any e-mail address must be entirely converted to lowercase -before any processing, instead of only its domain. +to mandate that, before any processing, e-mail address localparts must go +through a full case folding based on [the unicode mapping +file](https://www.unicode.org/Public/8.0.0/ucd/CaseFolding.txt), on top of +having their domain lowercased. + +This means that `Strauß@Example.com` must be considered as being the same e-mail +address as `strauss@example.com`. ## Other considered solutions @@ -33,17 +38,24 @@ However, [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) changes this: because hashing functions are case sensitive, we need both clients and identity servers to follow the same policy regarding case sensitivity. +An initial version of this proposal proposed to mandate lowercasing e-mail +addresses instead of case folding them, however it was pointed out that this +solution might not be the best and most future-proof one. + +Unicode normalisation was also looked at but judged unnecessary. + ## Tradeoffs Implementing this MSC in identity servers and homeservers might require the -databases of existing instances to be updated in a large part to convert the -email addresses of existing associations to lowercase, in order to avoid -conflicts. However, most of this update can usually be done by a single database -query (or a background job running at startup), so the UX improvement outweighs -this trouble. +databases of existing instances to be updated in a large part to case fold the +email addresses of existing associations, in order to avoid conflicts. However, +most of this update can usually be done by a background job running at startup, +so the UX improvement outweighs this trouble. ## Potential issues +### Conflicts with existing associations + Some users might already have two different accounts associated with the same e-mail address but with different cases. This appears to happen in a small number of cases, however, and can be dealt with by the identity server's or the @@ -58,6 +70,29 @@ like: 3. inform the user of the deletion by sending them an email notice to the email address +### Storing and querying + +Most database engines don't support case folding, therefore querying all +e-mail addresses matching a case folded e-mail address might not be trivial, +e.g. an identity server querying all associations for `strauss@example.com` when +processing a `/lookup` request would be expected to also get associations for +`Strauß@Example.com`. + +To address this issue, implementation maintainers are strongly encouraged to +make e-mail addresses go through a full case folding before storing them. + +### Implementing case folding + +The need for case folding in services on the Internet doesn't seem to be very +large currently (probably due to its young age), therefore there seem to be only +a few third-party implementation librairies out there. However, both +[Go](https://godoc.org/golang.org/x/text/cases#Fold), [Python +2](https://docs.python.org/2/library/stringprep.html#stringprep.map_table_b3) +and [Python 3](https://docs.python.org/3/library/stdtypes.html#str.casefold) +support it natively, and [a third-party JavaScript +implementation](https://github.com/ar-nelson/foldcase) exists which, although +young, seems to be working. + ## Footnotes [0]: This is specific to Sydent because of a bug it has where v1 lookups are From e658b1707088887fe1d422a00c609cd1e92a4ab6 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 16 Dec 2019 15:47:03 -0500 Subject: [PATCH 013/162] initial version of spec for key backups --- .../definitions/key_backup_data.yaml | 50 ++ api/client-server/key_backup.yaml | 774 ++++++++++++++++++ .../modules/end_to_end_encryption.rst | 125 +++ 3 files changed, 949 insertions(+) create mode 100644 api/client-server/definitions/key_backup_data.yaml create mode 100644 api/client-server/key_backup.yaml diff --git a/api/client-server/definitions/key_backup_data.yaml b/api/client-server/definitions/key_backup_data.yaml new file mode 100644 index 00000000..5111a663 --- /dev/null +++ b/api/client-server/definitions/key_backup_data.yaml @@ -0,0 +1,50 @@ +# Copyright 2019 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: object +title: KeyBackupData +description: "The key data" +properties: + first_message_index: + description: |- + The index of the first message in the session that the key can decrypt. + type: integer + example: 1 + forwarded_count: + description: |- + The number of times this key has been forwarded. + type: integer + example: 0 + is_verified: + description: |- + Whether the device backing up the key was verified the device that the key + is from. + type: boolean + example: false + session_data: + description: |- + Algorithm-dependent data. See the documentation for the backup + algorithms in `Server-side key backups`_ for more information on the + expected format of the data. + type: object + example: { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } +required: + - first_message_index + - forwarded_count + - is_verified + - session_data diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml new file mode 100644 index 00000000..c55b8f27 --- /dev/null +++ b/api/client-server/key_backup.yaml @@ -0,0 +1,774 @@ +# Copyright 2019 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +swagger: '2.0' +info: + title: "Matrix Client-Server Key Backup API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/%CLIENT_MAJOR_VERSION% +consumes: + - application/json +produces: + - application/json +securityDefinitions: + $ref: definitions/security.yaml +paths: + "/room_keys/version": + post: + summary: Create a new backup. + description: |- + Creates a new backup + operationId: postRoomKeysVersion + security: + - accessToken: [] + parameters: + - in: body + name: version + description: "The backup configuration" + schema: + type: object + properties: + algorithm: + description: The algorithm used for storing backups. + type: string + enum: ["m.megolm_backup.v1.curve25519-aes-sha2"] + example: "m.megolm_backup.v1.curve25519-aes-sha2" + auth_data: + description: |- + Algorithm-dependent data. See the documentation for the backup + algorithms in `Server-side key backups`_ for more information on the + expected format of the data. + type: object + example: { + "public_key": "abcdefg", + "signatures": { + "@alice:example.org": { + "ed25519:deviceid": "signature" + } + } + } + required: + - algorithm + - auth_data + responses: + 200: + description: + The version of the new backup. + schema: + type: object + properties: + version: + type: string + description: The backup version + example: "1" + required: + - version + tags: + - End-to-end encryption + "/room_keys/version/{version}": + get: + summary: Get information about an existing backup. + description: |- + Get information about an existing backup. + operationId: getRoomKeysVersion + security: + - accessToken: [] + parameters: + - in: path + type: string + name: version + description: |- + Optional. The backup version to get. If omitted, the current backup + is returned. + required: false + x-example: "1" + responses: + 200: + description: + The information about the requested backup. + schema: + type: object + properties: + algorithm: + type: string + description: The algorithm used for storing backups. + type: string + enum: ["m.megolm_backup.v1.curve25519-aes-sha2"] + example: "m.megolm_backup.v1.curve25519-aes-sha2" + auth_data: + description: |- + Algorithm-dependent data. See the documentation for the backup + algorithms in `Server-side key backups`_ for more information on the + expected format of the data. + type: object + example: { + "public_key": "abcdefg", + "signatures": { + "@alice:example.org": { + "ed25519:deviceid": "signature" + } + } + } + count: + description: The number of key stored in the backup. + type: integer + example: 42 + etag: + description: |- + An opaque string representing stored keys in the backup. + Clients can compare it with the ``etag`` value they received + in the request of their last key storage request. If not + equal, another client has modified the backup. + type: string + example: "anopaquestring" + version: + type: string + description: The backup version + example: "1" + required: + - algorithm + - auth_data + - count + - etag + - version + 404: + description: + The backup specified does not exist + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + tags: + - End-to-end encryption + put: + summary: Update information about an existing backup. + description: |- + Update information about an existing backup. Only ``auth_data`` can be modified. + operationId: putRoomKeysVersion + security: + - accessToken: [] + parameters: + - in: path + type: string + name: version + description: |- + The backup version to update. + required: true + x-example: "1" + - in: body + name: version + description: "The backup configuration" + schema: + type: object + properties: + algorithm: + description: |- + The algorithm used for storing backups. Must be the same as + the algorithm currently used by the backup. + type: string + enum: ["m.megolm_backup.v1.curve25519-aes-sha2"] + example: "m.megolm_backup.v1.curve25519-aes-sha2" + auth_data: + description: |- + Algorithm-dependent data. See the documentation for the backup + algorithms in `Server-side key backups`_ for more information on the + expected format of the data. + type: object + example: { + "public_key": "abcdefg", + "signatures": { + "@alice:example.org": { + "ed25519:deviceid": "signature" + } + } + } + version: + description: |- + The backup version. If present, must be the same as the + version in the path parameter. + type: string + example: "1" + required: + - algorithm + - auth_data + responses: + 200: + description: The update succeeded + schema: + type: object + properties: {} + 404: + description: The backup specified does not exist + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + tags: + - End-to-end encryption + "/room_keys/keys/{roomId}/{sessionId}": + put: + summary: Store a key in the backup + description: |- + Store a key in the backup. + operationId: postRoomKeysKeyRoomIdSessionId + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup in which to store the key. Must be the current backup. + required: true + x-example: "1" + - in: path + type: string + name: roomId + description: The ID of the room that the key is for. + required: true + x-example: "!roomid:example.org" + - in: path + type: string + name: sessionId + description: The ID of the megolm session that the key is for. + required: true + x-example: "sessionid" + - in: body + name: data + description: "The key data" + schema: + "$ref": "definitions/key_backup_data.yaml" + responses: + 200: + description: The update succeeded + schema: + type: object + properties: + etag: + description: |- + The new etag value representing stored keys in the backup. + See ``GET /room_keys/version/{version}`` for more details. + type: string + example: "abcdefg" + count: + description: The number of keys stored in the backup + type: integer + example: 10 + required: + - etag + - count + 403: + description: |- + The version specified does not match the current backup version. + The current version will be included in the ``current_version`` + field. + examples: + application/json: { + "errcode": "M_WRONG_ROOM_KEYS_VERSION", + "error": "Wrong backup version.", + "current_version": "42" + } + schema: + "$ref": "definitions/errors/error.yaml" + tags: + - End-to-end encryption + get: + summary: Retrieve a key from the backup + description: |- + Retrieve a key from the backup. + operationId: getRoomKeysKeyRoomIdSessionId + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup from which to retrieve the key + required: true + x-example: "1" + - in: path + type: string + name: roomId + description: The ID of the room that the requested key is for. + required: true + x-example: "!roomid:example.org" + - in: path + type: string + name: sessionId + description: The ID of the megolm session whose key is requested. + required: true + x-example: "sessionid" + responses: + 200: + description: The key data + schema: + "$ref": "definitions/key_backup_data.yaml" + 404: + description: The key or backup was not found. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Key not found." + } + schema: + "$ref": "definitions/errors/error.yaml" + delete: + summary: Delete a key from the backup + description: |- + Delete a key from the backup. + operationId: deleteRoomKeysKeyRoomIdSessionId + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup from which to delete the key + required: true + x-example: "1" + - in: path + type: string + name: roomId + description: The ID of the room that the specified key is for. + required: true + x-example: "!roomid:example.org" + - in: path + type: string + name: sessionId + description: The ID of the megolm session whose key is to be deleted. + required: true + x-example: "sessionid" + responses: + 200: + description: The update succeeded + schema: + type: object + properties: + etag: + description: |- + The new etag value representing stored keys in the backup. + See ``GET /room_keys/version/{version}`` for more details. + type: string + example: "abcdefg" + count: + description: The number of keys stored in the backup + type: integer + example: 10 + required: + - etag + - count + 404: + description: |- + The backup was not found. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + "/room_keys/keys/{roomId}": + put: + summary: Store several keys in the backup for a given room. + description: |- + Store a key in the backup. + operationId: postRoomKeysKeyRoomId + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup in which to store the keys. Must be the current backup. + required: true + x-example: "1" + - in: path + type: string + name: roomId + description: The ID of the room that the keys are for. + required: true + x-example: "!roomid:example.org" + - in: body + description: "The backup data" + name: backupData + schema: + type: object + properties: + sessions: + type: object + description: |- + A map of session IDs to key data. + additionalProperties: + allOf: + - $ref: "definitions/key_backup_data.yaml" + example: { + "sessionid1": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } + } + responses: + 200: + description: The update succeeded + schema: + type: object + properties: + etag: + description: |- + The new etag value representing stored keys in the backup. + See ``GET /room_keys/version/{version}`` for more details. + type: string + example: "abcdefg" + count: + description: The number of keys stored in the backup + type: integer + example: 10 + required: + - etag + - count + 403: + description: |- + The version specified does not match the current backup version. + The current version will be included in the ``current_version`` + field. + examples: + application/json: { + "errcode": "M_WRONG_ROOM_KEYS_VERSION", + "error": "Wrong backup version.", + "current_version": "42" + } + schema: + "$ref": "definitions/errors/error.yaml" + 404: + description: |- + The backup was not found. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + tags: + - End-to-end encryption + get: + summary: Retrieve the keys from the backup for a given room + description: |- + Retrieve the keys from the backup for a given room + operationId: getRoomKeysKeyRoomId + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup from which to retrieve the key + required: true + x-example: "1" + - in: path + type: string + name: roomId + description: The ID of the room that the requested key is for. + required: true + x-example: "!roomid:example.org" + responses: + 200: + description: |- + The key data. If no keys are found, then an object with an empty + ``sessions`` property will be returned (``{"sessions": {}}``). + schema: + type: object + properties: + sessions: + type: object + description: |- + A map of session IDs to key data. + additionalProperties: + allOf: + - $ref: "definitions/key_backup_data.yaml" + example: { + "sessionid1": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } + } + 404: + description: |- + The backup was not found. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + delete: + summary: Delete a key from the backup + description: |- + Delete a key from the backup. + operationId: deleteRoomKeysKeyRoomId + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup from which to delete the key + required: true + x-example: "1" + - in: path + type: string + name: roomId + description: The ID of the room that the specified key is for. + required: true + x-example: "!roomid:example.org" + responses: + 200: + description: The update succeeded + schema: + type: object + properties: + etag: + description: |- + The new etag value representing stored keys in the backup. + See ``GET /room_keys/version/{version}`` for more details. + type: string + example: "abcdefg" + count: + description: The number of keys stored in the backup + type: integer + example: 10 + required: + - etag + - count + 404: + description: |- + The backup was not found. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + "/room_keys/keys": + put: + summary: Store several keys in the backup. + description: |- + Store several keys in the backup. + operationId: postRoomKeysKey + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup in which to store the keys. Must be the current backup. + required: true + x-example: "1" + - in: body + description: "The backup data" + name: backupData + schema: + type: object + properties: + rooms: + type: object + description: |- + A map of room IDs to session IDs to key data. + additionalProperties: + type: object + additionalProperties: + allOf: + - $ref: "definitions/key_backup_data.yaml" + example: { + "!room:example.org": { + "sessions": { + "sessionid1": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } + } + } + } + responses: + 200: + description: The update succeeded + schema: + type: object + properties: + etag: + description: |- + The new etag value representing stored keys in the backup. + See ``GET /room_keys/version/{version}`` for more details. + type: string + example: "abcdefg" + count: + description: The number of keys stored in the backup + type: integer + example: 10 + required: + - etag + - count + 403: + description: |- + The version specified does not match the current backup version. + The current version will be included in the ``current_version`` + field. + examples: + application/json: { + "errcode": "M_WRONG_ROOM_KEYS_VERSION", + "error": "Wrong backup version.", + "current_version": "42" + } + schema: + "$ref": "definitions/errors/error.yaml" + 404: + description: |- + The backup was not found. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + tags: + - End-to-end encryption + get: + summary: Retrieve the keys from the backup for a given room + description: |- + Retrieve the keys from the backup for a given room + operationId: getRoomKeysKeyRoomId + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup from which to retrieve the keys. If omitted, the keys are + retrieved from the current backup. + x-example: "1" + - in: path + type: string + name: roomId + description: The ID of the room that the requested key is for. + required: true + x-example: "!roomid:example.org" + responses: + 200: + description: |- + The key data. If no keys are found, then an object with an empty + ``rooms`` property will be returned (``{"rooms": {}}``). + schema: + type: object + properties: + rooms: + type: object + description: |- + A map of room IDs to session IDs to key data. + additionalProperties: + type: object + additionalProperties: + allOf: + - $ref: "definitions/key_backup_data.yaml" + example: { + "!room:example.org": { + "sessions": { + "sessionid1": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } + } + } + } + 404: + description: The backup was not found. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version." + } + schema: + "$ref": "definitions/errors/error.yaml" + delete: + summary: Delete a key from the backup + description: |- + Delete a key from the backup. + operationId: deleteRoomKeysKeyRoomId + security: + - accessToken: [] + parameters: + - in: query + type: string + name: version + description: |- + The backup from which to delete the key + required: true + x-example: "1" + - in: path + type: string + name: roomId + description: The ID of the room that the specified key is for. + required: true + x-example: "!roomid:example.org" + responses: + 200: + description: The update succeeded + schema: + type: object + properties: + etag: + description: |- + The new etag value representing stored keys in the backup. + See ``GET /room_keys/version/{version}`` for more details. + type: string + example: "abcdefg" + count: + description: The number of keys stored in the backup + type: integer + example: 10 + required: + - etag + - count + 404: + description: |- + The backup was not found. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 7758e2c1..ce9ea4db 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -767,6 +767,126 @@ previously-received ``request`` message with the same ``request_id`` and A reasonable strategy is for a user's client to only send keys requested by the verified devices of the same user. +Server-side key backups +~~~~~~~~~~~~~~~~~~~~~~~ + +Devices may upload encrypted copies of keys to the server. When a device tries +to read a message that it does not have keys for, it may request the key from +the server and decrypt it. Backups are per-user, and users may replace backups +with new backups. + +In contrast with `Key requests`_, Server-side key backups do not require another +device to be online from which to request keys. However, as the session keys are +stored on the server encrypted, it requires users to enter a decryption key to +decrypt the session keys. + +To create a backup, a client will call ``POST /room_keys/version`` and define +how the keys are to be encrypted through the backup's ``auth_data``; other +clients can discover the backup by calling ``GET /room_keys/version``. Keys +are encrypted according to the backups ``auth_data`` and added to the backup by +calling ``PUT /room_keys/keys?version=$v`` or one of its variants, and can be +retrieved by calling ``GET /room_keys/keys?version=$v`` or one of its variants. +Backups can also be deleted using ``DELETE /room_keys/version``, or individual +keys can be deleted using ``DELETE /room_key/keys?version=$v`` or one of its +variants. + +Clients must only store keys in backups after they have ensured that the +``auth_data`` is trusted, either by checking the signatures on it, or by +deriving the public key from a private key that it obtained from a trusted +source. + +When a client uploads a key for a session that the server already has a key +for, the server will choose to either keep the existing key or replace it with +the new key based on the key metadata as follows: + +- if the keys have different values for ``is_verified``, then it will keep the + key that has ``is_verified`` set to ``true``; +- if they have the same values for ``is_verified``, then it will keep the key + with a lower ``first_message_index``; +- and finally, is ``is_verified`` and ``first_message_index`` are equal, then + it will keep the key with a lower ``forwarded_count``. + +Recovery key +<<<<<<<<<<<< + +If the recovery key (the private half of the backup encryption key) is +presented to the user to save, it is presented as a string constructed as +follows: + +1. The 256-bit curve25519 private key is prepended by the bytes ``0x8B`` and + ``0x01`` +2. All the bytes in the string above, including the two header bytes, are XORed + together to form a parity byte. This parity byte is appended to the byte + string. +3. The byte string is encoded using base58, using the same mapping as is used + for Bitcoin addresses. +4. A space should be added after every 4th character. + +When reading in a recovery key, clients must disregard whitespace, and perform +the reverse of steps 1 through 3. + +Backup algorithm: ``m.megolm_backup.v1.curve25519-aes-sha2`` +<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< + +When a backup is created with the ``algorithm`` set to +``m.megolm_backup.v1.curve25519-aes-sha2``, the ``auth_data`` should have the +following format: + +``AuthData`` + +.. table:: + :widths: auto + + ========== =========== ====================================================== + Parameter Type Description + ========== =========== ====================================================== + public_key string Required. The curve25519 public key used to encrypt + the backups, encoded in unpadded base64. + signatures {string: Optional. Signatures of the ``auth_data``, as Signed + {string: JSON + string}} + ========== =========== ====================================================== + +The ``session_data`` field in the backups is constructed as follows: + +1. Encode the session key to be backed up as a JSON object with the properties: + + .. table:: + :widths: auto + + =============================== ======== ========================================= + Parameter Type Description + =============================== ======== ========================================= + algorithm string Required. The end-to-end message + encryption algorithm that the key is + for. Must be ``m.megolm.v1.aes-sha2``. + sender_key string Required. Unpadded base64-encoded + device curve25519 key. + sender_claimed_keys {string: Required. Object containing the + string} identity key for the sending device. + forwarding_curve25519_key_chain [string] Required. Zero or more curve25519 keys + for devices who forwarded the session key. + session_key string Required. Unpadded base64-encoded + session key in `session-sharing format + `_. + =============================== ======== ========================================= + +2. Generate an ephemeral curve25519 key, and perform an ECDH with the ephemeral + key and the backup's public key to generate a shared secret. The public + half of the ephemeral key, encoded using unpadded base64, becomes the ``ephemeral`` + property of the ``session_data``. +3. Using the shared secret, generate 80 bytes by performing an HKDF using + SHA-256 as the hash, with a salt of 32 bytes of 0, and with the empty string + as the info. The first 32 bytes are used as the AES key, the next 32 bytes + are used as the MAC key, and the last 16 bytes are used as the AES + initialization vector. +4. Stringify the JSON object, and encrypt it using AES-CBC-256 with PKCS#7 + padding. This encrypted data, encoded using unpadded base64, becomes the + ``ciphertext`` property of the ``session_data``. +5. Pass the raw encrypted data (prior to base64 encoding) through HMAC-SHA-256 + using the MAC key generated above. The first 8 bytes of the resulting MAC + are base64-encoded, and become the ``mac`` property of the ``session_data``. + Key exports ~~~~~~~~~~~ @@ -1089,6 +1209,11 @@ Key management API {{keys_cs_http_api}} +Key Backup API +~~~~~~~~~~~~~~ + +{{key_backup_cs_http_api}} + .. anchor for link from /sync api spec .. |device_lists_sync| replace:: End-to-end encryption From 59e337187b97fa5eb1a8349477577e12019331bd Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 16 Dec 2019 15:54:42 -0500 Subject: [PATCH 014/162] remove duplicated line --- api/client-server/key_backup.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index c55b8f27..5d7050a8 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -106,7 +106,6 @@ paths: algorithm: type: string description: The algorithm used for storing backups. - type: string enum: ["m.megolm_backup.v1.curve25519-aes-sha2"] example: "m.megolm_backup.v1.curve25519-aes-sha2" auth_data: From 4cde800ea10df3e34c359a6e878e1252bd81bdba Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 16 Dec 2019 17:25:20 -0500 Subject: [PATCH 015/162] fix validation errors --- api/client-server/key_backup.yaml | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index 5d7050a8..3663731e 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -92,9 +92,9 @@ paths: type: string name: version description: |- - Optional. The backup version to get. If omitted, the current backup - is returned. - required: false + The backup version to get. When an empty string, the trailing slash + is optional, and the current backup is returned. + required: true x-example: "1" responses: 200: @@ -678,12 +678,6 @@ paths: The backup from which to retrieve the keys. If omitted, the keys are retrieved from the current backup. x-example: "1" - - in: path - type: string - name: roomId - description: The ID of the room that the requested key is for. - required: true - x-example: "!roomid:example.org" responses: 200: description: |- @@ -736,12 +730,6 @@ paths: The backup from which to delete the key required: true x-example: "1" - - in: path - type: string - name: roomId - description: The ID of the room that the specified key is for. - required: true - x-example: "!roomid:example.org" responses: 200: description: The update succeeded From 90bf2b3b46c17d44c36472a3aebed8e6734f0c87 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 16 Dec 2019 17:40:05 -0500 Subject: [PATCH 016/162] add changelog --- changelogs/client_server/2387.new | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/2387.new diff --git a/changelogs/client_server/2387.new b/changelogs/client_server/2387.new new file mode 100644 index 00000000..a709a5fa --- /dev/null +++ b/changelogs/client_server/2387.new @@ -0,0 +1 @@ +Add key backup (``/room_keys/*``) endpoints. From a62c817745bc393a2023810ac502251b5a7f5d7c Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 17 Dec 2019 11:07:04 -0500 Subject: [PATCH 017/162] Apply suggestions from code review Co-Authored-By: Matthew Hodgson --- api/client-server/definitions/key_backup_data.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/client-server/definitions/key_backup_data.yaml b/api/client-server/definitions/key_backup_data.yaml index 5111a663..8f8de999 100644 --- a/api/client-server/definitions/key_backup_data.yaml +++ b/api/client-server/definitions/key_backup_data.yaml @@ -23,12 +23,12 @@ properties: example: 1 forwarded_count: description: |- - The number of times this key has been forwarded. + The number of times this key has been forwarded via key-sharing between devices. type: integer example: 0 is_verified: description: |- - Whether the device backing up the key was verified the device that the key + Whether the device backing up the key verified the device that the key is from. type: boolean example: false From 2e2f1c1174e4487d79b23f2cb9604f62066dd0c6 Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Tue, 4 Feb 2020 16:50:50 +0000 Subject: [PATCH 018/162] Extend the scope of casefolding to the whole address --- proposals/2265-email-lowercase.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2265-email-lowercase.md b/proposals/2265-email-lowercase.md index 935e6f2c..5a1db682 100644 --- a/proposals/2265-email-lowercase.md +++ b/proposals/2265-email-lowercase.md @@ -1,4 +1,4 @@ -# Proposal for mandating case folding when processing e-mail address localparts +# Proposal for mandating case folding when processing e-mail addresses [RFC822](https://tools.ietf.org/html/rfc822#section-3.4.7) mandates that localparts in e-mail addresses must be processed with the original case @@ -22,8 +22,8 @@ Sydent. This proposal suggests changing the specification of the e-mail 3PID type in [the Matrix spec appendices](https://matrix.org/docs/spec/appendices#pid-types) -to mandate that, before any processing, e-mail address localparts must go -through a full case folding based on [the unicode mapping +to mandate that, before any processing, e-mail addresses must go through a full +case folding based on [the unicode mapping file](https://www.unicode.org/Public/8.0.0/ucd/CaseFolding.txt), on top of having their domain lowercased. From 41072fcaa90d94ded06fe27c467ff652832a832b Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 7 Feb 2020 15:28:19 -0500 Subject: [PATCH 019/162] Apply suggestions from code review Co-Authored-By: Matthew Hodgson --- api/client-server/key_backup.yaml | 4 ++-- specification/modules/end_to_end_encryption.rst | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index 3663731e..b409dfbf 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -67,7 +67,7 @@ paths: responses: 200: description: - The version of the new backup. + The version id of the new backup. schema: type: object properties: @@ -123,7 +123,7 @@ paths: } } count: - description: The number of key stored in the backup. + description: The number of keys stored in the backup. type: integer example: 42 etag: diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index ce9ea4db..63690924 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -786,6 +786,7 @@ clients can discover the backup by calling ``GET /room_keys/version``. Keys are encrypted according to the backups ``auth_data`` and added to the backup by calling ``PUT /room_keys/keys?version=$v`` or one of its variants, and can be retrieved by calling ``GET /room_keys/keys?version=$v`` or one of its variants. +Keys can only be written to the most recently created version of the backup. Backups can also be deleted using ``DELETE /room_keys/version``, or individual keys can be deleted using ``DELETE /room_key/keys?version=$v`` or one of its variants. From 3ee9a7d654050219f09487ce7b6bd9584827a6c2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 12 Feb 2020 21:03:11 -0700 Subject: [PATCH 020/162] Attempt to convert MSC2324 to reality We've been using this in practice for a while now, so we should document it more officially. MSC: https://github.com/matrix-org/matrix-doc/pull/2324 --- specification/proposals_intro.rst | 102 +++++++++++++++++++++++++++++- 1 file changed, 101 insertions(+), 1 deletion(-) diff --git a/specification/proposals_intro.rst b/specification/proposals_intro.rst index 82a4225b..3f70345f 100644 --- a/specification/proposals_intro.rst +++ b/specification/proposals_intro.rst @@ -230,7 +230,8 @@ follows: comments and in relevant rooms on Matrix. Discussion outside of GitHub should be summarised in a comment on the PR. - When a member of the Spec Core Team believes that no new discussion points are - being made, they will propose a motion for a final comment period (FCP), + being made, and the proposal has suitable evidence of working (see `implementing a + proposal`_ below), they will propose a motion for a final comment period (FCP), along with a *disposition* of either merge, close or postpone. This FCP is provided to allow a short period of time for any invested party to provide a final objection before a major decision is made. If sufficient reasoning is @@ -341,6 +342,105 @@ Closed proposal-closed A proposal which Obsolete obsolete A proposal which has been made obsolete by another proposal or decision elsewhere. =============================== ============================= ==================================== +Implementing a proposal +----------------------- + +As part of the proposal process the spec core team will require evidence of the MSC +working in order for it to move into FCP. This can usually be a branch/pull request +to whichever implementation of choice that proves the MSC works in practice, though +in some cases the MSC itself will be small enough to be considered proven. Where it's +unclear if a MSC will require an implementation proof, ask in `#matrix-spec:matrix.org +`_. + +Early adoption of a MSC/idea +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To help facilitate early releases of software dependent on a spec release, implementations +are required to use the following process to ensure that the official Matrix namespace +is not cluttered with development or testing data. + +.. Note:: + Proof of concept implementations that are solely left on a branch/unreleased and for + demonstrating that a particular MSC works do not have to follow this. Note that the + ``develop`` branch adopted by some projects is still subject to this process. + +The process for an implementation willing to be ahead of a spec release is: + +1. Have an idea for a feature. +2. Implement the feature using unstable endpoints, vendor prefixes, and unstable + feature flags as appropriate. + + * When using unstable endpoints, they MUST include a vendor prefix. For example: + `/_matrix/client/unstable/com.example/login`. Vendor prefixes throughout Matrix + always use the Java package naming convention. The MSC for the feature should + identify which preferred vendor prefix is to be used by early adopters. + * Unstable endpoints **do not** inherit from stable (e.g. `/r0`) APIs. Implementations + cannot assume that a particular endpoint will exist in the unstable namespace + even if the server advertises support for the feature. + * If the client needs to be sure the server supports the feature, an unstable + feature flag that MUST be vendor prefixed is to be used. This kind of flag shows + up in the ``unstable_features`` section of ``/versions`` as, for example, + ``com.example.new_login``. The MSC for the feature should identify which preferred + feature flag is to be used by early adopters. + * When using this approach correctly, the implementation can ship/release the + feature at any time, so long as the implementation is able to accept the technical + debt that results from needing to provide adequate backwards and forwards + compatibility. The implementation MUST support the flag disappearing and be + generally safe for users. Note that implementations early in the MSC review + process may also be required to provide backwards compatibility with earlier + editions of the proposal. + * If the implementation cannot support the technical debt (or if it's impossible + to provide forwards/backwards compatibility - e.g. a user authentication change + which can't be safely rolled back), the implementation should not attempt to + implement the feature and instead wait for a spec release. + * If at any point the idea changes, the feature flag should also change so that + implementations can adapt as needed. + +3. In parallel, or ahead of implementation, open an MSC and solicit review per above. +4. Before FCP can be called, the Spec Core Team will require evidence of the MSC + working as proposed. A typical example of this is an implementation of the MSC, + though the implementation does not need to be shipped anywhere and can therefore + avoid the forwards/backwards compatibility concerns mentioned here. +5. FCP is gone through, and assuming nothing is flagged the MSC lands. +6. A spec PR is written to incorporate the changes into Matrix. +7. A spec release happens. +8. Implementations switch to using stable prefixes (e.g.: ``/r0``) if the server + supports the specification version released. If the server doesn't advertise the + specification version, but does have the feature flag, unstable prefixes should + still be used. +9. A transition period of about 2 months starts immediately after the spec release, + before implementations start to loudly encourage other implementations to switch + to stable endpoints. For example, a server implementation should start asking + client implementations to support the stable endpoints 2 months after the spec + release, if they haven't already. The same applies in the reverse: if clients + cannot switch to stable prefixes because server implementations haven't started + supporting the new spec release, some noise should be raised in the general direction + of the implementation. + + * Please be considerate when being noisy to implementations. A gentle reminder + in their issue tracker is generally good enough. + +.. Note:: + MSCs MUST still describe what the stable endpoints/feature looks like with a note + towards the bottom for what the unstable feature flag/prefixes are. For example, + a MSC would propose `/_matrix/client/r0/new/endpoint`, not `/_matrix/client/unstable/ + com.example/new/endpoint`. + +In summary: + +* Implementations MUST NOT use stable endpoints before the MSC is in the spec. This + includes NOT using stable endpoints before a spec release happens, but after FCP has + passed. +* Implementations are able to ship features that are exposed to users by default before + an MSC has been merged to the spec, provided they follow the process above. +* Implementations SHOULD be wary of the technical debt they are incurring by moving faster + than the spec. +* The vendor prefix is chosen by the developer of the feature, using the Java package + naming convention. The foundation's preferred vendor prefix is `org.matrix`. +* The vendor prefixes, unstable feature flags, and unstable endpoints should be included + in the MSC, though the MSC MUST be written in a way that proposes new stable endpoints. + Typically this is solved by a small table at the bottom mapping the various values + from stable to unstable. Proposal Tracking ----------------- From da795e582478da464defd8f68d2635ea1c41eabc Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 5 Mar 2020 17:24:29 -0700 Subject: [PATCH 021/162] Apply suggestions from code review Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- specification/proposals_intro.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/proposals_intro.rst b/specification/proposals_intro.rst index 3f70345f..c3b312ce 100644 --- a/specification/proposals_intro.rst +++ b/specification/proposals_intro.rst @@ -385,14 +385,14 @@ The process for an implementation willing to be ahead of a spec release is: * When using this approach correctly, the implementation can ship/release the feature at any time, so long as the implementation is able to accept the technical debt that results from needing to provide adequate backwards and forwards - compatibility. The implementation MUST support the flag disappearing and be + compatibility. The implementation MUST support the flag (and server-side implementation) disappearing and be generally safe for users. Note that implementations early in the MSC review process may also be required to provide backwards compatibility with earlier editions of the proposal. * If the implementation cannot support the technical debt (or if it's impossible to provide forwards/backwards compatibility - e.g. a user authentication change which can't be safely rolled back), the implementation should not attempt to - implement the feature and instead wait for a spec release. + implement the feature and should instead wait for a spec release. * If at any point the idea changes, the feature flag should also change so that implementations can adapt as needed. @@ -429,7 +429,7 @@ The process for an implementation willing to be ahead of a spec release is: In summary: * Implementations MUST NOT use stable endpoints before the MSC is in the spec. This - includes NOT using stable endpoints before a spec release happens, but after FCP has + includes NOT using stable endpoints in the period between completion of FCP and release of the spec. passed. * Implementations are able to ship features that are exposed to users by default before an MSC has been merged to the spec, provided they follow the process above. From 79b2e5263d296a735d3346bf4eb8882841671fb6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sat, 7 Mar 2020 14:05:05 -0700 Subject: [PATCH 022/162] Update specification/proposals_intro.rst Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- specification/proposals_intro.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/proposals_intro.rst b/specification/proposals_intro.rst index c3b312ce..5ee82dc3 100644 --- a/specification/proposals_intro.rst +++ b/specification/proposals_intro.rst @@ -393,7 +393,7 @@ The process for an implementation willing to be ahead of a spec release is: to provide forwards/backwards compatibility - e.g. a user authentication change which can't be safely rolled back), the implementation should not attempt to implement the feature and should instead wait for a spec release. - * If at any point the idea changes, the feature flag should also change so that + * If at any point after early release, the idea changes in a backwards-incompatible way, the feature flag should also change so that implementations can adapt as needed. 3. In parallel, or ahead of implementation, open an MSC and solicit review per above. From c69a747bae52296f8a7fe930090c64c98227cd73 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Mon, 9 Mar 2020 12:31:06 -0400 Subject: [PATCH 023/162] Convert the initial Google Doc to Markdown. --- proposals/xxxx-ui-interactive-auth-for-sso.md | 262 ++++++++++++++++++ 1 file changed, 262 insertions(+) create mode 100644 proposals/xxxx-ui-interactive-auth-for-sso.md diff --git a/proposals/xxxx-ui-interactive-auth-for-sso.md b/proposals/xxxx-ui-interactive-auth-for-sso.md new file mode 100644 index 00000000..fc7413fe --- /dev/null +++ b/proposals/xxxx-ui-interactive-auth-for-sso.md @@ -0,0 +1,262 @@ +# User-Interactive Auth for SSO-backed homeservers + +## Background + +Certain endpoints, such as `DELETE /_matrix/client/r0/devices/{deviceId}` and +`POST /_matrix/client/r0/account/3pid/add`, require the user to reconfirm their +identity, as a guard against a leaked access token being used to take over an +entire account. + +On a normal homeserver, this is done by prompting the user to enter their +password. However, on a homeserver where users authenticate via a single-sign-on +system, the user doesn't have a password, so this doesn't work. Instead we need +to delegate that check to the SSO system. + +At the protocol level, this means adding support for SSO to +[user-interactive auth](https://matrix.org/docs/spec/client_server/r0.6.0#user-interactive-authentication-api). + +### The current implementation + +Or, "how UI Auth works, in practice": + +When the client calls one of the protected endpoints, it initially returns a 401 +response. For example: + +``` +POST /_matrix/client/r0/delete_devices HTTP/1.1 +Content-Type: application/json + +{ + "devices": ["FSVVTZRRAA"] +} + +HTTP/1.1 401 Unauthorized +Content-Type: application/json + +{ + "flows": [ + { + "stages": [ + "m.login.password" + ] + } + ], + "params": {}, + "session": "dTKfsLHSAJeAhqfxUsvrIVJd" +} +``` + +The client: + +* inspects the "flows" list +* discovers there is a flow it knows how to follow +* carries out the first "stage" of that flow (m.login.password) + +ie, the client prompts the user to enter a password. + +The client then resubmits with an additional 'auth' param, with "type" giving +the name of the authentication type it has just carried out. That completes the +authentication flow, so the server is now happy, and returns a 200: + +``` +POST /_matrix/client/r0/delete_devices HTTP/1.1 +Content-Type: application/json + +{ + "devices": ["FSVVTZRRAA"], + "auth": { + "session":"dTKfsLHSAJeAhqfxUsvrIVJd", + "type":"m.login.password", + "identifier":{"type":"m.id.user", "user":"@userid:matrix.org"}, + "password":"" + } +} + + +HTTP/1.1 200 OK +Content-Type: application/json +Content-Length: 179 + +{} +``` + +## Proposal + +We add an `m.login.sso` authentication type to the UI auth spec. There are no +additional params as part of this auth type. + +1. If the client wants to complete that authentication type, it opens a browser + window for `/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>` + with session set to the UI-Auth session id (from the "auth" dict). + + The homeserver returns a page which says words to the effect of "A client is + trying to remove a device/add an email address/take over your account. To + confirm this action, **re-authenticate with single sign-on**. If you did not + expect this, your account may be compromised!" + + See security section below. +2. The link, once the user clicks on it, goes to the SSO provider's page. +3. The SSO provider validates the user, and redirects the browser back to the + homeserver. +4. The homeserver validates the response from the SSO provider, updates the + user-interactive auth session to show that the SSO has completed, serves the + fallback auth completion page as specced: + https://matrix.org/docs/spec/client_server/r0.6.0#fallback +5. The client resubmits its original request, with its original session id, + which now hopefully completes. + +Note that the post-SSO URL on the homeserver is left up to the homeserver +implementation rather than forming part of the spec. + +* SAML2 servers typically only support one URL per service provider, so in + practice it will need to be the same as that already used for the login flow + (for synapse, it's /_matrix/saml2/authn_response) - and the server needs to + be able to figure out if it's doing SSO for a login attempt or an SSO + attempt. +* CAS doesn't have the same restriction. + +### Example flow: + +0. Client submits the request, which the server says requires SSO: + + ``` + POST /_matrix/client/r0/delete_devices HTTP/1.1 + Content-Type: application/json + Authorization: Bearer xyzzy + + { + "devices": ["FSVVTZRRAA"] + } + + HTTP/1.1 401 Unauthorized + Content-Type: application/json + + { + "flows": [ + { + "stages": [ + "m.login.sso" + ] + } + ], + "params": {}, + "session": "dTKfsLHSAJeAhqfxUsvrIVJd" + } + ``` + +1. Client opens a browser window for the fallback endpoint: + + ``` + GET /_matrix/client/r0/auth/m.login.sso/fallback/web + ?session=dTKfsLHSAJeAhqfxUsvrIVJd HTTP/1.1 + + HTTP/1.1 200 OK + + + can delete device pls? + clicky + + ``` + +2. The user clicks the confirmation link which goes to the SSO provider's site: + + ``` + GET https://sso_provider/validate?SAMLRequest= HTTP/1.1 + + ... SAML/CAS stuff + ``` + +3. The SSO provider validates the user and ends up redirecting the browser back + to the homeserver. (The example below shows a 302 for simplicity but SAML normally uses a 200 with an html
, with javascript to automatically submit it.) + + ``` + HTTP/1.1 302 Moved + Location: https://homeserver/_matrix/saml2/authn_response? + SAMLResponse= + ``` + +4. The browser sends the SSO response to the homeserver, which validates it and + shows the fallback auth completion page: + + ``` + GET /_matrix/saml2/authn_response?SAMLResponse= + + + HTTP/1.1 200 OK + + + +

Thank you.

+

You may now close this window and return to the application.

+ ``` + +5. The client closes the browser popup if it is still open, and resubmits its + original request, which now succeeds: + + ``` + POST /_matrix/client/r0/delete_devices HTTP/1.1 + Content-Type: application/json + Authorization: Bearer xyzzy + + { + "auth": { + "session": "dTKfsLHSAJeAhqfxUsvrIVJd" + } + } + + HTTP/1.1 200 OK + Content-Type: application/json + + {} + ``` + +A few notes: + +* The security of this relies on UI-Auth sessions only being used for the same + request as they were initiated for. I don't think that's currently enforced. +* We might consider an alternative client flow where the fallback auth ends up + redirecting to a given URI, instead of doing javascript postMessage foo. I + think that's an orthogonal change to the fallback auth though. +* In theory, any clients that already implement the fallback process for + unknown authentication types will work fine without modification. + Unfortunately, I don't think Riot (on any platform) is among them. + +## Security considerations + +Recall that the thing we are trying to guard against here is a single leaked +access-token being used to take over an entire account. So let's assume the +attacker has got hold of an access token for your account. What happens if we + skip the confirmation step? + +The attacker, who has your access token, starts a UI-Auth session to add his +email address to your account. + +He then sends you a link "hey, check out this cool video!"; the link leads (via +an innocent-looking url shortener) to +`/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>`, with the ID of +the session that he just created. + +Recall that we're skipping the confirmation step, so the server redirects +straight to the SSO provider. + +It's common for SSO providers to redirect straight back to the app if you've +recently authenticated with them; even in the best case, the SSO provider shows +some innocent message along the lines of "Confirm that you want to sign in to +". + +So the SSO completes, and the attacker's session is validated. + +By telling the user what's about to happen as clearly as we can, and making them +confirm, we can mitigate this problem. + +## Unstable prefix + +We should use a vendor prefix here until this hits the spec. + +`org.matrix.login.sso` ? \ No newline at end of file From f48bbd32788ec8734fbaaa85728197632aeffd61 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Mon, 9 Mar 2020 14:00:28 -0400 Subject: [PATCH 024/162] Attempt to clarify the MSC. --- proposals/xxxx-ui-interactive-auth-for-sso.md | 90 ++++++++++--------- 1 file changed, 47 insertions(+), 43 deletions(-) diff --git a/proposals/xxxx-ui-interactive-auth-for-sso.md b/proposals/xxxx-ui-interactive-auth-for-sso.md index fc7413fe..5b6f396d 100644 --- a/proposals/xxxx-ui-interactive-auth-for-sso.md +++ b/proposals/xxxx-ui-interactive-auth-for-sso.md @@ -1,6 +1,4 @@ -# User-Interactive Auth for SSO-backed homeservers - -## Background +# User-Interactive Auth for SSO-backed homeserver Certain endpoints, such as `DELETE /_matrix/client/r0/devices/{deviceId}` and `POST /_matrix/client/r0/account/3pid/add`, require the user to reconfirm their @@ -9,15 +7,17 @@ entire account. On a normal homeserver, this is done by prompting the user to enter their password. However, on a homeserver where users authenticate via a single-sign-on -system, the user doesn't have a password, so this doesn't work. Instead we need -to delegate that check to the SSO system. +system, the user doesn't have a password registered with the homeserver. Instead +we need to delegate that check to the SSO system. At the protocol level, this means adding support for SSO to [user-interactive auth](https://matrix.org/docs/spec/client_server/r0.6.0#user-interactive-authentication-api). -### The current implementation +In theory, any clients that already implement the fallback process for unknown +authentication types will work fine without modification. It is unknown whether +this is widely supported among clients. -Or, "how UI Auth works, in practice": +### UI Auth Overview When the client calls one of the protected endpoints, it initially returns a 401 response. For example: @@ -52,7 +52,7 @@ The client: * discovers there is a flow it knows how to follow * carries out the first "stage" of that flow (m.login.password) -ie, the client prompts the user to enter a password. +In this example, the client prompts the user to enter a password. The client then resubmits with an additional 'auth' param, with "type" giving the name of the authentication type it has just carried out. That completes the @@ -89,24 +89,25 @@ additional params as part of this auth type. window for `/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>` with session set to the UI-Auth session id (from the "auth" dict). - The homeserver returns a page which says words to the effect of "A client is - trying to remove a device/add an email address/take over your account. To - confirm this action, **re-authenticate with single sign-on**. If you did not - expect this, your account may be compromised!" + The homeserver returns a page which asks for the user's confirmation before + proceeding. See the security considerations section below for why this is + necessary. For example, the page could say words to the effect of: - See security section below. + > A client is trying to remove a device/add an email address/take over your + > account. To confirm this action, **re-authenticate with single sign-on**. + > If you did not expect this, your account may be compromised! 2. The link, once the user clicks on it, goes to the SSO provider's page. 3. The SSO provider validates the user, and redirects the browser back to the homeserver. 4. The homeserver validates the response from the SSO provider, updates the - user-interactive auth session to show that the SSO has completed, serves the - fallback auth completion page as specced: - https://matrix.org/docs/spec/client_server/r0.6.0#fallback + user-interactive auth session to show that the SSO has completed, + [serves the fallback auth completion page as specced](https://matrix.org/docs/spec/client_server/r0.6.0#fallback). 5. The client resubmits its original request, with its original session id, - which now hopefully completes. + which now should complete. Note that the post-SSO URL on the homeserver is left up to the homeserver -implementation rather than forming part of the spec. +implementation rather than forming part of the specification, choices might be +limited by the chosen SSO implementation, for example: * SAML2 servers typically only support one URL per service provider, so in practice it will need to be the same as that already used for the login flow @@ -216,47 +217,50 @@ implementation rather than forming part of the spec. {} ``` -A few notes: +## Alternatives -* The security of this relies on UI-Auth sessions only being used for the same - request as they were initiated for. I don't think that's currently enforced. -* We might consider an alternative client flow where the fallback auth ends up - redirecting to a given URI, instead of doing javascript postMessage foo. I - think that's an orthogonal change to the fallback auth though. -* In theory, any clients that already implement the fallback process for - unknown authentication types will work fine without modification. - Unfortunately, I don't think Riot (on any platform) is among them. +An alternative client flow where the fallback auth ends up redirecting to a +given URI, instead of doing JavaScript postMessage foo could be considered. +This is probably an orthogonal change to the fallback auth though. ## Security considerations +### Why we need user to confirm before the SSO flow + Recall that the thing we are trying to guard against here is a single leaked access-token being used to take over an entire account. So let's assume the -attacker has got hold of an access token for your account. What happens if we - skip the confirmation step? +attacker has got hold of an access token for your account. What happens if the +confirmation step is skipped? -The attacker, who has your access token, starts a UI-Auth session to add his -email address to your account. +The attacker, who has your access token, starts a UI Authentication session to +add their email address to your account. -He then sends you a link "hey, check out this cool video!"; the link leads (via -an innocent-looking url shortener) to +They then sends you a link "hey, check out this cool video!"; the link leads (via +an innocent-looking URL shortener or some other phishing technique) to `/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>`, with the ID of the session that he just created. -Recall that we're skipping the confirmation step, so the server redirects -straight to the SSO provider. +Since there is no confirmation step, the server redirects directly to the SSO +provider. It's common for SSO providers to redirect straight back to the app if you've recently authenticated with them; even in the best case, the SSO provider shows -some innocent message along the lines of "Confirm that you want to sign in to -". +an innocent message along the lines of "Confirm that you want to sign in to +". -So the SSO completes, and the attacker's session is validated. +After redirecting back to the homeserver, the SSO is completed and the +attacker's session is validated. They are now able to make their malicious +change to your account. -By telling the user what's about to happen as clearly as we can, and making them -confirm, we can mitigate this problem. +This problem can be mitigated by clearly telling the user what is about to happen. -## Unstable prefix +### Reusing UI-Auth sessions -We should use a vendor prefix here until this hits the spec. +The security of this relies on UI-Auth sessions only being used for the same +request as they were initiated for. It is not believed that this is currently +enforced. + +## Unstable prefix -`org.matrix.login.sso` ? \ No newline at end of file +A vendor prefix of `org.matrix.login.sso` (instead of `m.login.sso` is proposed +until this is part of the specification. From 4d177753e1e8043b92f9bcec0c4501b9a2802a63 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Mon, 9 Mar 2020 15:05:37 -0400 Subject: [PATCH 025/162] Move proposal to proper proposal number. --- ...active-auth-for-sso.md => 2454-ui-interactive-auth-for-sso.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{xxxx-ui-interactive-auth-for-sso.md => 2454-ui-interactive-auth-for-sso.md} (100%) diff --git a/proposals/xxxx-ui-interactive-auth-for-sso.md b/proposals/2454-ui-interactive-auth-for-sso.md similarity index 100% rename from proposals/xxxx-ui-interactive-auth-for-sso.md rename to proposals/2454-ui-interactive-auth-for-sso.md From 570398e045476fcf4e289367d4856d44b0882b67 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Tue, 10 Mar 2020 10:00:58 -0400 Subject: [PATCH 026/162] Remove section on the how the authentication currently works and just reference the specification. --- proposals/2454-ui-interactive-auth-for-sso.md | 65 +------------------ 1 file changed, 1 insertion(+), 64 deletions(-) diff --git a/proposals/2454-ui-interactive-auth-for-sso.md b/proposals/2454-ui-interactive-auth-for-sso.md index 5b6f396d..bfad25a9 100644 --- a/proposals/2454-ui-interactive-auth-for-sso.md +++ b/proposals/2454-ui-interactive-auth-for-sso.md @@ -11,75 +11,12 @@ system, the user doesn't have a password registered with the homeserver. Instead we need to delegate that check to the SSO system. At the protocol level, this means adding support for SSO to -[user-interactive auth](https://matrix.org/docs/spec/client_server/r0.6.0#user-interactive-authentication-api). +[user-interactive authentication API](https://matrix.org/docs/spec/client_server/r0.6.0#user-interactive-authentication-api). In theory, any clients that already implement the fallback process for unknown authentication types will work fine without modification. It is unknown whether this is widely supported among clients. -### UI Auth Overview - -When the client calls one of the protected endpoints, it initially returns a 401 -response. For example: - -``` -POST /_matrix/client/r0/delete_devices HTTP/1.1 -Content-Type: application/json - -{ - "devices": ["FSVVTZRRAA"] -} - -HTTP/1.1 401 Unauthorized -Content-Type: application/json - -{ - "flows": [ - { - "stages": [ - "m.login.password" - ] - } - ], - "params": {}, - "session": "dTKfsLHSAJeAhqfxUsvrIVJd" -} -``` - -The client: - -* inspects the "flows" list -* discovers there is a flow it knows how to follow -* carries out the first "stage" of that flow (m.login.password) - -In this example, the client prompts the user to enter a password. - -The client then resubmits with an additional 'auth' param, with "type" giving -the name of the authentication type it has just carried out. That completes the -authentication flow, so the server is now happy, and returns a 200: - -``` -POST /_matrix/client/r0/delete_devices HTTP/1.1 -Content-Type: application/json - -{ - "devices": ["FSVVTZRRAA"], - "auth": { - "session":"dTKfsLHSAJeAhqfxUsvrIVJd", - "type":"m.login.password", - "identifier":{"type":"m.id.user", "user":"@userid:matrix.org"}, - "password":"" - } -} - - -HTTP/1.1 200 OK -Content-Type: application/json -Content-Length: 179 - -{} -``` - ## Proposal We add an `m.login.sso` authentication type to the UI auth spec. There are no From 78e08c1987460cd14eba27f503b95b6dc4083a29 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Tue, 10 Mar 2020 10:11:08 -0400 Subject: [PATCH 027/162] Attempt to clarify the proposed changes. --- proposals/2454-ui-interactive-auth-for-sso.md | 38 +++++++++++-------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/proposals/2454-ui-interactive-auth-for-sso.md b/proposals/2454-ui-interactive-auth-for-sso.md index bfad25a9..d3cab51b 100644 --- a/proposals/2454-ui-interactive-auth-for-sso.md +++ b/proposals/2454-ui-interactive-auth-for-sso.md @@ -1,4 +1,4 @@ -# User-Interactive Auth for SSO-backed homeserver +# User-Interactive Authentication for SSO-backed homeserver Certain endpoints, such as `DELETE /_matrix/client/r0/devices/{deviceId}` and `POST /_matrix/client/r0/account/3pid/add`, require the user to reconfirm their @@ -10,17 +10,21 @@ password. However, on a homeserver where users authenticate via a single-sign-on system, the user doesn't have a password registered with the homeserver. Instead we need to delegate that check to the SSO system. -At the protocol level, this means adding support for SSO to +At the protocol level, this means adding support for SSO to the [user-interactive authentication API](https://matrix.org/docs/spec/client_server/r0.6.0#user-interactive-authentication-api). -In theory, any clients that already implement the fallback process for unknown -authentication types will work fine without modification. It is unknown whether -this is widely supported among clients. +In theory, once SSO is added as a possible flow for authentication any clients +that already implement the [fallback process for unknown authentication types](https://matrix.org/docs/spec/client_server/r0.6.0#fallback) +will work fine without modification. It is unknown whether this is widely +supported among clients. ## Proposal -We add an `m.login.sso` authentication type to the UI auth spec. There are no -additional params as part of this auth type. +An [additional authentication type](https://matrix.org/docs/spec/client_server/r0.6.0#authentication-types) +of `m.login.sso` is added to the user-interactive authentication specification. +There are no additional parameters as part of this authentication type. + +When choosing this authentication flow, the following should occur: 1. If the client wants to complete that authentication type, it opens a browser window for `/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>` @@ -91,8 +95,9 @@ limited by the chosen SSO implementation, for example: HTTP/1.1 200 OK - can delete device pls? - clicky + A client is trying to remove a device from your account. To confirm this + action, re-authenticate with single sign-on. + If you did not expect this, your account may be compromised! ``` @@ -101,11 +106,12 @@ limited by the chosen SSO implementation, for example: ``` GET https://sso_provider/validate?SAMLRequest= HTTP/1.1 - ... SAML/CAS stuff + ``` 3. The SSO provider validates the user and ends up redirecting the browser back - to the homeserver. (The example below shows a 302 for simplicity but SAML normally uses a 200 with an html , with javascript to automatically submit it.) + to the homeserver. The example below shows a 302 for simplicity, this might + vary based on SSO implementation. ``` HTTP/1.1 302 Moved @@ -191,13 +197,13 @@ change to your account. This problem can be mitigated by clearly telling the user what is about to happen. -### Reusing UI-Auth sessions +### Reusing User Interactive Authentication sessions -The security of this relies on UI-Auth sessions only being used for the same -request as they were initiated for. It is not believed that this is currently -enforced. +The security of this relies on User Interactive Authentication sessions only +being used for the same request as they were initiated for. It is not believed +that this is currently enforced. ## Unstable prefix -A vendor prefix of `org.matrix.login.sso` (instead of `m.login.sso` is proposed +A vendor prefix of `org.matrix.login.sso` is proposed (instead of `m.login.sso`) until this is part of the specification. From 1facf7ff12834ad06fcaff9ee5e2cdf9d1302837 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Thu, 12 Mar 2020 14:14:20 -0400 Subject: [PATCH 028/162] Add an MSC proposal for how to handle sessions when modifying a password. --- ...sword-modification-invalidating-devices.md | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 proposals/2457-password-modification-invalidating-devices.md diff --git a/proposals/2457-password-modification-invalidating-devices.md b/proposals/2457-password-modification-invalidating-devices.md new file mode 100644 index 00000000..910e7b62 --- /dev/null +++ b/proposals/2457-password-modification-invalidating-devices.md @@ -0,0 +1,53 @@ +# Invalidating devices during password modification + +There are multiple use cases for why a user might want to modify their password: + +* Adopting a password manager (to use a unique password or more secure password). +* Password rotation. +* Re-secure a compromised account. +* ... probably tons of others ... + +These can be summarized into two groups: + +1. "My account has been compromised and I need to re-secure it." +2. "I just want to change my password." + +The [current Matrix specification](https://matrix.org/docs/spec/client_server/r0.6.0#post-matrix-client-r0-account-password) +does not provide a way to differentiate between these use cases. It gives no +guidance into what should happen to other sessions / devices when a password is +modified and leaves it up to the implementation. + +It is reasonable for a client to want to specify this behavior to offer two +different workflows: + +1. Modify a password and log all other devices out (for use when an account has + been compromised). +2. Modify a password and do not touch any session data (for use in a + non-malicious situations). + +Alternately a client may default to whichever workflow is best for their users. + +## Proposal + +An optional field is added to the JSON body body of the [password reset endpoint](https://matrix.org/docs/spec/client_server/r0.6.0#post-matrix-client-r0-account-password) +called `logout_devices`. This is a boolean flag (defaulting to `true`) that +signals to whether other devices and sessions should be invalidated after +modifying the password. + +## Potential issues + +While Synapse defaults to the behavior of invalidating other devices and +sessions this may not be true of other implementations. Thus, a default of +`true` may not be backwards compatible. It might be more prudent to specify that +the behavior of not specifying the `logout_devices` flag is undefined. + +## Alternatives + +Provide a new endpoint in a future version that supports an additional field (as +above), using a new endpoint would avoid backwards compatibility issues. + +## Security considerations + +By defaulting to invalidating devices and sessions the security considerations +of this endpoint should remain intact. A client will need to be modified to +choose to keep other devices active. From 61b306f8a8a63005f2ab131d32cbf1d5409ddada Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Fri, 13 Mar 2020 07:12:57 -0400 Subject: [PATCH 029/162] Remove concerns about backwards compatibility. --- ...-password-modification-invalidating-devices.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/proposals/2457-password-modification-invalidating-devices.md b/proposals/2457-password-modification-invalidating-devices.md index 910e7b62..9af6e229 100644 --- a/proposals/2457-password-modification-invalidating-devices.md +++ b/proposals/2457-password-modification-invalidating-devices.md @@ -36,15 +36,18 @@ modifying the password. ## Potential issues -While Synapse defaults to the behavior of invalidating other devices and -sessions this may not be true of other implementations. Thus, a default of -`true` may not be backwards compatible. It might be more prudent to specify that -the behavior of not specifying the `logout_devices` flag is undefined. +The specification states: + +> The homeserver SHOULD NOT revoke the access token provided in the request, +> however all other access tokens for the user should be revoked if the request +> succeeds. + +Defaulting `logout_devices` to `true` should be backwards compatible. ## Alternatives -Provide a new endpoint in a future version that supports an additional field (as -above), using a new endpoint would avoid backwards compatibility issues. +A new endpoint could be provided in a future version of the specification that +supports an additional field (as described above). ## Security considerations From 2d2731efff99ddeb67bdb16a716ed1eba8660189 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Mon, 16 Mar 2020 10:42:17 -0400 Subject: [PATCH 030/162] Remove duplicated word. Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- proposals/2457-password-modification-invalidating-devices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2457-password-modification-invalidating-devices.md b/proposals/2457-password-modification-invalidating-devices.md index 9af6e229..9b59156c 100644 --- a/proposals/2457-password-modification-invalidating-devices.md +++ b/proposals/2457-password-modification-invalidating-devices.md @@ -29,7 +29,7 @@ Alternately a client may default to whichever workflow is best for their users. ## Proposal -An optional field is added to the JSON body body of the [password reset endpoint](https://matrix.org/docs/spec/client_server/r0.6.0#post-matrix-client-r0-account-password) +An optional field is added to the JSON body of the [password reset endpoint](https://matrix.org/docs/spec/client_server/r0.6.0#post-matrix-client-r0-account-password) called `logout_devices`. This is a boolean flag (defaulting to `true`) that signals to whether other devices and sessions should be invalidated after modifying the password. From e67ba0cd05e9021386253bace2be136f3cec0a11 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 20 Mar 2020 16:49:53 -0400 Subject: [PATCH 031/162] add note indicating similarity between key backup format and key exports also copy description of forwarding_curve25519_key_chain from key exports, since it's a better description --- specification/modules/end_to_end_encryption.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 63690924..a1ce68f6 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -861,12 +861,14 @@ The ``session_data`` field in the backups is constructed as follows: algorithm string Required. The end-to-end message encryption algorithm that the key is for. Must be ``m.megolm.v1.aes-sha2``. + forwarding_curve25519_key_chain [string] Required. Chain of Curve25519 keys + through which this session was + forwarded, via + `m.forwarded_room_key`_ events. sender_key string Required. Unpadded base64-encoded device curve25519 key. sender_claimed_keys {string: Required. Object containing the string} identity key for the sending device. - forwarding_curve25519_key_chain [string] Required. Zero or more curve25519 keys - for devices who forwarded the session key. session_key string Required. Unpadded base64-encoded session key in `session-sharing format `_. @@ -961,6 +963,9 @@ described as follows: session_key string Required. The key for the session. =============================== =========== ==================================== +This is similar to the format before encryption used for the session keys in +`Server-side key backups`_ but adds the ``room_id`` and ``session_id`` fields. + Example: .. code:: json From 30586ed98c6862f3822616cee79ad15a406591fa Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 20 Mar 2020 16:55:56 -0400 Subject: [PATCH 032/162] remove extra space --- specification/modules/end_to_end_encryption.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index a1ce68f6..6945ce42 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -868,7 +868,7 @@ The ``session_data`` field in the backups is constructed as follows: sender_key string Required. Unpadded base64-encoded device curve25519 key. sender_claimed_keys {string: Required. Object containing the - string} identity key for the sending device. + string} identity key for the sending device. session_key string Required. Unpadded base64-encoded session key in `session-sharing format `_. From eb17e5d826008a82ca197866a03761ddb6fe9480 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 14 Feb 2020 16:20:10 -0500 Subject: [PATCH 033/162] switch to symmetric algorithm --- proposals/1946-secure_server-side_storage.md | 60 ++++++++++---------- 1 file changed, 29 insertions(+), 31 deletions(-) diff --git a/proposals/1946-secure_server-side_storage.md b/proposals/1946-secure_server-side_storage.md index fd907e53..59f02d8b 100644 --- a/proposals/1946-secure_server-side_storage.md +++ b/proposals/1946-secure_server-side_storage.md @@ -123,38 +123,33 @@ and the key descriptions for the keys would be: #### Encryption algorithms -##### `m.secret_storage.v1.curve25519-aes-sha2` - -The public key is stored in the `pubkey` property of the `m.secret_storage.key.[key -ID]` account_data as a base64-encoded string. - -The data is encrypted and MACed as follows: - -1. Generate an ephemeral curve25519 key, and perform an ECDH with the ephemeral - key and the public key to generate a shared secret. The public half of the - ephemeral key, encoded using base64, becomes the `ephemeral` property. -2. Using the shared secret, generate 80 bytes by performing an HKDF using - SHA-256 as the hash, with a salt of 32 bytes of 0, and with the empty string - as the info. The first 32 bytes are used as the AES key, the next 32 bytes - are used as the MAC key, and the last 16 bytes are used as the AES - initialization vector. -4. Encrypt the data using AES-CBC-256 with PKCS#7 padding. This encrypted - data, encoded using base64, becomes the `ciphertext` property. -5. Pass the raw encrypted data (prior to base64 encoding) through HMAC-SHA-256 - using the MAC key generated above. The first 8 bytes of the resulting MAC - are base64-encoded, and become the `mac` property. - -(The key HKDF, AES, and HMAC steps are the same as what are used for encryption -in olm and megolm.) - -For example, the `m.secret_storage.key.[key ID]` for a key using this algorithm +##### `m.secret_storage.v1.aes-hmac-sha2` + +Secrets are encrypted using AES-CTR-256 and MACed using HMAC-SHA-256. The data +is encrypted and MACed as follows: + +1. Given the secret storage key, generate 64 bytes by performing an HKDF with + SHA-256 as the hash, a salt of 32 bytes of 0, and with the secret name as + the info. The first 32 bytes are used as the AES key, and the next 32 bytes + are used as the MAC key +2. Generate 16 random bytes, set bit 63 to 0 (in order to work around + differences in AES-CTR implementations), and use this as the AES + initialization vector. This becomes the `iv` property, encoded using base64. +3. Encrypt the data using AES-CTR-256 using the AES key generated above. This + encrypted data, encoded using base64, becomes the `ciphertext` property. +4. Pass the raw encrypted data (prior to base64 encoding) through HMAC-SHA-256 + using the MAC key generated above. The resulting MAC is base64-encoded and + becomes the `mac` property. + +(We use AES-CTR to match file encryption and key exports.) + +For example, the `m.secret_storage.key.key_id` for a key using this algorithm could look like: ```json { "name": "m.default", - "algorithm": "m.secret_storage.v1.curve25519-aes-sha2", - "pubkey": "base64+public+key" + "algorithm": "m.secret_storage.v1.aes-hmac-sha2" } ``` @@ -164,8 +159,8 @@ and data encrypted using this algorithm could look like this: { "encrypted": { "key_id": { + "iv": "16+bytes+base64", "ciphertext": "base64+encoded+encrypted+data", - "ephemeral": "base64+ephemeral+key", "mac": "base64+encoded+mac" } } @@ -174,7 +169,7 @@ and data encrypted using this algorithm could look like this: ###### Keys -When a user is given a raw key for `m.secret_storage.v1.curve25519-aes-sha2`, +When a user is given a raw key for `m.secret_storage.v1.aes-hmac-sha2`, it will be encoded as follows (this is the same as what is proposed in MSC1703): * prepend the two bytes 0x8b and 0x01 to the key @@ -200,7 +195,8 @@ ID]` account-data: "passphrase": { "algorithm": "m.pbkdf2", "salt": "MmMsAlty", - "iterations": 100000 + "iterations": 100000, + "bits": 256 }, ... } @@ -209,7 +205,9 @@ ID]` account-data: **`m.pbkdf2`** The key is generated using PBKDF2 using the salt given in the `salt` parameter, -and the number of iterations given in the `iterations` parameter. +and the number of iterations given in the `iterations` parameter. The key size +that is generated is given by the `bits` parameter, or 256 bits if no `bits` +parameter is given. ### Sharing From f7ebda040e4c9d476da81cd6bd48e56b3affa805 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 24 Mar 2020 15:35:32 -0400 Subject: [PATCH 034/162] update some examples to use new algorithm name --- proposals/1946-secure_server-side_storage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/1946-secure_server-side_storage.md b/proposals/1946-secure_server-side_storage.md index 59f02d8b..61950b5f 100644 --- a/proposals/1946-secure_server-side_storage.md +++ b/proposals/1946-secure_server-side_storage.md @@ -43,7 +43,7 @@ A key with ID `abcdefg` is stored in `m.secret_storage.key.abcdefg` ```json { "name": "Some key", - "algorithm": "m.secret_storage.v1.curve25519-aes-sha2", + "algorithm": "m.secret_storage.v1.aes-hmac-sha2", // ... other properties according to algorithm } ``` @@ -106,7 +106,7 @@ and the key descriptions for the keys would be: ```json { "name": "Some key", - "algorithm": "m.secret_storage.v1.curve25519-aes-sha2", + "algorithm": "m.secret_storage.v1.aes-hmac-sha2", // ... other properties according to algorithm } ``` @@ -116,7 +116,7 @@ and the key descriptions for the keys would be: ```json { "name": "Some other key", - "algorithm": "m.secret_storage.v1.curve25519-aes-sha2", + "algorithm": "m.secret_storage.v1.aes-hmac-sha2", // ... other properties according to algorithm } ``` From 76dd16b854b8b0d94fcced863fa0751d2fff4bcd Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 24 Mar 2020 16:18:44 -0400 Subject: [PATCH 035/162] add proposal to switch to symmetric crypto --- proposals/1946-secure_server-side_storage.md | 6 ++ proposals/xxxx-symmetric-ssss.md | 76 ++++++++++++++++++++ 2 files changed, 82 insertions(+) create mode 100644 proposals/xxxx-symmetric-ssss.md diff --git a/proposals/1946-secure_server-side_storage.md b/proposals/1946-secure_server-side_storage.md index 61950b5f..5263e8c7 100644 --- a/proposals/1946-secure_server-side_storage.md +++ b/proposals/1946-secure_server-side_storage.md @@ -8,6 +8,12 @@ decryption key for the backups on the server, or cross-signing ([MSC1756](https://github.com/matrix-org/matrix-doc/pull/1756)) can store the signing keys. This proposal presents a standardized way of storing such data. +## Changes + +- [MSCxxxx](https://github.com/matrix-org/matrix-doc/pull/xxxx) changed the + encryption algorithm used from an asymmetric algorithm (Curve25519) to a + symmetric algorithm (AES). + ## Proposal Secrets are data that clients need to use and that are sent through or stored diff --git a/proposals/xxxx-symmetric-ssss.md b/proposals/xxxx-symmetric-ssss.md new file mode 100644 index 00000000..bde9fa95 --- /dev/null +++ b/proposals/xxxx-symmetric-ssss.md @@ -0,0 +1,76 @@ +# Symmetric SSSS + +[MSC1946 (Secure Secret Storage and +Sharing)](https://github.com/matrix-org/matrix-doc/pull/1946) proposed a way of +storing encrypted secrets on the server. In the proposal, secrets were +encrypted using a Curve25519 key, which was chosen to allow easier migration +from key backups that we created before the backup key was stored using it. + +However this does not provide any guarantees that data stored using the +proposal came from a trusted source. To remedy this, we propose to change the +encryption to use AES with a MAC to ensure that only someone who knows the key +is able to store data. + +## Proposal + +* The `m.secret_storage.v1.curve25519-aes-sha2` method proposed in MSC1946 is + removed. + +* A new method, `m.secret_storage.v1.aes-hmac-sha2`, is added. With this + method, the Secret Storage key may be any size (though 256 bits is + recommended), and data is encrypted as follows: + + 1. Given the secret storage key, generate 64 bytes by performing an HKDF with + SHA-256 as the hash, a salt of 32 bytes of 0, and with the secret name as + the info. The first 32 bytes are used as the AES key, and the next 32 bytes + are used as the MAC key + 2. Generate 16 random bytes, set bit 63 to 0 (in order to work around + differences in AES-CTR implementations), and use this as the AES + initialization vector. This becomes the `iv` property, encoded using base64. + 3. Encrypt the data using AES-CTR-256 using the AES key generated above. This + encrypted data, encoded using base64, becomes the `ciphertext` property. + 4. Pass the raw encrypted data (prior to base64 encoding) through HMAC-SHA-256 + using the MAC key generated above. The resulting MAC is base64-encoded and + becomes the `mac` property. + + (We use AES-CTR to match file encryption and key exports.) + + If the key Secret Storage key is generated from a passphrase, information + about how to generate the key is stored in the `passphrase` property of the + key's account-data in a similar manner to what was done with the original + `m.secret_storage.v1.curve25519-aes-sha2` method, except that there is an + optional `bits` parameter that defaults to 256, and indicates the number of + bits that should be generated from PBKDF2 (in other words, the size of the + key). + +* The `passthrough` property specified in the "Enconding the recovery key for + server-side storage via MSC1946" section of MSC1219 is removed. The primary + purpose of that property was to allow easy migration of pre-MSC1946 backups, + so that users could reuse the backup recovery key as the Secret Storage key + without needing to re-enter the recovery key. However, since we are now + using a symmetric encryption algorithm, the client needs to know the key that + is used to encrypt, so the purpose of the field cannot be fulfilled. + + +## Potential issues + +Users who have data stored using the old encryption algorithm will need their +data migrated. Clients that support the old algorithm but not the new +algorithm will not be able to use the migrated secrets until they are updated +with the new algorithms. This should not be a major problem because the only +clients that are known to have implemented the old algorithm are Riot +Web/Android/iOS, and they have been upgraded to implement the new algorithm. + + +## Alternatives + +Rather than switching to a symmetric encryption algorithm, we could stay with +an asymmetric encryption algorithm, and add on a method to authenticate the +data. However, it is much safer to use well-known cryptographic methods rather +than trying to invent something new. Since the main reason for using an +asymmetric scheme was to ease migration from older key backups without +requiring the user to re-enter the key, but this is no longer possible due to +the need to authenticate the data using the Secret Storage key, there is no +reason to stay with an asymmetric algorithm. It is also better to use +cryptographic methods already used in Matrix where possible, rather than +introducing something new. From 4b21e44e5f1f4dd52afa9555ba509f4755e7de37 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 24 Mar 2020 16:24:09 -0400 Subject: [PATCH 036/162] use the MSC number --- proposals/1946-secure_server-side_storage.md | 2 +- proposals/{xxxx-symmetric-ssss.md => 2472-symmetric-ssss.md} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename proposals/{xxxx-symmetric-ssss.md => 2472-symmetric-ssss.md} (100%) diff --git a/proposals/1946-secure_server-side_storage.md b/proposals/1946-secure_server-side_storage.md index 5263e8c7..a73fee4c 100644 --- a/proposals/1946-secure_server-side_storage.md +++ b/proposals/1946-secure_server-side_storage.md @@ -10,7 +10,7 @@ signing keys. This proposal presents a standardized way of storing such data. ## Changes -- [MSCxxxx](https://github.com/matrix-org/matrix-doc/pull/xxxx) changed the +- [MSC2472](https://github.com/matrix-org/matrix-doc/pull/2472) changed the encryption algorithm used from an asymmetric algorithm (Curve25519) to a symmetric algorithm (AES). diff --git a/proposals/xxxx-symmetric-ssss.md b/proposals/2472-symmetric-ssss.md similarity index 100% rename from proposals/xxxx-symmetric-ssss.md rename to proposals/2472-symmetric-ssss.md From eb48863c4044105b86d62997d88a086eda22e690 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Thu, 26 Mar 2020 11:47:32 -0400 Subject: [PATCH 037/162] Markdown formatting. Co-Authored-By: Hubert Chathi --- proposals/2454-ui-interactive-auth-for-sso.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2454-ui-interactive-auth-for-sso.md b/proposals/2454-ui-interactive-auth-for-sso.md index d3cab51b..3532a2fc 100644 --- a/proposals/2454-ui-interactive-auth-for-sso.md +++ b/proposals/2454-ui-interactive-auth-for-sso.md @@ -52,7 +52,7 @@ limited by the chosen SSO implementation, for example: * SAML2 servers typically only support one URL per service provider, so in practice it will need to be the same as that already used for the login flow - (for synapse, it's /_matrix/saml2/authn_response) - and the server needs to + (for synapse, it's `/_matrix/saml2/authn_response`) - and the server needs to be able to figure out if it's doing SSO for a login attempt or an SSO attempt. * CAS doesn't have the same restriction. From 745f8c09df6f948f9083449b2b0848444776855e Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Fri, 27 Mar 2020 13:59:37 -0400 Subject: [PATCH 038/162] Fix incorrect statement about the current spec's guidance. --- .../2457-password-modification-invalidating-devices.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2457-password-modification-invalidating-devices.md b/proposals/2457-password-modification-invalidating-devices.md index 9b59156c..ccde2ac6 100644 --- a/proposals/2457-password-modification-invalidating-devices.md +++ b/proposals/2457-password-modification-invalidating-devices.md @@ -13,9 +13,9 @@ These can be summarized into two groups: 2. "I just want to change my password." The [current Matrix specification](https://matrix.org/docs/spec/client_server/r0.6.0#post-matrix-client-r0-account-password) -does not provide a way to differentiate between these use cases. It gives no -guidance into what should happen to other sessions / devices when a password is -modified and leaves it up to the implementation. +does not provide a way to differentiate between these use cases. It currently +specifies behavior that fits well into the first use-case above: that the +sessions except the current session should be revoked. It is reasonable for a client to want to specify this behavior to offer two different workflows: From 943d9386bb3d7e87e439eed10b5fd5c0057e360b Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 30 Mar 2020 17:43:46 -0400 Subject: [PATCH 039/162] remove requirement to sign the key, since it is no longer used to trust it --- proposals/1946-secure_server-side_storage.md | 12 ++---------- proposals/2472-symmetric-ssss.md | 4 ++++ 2 files changed, 6 insertions(+), 10 deletions(-) diff --git a/proposals/1946-secure_server-side_storage.md b/proposals/1946-secure_server-side_storage.md index a73fee4c..ca4230de 100644 --- a/proposals/1946-secure_server-side_storage.md +++ b/proposals/1946-secure_server-side_storage.md @@ -38,9 +38,8 @@ Each key has an ID, and the description of the key is stored in the user's account_data using the event type `m.secret_storage.key.[key ID]`. The contents of the account data for the key will include an `algorithm` property, which indicates the encryption algorithm used, as well as a `name` property, which is -a human-readable name. The contents will be signed as signed JSON using the -user's master cross-signing key. Other properties depend on the encryption -algorithm, and are described below. +a human-readable name. Other properties depend on the encryption algorithm, +and are described below. Example: @@ -61,13 +60,6 @@ secrets that the user would expect to be available on all their clients. Unless the user specifies otherwise, clients will try to use the default key to decrypt secrets. -Clients MUST ensure that the key is trusted before using it to encrypt secrets. -One way to do that is to have the client that creates the key sign the key -description (as signed JSON) using the user's master cross-signing key. -Another way to do that is to prompt the user to enter the passphrase used to -generate the encryption key and ensure that the generated private key -corresponds to the public key. - #### Secret storage Encrypted data is stored in the user's account_data using the event type diff --git a/proposals/2472-symmetric-ssss.md b/proposals/2472-symmetric-ssss.md index bde9fa95..4827e2cd 100644 --- a/proposals/2472-symmetric-ssss.md +++ b/proposals/2472-symmetric-ssss.md @@ -51,6 +51,10 @@ is able to store data. using a symmetric encryption algorithm, the client needs to know the key that is used to encrypt, so the purpose of the field cannot be fulfilled. +* Signing the Secret Storage key with the user's master cross-signing key is no + longer required. The key is trusted on the basis of the user entering the + key/passphrase. + ## Potential issues From 4a3f9ec63ade930eaafad5b62d9ccb912893ba9a Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 30 Mar 2020 17:44:06 -0400 Subject: [PATCH 040/162] add information to check the key --- proposals/1946-secure_server-side_storage.md | 12 +++++++++++- proposals/2472-symmetric-ssss.md | 8 ++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/proposals/1946-secure_server-side_storage.md b/proposals/1946-secure_server-side_storage.md index ca4230de..5e24b3ae 100644 --- a/proposals/1946-secure_server-side_storage.md +++ b/proposals/1946-secure_server-side_storage.md @@ -141,13 +141,23 @@ is encrypted and MACed as follows: (We use AES-CTR to match file encryption and key exports.) +For the purposes of allowing clients to check whether a user has correctly +entered the key, clients should: + + 1. encrypt and MAC a message consisting of 32 bytes of 0 as described above, + using the empty string as the info parameter to the HKDF in step 1. + 2. store the `iv` and `mac` in the `m.secret_storage.key.[key ID]` + account-data. + For example, the `m.secret_storage.key.key_id` for a key using this algorithm could look like: ```json { "name": "m.default", - "algorithm": "m.secret_storage.v1.aes-hmac-sha2" + "algorithm": "m.secret_storage.v1.aes-hmac-sha2", + "iv": "random+data", + "mac": "mac+of+encrypted+zeros" } ``` diff --git a/proposals/2472-symmetric-ssss.md b/proposals/2472-symmetric-ssss.md index 4827e2cd..7fe9f179 100644 --- a/proposals/2472-symmetric-ssss.md +++ b/proposals/2472-symmetric-ssss.md @@ -43,6 +43,14 @@ is able to store data. bits that should be generated from PBKDF2 (in other words, the size of the key). +* For the purposes of allowing clients to check whether a user has correctly + entered the key, clients should: + + 1. encrypt and MAC a message consisting of 32 bytes of 0 as described above, + using the empty string as the info parameter to the HKDF in step 1. + 2. store the `iv` and `mac` in the `m.secret_storage.key.[key ID]` + account-data. + * The `passthrough` property specified in the "Enconding the recovery key for server-side storage via MSC1946" section of MSC1219 is removed. The primary purpose of that property was to allow easy migration of pre-MSC1946 backups, From 4cfcda57fb3676d6de6145bb9bd1f9de60645559 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Wed, 1 Apr 2020 10:48:05 -0400 Subject: [PATCH 041/162] Clarify that the only new item here is the new authentication type. --- proposals/2454-ui-interactive-auth-for-sso.md | 31 ++++++++++++++++--- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/proposals/2454-ui-interactive-auth-for-sso.md b/proposals/2454-ui-interactive-auth-for-sso.md index 3532a2fc..fff96dcf 100644 --- a/proposals/2454-ui-interactive-auth-for-sso.md +++ b/proposals/2454-ui-interactive-auth-for-sso.md @@ -22,11 +22,27 @@ supported among clients. An [additional authentication type](https://matrix.org/docs/spec/client_server/r0.6.0#authentication-types) of `m.login.sso` is added to the user-interactive authentication specification. -There are no additional parameters as part of this authentication type. + +There are no additional parameters as part of this authentication type. As per +the user-interactive authentication specification the only parameter include in +the `auth` dictionary should be the session ID from the homeserver, e.g.: + +```json +{ + "auth": { + "session": "" + } +} +``` + +### Detailed fallback authentication flow: + +The following is a re-iteration of the [fallback authentication flow](https://matrix.org/docs/spec/client_server/r0.6.0#fallback), +but with details filled in for the proposed new authentication type. When choosing this authentication flow, the following should occur: -1. If the client wants to complete that authentication type, it opens a browser +1. If the client wants to complete authentication using SSO, it opens a browser window for `/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>` with session set to the UI-Auth session id (from the "auth" dict). @@ -59,6 +75,10 @@ limited by the chosen SSO implementation, for example: ### Example flow: +A more complete example is provided below in which a user attempts to delete +a device and is pushed into the user interactive authentication process with +SSO being the only possible flow. + 0. Client submits the request, which the server says requires SSO: ``` @@ -163,7 +183,7 @@ limited by the chosen SSO implementation, for example: ## Alternatives An alternative client flow where the fallback auth ends up redirecting to a -given URI, instead of doing JavaScript postMessage foo could be considered. +given URI, instead of doing JavaScript `postMessage` foo could be considered. This is probably an orthogonal change to the fallback auth though. ## Security considerations @@ -200,8 +220,9 @@ This problem can be mitigated by clearly telling the user what is about to happe ### Reusing User Interactive Authentication sessions The security of this relies on User Interactive Authentication sessions only -being used for the same request as they were initiated for. It is not believed -that this is currently enforced. +being used for the same request as they were initiated for. This security is not +only a concern for the proposed SSO authentication type. It is not believed +that this is currently enforced in implementations. ## Unstable prefix From f6879c897bace91ccfe0cb102f490e9dc5619cde Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Wed, 15 Apr 2020 07:39:00 -0400 Subject: [PATCH 042/162] Fix minor grammatical fixes. Co-Authored-By: Hubert Chathi --- proposals/2454-ui-interactive-auth-for-sso.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2454-ui-interactive-auth-for-sso.md b/proposals/2454-ui-interactive-auth-for-sso.md index fff96dcf..e39facfb 100644 --- a/proposals/2454-ui-interactive-auth-for-sso.md +++ b/proposals/2454-ui-interactive-auth-for-sso.md @@ -13,7 +13,7 @@ we need to delegate that check to the SSO system. At the protocol level, this means adding support for SSO to the [user-interactive authentication API](https://matrix.org/docs/spec/client_server/r0.6.0#user-interactive-authentication-api). -In theory, once SSO is added as a possible flow for authentication any clients +In theory, once SSO is added as a possible flow for authentication, any clients that already implement the [fallback process for unknown authentication types](https://matrix.org/docs/spec/client_server/r0.6.0#fallback) will work fine without modification. It is unknown whether this is widely supported among clients. @@ -24,7 +24,7 @@ An [additional authentication type](https://matrix.org/docs/spec/client_server/r of `m.login.sso` is added to the user-interactive authentication specification. There are no additional parameters as part of this authentication type. As per -the user-interactive authentication specification the only parameter include in +the user-interactive authentication specification, the only parameter included in the `auth` dictionary should be the session ID from the homeserver, e.g.: ```json @@ -57,7 +57,7 @@ When choosing this authentication flow, the following should occur: 3. The SSO provider validates the user, and redirects the browser back to the homeserver. 4. The homeserver validates the response from the SSO provider, updates the - user-interactive auth session to show that the SSO has completed, + user-interactive auth session to show that the SSO has completed, and [serves the fallback auth completion page as specced](https://matrix.org/docs/spec/client_server/r0.6.0#fallback). 5. The client resubmits its original request, with its original session id, which now should complete. From 0ae597626af258c133ac285220754ccbd663bae6 Mon Sep 17 00:00:00 2001 From: David Vo Date: Fri, 1 May 2020 00:32:15 +1000 Subject: [PATCH 043/162] Fix .m.rule.contains_user_name push rule to highlight Signed-off-by: David Vo --- specification/modules/push.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/specification/modules/push.rst b/specification/modules/push.rst index c9489987..ebf1aef2 100644 --- a/specification/modules/push.rst +++ b/specification/modules/push.rst @@ -473,6 +473,9 @@ Definition (as a ``content`` rule): { "set_tweak": "sound", "value": "default" + }, + { + "set_tweak": "highlight" } ] } From b8efb5ac4701554c5eda7ca7ba3a8f2a880c93b5 Mon Sep 17 00:00:00 2001 From: David Vo Date: Fri, 1 May 2020 15:27:20 +1000 Subject: [PATCH 044/162] Add changelog for 2519 Signed-off-by: David Vo --- changelogs/client_server/newsfragments/2519.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2519.clarification diff --git a/changelogs/client_server/newsfragments/2519.clarification b/changelogs/client_server/newsfragments/2519.clarification new file mode 100644 index 00000000..eb3fb6a8 --- /dev/null +++ b/changelogs/client_server/newsfragments/2519.clarification @@ -0,0 +1 @@ +Fix the ``.m.rule.contains_user_name`` default push rule to set the highlight tweak, matching Synapse and users' expectations. From 411b3f432b57e9f1f6d3a170cd159020fb26bd74 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 May 2020 10:25:47 -0600 Subject: [PATCH 045/162] Update changelog --- changelogs/client_server/newsfragments/2519.clarification | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/client_server/newsfragments/2519.clarification b/changelogs/client_server/newsfragments/2519.clarification index eb3fb6a8..4b32da1e 100644 --- a/changelogs/client_server/newsfragments/2519.clarification +++ b/changelogs/client_server/newsfragments/2519.clarification @@ -1 +1 @@ -Fix the ``.m.rule.contains_user_name`` default push rule to set the highlight tweak, matching Synapse and users' expectations. +Fix the ``.m.rule.contains_user_name`` default push rule to set the highlight tweak. From 958e1b4a2e2ffd7dd4210818daf88385325ac0c4 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Mon, 4 May 2020 14:36:52 -0400 Subject: [PATCH 046/162] Make the spec changes for MSC 2457. --- api/client-server/registration.yaml | 8 ++++++-- changelogs/client_server/newsfragments/2523.feature | 1 + 2 files changed, 7 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2523.feature diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index 8114299e..ecc7a6fa 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -328,8 +328,7 @@ paths: The homeserver may change the flows available depending on whether a valid access token is provided. The homeserver SHOULD NOT revoke the - access token provided in the request, however all other access tokens - for the user should be revoked if the request succeeds. + access token provided in the request. security: - accessToken: [] operationId: changePassword @@ -343,6 +342,11 @@ paths: type: string description: The new password for the account. example: "ihatebananas" + logout_devices: + type: boolean + description: |- + Whether other access tokens should be revoked if the request succeeds. Defaults to true. + example: true auth: description: |- Additional authentication information for the user-interactive authentication API. diff --git a/changelogs/client_server/newsfragments/2523.feature b/changelogs/client_server/newsfragments/2523.feature new file mode 100644 index 00000000..6f690ea4 --- /dev/null +++ b/changelogs/client_server/newsfragments/2523.feature @@ -0,0 +1 @@ +Optionally invalidate other access tokens during password modification per `MSC 2457 `_. From 0c582ea8c71658b71e76210d30f2ece5c3ea8644 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Mon, 4 May 2020 16:21:19 -0400 Subject: [PATCH 047/162] Apply suggestions from code review Co-authored-by: Travis Ralston --- api/client-server/registration.yaml | 3 ++- changelogs/client_server/newsfragments/2523.feature | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index ecc7a6fa..c3f544c1 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -345,7 +345,8 @@ paths: logout_devices: type: boolean description: |- - Whether other access tokens should be revoked if the request succeeds. Defaults to true. + Whether the other access tokens, and their associated devices, for the user should be + revoked if the request succeeds. Defaults to true. example: true auth: description: |- diff --git a/changelogs/client_server/newsfragments/2523.feature b/changelogs/client_server/newsfragments/2523.feature index 6f690ea4..e45d1c2f 100644 --- a/changelogs/client_server/newsfragments/2523.feature +++ b/changelogs/client_server/newsfragments/2523.feature @@ -1 +1 @@ -Optionally invalidate other access tokens during password modification per `MSC 2457 `_. +Optionally invalidate other access tokens during password modification per `MSC2457 `_. From 1e330c942307259ffdf076940ca70b7ba33da83e Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Mon, 4 May 2020 16:23:31 -0400 Subject: [PATCH 048/162] Clarify revocation behavior. --- api/client-server/registration.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index c3f544c1..50ce4a96 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -328,7 +328,8 @@ paths: The homeserver may change the flows available depending on whether a valid access token is provided. The homeserver SHOULD NOT revoke the - access token provided in the request. + access token provided in the request. Whether other access tokens for + the user are revoked depends on the request parameters. security: - accessToken: [] operationId: changePassword From e89521d1951976cb2a3bca9509e5e501b7fd6325 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 4 May 2020 22:08:22 -0400 Subject: [PATCH 049/162] some fixes (spelling, RST, and naming) --- api/client-server/keys.yaml | 2 +- specification/modules/end_to_end_encryption.rst | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/api/client-server/keys.yaml b/api/client-server/keys.yaml index 69e39def..e172ea8a 100644 --- a/api/client-server/keys.yaml +++ b/api/client-server/keys.yaml @@ -101,7 +101,7 @@ paths: responses: 200: description: - The provided keys were sucessfully uploaded. + The provided keys were successfully uploaded. schema: type: object properties: diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 4b433f17..0223d9f5 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -565,9 +565,9 @@ The process between Alice and Bob verifying each other would be: they match or not. #. Assuming they match, Alice and Bob's devices calculate the HMAC of their own device keys and a comma-separated sorted list of of the key IDs that they wish the other user - to verify, using SHA-256 as the hash function. HMAC is defined in [RFC 2104](https://tools.ietf.org/html/rfc2104). + to verify, using SHA-256 as the hash function. HMAC is defined in `RFC 2104 `_. The key for the HMAC is different for each item and is calculated by generating - 32 bytes (256 bits) using `the key verification HKDF <#SAS-HKDF>`_. + 32 bytes (256 bits) using `the key verification HKDF <#sas-hkdf>`_. #. Alice's device sends Bob's device a ``m.key.verification.mac`` message containing the MAC of Alice's device keys and the MAC of her key IDs to be verified. Bob's device does the same for Bob's device keys and key IDs concurrently with Alice. @@ -653,14 +653,14 @@ are used in addition to those already specified: {{m_key_verification_mac_event}} -.. _`SAS-HKDF`: +.. _sas-hkdf: HKDF calculation <<<<<<<<<<<<<<<< -In all of the SAS methods, HKDF is as defined in [RFC 5869](https://tools.ietf.org/html/rfc5869) +In all of the SAS methods, HKDF is as defined in `RFC 5869 `_ and uses the previously agreed-upon hash function for the hash function. The shared -secret is supplied as the input keying material. No salt is used, and the input +secret is supplied as the input keying material. No salt is used, and the info parameter is the concatenation of: * The string ``MATRIX_KEY_VERIFICATION_SAS``. @@ -677,7 +677,7 @@ parameter is the concatenation of: For verification of each party's device keys, HKDF is as defined in RFC 5869 and uses SHA-256 as the hash function. The shared secret is supplied as the input keying -material. No salt is used, and in the input parameter is the concatenation of: +material. No salt is used, and in the info parameter is the concatenation of: * The string ``MATRIX_KEY_VERIFICATION_MAC``. * The Matrix ID of the user whose key is being MAC-ed. @@ -691,7 +691,7 @@ material. No salt is used, and in the input parameter is the concatenation of: SAS method: ``decimal`` <<<<<<<<<<<<<<<<<<<<<<< -Generate 5 bytes using `HKDF <#SAS-HKDF>`_ then take sequences of 13 bits to +Generate 5 bytes using `HKDF <#sas-hkdf>`_ then take sequences of 13 bits to convert to decimal numbers (resulting in 3 numbers between 0 and 8191 inclusive each). Add 1000 to each calculated number. @@ -708,7 +708,7 @@ such as dashes, or with the numbers on individual lines. SAS method: ``emoji`` <<<<<<<<<<<<<<<<<<<<< -Generate 6 bytes using `HKDF <#SAS-HKDF>`_ then split the first 42 bits into +Generate 6 bytes using `HKDF <#sas-hkdf>`_ then split the first 42 bits into 7 groups of 6 bits, similar to how one would base64 encode something. Convert each group of 6 bits to a number and use the following table to get the corresponding emoji: From 6e339542562dfc00c77b2abc91d18b0476f48854 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Tue, 5 May 2020 12:11:51 +0200 Subject: [PATCH 050/162] client-server: Mark the event_id when putting room events as required. --- api/client-server/room_send.yaml | 2 ++ api/client-server/room_state.yaml | 2 ++ 2 files changed, 4 insertions(+) diff --git a/api/client-server/room_send.yaml b/api/client-server/room_send.yaml index 6963f76c..fc8f3339 100644 --- a/api/client-server/room_send.yaml +++ b/api/client-server/room_send.yaml @@ -85,5 +85,7 @@ paths: type: string description: |- A unique identifier for the event. + required: + - event_id tags: - Room participation diff --git a/api/client-server/room_state.yaml b/api/client-server/room_state.yaml index 62168f26..640e2009 100644 --- a/api/client-server/room_state.yaml +++ b/api/client-server/room_state.yaml @@ -92,6 +92,8 @@ paths: type: string description: |- A unique identifier for the event. + required: + - event_id 403: description: |- The sender doesn't have permission to send the event into the room. From 3054ac572fb518499dd53eb21334788d9d3def68 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Damir=20Jeli=C4=87?= Date: Tue, 5 May 2020 12:17:54 +0200 Subject: [PATCH 051/162] changelogs: Fragment for the event id when putting room events clarification. --- changelogs/client_server/newsfragments/2525.clarification | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 changelogs/client_server/newsfragments/2525.clarification diff --git a/changelogs/client_server/newsfragments/2525.clarification b/changelogs/client_server/newsfragments/2525.clarification new file mode 100644 index 00000000..e93444bf --- /dev/null +++ b/changelogs/client_server/newsfragments/2525.clarification @@ -0,0 +1,2 @@ +Clarification that the event ids are required in the response when putting room +events. From b0e8fd648df7fd9119a0e472710be3e46d3c6483 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 5 May 2020 09:21:58 -0600 Subject: [PATCH 052/162] Apply suggestions from code review --- changelogs/client_server/newsfragments/2525.clarification | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/changelogs/client_server/newsfragments/2525.clarification b/changelogs/client_server/newsfragments/2525.clarification index e93444bf..df1fc538 100644 --- a/changelogs/client_server/newsfragments/2525.clarification +++ b/changelogs/client_server/newsfragments/2525.clarification @@ -1,2 +1 @@ -Clarification that the event ids are required in the response when putting room -events. +Clarify that an ``event_id`` is returned when sending events. From 9980b83dd4bd120a055e8544240121b38cbb628f Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 15:14:24 -0400 Subject: [PATCH 053/162] add changelog --- changelogs/client_server/newsfragments/2524.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2524.clarification diff --git a/changelogs/client_server/newsfragments/2524.clarification b/changelogs/client_server/newsfragments/2524.clarification new file mode 100644 index 00000000..cf36c3ba --- /dev/null +++ b/changelogs/client_server/newsfragments/2524.clarification @@ -0,0 +1 @@ +Fix various wording and markup issues in the end-to-end encryption section. From 6ede023b35e735f262886d8f02dc3523527443e9 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 16:06:21 -0400 Subject: [PATCH 054/162] make the changelog match others so they get combined --- changelogs/client_server/newsfragments/2524.clarification | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/client_server/newsfragments/2524.clarification b/changelogs/client_server/newsfragments/2524.clarification index cf36c3ba..902a9c3f 100644 --- a/changelogs/client_server/newsfragments/2524.clarification +++ b/changelogs/client_server/newsfragments/2524.clarification @@ -1 +1 @@ -Fix various wording and markup issues in the end-to-end encryption section. +Fix various spelling errors throughout the specification. From e0b4a3c9121689e0ec1cbdf9351236d8004166d4 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 17:02:16 -0400 Subject: [PATCH 055/162] Apply suggestions from code review Co-authored-by: Travis Ralston --- api/client-server/key_backup.yaml | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index b409dfbf..543a1992 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -31,14 +31,14 @@ paths: post: summary: Create a new backup. description: |- - Creates a new backup + Creates a new backup. operationId: postRoomKeysVersion security: - accessToken: [] parameters: - in: body name: version - description: "The backup configuration" + description: "The backup configuration." schema: type: object properties: @@ -73,7 +73,7 @@ paths: properties: version: type: string - description: The backup version + description: The backup version. example: "1" required: - version @@ -146,12 +146,12 @@ paths: - version 404: description: - The backup specified does not exist + The backup specified does not exist. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" tags: @@ -209,12 +209,12 @@ paths: - auth_data responses: 200: - description: The update succeeded + description: The update succeeded. schema: type: object properties: {} 404: - description: The backup specified does not exist + description: The backup specified does not exist. examples: application/json: { "errcode": "M_NOT_FOUND", @@ -226,7 +226,7 @@ paths: - End-to-end encryption "/room_keys/keys/{roomId}/{sessionId}": put: - summary: Store a key in the backup + summary: Store a key in the backup. description: |- Store a key in the backup. operationId: postRoomKeysKeyRoomIdSessionId @@ -254,12 +254,12 @@ paths: x-example: "sessionid" - in: body name: data - description: "The key data" + description: "The key data." schema: "$ref": "definitions/key_backup_data.yaml" responses: 200: - description: The update succeeded + description: The update succeeded. schema: type: object properties: From 8ff1c26ef22449fad5526a8861304a88826ae48d Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 17:28:54 -0400 Subject: [PATCH 056/162] fix indentation --- .../definitions/key_backup_data.yaml | 8 +- api/client-server/key_backup.yaml | 142 +++++++++--------- 2 files changed, 75 insertions(+), 75 deletions(-) diff --git a/api/client-server/definitions/key_backup_data.yaml b/api/client-server/definitions/key_backup_data.yaml index 8f8de999..6a3b4042 100644 --- a/api/client-server/definitions/key_backup_data.yaml +++ b/api/client-server/definitions/key_backup_data.yaml @@ -39,10 +39,10 @@ properties: expected format of the data. type: object example: { - "ephemeral": "base64+ephemeral+key", - "ciphertext": "base64+ciphertext+of+JSON+data", - "mac": "base64+mac+of+ciphertext" - } + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } required: - first_message_index - forwarded_count diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index 543a1992..bdca769e 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -54,13 +54,13 @@ paths: expected format of the data. type: object example: { - "public_key": "abcdefg", - "signatures": { - "@alice:example.org": { - "ed25519:deviceid": "signature" - } + "public_key": "abcdefg", + "signatures": { + "@alice:example.org": { + "ed25519:deviceid": "signature" } } + } required: - algorithm - auth_data @@ -115,13 +115,13 @@ paths: expected format of the data. type: object example: { - "public_key": "abcdefg", - "signatures": { - "@alice:example.org": { - "ed25519:deviceid": "signature" - } + "public_key": "abcdefg", + "signatures": { + "@alice:example.org": { + "ed25519:deviceid": "signature" } } + } count: description: The number of keys stored in the backup. type: integer @@ -191,13 +191,13 @@ paths: expected format of the data. type: object example: { - "public_key": "abcdefg", - "signatures": { - "@alice:example.org": { - "ed25519:deviceid": "signature" - } + "public_key": "abcdefg", + "signatures": { + "@alice:example.org": { + "ed25519:deviceid": "signature" } } + } version: description: |- The backup version. If present, must be the same as the @@ -283,10 +283,10 @@ paths: field. examples: application/json: { - "errcode": "M_WRONG_ROOM_KEYS_VERSION", - "error": "Wrong backup version.", - "current_version": "42" - } + "errcode": "M_WRONG_ROOM_KEYS_VERSION", + "error": "Wrong backup version.", + "current_version": "42" + } schema: "$ref": "definitions/errors/error.yaml" tags: @@ -327,9 +327,9 @@ paths: description: The key or backup was not found. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Key not found." - } + "errcode": "M_NOT_FOUND", + "error": "Key not found." + } schema: "$ref": "definitions/errors/error.yaml" delete: @@ -383,9 +383,9 @@ paths: The backup was not found. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" "/room_keys/keys/{roomId}": @@ -424,12 +424,12 @@ paths: allOf: - $ref: "definitions/key_backup_data.yaml" example: { - "sessionid1": { - "ephemeral": "base64+ephemeral+key", - "ciphertext": "base64+ciphertext+of+JSON+data", - "mac": "base64+mac+of+ciphertext" - } + "sessionid1": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" } + } responses: 200: description: The update succeeded @@ -467,9 +467,9 @@ paths: The backup was not found. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" tags: @@ -511,20 +511,20 @@ paths: allOf: - $ref: "definitions/key_backup_data.yaml" example: { - "sessionid1": { - "ephemeral": "base64+ephemeral+key", - "ciphertext": "base64+ciphertext+of+JSON+data", - "mac": "base64+mac+of+ciphertext" - } + "sessionid1": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" } + } 404: description: |- The backup was not found. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" delete: @@ -572,9 +572,9 @@ paths: The backup was not found. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" "/room_keys/keys": @@ -609,16 +609,16 @@ paths: allOf: - $ref: "definitions/key_backup_data.yaml" example: { - "!room:example.org": { - "sessions": { - "sessionid1": { - "ephemeral": "base64+ephemeral+key", - "ciphertext": "base64+ciphertext+of+JSON+data", - "mac": "base64+mac+of+ciphertext" - } + "!room:example.org": { + "sessions": { + "sessionid1": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" } } } + } responses: 200: description: The update succeeded @@ -645,10 +645,10 @@ paths: field. examples: application/json: { - "errcode": "M_WRONG_ROOM_KEYS_VERSION", - "error": "Wrong backup version.", - "current_version": "42" - } + "errcode": "M_WRONG_ROOM_KEYS_VERSION", + "error": "Wrong backup version.", + "current_version": "42" + } schema: "$ref": "definitions/errors/error.yaml" 404: @@ -656,9 +656,9 @@ paths: The backup was not found. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" tags: @@ -696,23 +696,23 @@ paths: allOf: - $ref: "definitions/key_backup_data.yaml" example: { - "!room:example.org": { - "sessions": { - "sessionid1": { - "ephemeral": "base64+ephemeral+key", - "ciphertext": "base64+ciphertext+of+JSON+data", - "mac": "base64+mac+of+ciphertext" - } + "!room:example.org": { + "sessions": { + "sessionid1": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" } } } + } 404: description: The backup was not found. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version." - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version." + } schema: "$ref": "definitions/errors/error.yaml" delete: @@ -754,8 +754,8 @@ paths: The backup was not found. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" From 5f3ee44cfbcfbdf625170788ad7fb8d91627d5fc Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 18:09:17 -0400 Subject: [PATCH 057/162] add more suggestions from review --- api/client-server/key_backup.yaml | 14 ++++++++++---- specification/modules/end_to_end_encryption.rst | 4 ++-- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index bdca769e..ae729d62 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -73,7 +73,7 @@ paths: properties: version: type: string - description: The backup version. + description: The backup version. This is an opaque string. example: "1" required: - version @@ -92,8 +92,10 @@ paths: type: string name: version description: |- - The backup version to get. When an empty string, the trailing slash - is optional, and the current backup is returned. + The backup version to get, as returned in the ``version`` parameter + of the response in `POST /_matrix/client/r0/room_keys/version`_ or + this endpoint. When an empty string, the trailing slash is + optional, and the current backup is returned. required: true x-example: "1" responses: @@ -168,7 +170,11 @@ paths: type: string name: version description: |- - The backup version to update. + The backup version to update, as returned in the ``version`` + parameter in the response of `POST + /_matrix/client/r0/room_keys/version`_ or `GET + /_matrix/client/r0/room_keys/version/{version}`_. Unlike with the + ``GET`` version of this endpoint, this may not be empty. required: true x-example: "1" - in: body diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 6945ce42..8e4b0739 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -819,8 +819,8 @@ follows: 2. All the bytes in the string above, including the two header bytes, are XORed together to form a parity byte. This parity byte is appended to the byte string. -3. The byte string is encoded using base58, using the same mapping as is used - for Bitcoin addresses. +3. The byte string is encoded using base58, using the same `mapping as is used + for Bitcoin addresses `_. 4. A space should be added after every 4th character. When reading in a recovery key, clients must disregard whitespace, and perform From 3d1c33ed1b0f788135d5e03cbf49dafa04919698 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 18:09:35 -0400 Subject: [PATCH 058/162] move API definitions into backup section --- specification/modules/end_to_end_encryption.rst | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 8e4b0739..48a2d6ca 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -890,6 +890,8 @@ The ``session_data`` field in the backups is constructed as follows: using the MAC key generated above. The first 8 bytes of the resulting MAC are base64-encoded, and become the ``mac`` property of the ``session_data``. +{{key_backup_cs_http_api}} + Key exports ~~~~~~~~~~~ @@ -1215,11 +1217,6 @@ Key management API {{keys_cs_http_api}} -Key Backup API -~~~~~~~~~~~~~~ - -{{key_backup_cs_http_api}} - .. anchor for link from /sync api spec .. |device_lists_sync| replace:: End-to-end encryption From fffed943bcdfc3b016a5c03ce4fa66c35160220b Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 19:18:04 -0400 Subject: [PATCH 059/162] proposal to add endpoint to delete backups --- proposals/xxxx-add-delete-backup.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 proposals/xxxx-add-delete-backup.md diff --git a/proposals/xxxx-add-delete-backup.md b/proposals/xxxx-add-delete-backup.md new file mode 100644 index 00000000..b84de399 --- /dev/null +++ b/proposals/xxxx-add-delete-backup.md @@ -0,0 +1,14 @@ +# Add ability to delete key backups + +[MSC1219](https://github.com/matrix-org/matrix-doc/issues/1219) defined a +mechanism for key backups. However, it inadvertently omitted the endpoint to +delete an entire key backup. This proposal adds the endpoint. + +## Proposal + +An endpoint is added, `DELETE /room_keys/version/{version}`, that deletes a +backup version. Both the information about the key backup, as well as all keys +associated with the backup should be deleted. Like `POST +/room_keys/version/{version}`, and unlike `GET /room_keys/version/{version}`, +`{version}` cannot be empty, to ensure that the wrong backup is not +accidentally deleted. From 0b39308a139276a3ed02297bac4e9a5f99032a3e Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 19:20:20 -0400 Subject: [PATCH 060/162] rename to use msc number --- .../{xxxx-add-delete-backup.md => 2526-add-delete-backup.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{xxxx-add-delete-backup.md => 2526-add-delete-backup.md} (93%) diff --git a/proposals/xxxx-add-delete-backup.md b/proposals/2526-add-delete-backup.md similarity index 93% rename from proposals/xxxx-add-delete-backup.md rename to proposals/2526-add-delete-backup.md index b84de399..e4d589e3 100644 --- a/proposals/xxxx-add-delete-backup.md +++ b/proposals/2526-add-delete-backup.md @@ -1,4 +1,4 @@ -# Add ability to delete key backups +# MSC2526: Add ability to delete key backups [MSC1219](https://github.com/matrix-org/matrix-doc/issues/1219) defined a mechanism for key backups. However, it inadvertently omitted the endpoint to From 0145191d43d4c9021bc43ad3bb36caae468d6537 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 5 May 2020 20:00:45 -0400 Subject: [PATCH 061/162] add links to endpoints and add delete endpoint --- api/client-server/key_backup.yaml | 37 +++++++++++++++++++ .../modules/end_to_end_encryption.rst | 21 ++++++----- 2 files changed, 49 insertions(+), 9 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index ae729d62..387621e6 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -230,6 +230,43 @@ paths: "$ref": "definitions/errors/error.yaml" tags: - End-to-end encryption + delete: + summary: Delete an existing key backup. + description: |- + Delete an existing key backup. Both the information about the backup, + as well as all key data related to the backup will be deleted. + operationId: deleteRoomKeysVersion + security: + - accessToken: [] + parameters: + - in: path + type: string + name: version + description: |- + The backup version to delete, as returned in the ``version`` + parameter in the response of `POST + /_matrix/client/r0/room_keys/version`_ or `GET + /_matrix/client/r0/room_keys/version/{version}`_. Unlike with the + ``GET`` version of this endpoint, this may not be empty. + required: true + x-example: "1" + responses: + 200: + description: The delete succeeded. + schema: + type: object + properties: {} + 404: + description: The backup specified does not exist. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + tags: + - End-to-end encryption "/room_keys/keys/{roomId}/{sessionId}": put: summary: Store a key in the backup. diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 48a2d6ca..d6c87bc3 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -780,15 +780,18 @@ device to be online from which to request keys. However, as the session keys are stored on the server encrypted, it requires users to enter a decryption key to decrypt the session keys. -To create a backup, a client will call ``POST /room_keys/version`` and define -how the keys are to be encrypted through the backup's ``auth_data``; other -clients can discover the backup by calling ``GET /room_keys/version``. Keys -are encrypted according to the backups ``auth_data`` and added to the backup by -calling ``PUT /room_keys/keys?version=$v`` or one of its variants, and can be -retrieved by calling ``GET /room_keys/keys?version=$v`` or one of its variants. -Keys can only be written to the most recently created version of the backup. -Backups can also be deleted using ``DELETE /room_keys/version``, or individual -keys can be deleted using ``DELETE /room_key/keys?version=$v`` or one of its +To create a backup, a client will call `POST +/_matrix/client/r0/room_keys/version`_ and define how the keys are to be +encrypted through the backup's ``auth_data``; other clients can discover the +backup by calling `GET /_matrix/client/r0/room_keys/version/{version}`_, +setting ``{version}`` to the empty string. Keys are encrypted according to the +backups ``auth_data`` and added to the backup by calling `PUT +/_matrix/client/r0/room_keys/keys`_ or one of its variants, and can be +retrieved by calling `GET /_matrix/client/r0/room_keys/keys`_ or one of its +variants. Keys can only be written to the most recently created version of the +backup. Backups can also be deleted using `DELETE +/_matrix/client/r0/room_keys/version/{version}`_, or individual keys can be +deleted using `DELETE /_matrix/client/r0/room_keys/keys`_ or one of its variants. Clients must only store keys in backups after they have ensured that the From 6b8b31ce638d82fed94957bb1b6c33bc71fd92cb Mon Sep 17 00:00:00 2001 From: Gnuxie <50846879+Gnuxie@users.noreply.github.com> Date: Wed, 6 May 2020 17:10:39 +0100 Subject: [PATCH 062/162] correct use of required annotation in json-schema https://tools.ietf.org/html/draft-handrews-json-schema-validation-02#section-6.5.3 --- api/server-server/definitions/keys.yaml | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/api/server-server/definitions/keys.yaml b/api/server-server/definitions/keys.yaml index 8bc6c563..1e025a52 100644 --- a/api/server-server/definitions/keys.yaml +++ b/api/server-server/definitions/keys.yaml @@ -20,7 +20,6 @@ properties: server_name: type: string description: DNS name of the homeserver. - required: true example: "example.org" verify_keys: type: object @@ -31,7 +30,6 @@ properties: algorithm and ``abc123`` being the version in the example below). Together, this forms the Key ID. The version must have characters matching the regular expression ``[a-zA-Z0-9_]``. - required: true additionalProperties: type: object title: Verify Key @@ -44,8 +42,8 @@ properties: key: type: string description: The `Unpadded Base64`_ encoded key. - required: true example: "VGhpcyBzaG91bGQgYmUgYSByZWFsIGVkMjU1MTkgcGF5bG9hZA" + required: ["key"] old_verify_keys: type: object description: |- @@ -69,13 +67,12 @@ properties: type: integer format: int64 description: POSIX timestamp in milliseconds for when this key expired. - required: true example: 1532645052628 key: type: string description: The `Unpadded Base64`_ encoded key. - required: true example: "VGhpcyBzaG91bGQgYmUgeW91ciBvbGQga2V5J3MgZWQyNTUxOSBwYXlsb2FkLg" + required: ["expired_ts", "key"] signatures: type: object description: Digital signatures for this object signed using the ``verify_keys``. @@ -103,3 +100,4 @@ properties: publishes a key which is valid for a significant amount of time without a way for the homeserver owner to revoke it. example: 1052262000000 +required: ["server_name", "verify_keys"] From a896729ac2ec68b794cbf8ee3b8e29e3ccfb6078 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 6 May 2020 17:11:41 -0400 Subject: [PATCH 063/162] some more clarifications and indentation fixes --- api/client-server/key_backup.yaml | 24 ++++++++++++++----- .../modules/end_to_end_encryption.rst | 10 ++++---- 2 files changed, 23 insertions(+), 11 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index 387621e6..0f7ff25b 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -219,13 +219,25 @@ paths: schema: type: object properties: {} + 400: + description: |- + A parameter was incorrect. For example, the ``algorithm`` does not + match the current backup algorithm, or the ``version`` in the body + does not match the ``version`` in the path. + examples: + application/json: { + "errcode": "M_INVALID_PARAM", + "error": "Algorithm does not match" + } + schema: + "$ref": "definitions/errors/error.yaml" 404: description: The backup specified does not exist. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" tags: @@ -260,9 +272,9 @@ paths: description: The backup specified does not exist. examples: application/json: { - "errcode": "M_NOT_FOUND", - "error": "Unknown backup version" - } + "errcode": "M_NOT_FOUND", + "error": "Unknown backup version" + } schema: "$ref": "definitions/errors/error.yaml" tags: diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index d6c87bc3..8f26477a 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -846,9 +846,8 @@ following format: ========== =========== ====================================================== public_key string Required. The curve25519 public key used to encrypt the backups, encoded in unpadded base64. - signatures {string: Optional. Signatures of the ``auth_data``, as Signed - {string: JSON - string}} + signatures Signatures Optional. Signatures of the ``auth_data``, as Signed + JSON ========== =========== ====================================================== The ``session_data`` field in the backups is constructed as follows: @@ -870,8 +869,9 @@ The ``session_data`` field in the backups is constructed as follows: `m.forwarded_room_key`_ events. sender_key string Required. Unpadded base64-encoded device curve25519 key. - sender_claimed_keys {string: Required. Object containing the - string} identity key for the sending device. + sender_claimed_keys {string: Required. A map from algorithm name + string} (``ed25519``) to the identity key + for the sending device. session_key string Required. Unpadded base64-encoded session key in `session-sharing format `_. From 6613cd89a668f400a82a8a9f36b961768d77090f Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Thu, 7 May 2020 15:42:04 +0100 Subject: [PATCH 064/162] 2454-ui-interactive-auth-for-sso.md: markup fix --- proposals/2454-ui-interactive-auth-for-sso.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2454-ui-interactive-auth-for-sso.md b/proposals/2454-ui-interactive-auth-for-sso.md index e39facfb..18112c5f 100644 --- a/proposals/2454-ui-interactive-auth-for-sso.md +++ b/proposals/2454-ui-interactive-auth-for-sso.md @@ -209,7 +209,7 @@ provider. It's common for SSO providers to redirect straight back to the app if you've recently authenticated with them; even in the best case, the SSO provider shows an innocent message along the lines of "Confirm that you want to sign in to -". +\". After redirecting back to the homeserver, the SSO is completed and the attacker's session is validated. They are now able to make their malicious From 9065d1aa143e06a42fd36e59d923d0756ae6225d Mon Sep 17 00:00:00 2001 From: Ben Parsons Date: Thu, 7 May 2020 15:45:33 +0100 Subject: [PATCH 065/162] proposal to use existing events as captions for images --- proposals/2529-text-messages-as-captions.md | 37 +++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 proposals/2529-text-messages-as-captions.md diff --git a/proposals/2529-text-messages-as-captions.md b/proposals/2529-text-messages-as-captions.md new file mode 100644 index 00000000..df76e325 --- /dev/null +++ b/proposals/2529-text-messages-as-captions.md @@ -0,0 +1,37 @@ +# Use existing m.room.message/m.text events as captions for images + +## Background + +There is a demand to be able to apply a text caption to an image, as is possible in other chat platforms. In Matrix this is not possible, so people will generally send two events: one `m.image`, then a `m.text` event immediately afterward to simulate a caption. + +Better would be to able to explicitly mark an event as a caption. + +## Proposal + +Allow an optional `m.relates_to` field in the `content` field of a text message event. + +Example: + +``` +... + "content": { + "body": "Caption text", + "msgtype": "m.text", + "m.relates_to": { + "event_id": "$(some image event)", + "rel_type": "m.caption" + } + }, +``` + +If a client recognises the `rel_type`, they can render the caption with the image rather than as a separate message in the timeline. + +The benefit of this is that if a client doesn't support or recognise the `m.caption`, it can ignore the relation and just render the message inline. + +This would not require aggregation from the server since there will always be a need to send the event separately anyway. + +## Potential issues + +* Not sure how this relates to the broader questions discussed in MSC1849 +* This is catering to a narrow use-case requirement. There may be a more general solution available +* Would MSC1767 (extensible events) obsolete this? From c704da1449eb18127a4fa4b8028f6dbfb3e1fd9d Mon Sep 17 00:00:00 2001 From: Ben Parsons Date: Thu, 7 May 2020 15:49:18 +0100 Subject: [PATCH 066/162] remove proposal --- proposals/2529-text-messages-as-captions.md | 37 --------------------- 1 file changed, 37 deletions(-) delete mode 100644 proposals/2529-text-messages-as-captions.md diff --git a/proposals/2529-text-messages-as-captions.md b/proposals/2529-text-messages-as-captions.md deleted file mode 100644 index df76e325..00000000 --- a/proposals/2529-text-messages-as-captions.md +++ /dev/null @@ -1,37 +0,0 @@ -# Use existing m.room.message/m.text events as captions for images - -## Background - -There is a demand to be able to apply a text caption to an image, as is possible in other chat platforms. In Matrix this is not possible, so people will generally send two events: one `m.image`, then a `m.text` event immediately afterward to simulate a caption. - -Better would be to able to explicitly mark an event as a caption. - -## Proposal - -Allow an optional `m.relates_to` field in the `content` field of a text message event. - -Example: - -``` -... - "content": { - "body": "Caption text", - "msgtype": "m.text", - "m.relates_to": { - "event_id": "$(some image event)", - "rel_type": "m.caption" - } - }, -``` - -If a client recognises the `rel_type`, they can render the caption with the image rather than as a separate message in the timeline. - -The benefit of this is that if a client doesn't support or recognise the `m.caption`, it can ignore the relation and just render the message inline. - -This would not require aggregation from the server since there will always be a need to send the event separately anyway. - -## Potential issues - -* Not sure how this relates to the broader questions discussed in MSC1849 -* This is catering to a narrow use-case requirement. There may be a more general solution available -* Would MSC1767 (extensible events) obsolete this? From 78f8d1322f220018d51281885ca237a724831366 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Fri, 8 May 2020 11:27:11 -0400 Subject: [PATCH 067/162] Add MSC2454 to the specification. --- .../client_server/newsfragments/2532.feature | 1 + specification/client_server_api.rst | 33 +++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 changelogs/client_server/newsfragments/2532.feature diff --git a/changelogs/client_server/newsfragments/2532.feature b/changelogs/client_server/newsfragments/2532.feature new file mode 100644 index 00000000..cf74a289 --- /dev/null +++ b/changelogs/client_server/newsfragments/2532.feature @@ -0,0 +1 @@ +Add User-Interactive Authentication for SSO-backed homeserver per `MSC2454 `_. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index eb32d3b4..581f0c28 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -643,6 +643,7 @@ This specification defines the following auth types: - ``m.login.password`` - ``m.login.recaptcha`` - ``m.login.oauth2`` + - ``m.login.sso`` - ``m.login.email.identity`` - ``m.login.msisdn`` - ``m.login.token`` @@ -782,6 +783,38 @@ the auth code. Homeservers can choose any path for the ``redirect URI``. Once the OAuth flow has completed, the client retries the request with the session only, as above. +Single Sign-On +<<<<<<<<<<<<<< +:Type: + ``m.login.sso`` +:Description: + Authentication is supported by authorising with an external single sign-on + provider. + +A client wanting to complete authentication using SSO should use the +`Fallback`_ authentication flow by opening a browser window for +``/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>`` with the +session parameter set to the session ID provied by the server. + +The homeserver should return a page which asks for the user's confirmation +before proceeding. For example, the page could say words to the effect of: + + A client is trying to remove a device/add an email address/take over your + account. To confirm this action, re-authenticate with single sign-on. If you + did not expect this, your account may be compromised! + +Once the user has confirmed they should be redirected to the single sign-on +provider's login page. Once the provider has validated the user, the browser is +redirected back to the homeserver. + +The homeserver then validates the response from the single sign-on provider and +updates the user-interactive authentication session to mark the single sign-on +stage has been completed. The browser is shown the fallback authentication +completion page. + +Once the flow has completed, the client retries the request with the session +only, as above. + Email-based (identity / homeserver) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< :Type: From e85f6c31a3f6d2f93de402347edae4500d95b383 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Mon, 11 May 2020 15:14:11 -0400 Subject: [PATCH 068/162] Fix a typo found in review. Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- specification/client_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 581f0c28..fce879a2 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -794,7 +794,7 @@ Single Sign-On A client wanting to complete authentication using SSO should use the `Fallback`_ authentication flow by opening a browser window for ``/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>`` with the -session parameter set to the session ID provied by the server. +session parameter set to the session ID provided by the server. The homeserver should return a page which asks for the user's confirmation before proceeding. For example, the page could say words to the effect of: From 3556b8457f5bde619c6e81c9b996c43fc86fb78d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 11 May 2020 21:05:26 -0600 Subject: [PATCH 069/162] Add changelog for 2527 --- changelogs/server_server/newsfragments/2527.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/server_server/newsfragments/2527.clarification diff --git a/changelogs/server_server/newsfragments/2527.clarification b/changelogs/server_server/newsfragments/2527.clarification new file mode 100644 index 00000000..329d5da2 --- /dev/null +++ b/changelogs/server_server/newsfragments/2527.clarification @@ -0,0 +1 @@ +Clarify which fields are required on the key server endpoints. From f156938821cba90bb555af45dac28c01b460524f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 12 May 2020 16:10:14 -0600 Subject: [PATCH 070/162] Update v6 room proposal for modern activity --- proposals/2240-rooms-v6.md | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/proposals/2240-rooms-v6.md b/proposals/2240-rooms-v6.md index a67a1ad7..fddbdc9c 100644 --- a/proposals/2240-rooms-v6.md +++ b/proposals/2240-rooms-v6.md @@ -1,21 +1,14 @@ # MSC2240: Room Version 6 -A new room version, `6`, is proposed using [room version 5](https://matrix.org/docs/spec/rooms/v5.html) -as a starting point and incorporating the following MSCs: +A new room version, `6`, is proposed using [room version 5](https://matrix.org/docs/spec/rooms/v5.html) as a base +and incorporating the following MSCs: -* [MSC2174](https://github.com/matrix-org/matrix-doc/pull/2174) - Moving the `redacts` key. -* [MSC2175](https://github.com/matrix-org/matrix-doc/pull/2175) - Removing the `creator` from create events. -* [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) - Modern redaction rules. * [MSC2209](https://github.com/matrix-org/matrix-doc/pull/2209) - Including notifications in power level auth rules. -* [MSC2212](https://github.com/matrix-org/matrix-doc/pull/2212) - Third party user power levels. -* [MSC2213](https://github.com/matrix-org/matrix-doc/pull/2213) - Rejoinability of invite-only rooms. -* [MSC1849](https://github.com/matrix-org/matrix-doc/pull/1849) - Event aggregations/relationships. +* [MSC2212](https://github.com/matrix-org/matrix-doc/pull/2432) - Alias event authorisation and redaction. +* PLACEHOLDER - Something to do with integers. -MSCs which require a new room version and are excluded from v6 are: -* [MSC2214](https://github.com/matrix-org/matrix-doc/pull/2214) - Joining upgraded private rooms. - This MSC is excluded due to the solution not being fully worked out. As this v6 proposal progresses, - it is possible for it to be included in the future. +Though other MSCs are capable of being included in this version, they do not have sufficient implementation to be +considered stable enough for v6 rooms. A future room version may still include them. - -Room version 6 upon being added to the specification shall be considered stable. No other room versions -are affected by this MSC. +Room version 6 upon being added to the specification shall be considered stable. No other room versions are affected +by this MSC. From 42ae5c966a1256e3ff2ec1945c45529dc92fae68 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 12 May 2020 19:02:39 -0600 Subject: [PATCH 071/162] Clarify the limits of prev_events and auth_events Fixes https://github.com/matrix-org/matrix-doc/issues/2307 These restrictions are due to a Synapse change which never made it to the spec prior to the MSC process. This is therefore documenting an omission. Synapse's numbers are verified here: https://github.com/matrix-org/synapse/blob/207b1737ee0acd226359d59ce3b7f7d46111b1c8/synapse/handlers/federation.py#L1183-L1213 ... and Synapse's handling of the backfill endpoint is described here: https://github.com/matrix-org/synapse/blob/207b1737ee0acd226359d59ce3b7f7d46111b1c8/synapse/handlers/federation.py#L829-L837 This also includes a mention that https://github.com/matrix-org/matrix-doc/pull/1594 should render the `auth_events` limitation unnecessary. No changes are required to the the "checks on receipt of a PDU" section of the server-server spec as it starts with "must be a valid event", to which an event is invalid if it contains too many prev or auth events. --- api/server-server/backfill.yaml | 8 ++++++++ api/server-server/definitions/pdu_v3.yaml | 6 ++++++ api/server-server/definitions/pdu_v4.yaml | 6 ++++++ api/server-server/definitions/unsigned_pdu_base.yaml | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/api/server-server/backfill.yaml b/api/server-server/backfill.yaml index 2ed6298c..d53de6c0 100644 --- a/api/server-server/backfill.yaml +++ b/api/server-server/backfill.yaml @@ -63,6 +63,14 @@ paths: description: |- A transaction containing the PDUs that preceded the given event(s), including the given event(s), up to the given limit. + + .. Note:: + Though the PDU definitions require that ``prev_events`` and ``auth_events`` be limited + in number, the response of backfill MUST NOT be validated on these specific restrictions. + + Due to historical reasons, it is possible that events which were previously accepted + would now be rejected by these limitations. The events should be rejected per usual by + the ``/send``, ``/get_missing_events``, and remaining endpoints. schema: $ref: "definitions/unlimited_pdu_transaction.yaml" "/get_missing_events/{roomId}": diff --git a/api/server-server/definitions/pdu_v3.yaml b/api/server-server/definitions/pdu_v3.yaml index 38105098..3e69d941 100644 --- a/api/server-server/definitions/pdu_v3.yaml +++ b/api/server-server/definitions/pdu_v3.yaml @@ -32,6 +32,10 @@ allOf: description: |- Event IDs for the authorization events that would allow this event to be in the room. + + Must contain less than or equal to 10 events. Note that if the relevant + auth event selection rules are used, this restriction should never be + encountered. example: ["$base64EncodedHash", "$AnotherEvent"] prev_events: type: array @@ -41,6 +45,8 @@ allOf: description: |- Event IDs for the most recent events in the room that the homeserver was aware of when it made this event. + + Must contain less than or equal to 20 events. example: ["$base64EncodedHash", "$AnotherEvent"] hashes: type: object diff --git a/api/server-server/definitions/pdu_v4.yaml b/api/server-server/definitions/pdu_v4.yaml index a045e657..d22956c6 100644 --- a/api/server-server/definitions/pdu_v4.yaml +++ b/api/server-server/definitions/pdu_v4.yaml @@ -32,6 +32,10 @@ allOf: description: |- Event IDs for the authorization events that would allow this event to be in the room. + + Must contain less than or equal to 10 events. Note that if the relevant + auth event selection rules are used, this restriction should never be + encountered. example: ["$URLsafe-base64EncodedHash", "$Another_Event"] prev_events: type: array @@ -41,6 +45,8 @@ allOf: description: |- Event IDs for the most recent events in the room that the homeserver was aware of when it made this event. + + Must contain less than or equal to 20 events. example: ["$URLsafe-base64EncodedHash", "$Another_Event"] required: - auth_events diff --git a/api/server-server/definitions/unsigned_pdu_base.yaml b/api/server-server/definitions/unsigned_pdu_base.yaml index 283e4fed..f1485b55 100644 --- a/api/server-server/definitions/unsigned_pdu_base.yaml +++ b/api/server-server/definitions/unsigned_pdu_base.yaml @@ -53,6 +53,8 @@ properties: description: |- Event IDs and reference hashes for the most recent events in the room that the homeserver was aware of when it made this event. + + Must contain less than or equal to 20 events. items: type: array maxItems: 2 @@ -84,6 +86,10 @@ properties: description: |- Event IDs and reference hashes for the authorization events that would allow this event to be in the room. + + Must contain less than or equal to 10 events. Note that if the relevant + auth event selection rules are used, this restriction should never be + encountered. items: type: array maxItems: 2 From 9599c3bc99bd1e7a2914d868f7349da4e9707b6f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 12 May 2020 19:03:53 -0600 Subject: [PATCH 072/162] changelog --- changelogs/server_server/newsfragments/2538.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/server_server/newsfragments/2538.clarification diff --git a/changelogs/server_server/newsfragments/2538.clarification b/changelogs/server_server/newsfragments/2538.clarification new file mode 100644 index 00000000..be6f769e --- /dev/null +++ b/changelogs/server_server/newsfragments/2538.clarification @@ -0,0 +1 @@ +Clarify the limits of ``prev_event``s and ``auth_events`` for PDUs. From 05228bf40be9eb89b225924f74b497119d1dac62 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 12 May 2020 19:13:16 -0600 Subject: [PATCH 073/162] ``s -> s`` --- changelogs/server_server/newsfragments/2538.clarification | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/server_server/newsfragments/2538.clarification b/changelogs/server_server/newsfragments/2538.clarification index be6f769e..4b709a8d 100644 --- a/changelogs/server_server/newsfragments/2538.clarification +++ b/changelogs/server_server/newsfragments/2538.clarification @@ -1 +1 @@ -Clarify the limits of ``prev_event``s and ``auth_events`` for PDUs. +Clarify the limits of ``prev_events`` and ``auth_events`` for PDUs. From 7797c187275c5e340b2d4ef54aa8d4ef331a5012 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Wed, 13 May 2020 11:22:51 -0400 Subject: [PATCH 074/162] Add MSC2540 for stricter validation of event JSON. --- proposals/2540-stricter-event-validation.md | 55 +++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 proposals/2540-stricter-event-validation.md diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md new file mode 100644 index 00000000..9763b4f4 --- /dev/null +++ b/proposals/2540-stricter-event-validation.md @@ -0,0 +1,55 @@ +# MSC2540: Stricter event validation: JSON compliance + +## Background + +There has been [prior discussions](https://github.com/matrix-org/matrix-doc/issues/1646) +about validating events more strictly. This MSC proposes fixing a small piece of +this: JSON compliance. + +The [Canonical JSON](https://matrix.org/docs/spec/appendices#canonical-json) +specification requires that numbers that are serialized in JSON are integers in +the range of [-2 ^ 53 + 1, 2 ^ 53 - 1], which matches the requirements of +[section 6 of RFC 7159](https://tools.ietf.org/html/rfc7159). Note that it is +not explicit, but all floats are invalid. + +It is worth mentioning that there are common extensions to JSON which produce +invalid JSON according to the Matrix specification, some programming langauges +even support these by default. One common additional feature is handling +"special" float values: `Infinity`, `-Infinity`, and `NaN`. + + +## Proposal + +In a future room version, Matrix server implementations should strictly enforce +the Canonical JSON specification for events. + +The rationale for doing this in a future room version is to avoid a split brain +room -- where some federated servers believe an event is valid and others reject +it as invalid. Rooms will be able to opt into this behavior as part of a room +version upgrade. + + +## Potential issues + +N/A + + +## Alternatives + +It could be argued that this MSC is unnecessary since it does not add any new +requirements for handling of JSON data. Unfortunately starting to enforce these +requirements in current rooms could cause federation to break as homeservers +will disagree on whether events are valid. + + +## Security considerations + +N/A + + +## Unstable prefix + +A room versions of `org.matrix.strict_canonicaljson` until a future room version +is available. This room version will use +[room version 5](https://matrix.org/docs/spec/rooms/v5) as base and include the +above modifications. From 11587a53849b4e898fd13b8051a8417619590879 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Wed, 13 May 2020 11:25:31 -0400 Subject: [PATCH 075/162] Small clarification. --- proposals/2540-stricter-event-validation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index 9763b4f4..81acd970 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -21,7 +21,7 @@ even support these by default. One common additional feature is handling ## Proposal In a future room version, Matrix server implementations should strictly enforce -the Canonical JSON specification for events. +the JSON compliance of the Canonical JSON specification for events. The rationale for doing this in a future room version is to avoid a split brain room -- where some federated servers believe an event is valid and others reject From d8e6e17bd378481e7f411f83c1fe30b001b9fb18 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 13 May 2020 09:42:14 -0600 Subject: [PATCH 076/162] Incorporate integers MSC --- proposals/2240-rooms-v6.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2240-rooms-v6.md b/proposals/2240-rooms-v6.md index fddbdc9c..9555f97c 100644 --- a/proposals/2240-rooms-v6.md +++ b/proposals/2240-rooms-v6.md @@ -5,7 +5,7 @@ and incorporating the following MSCs: * [MSC2209](https://github.com/matrix-org/matrix-doc/pull/2209) - Including notifications in power level auth rules. * [MSC2212](https://github.com/matrix-org/matrix-doc/pull/2432) - Alias event authorisation and redaction. -* PLACEHOLDER - Something to do with integers. +* [MSC2540](https://github.com/matrix-org/matrix-doc/pull/2540) - Integers in canonical JSON compliance. Though other MSCs are capable of being included in this version, they do not have sufficient implementation to be considered stable enough for v6 rooms. A future room version may still include them. From bbbd9c4c33a5641da7bf247d9f7967278ef0802c Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Wed, 13 May 2020 12:50:09 -0400 Subject: [PATCH 077/162] Update wording to avoid demand-style language. Co-authored-by: Travis Ralston --- proposals/2540-stricter-event-validation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index 81acd970..d5d31b46 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -20,7 +20,7 @@ even support these by default. One common additional feature is handling ## Proposal -In a future room version, Matrix server implementations should strictly enforce +In a future room version, Matrix server implementations are to strictly enforce the JSON compliance of the Canonical JSON specification for events. The rationale for doing this in a future room version is to avoid a split brain From 03588cbf0dfe47b8eb6bcc0dbb53a1c39115044a Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Wed, 13 May 2020 14:02:50 -0400 Subject: [PATCH 078/162] Be clearer about errors and what to do with current room versions. --- proposals/2540-stricter-event-validation.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index d5d31b46..12996333 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -20,18 +20,26 @@ even support these by default. One common additional feature is handling ## Proposal -In a future room version, Matrix server implementations are to strictly enforce -the JSON compliance of the Canonical JSON specification for events. +In a future room version, homeserver implementations are to strictly enforce +the JSON compliance of the Canonical JSON specification for events. Events that +do not abide by these rules should be rejected with the error code `M_NOT_JSON`. The rationale for doing this in a future room version is to avoid a split brain room -- where some federated servers believe an event is valid and others reject it as invalid. Rooms will be able to opt into this behavior as part of a room version upgrade. +Homeserver implementations are not to strictly enforce this JSON compliance in +[stable room versions](https://matrix.org/docs/spec/#complete-list-of-room-versions). +The rationale is essentially the same as why a future room version is necessary, +this ensures that all federated servers treat the same events as valid. + ## Potential issues -N/A +Homeserver implementations might include JSON parsers which are stricter than +others. It may not be worthwhile or reasonable to loosen those restrictions for +stable room versions. ## Alternatives From 1bae8ea6333fbc78e1bed50dfde35ad0d7fa8bbf Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 13 May 2020 19:24:34 -0400 Subject: [PATCH 079/162] we actually don't support the GET /room_key/versions/ with trailing slash --- api/client-server/key_backup.yaml | 77 ++++++++++++++++++++++++++++--- 1 file changed, 71 insertions(+), 6 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index 0f7ff25b..2735c237 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -79,6 +79,74 @@ paths: - version tags: - End-to-end encryption + "/room_keys/version": + get: + summary: Get information about the latest backup version. + description: |- + Get information about the latest backup version. + operationId: getRoomKeysVersionCurrent + security: + - accessToken: [] + responses: + 200: + description: + The information about the backup. + schema: + type: object + properties: + algorithm: + type: string + description: The algorithm used for storing backups. + enum: ["m.megolm_backup.v1.curve25519-aes-sha2"] + example: "m.megolm_backup.v1.curve25519-aes-sha2" + auth_data: + description: |- + Algorithm-dependent data. See the documentation for the backup + algorithms in `Server-side key backups`_ for more information on the + expected format of the data. + type: object + example: { + "public_key": "abcdefg", + "signatures": { + "@alice:example.org": { + "ed25519:deviceid": "signature" + } + } + } + count: + description: The number of keys stored in the backup. + type: integer + example: 42 + etag: + description: |- + An opaque string representing stored keys in the backup. + Clients can compare it with the ``etag`` value they received + in the request of their last key storage request. If not + equal, another client has modified the backup. + type: string + example: "anopaquestring" + version: + type: string + description: The backup version + example: "1" + required: + - algorithm + - auth_data + - count + - etag + - version + 404: + description: + No backup exists. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "No current backup version" + } + schema: + "$ref": "definitions/errors/error.yaml" + tags: + - End-to-end encryption "/room_keys/version/{version}": get: summary: Get information about an existing backup. @@ -94,8 +162,7 @@ paths: description: |- The backup version to get, as returned in the ``version`` parameter of the response in `POST /_matrix/client/r0/room_keys/version`_ or - this endpoint. When an empty string, the trailing slash is - optional, and the current backup is returned. + this endpoint. required: true x-example: "1" responses: @@ -173,8 +240,7 @@ paths: The backup version to update, as returned in the ``version`` parameter in the response of `POST /_matrix/client/r0/room_keys/version`_ or `GET - /_matrix/client/r0/room_keys/version/{version}`_. Unlike with the - ``GET`` version of this endpoint, this may not be empty. + /_matrix/client/r0/room_keys/version/{version}`_. required: true x-example: "1" - in: body @@ -258,8 +324,7 @@ paths: The backup version to delete, as returned in the ``version`` parameter in the response of `POST /_matrix/client/r0/room_keys/version`_ or `GET - /_matrix/client/r0/room_keys/version/{version}`_. Unlike with the - ``GET`` version of this endpoint, this may not be empty. + /_matrix/client/r0/room_keys/version/{version}`_. required: true x-example: "1" responses: From bd95568c8cb7ba69274aea2f049113c53dcbde31 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 13 May 2020 19:27:36 -0400 Subject: [PATCH 080/162] remove duplicate key --- api/client-server/key_backup.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index 2735c237..f4e83bae 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -79,7 +79,6 @@ paths: - version tags: - End-to-end encryption - "/room_keys/version": get: summary: Get information about the latest backup version. description: |- From 6757c60d35cba656972c8e979f83ba3514b17350 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Thu, 14 May 2020 07:22:48 -0400 Subject: [PATCH 081/162] Specify the current stable room versions. Co-authored-by: Travis Ralston --- proposals/2540-stricter-event-validation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index 12996333..64a64dde 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -30,7 +30,7 @@ it as invalid. Rooms will be able to opt into this behavior as part of a room version upgrade. Homeserver implementations are not to strictly enforce this JSON compliance in -[stable room versions](https://matrix.org/docs/spec/#complete-list-of-room-versions). +[room versions 1, 2, 3, 4, and 5](https://matrix.org/docs/spec/#complete-list-of-room-versions). The rationale is essentially the same as why a future room version is necessary, this ensures that all federated servers treat the same events as valid. From 560eb6df71d974f71038c67f004092e9beca3076 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 14 May 2020 10:44:44 -0600 Subject: [PATCH 082/162] Fix MSC reference --- proposals/2240-rooms-v6.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2240-rooms-v6.md b/proposals/2240-rooms-v6.md index 9555f97c..2b3f1163 100644 --- a/proposals/2240-rooms-v6.md +++ b/proposals/2240-rooms-v6.md @@ -4,7 +4,7 @@ A new room version, `6`, is proposed using [room version 5](https://matrix.org/d and incorporating the following MSCs: * [MSC2209](https://github.com/matrix-org/matrix-doc/pull/2209) - Including notifications in power level auth rules. -* [MSC2212](https://github.com/matrix-org/matrix-doc/pull/2432) - Alias event authorisation and redaction. +* [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) - Alias event authorisation and redaction. * [MSC2540](https://github.com/matrix-org/matrix-doc/pull/2540) - Integers in canonical JSON compliance. Though other MSCs are capable of being included in this version, they do not have sufficient implementation to be From 0b6301ac80b95473ddaad44b644bbf7d5867ae92 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Fri, 15 May 2020 10:51:41 -0400 Subject: [PATCH 083/162] Fix grammar and typos from review. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- proposals/2540-stricter-event-validation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index 64a64dde..94ec481e 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -13,7 +13,7 @@ the range of [-2 ^ 53 + 1, 2 ^ 53 - 1], which matches the requirements of not explicit, but all floats are invalid. It is worth mentioning that there are common extensions to JSON which produce -invalid JSON according to the Matrix specification, some programming langauges +invalid JSON according to the Matrix specification; some programming langauges even support these by default. One common additional feature is handling "special" float values: `Infinity`, `-Infinity`, and `NaN`. @@ -31,7 +31,7 @@ version upgrade. Homeserver implementations are not to strictly enforce this JSON compliance in [room versions 1, 2, 3, 4, and 5](https://matrix.org/docs/spec/#complete-list-of-room-versions). -The rationale is essentially the same as why a future room version is necessary, +The rationale is essentially the same as why a future room version is necessary: this ensures that all federated servers treat the same events as valid. @@ -57,7 +57,7 @@ N/A ## Unstable prefix -A room versions of `org.matrix.strict_canonicaljson` until a future room version +A room version of `org.matrix.strict_canonicaljson` until a future room version is available. This room version will use [room version 5](https://matrix.org/docs/spec/rooms/v5) as base and include the above modifications. From 007c8b57992137c19e2ded990e141bf0b9157eb1 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Fri, 15 May 2020 10:52:29 -0400 Subject: [PATCH 084/162] Clarify the range of valid values. --- proposals/2540-stricter-event-validation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index 94ec481e..3027be56 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -8,7 +8,7 @@ this: JSON compliance. The [Canonical JSON](https://matrix.org/docs/spec/appendices#canonical-json) specification requires that numbers that are serialized in JSON are integers in -the range of [-2 ^ 53 + 1, 2 ^ 53 - 1], which matches the requirements of +the inclusive range of `[-2 ^ 53 + 1, 2 ^ 53 - 1]`, which matches the requirements of [section 6 of RFC 7159](https://tools.ietf.org/html/rfc7159). Note that it is not explicit, but all floats are invalid. From e5fa76c8edc526bad233000019e9da8da17ce232 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Fri, 15 May 2020 10:53:26 -0400 Subject: [PATCH 085/162] Clarify range a bit more. --- proposals/2540-stricter-event-validation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index 3027be56..3d58411d 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -8,7 +8,7 @@ this: JSON compliance. The [Canonical JSON](https://matrix.org/docs/spec/appendices#canonical-json) specification requires that numbers that are serialized in JSON are integers in -the inclusive range of `[-2 ^ 53 + 1, 2 ^ 53 - 1]`, which matches the requirements of +the inclusive range of `[-(2^53) + 1, (2^53) - 1]`, which matches the requirements of [section 6 of RFC 7159](https://tools.ietf.org/html/rfc7159). Note that it is not explicit, but all floats are invalid. From bac9afa124b47ed722a74c00ef7c65d83308beb4 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 15 May 2020 12:14:46 -0400 Subject: [PATCH 086/162] document error codes and remove a statement that isn't true --- proposals/2526-add-delete-backup.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/2526-add-delete-backup.md b/proposals/2526-add-delete-backup.md index e4d589e3..b41fe5a3 100644 --- a/proposals/2526-add-delete-backup.md +++ b/proposals/2526-add-delete-backup.md @@ -8,7 +8,7 @@ delete an entire key backup. This proposal adds the endpoint. An endpoint is added, `DELETE /room_keys/version/{version}`, that deletes a backup version. Both the information about the key backup, as well as all keys -associated with the backup should be deleted. Like `POST -/room_keys/version/{version}`, and unlike `GET /room_keys/version/{version}`, -`{version}` cannot be empty, to ensure that the wrong backup is not -accidentally deleted. +associated with the backup should be deleted. If the specified version was +previously deleted, the endpoint succeeds, returning an HTTP code of 200. If +the specified version never existed, the endpoint returns an HTTP code of 404 +with a Matrix `errcode` of `M_NOT_FOUND`. From fae1165e1cc7eda27140267c08e6ee8f124142a1 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 15 May 2020 12:17:37 -0400 Subject: [PATCH 087/162] spec what to do if you try to delete a backup that's already deleted --- api/client-server/key_backup.yaml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index f4e83bae..1a79db31 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -328,12 +328,16 @@ paths: x-example: "1" responses: 200: - description: The delete succeeded. + description: |- + The delete succeeded, or the specified backup was previously + deleted. schema: type: object properties: {} 404: - description: The backup specified does not exist. + description: |- + The backup specified does not exist. If the backup was previously + deleted, the call should succeed rather than returning an error. examples: application/json: { "errcode": "M_NOT_FOUND", From d24f15a3a91b19429d65dd516e8cded1c7a2b44f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 May 2020 13:41:05 -0600 Subject: [PATCH 088/162] Spec soft-logout per MSC1466 MSC: https://github.com/matrix-org/matrix-doc/issues/1466 --- api/client-server/registration.yaml | 5 ++++- .../client_server/newsfragments/2545.feature | 1 + specification/client_server_api.rst | 22 +++++++++++++++++++ 3 files changed, 27 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/2545.feature diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index 50ce4a96..a6d5d275 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -346,8 +346,11 @@ paths: logout_devices: type: boolean description: |- - Whether the other access tokens, and their associated devices, for the user should be + Whether the other access tokens, and their associated devices, for the user should be revoked if the request succeeds. Defaults to true. + + When ``false``, the server can still take advantage of `the soft logout method <#soft-logout>`_ + for the user's remaining devices. example: true auth: description: |- diff --git a/changelogs/client_server/newsfragments/2545.feature b/changelogs/client_server/newsfragments/2545.feature new file mode 100644 index 00000000..40c066f4 --- /dev/null +++ b/changelogs/client_server/newsfragments/2545.feature @@ -0,0 +1 @@ +Add soft-logout support per `MSC1466 `_. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index fce879a2..3c99d305 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -123,6 +123,10 @@ The common error codes are: :``M_UNKNOWN_TOKEN``: The access token specified was not recognised. + An additional response parameter, ``soft_logout``, might be present on the response + for 401 HTTP status codes. See `the soft logout section <#soft-logout>`_ for more + information. + :``M_MISSING_TOKEN``: No access token was specified for the request. @@ -404,6 +408,24 @@ should pass the ``device_id`` in the request body. If the client sets the to that device. There is therefore at most one active access token assigned to each device at any one time. +Soft logout +~~~~~~~~~~~ + + + +When a requests fail due to a 401 status code per above, the server can +include an extra response parameter, ``soft_logout``, to indicate if the +device information has been retained by the server. This defaults to ``false``, +implying the server has deleted the device alongside the access token. + +When ``soft_logout`` is true, the client can acquire a new access token by +specifying the device ID it is already using to the login API. In most cases +a ``soft_logout: true`` response indicates that the user's session has expired +on the server-side and the user simply needs to provide their credentials again. + +If ``soft_logout`` is ``false``, the client will not be able to reuse the device +information it already has - the server has destroyed the session. + User-Interactive Authentication API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From bd26a41114719c59c43befbc944301d49bfa4603 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 May 2020 13:42:08 -0600 Subject: [PATCH 089/162] off by one error --- .../client_server/newsfragments/{2545.feature => 2546.feature} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changelogs/client_server/newsfragments/{2545.feature => 2546.feature} (100%) diff --git a/changelogs/client_server/newsfragments/2545.feature b/changelogs/client_server/newsfragments/2546.feature similarity index 100% rename from changelogs/client_server/newsfragments/2545.feature rename to changelogs/client_server/newsfragments/2546.feature From b4beb05dde0d255d0738a11718fc0bc042d53f21 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 May 2020 13:44:21 -0600 Subject: [PATCH 090/162] Remove excess whitespace --- specification/client_server_api.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 3c99d305..4082fceb 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -411,8 +411,6 @@ each device at any one time. Soft logout ~~~~~~~~~~~ - - When a requests fail due to a 401 status code per above, the server can include an extra response parameter, ``soft_logout``, to indicate if the device information has been retained by the server. This defaults to ``false``, From 54102b8d36d31ade99ad1b8023849d1a54a5dcc3 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 May 2020 13:50:17 -0600 Subject: [PATCH 091/162] Clarify wording per review suggestions --- specification/proposals_intro.rst | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/specification/proposals_intro.rst b/specification/proposals_intro.rst index e6cebcf4..7c8e4efd 100644 --- a/specification/proposals_intro.rst +++ b/specification/proposals_intro.rst @@ -382,17 +382,16 @@ in some cases the MSC itself will be small enough to be considered proven. Where unclear if a MSC will require an implementation proof, ask in `#matrix-spec:matrix.org `_. -Early adoption of a MSC/idea -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Early release of a MSC/idea +~~~~~~~~~~~~~~~~~~~~~~~~~~~ To help facilitate early releases of software dependent on a spec release, implementations are required to use the following process to ensure that the official Matrix namespace is not cluttered with development or testing data. .. Note:: - Proof of concept implementations that are solely left on a branch/unreleased and for - demonstrating that a particular MSC works do not have to follow this. Note that the - ``develop`` branch adopted by some projects is still subject to this process. + Proof of concept implementations that are solely left on a branch/unreleased for + demonstrating that a particular MSC works do not have to follow this. The process for an implementation willing to be ahead of a spec release is: @@ -404,9 +403,9 @@ The process for an implementation willing to be ahead of a spec release is: `/_matrix/client/unstable/com.example/login`. Vendor prefixes throughout Matrix always use the Java package naming convention. The MSC for the feature should identify which preferred vendor prefix is to be used by early adopters. - * Unstable endpoints **do not** inherit from stable (e.g. `/r0`) APIs. Implementations - cannot assume that a particular endpoint will exist in the unstable namespace - even if the server advertises support for the feature. + * Note that unstable namespaces do not automatically inherit endpoints from stable + namespaces: for example, the fact that ``/_matrix/client/r0/sync`` exists does + not imply that ``/_matrix/client/unstable/com.example/sync`` exists. * If the client needs to be sure the server supports the feature, an unstable feature flag that MUST be vendor prefixed is to be used. This kind of flag shows up in the ``unstable_features`` section of ``/versions`` as, for example, @@ -431,7 +430,7 @@ The process for an implementation willing to be ahead of a spec release is: working as proposed. A typical example of this is an implementation of the MSC, though the implementation does not need to be shipped anywhere and can therefore avoid the forwards/backwards compatibility concerns mentioned here. -5. FCP is gone through, and assuming nothing is flagged the MSC lands. +5. The FCP process is completed, and assuming nothing is flagged the MSC lands. 6. A spec PR is written to incorporate the changes into Matrix. 7. A spec release happens. 8. Implementations switch to using stable prefixes (e.g.: ``/r0``) if the server From a2a1694c35e830cc7271cdd2e6cb242e62f037a4 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 May 2020 14:10:06 -0600 Subject: [PATCH 092/162] Spec v2 send_join and send_leave endpoints MSC: https://github.com/matrix-org/matrix-doc/pull/1802 Fixes https://github.com/matrix-org/matrix-doc/issues/2541 This also adds the v2 invite endpoint to the ACL protected list as that appears to be an omission. --- .../{joins.yaml => joins-v1.yaml} | 7 +- api/server-server/joins-v2.yaml | 176 ++++++++++++++++++ .../{leaving.yaml => leaving-v1.yaml} | 7 +- api/server-server/leaving-v2.yaml | 140 ++++++++++++++ .../server_server/newsfragments/2547.new | 1 + specification/server_server_api.rst | 11 +- 6 files changed, 338 insertions(+), 4 deletions(-) rename api/server-server/{joins.yaml => joins-v1.yaml} (98%) create mode 100644 api/server-server/joins-v2.yaml rename api/server-server/{leaving.yaml => leaving-v1.yaml} (97%) create mode 100644 api/server-server/leaving-v2.yaml create mode 100644 changelogs/server_server/newsfragments/2547.new diff --git a/api/server-server/joins.yaml b/api/server-server/joins-v1.yaml similarity index 98% rename from api/server-server/joins.yaml rename to api/server-server/joins-v1.yaml index 1b5f4632..8defda20 100644 --- a/api/server-server/joins.yaml +++ b/api/server-server/joins-v1.yaml @@ -1,4 +1,5 @@ # Copyright 2018 New Vector Ltd +# Copyright 2020 The Matrix.org Foundation C.I.C. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -165,6 +166,10 @@ paths: put: summary: Submit a signed join event to a resident server description: |- + .. Note:: + Servers should instead prefer to use the v2 ``/send_join`` + endpoint. + Submits a signed join event to the resident server for it to accept it into the room's graph. Note that events have a different format depending on the room version - check @@ -172,7 +177,7 @@ paths: **The request and response body here describes the common event fields in more detail and may be missing other required fields for a PDU.** - operationId: sendJoin + operationId: sendJoinV1 security: - signedRequest: [] parameters: diff --git a/api/server-server/joins-v2.yaml b/api/server-server/joins-v2.yaml new file mode 100644 index 00000000..585ab4b8 --- /dev/null +++ b/api/server-server/joins-v2.yaml @@ -0,0 +1,176 @@ +# Copyright 2018 New Vector Ltd +# Copyright 2020 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +swagger: '2.0' +info: + title: "Matrix Federation Join Room API" + version: "1.0.0" +host: localhost:8448 +schemes: + - https +basePath: /_matrix/federation/v2 +consumes: + - application/json +produces: + - application/json +securityDefinitions: + $ref: definitions/security.yaml +paths: + # Note: there is no v2 of make_join (yet) + "/send_join/{roomId}/{eventId}": + put: + summary: Submit a signed join event to a resident server + description: |- + .. Note:: + This API is nearly identical to the v1 API with the + exception of the response format being fixed. + + This endpoint is preferred over the v1 API as it provides + a more standarised response format. Senders which receive + a 400, 404, or other status code which indicates this endpoint + is not available should retry using the v1 API instead. + + Submits a signed join event to the resident server for it + to accept it into the room's graph. Note that events have + a different format depending on the room version - check + the `room version specification`_ for precise event formats. + **The request and response body here describes the common + event fields in more detail and may be missing other required + fields for a PDU.** + operationId: sendJoinV2 + security: + - signedRequest: [] + parameters: + - in: path + name: roomId + type: string + description: The room ID that is about to be joined. + required: true + x-example: "!abc123:matrix.org" + - in: path + name: eventId + type: string + description: The event ID for the join event. + required: true + x-example: "$abc123:example.org" + - in: body + name: body + type: object + required: true + schema: + type: object + properties: + sender: + type: string + description: The user ID of the joining member. + example: "@someone:example.org" + origin: + type: string + description: The name of the joining homeserver. + example: "matrix.org" + origin_server_ts: + type: integer + format: int64 + description: A timestamp added by the joining homeserver. + example: 1234567890 + type: + type: string + description: The value ``m.room.member``. + example: "m.room.member" + state_key: + type: string + description: The user ID of the joining member. + example: "@someone:example.org" + content: + type: object + title: Membership Event Content + description: The content of the event. + example: {"membership": "join"} + properties: + membership: + type: string + description: The value ``join``. + example: "join" + required: ['membership'] + required: + - state_key + - sender + - origin + - origin_server_ts + - type + - content + example: { + "$ref": "examples/minimal_pdu.json", + "type": "m.room.member", + "state_key": "@someone:example.org", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", + "content": { + "membership": "join" + } + } + responses: + 200: + description: |- + The full state for the room, having accepted the join event. + schema: + type: object + title: Room State + description: The state for the room. + properties: + origin: + type: string + description: The resident server's DNS name. + auth_chain: + type: array + description: |- + The auth chain. Note that events have a different format depending on + the room version - check the `room version specification`_ for precise + event formats. + items: + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ that make up the auth chain. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. + schema: + type: object + properties: [] + example: + $ref: "examples/minimal_pdu.json" + state: + type: array + description: |- + The room state. The event format varies depending on the room version - + check the `room version specification`_ for precise event formats. + items: + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ for the fully resolved state of the room. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. + schema: + type: object + properties: [] + example: + $ref: "examples/minimal_pdu.json" + required: ["auth_chain", "state", "origin"] + examples: + application/json: { + "origin": "matrix.org", + "auth_chain": [{"$ref": "examples/minimal_pdu.json"}], + "state": [{"$ref": "examples/minimal_pdu.json"}] + } diff --git a/api/server-server/leaving.yaml b/api/server-server/leaving-v1.yaml similarity index 97% rename from api/server-server/leaving.yaml rename to api/server-server/leaving-v1.yaml index e0882fe8..cd07406e 100644 --- a/api/server-server/leaving.yaml +++ b/api/server-server/leaving-v1.yaml @@ -1,4 +1,5 @@ # Copyright 2018 New Vector Ltd +# Copyright 2020 The Matrix.org Foundation C.I.C. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -142,6 +143,10 @@ paths: put: summary: Submit a signed leave event to a resident server description: |- + .. Note:: + Servers should instead prefer to use the v2 ``/send_leave`` + endpoint. + Submits a signed leave event to the resident server for it to accept it into the room's graph. Note that events have a different format depending on the room version - check @@ -149,7 +154,7 @@ paths: **The request and response body here describes the common event fields in more detail and may be missing other required fields for a PDU.** - operationId: sendLeave + operationId: sendLeaveV1 security: - signedRequest: [] parameters: diff --git a/api/server-server/leaving-v2.yaml b/api/server-server/leaving-v2.yaml new file mode 100644 index 00000000..ed16773e --- /dev/null +++ b/api/server-server/leaving-v2.yaml @@ -0,0 +1,140 @@ +# Copyright 2018 New Vector Ltd +# Copyright 2020 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +swagger: '2.0' +info: + title: "Matrix Federation Leave Room API" + version: "1.0.0" +host: localhost:8448 +schemes: + - https +basePath: /_matrix/federation/v2 +consumes: + - application/json +produces: + - application/json +securityDefinitions: + $ref: definitions/security.yaml +paths: + # Note: there is no v2 of make_leave (yet) + "/send_leave/{roomId}/{eventId}": + put: + summary: Submit a signed leave event to a resident server + description: |- + .. Note:: + This API is nearly identical to the v1 API with the + exception of the response format being fixed. + + This endpoint is preferred over the v1 API as it provides + a more standarised response format. Senders which receive + a 400, 404, or other status code which indicates this endpoint + is not available should retry using the v1 API instead. + + Submits a signed leave event to the resident server for it + to accept it into the room's graph. Note that events have + a different format depending on the room version - check + the `room version specification`_ for precise event formats. + **The request and response body here describes the common + event fields in more detail and may be missing other required + fields for a PDU.** + operationId: sendLeaveV2 + security: + - signedRequest: [] + parameters: + - in: path + name: roomId + type: string + description: The room ID that is about to be left. + required: true + x-example: "!abc123:matrix.org" + - in: path + name: eventId + type: string + description: The event ID for the leave event. + required: true + x-example: "$abc123:example.org" + - in: body + name: body + type: object + required: true + schema: + type: object + properties: + sender: + type: string + description: The user ID of the leaving member. + example: "@someone:example.org" + origin: + type: string + description: The name of the leaving homeserver. + example: "matrix.org" + origin_server_ts: + type: integer + format: int64 + description: A timestamp added by the leaving homeserver. + example: 1234567890 + type: + type: string + description: The value ``m.room.member``. + example: "m.room.member" + state_key: + type: string + description: The user ID of the leaving member. + example: "@someone:example.org" + content: + type: object + title: Membership Event Content + description: The content of the event. + example: {"membership": "leave"} + properties: + membership: + type: string + description: The value ``leave``. + example: "leave" + required: ['membership'] + depth: + type: integer + description: This field must be present but is ignored; it may be 0. + example: 12 + required: + - state_key + - sender + - origin + - origin_server_ts + - type + - depth + - content + example: { + "$ref": "examples/minimal_pdu.json", + "type": "m.room.member", + "state_key": "@someone:example.org", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", + "content": { + "membership": "leave" + } + } + responses: + 200: + description: |- + An empty response to indicate the event was accepted into the graph by + the receiving homeserver. + schema: + type: object + title: Empty Object + description: An empty object. + examples: + application/json: {} diff --git a/changelogs/server_server/newsfragments/2547.new b/changelogs/server_server/newsfragments/2547.new new file mode 100644 index 00000000..c558fb5b --- /dev/null +++ b/changelogs/server_server/newsfragments/2547.new @@ -0,0 +1 @@ +Add new v2 ``/send_join`` and ``/send_leave`` endpoints per `MSC1802 `_. diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 655a8cfc..b0a2bc1a 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -780,7 +780,9 @@ and responds to the joining server with the full set of state for the newly-joined room. The resident server must also send the event to other servers participating in the room. -{{joins_ss_http_api}} +{{joins_v1_ss_http_api}} + +{{joins_v2_ss_http_api}} .. TODO-spec - (paul) I don't really understand why the full auth_chain events are given @@ -817,7 +819,9 @@ signs the event and replaces the ``event_id`` with it's own. This is then sent t the resident server via ``/send_leave``. The resident server will then send the event to other servers in the room. -{{leaving_ss_http_api}} +{{leaving_v1_ss_http_api}} + +{{leaving_v2_ss_http_api}} Third-party invites ------------------- @@ -1071,8 +1075,11 @@ The following endpoint prefixes MUST be protected: * ``/_matrix/federation/v1/make_join`` * ``/_matrix/federation/v1/make_leave`` * ``/_matrix/federation/v1/send_join`` +* ``/_matrix/federation/v2/send_join`` * ``/_matrix/federation/v1/send_leave`` +* ``/_matrix/federation/v2/send_leave`` * ``/_matrix/federation/v1/invite`` +* ``/_matrix/federation/v2/invite`` * ``/_matrix/federation/v1/state`` * ``/_matrix/federation/v1/state_ids`` * ``/_matrix/federation/v1/backfill`` From 19cac033058898eaf1d90b2b16651ad70ff94f3c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 May 2020 22:45:18 -0600 Subject: [PATCH 093/162] Update specification/client_server_api.rst Co-authored-by: Hubert Chathi --- specification/client_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 4082fceb..62550d0a 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -411,7 +411,7 @@ each device at any one time. Soft logout ~~~~~~~~~~~ -When a requests fail due to a 401 status code per above, the server can +When a request fails due to a 401 status code per above, the server can include an extra response parameter, ``soft_logout``, to indicate if the device information has been retained by the server. This defaults to ``false``, implying the server has deleted the device alongside the access token. From 28d460fb62c61f963181c890dc933b8289f4a1cc Mon Sep 17 00:00:00 2001 From: Dominic Fischer Date: Sat, 16 May 2020 21:35:26 +0100 Subject: [PATCH 094/162] Fix Typo --- api/client-server/admin.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/client-server/admin.yaml b/api/client-server/admin.yaml index 09942a10..8794a65c 100644 --- a/api/client-server/admin.yaml +++ b/api/client-server/admin.yaml @@ -82,7 +82,7 @@ paths: devices: type: object description: |- - Each key is an identitfier for one of the user's devices. + Each key is an identifier for one of the user's devices. additionalProperties: type: object title: DeviceInfo From c3582e5fc6206f25ab06e4590d2fa54835f3d661 Mon Sep 17 00:00:00 2001 From: Dominic Fischer Date: Sat, 16 May 2020 21:41:09 +0100 Subject: [PATCH 095/162] Add changelog --- changelogs/client_server/newsfragments/2553.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2553.clarification diff --git a/changelogs/client_server/newsfragments/2553.clarification b/changelogs/client_server/newsfragments/2553.clarification new file mode 100644 index 00000000..3ccb2333 --- /dev/null +++ b/changelogs/client_server/newsfragments/2553.clarification @@ -0,0 +1 @@ +Fix various typos throughout the specification. From a7e7496cd6d709bc30d858d48e4f86773214df8e Mon Sep 17 00:00:00 2001 From: Nicolas Werner Date: Sun, 17 May 2020 22:18:00 +0200 Subject: [PATCH 096/162] Fix some numbers in the spec text. Odds were not written as an exponent. Special tag names count was not updated. Fixes #2550 Signed-off-by: Nicolas Werner --- changelogs/client_server/newsfragments/2554.clarification | 1 + specification/modules/end_to_end_encryption.rst | 2 +- specification/modules/tags.rst | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2554.clarification diff --git a/changelogs/client_server/newsfragments/2554.clarification b/changelogs/client_server/newsfragments/2554.clarification new file mode 100644 index 00000000..826fafd8 --- /dev/null +++ b/changelogs/client_server/newsfragments/2554.clarification @@ -0,0 +1 @@ +Fix some numbers in the specification to match their explanation text. diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 0223d9f5..7411abe4 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -513,7 +513,7 @@ received the other party's part. Thus an attacker essentially only has one attem attack the Diffie-Hellman exchange, and hence we can verify fewer bits while still achieving a high degree of security: if we verify n bits, then an attacker has a 1 in 2\ :sup:`n` chance of success. For example, if we verify 40 bits, then an attacker has -a 1 in 1,099,511,627,776 chance (or less than 1 in 1012 chance) of success. A failed +a 1 in 1,099,511,627,776 chance (or less than 1 in 10\ :sup:`12` chance) of success. A failed attack would result in a mismatched Short Authentication String, alerting users to the attack. diff --git a/specification/modules/tags.rst b/specification/modules/tags.rst index 9d0cd554..33948509 100644 --- a/specification/modules/tags.rst +++ b/specification/modules/tags.rst @@ -55,7 +55,7 @@ The tag namespace is defined as follows: display name directly). These non-namespaced tags are supported for historical reasons. New tags should use one of the defined namespaces above. -Two special names are listed in the specification: +Three special names are listed in the specification: The following tags are defined in the ``m.*`` namespace: * ``m.favourite``: The user's favourite rooms. These should be shown with higher precedence than other rooms. From ddeb695b9740a01140e215c90d1cb318d48f6795 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 18 May 2020 00:19:30 -0600 Subject: [PATCH 097/162] Use 'several' in place of counting --- specification/modules/tags.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/tags.rst b/specification/modules/tags.rst index 33948509..a5e23770 100644 --- a/specification/modules/tags.rst +++ b/specification/modules/tags.rst @@ -55,7 +55,7 @@ The tag namespace is defined as follows: display name directly). These non-namespaced tags are supported for historical reasons. New tags should use one of the defined namespaces above. -Three special names are listed in the specification: +Several special names are listed in the specification: The following tags are defined in the ``m.*`` namespace: * ``m.favourite``: The user's favourite rooms. These should be shown with higher precedence than other rooms. From c2d5516517559ceaa4e6bfa56024392adfa7472d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 18 May 2020 00:24:05 -0600 Subject: [PATCH 098/162] Clobber changelogs --- changelogs/client_server/newsfragments/2553.clarification | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/client_server/newsfragments/2553.clarification b/changelogs/client_server/newsfragments/2553.clarification index 3ccb2333..902a9c3f 100644 --- a/changelogs/client_server/newsfragments/2553.clarification +++ b/changelogs/client_server/newsfragments/2553.clarification @@ -1 +1 @@ -Fix various typos throughout the specification. +Fix various spelling errors throughout the specification. From 77d83b9ce503b9874bad98180eb233789b4bcd6a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 19 May 2020 08:05:47 -0600 Subject: [PATCH 099/162] Clarify that soft logout is about client state --- specification/client_server_api.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 62550d0a..11a1e3cd 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -412,17 +412,17 @@ Soft logout ~~~~~~~~~~~ When a request fails due to a 401 status code per above, the server can -include an extra response parameter, ``soft_logout``, to indicate if the -device information has been retained by the server. This defaults to ``false``, -implying the server has deleted the device alongside the access token. +include an extra response parameter, ``soft_logout``, to indicate if the client's +persisted information can be retained. This defaults to ``false``, indicating +that the server has destroyed the session. Any persisted state held by the client, +such as encryption keys and device information, will not be able to be reused. When ``soft_logout`` is true, the client can acquire a new access token by specifying the device ID it is already using to the login API. In most cases a ``soft_logout: true`` response indicates that the user's session has expired on the server-side and the user simply needs to provide their credentials again. -If ``soft_logout`` is ``false``, the client will not be able to reuse the device -information it already has - the server has destroyed the session. +In either case, the client's previously known access token will no longer function. User-Interactive Authentication API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From d419e3b38a223ab2ad56442997a6578a5341f5ae Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 19 May 2020 08:08:31 -0600 Subject: [PATCH 100/162] Remove extraneous spaces --- api/server-server/joins-v1.yaml | 4 ++-- api/server-server/joins-v2.yaml | 2 +- api/server-server/leaving-v1.yaml | 4 ++-- api/server-server/leaving-v2.yaml | 2 +- api/server-server/transactions.yaml | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/api/server-server/joins-v1.yaml b/api/server-server/joins-v1.yaml index 8defda20..86dd1722 100644 --- a/api/server-server/joins-v1.yaml +++ b/api/server-server/joins-v1.yaml @@ -63,7 +63,7 @@ paths: 200: description: |- A template to be used for the rest of the `Joining Rooms`_ handshake. Note that - events have a different format depending on the room version - check the + events have a different format depending on the room version - check the `room version specification`_ for precise event formats. **The response body here describes the common event fields in more detail and may be missing other required fields for a PDU.** @@ -172,7 +172,7 @@ paths: Submits a signed join event to the resident server for it to accept it into the room's graph. Note that events have - a different format depending on the room version - check + a different format depending on the room version - check the `room version specification`_ for precise event formats. **The request and response body here describes the common event fields in more detail and may be missing other required diff --git a/api/server-server/joins-v2.yaml b/api/server-server/joins-v2.yaml index 585ab4b8..d9569684 100644 --- a/api/server-server/joins-v2.yaml +++ b/api/server-server/joins-v2.yaml @@ -44,7 +44,7 @@ paths: Submits a signed join event to the resident server for it to accept it into the room's graph. Note that events have - a different format depending on the room version - check + a different format depending on the room version - check the `room version specification`_ for precise event formats. **The request and response body here describes the common event fields in more detail and may be missing other required diff --git a/api/server-server/leaving-v1.yaml b/api/server-server/leaving-v1.yaml index cd07406e..2d7785f6 100644 --- a/api/server-server/leaving-v1.yaml +++ b/api/server-server/leaving-v1.yaml @@ -54,7 +54,7 @@ paths: 200: description: |- A template to be used to call ``/send_leave``. Note that - events have a different format depending on the room version - check the + events have a different format depending on the room version - check the `room version specification`_ for precise event formats. **The response body here describes the common event fields in more detail and may be missing other required fields for a PDU.** @@ -149,7 +149,7 @@ paths: Submits a signed leave event to the resident server for it to accept it into the room's graph. Note that events have - a different format depending on the room version - check + a different format depending on the room version - check the `room version specification`_ for precise event formats. **The request and response body here describes the common event fields in more detail and may be missing other required diff --git a/api/server-server/leaving-v2.yaml b/api/server-server/leaving-v2.yaml index ed16773e..ac3111ed 100644 --- a/api/server-server/leaving-v2.yaml +++ b/api/server-server/leaving-v2.yaml @@ -44,7 +44,7 @@ paths: Submits a signed leave event to the resident server for it to accept it into the room's graph. Note that events have - a different format depending on the room version - check + a different format depending on the room version - check the `room version specification`_ for precise event formats. **The request and response body here describes the common event fields in more detail and may be missing other required diff --git a/api/server-server/transactions.yaml b/api/server-server/transactions.yaml index 9cc8be75..b34c1902 100644 --- a/api/server-server/transactions.yaml +++ b/api/server-server/transactions.yaml @@ -38,7 +38,7 @@ paths: The sending server must wait and retry for a 200 OK response before sending a transaction with a different ``txnId`` to the receiving server. - Note that events have a different format depending on the room version - check + Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. operationId: sendTransaction security: From 7c0046ffc75ba1e6e82e6331039e0cd8c45a037f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 19 May 2020 08:09:19 -0600 Subject: [PATCH 101/162] describes -> describe --- api/server-server/leaving-v2.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/server-server/leaving-v2.yaml b/api/server-server/leaving-v2.yaml index ac3111ed..3e82c414 100644 --- a/api/server-server/leaving-v2.yaml +++ b/api/server-server/leaving-v2.yaml @@ -46,7 +46,7 @@ paths: to accept it into the room's graph. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. - **The request and response body here describes the common + **The request and response body here describe the common event fields in more detail and may be missing other required fields for a PDU.** operationId: sendLeaveV2 From 05e83f2e7d7c09cd1518b4dc773d9baf4eeb3080 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 19 May 2020 09:54:07 -0600 Subject: [PATCH 102/162] Fix more describes -> describe --- api/server-server/joins-v1.yaml | 2 +- api/server-server/joins-v2.yaml | 2 +- api/server-server/leaving-v1.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/api/server-server/joins-v1.yaml b/api/server-server/joins-v1.yaml index 86dd1722..39287264 100644 --- a/api/server-server/joins-v1.yaml +++ b/api/server-server/joins-v1.yaml @@ -174,7 +174,7 @@ paths: to accept it into the room's graph. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. - **The request and response body here describes the common + **The request and response body here describe the common event fields in more detail and may be missing other required fields for a PDU.** operationId: sendJoinV1 diff --git a/api/server-server/joins-v2.yaml b/api/server-server/joins-v2.yaml index d9569684..a98d20fd 100644 --- a/api/server-server/joins-v2.yaml +++ b/api/server-server/joins-v2.yaml @@ -46,7 +46,7 @@ paths: to accept it into the room's graph. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. - **The request and response body here describes the common + **The request and response body here describe the common event fields in more detail and may be missing other required fields for a PDU.** operationId: sendJoinV2 diff --git a/api/server-server/leaving-v1.yaml b/api/server-server/leaving-v1.yaml index 2d7785f6..c92b7920 100644 --- a/api/server-server/leaving-v1.yaml +++ b/api/server-server/leaving-v1.yaml @@ -151,7 +151,7 @@ paths: to accept it into the room's graph. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. - **The request and response body here describes the common + **The request and response body here describe the common event fields in more detail and may be missing other required fields for a PDU.** operationId: sendLeaveV1 From 10e4ad72ca7bd6ed36b1611e0723fcf494c34306 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 19 May 2020 11:58:28 -0600 Subject: [PATCH 103/162] Clarify retention of information for soft logout Co-authored-by: Hubert Chathi --- specification/client_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 11a1e3cd..f4cc7478 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -415,7 +415,7 @@ When a request fails due to a 401 status code per above, the server can include an extra response parameter, ``soft_logout``, to indicate if the client's persisted information can be retained. This defaults to ``false``, indicating that the server has destroyed the session. Any persisted state held by the client, -such as encryption keys and device information, will not be able to be reused. +such as encryption keys and device information, must not be reused and must be discarded. When ``soft_logout`` is true, the client can acquire a new access token by specifying the device ID it is already using to the login API. In most cases From f5ebe33a9ce858ad3bde26a35bd15437e4eb3ee5 Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Tue, 19 May 2020 14:35:11 -0400 Subject: [PATCH 104/162] Reword how bad values are handled. --- proposals/2540-stricter-event-validation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index 3d58411d..ac28efc4 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -22,7 +22,7 @@ even support these by default. One common additional feature is handling In a future room version, homeserver implementations are to strictly enforce the JSON compliance of the Canonical JSON specification for events. Events that -do not abide by these rules should be rejected with the error code `M_NOT_JSON`. +do not abide by these rules should be treated like any other bad JSON value. The rationale for doing this in a future room version is to avoid a split brain room -- where some federated servers believe an event is valid and others reject From 9765116c47cf1045f0499bd3e190405fee8e05e0 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 20 May 2020 10:11:30 +0100 Subject: [PATCH 105/162] Make backfill wording clearer --- api/server-server/backfill.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/server-server/backfill.yaml b/api/server-server/backfill.yaml index d53de6c0..e98c9e90 100644 --- a/api/server-server/backfill.yaml +++ b/api/server-server/backfill.yaml @@ -32,8 +32,8 @@ paths: summary: Retrieves the events which precede the given event description: |- Retrieves a sliding-window history of previous PDUs that occurred in the given room. - Starting from the PDU ID(s) given in the ``v`` argument, the PDUs that preceded it - are retrieved, up to the total number given by the ``limit``. + Starting from the PDU ID(s) given in the ``v`` argument, the PDUs given in ``v`` and + the PDUs that preceded it are retrieved, up to the total number given by the ``limit``. operationId: backfillRoom security: - signedRequest: [] From fc91946e5a4b5ea888912420ed3ab074dfc104f9 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 20 May 2020 11:07:35 +0100 Subject: [PATCH 106/162] Update api/server-server/backfill.yaml Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- api/server-server/backfill.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/server-server/backfill.yaml b/api/server-server/backfill.yaml index e98c9e90..4bf62663 100644 --- a/api/server-server/backfill.yaml +++ b/api/server-server/backfill.yaml @@ -33,7 +33,7 @@ paths: description: |- Retrieves a sliding-window history of previous PDUs that occurred in the given room. Starting from the PDU ID(s) given in the ``v`` argument, the PDUs given in ``v`` and - the PDUs that preceded it are retrieved, up to the total number given by the ``limit``. + the PDUs that preceded them are retrieved, up to the total number given by the ``limit``. operationId: backfillRoom security: - signedRequest: [] From 07716711f14378a08f0d45064c49ecfde8c9e66b Mon Sep 17 00:00:00 2001 From: Patrick Cloke Date: Wed, 20 May 2020 08:58:32 -0400 Subject: [PATCH 107/162] Give more guidance on how invalid events should be handled. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- proposals/2540-stricter-event-validation.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/proposals/2540-stricter-event-validation.md b/proposals/2540-stricter-event-validation.md index ac28efc4..31c4c39b 100644 --- a/proposals/2540-stricter-event-validation.md +++ b/proposals/2540-stricter-event-validation.md @@ -21,8 +21,10 @@ even support these by default. One common additional feature is handling ## Proposal In a future room version, homeserver implementations are to strictly enforce -the JSON compliance of the Canonical JSON specification for events. Events that -do not abide by these rules should be treated like any other bad JSON value. +the JSON compliance of the Canonical JSON specification for events. +Non-compliant events should be treated like any other malformed event, +for example by rejecting the request with an HTTP 400 error with `M_BAD_JSON`, +or by discarding the event. The rationale for doing this in a future room version is to avoid a split brain room -- where some federated servers believe an event is valid and others reject From 3875d50de80463d32b777975a94a16a99f4bffd7 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 07:50:39 -0600 Subject: [PATCH 108/162] Add missing changelog --- changelogs/server_server/newsfragments/2559.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/server_server/newsfragments/2559.clarification diff --git a/changelogs/server_server/newsfragments/2559.clarification b/changelogs/server_server/newsfragments/2559.clarification new file mode 100644 index 00000000..cdb9cfe0 --- /dev/null +++ b/changelogs/server_server/newsfragments/2559.clarification @@ -0,0 +1 @@ +Clarify which events are targeted by backfill. From 74b627ac0b824a5fa40cf1a943a684afbf7a5516 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 17:42:10 -0600 Subject: [PATCH 109/162] Fix the response format of the `/send` endpoint Fixes https://github.com/matrix-org/matrix-doc/issues/2236 --- api/server-server/transactions.yaml | 64 ++++++++----------- .../newsfragments/2560.clarification | 1 + 2 files changed, 28 insertions(+), 37 deletions(-) create mode 100644 changelogs/server_server/newsfragments/2560.clarification diff --git a/api/server-server/transactions.yaml b/api/server-server/transactions.yaml index b34c1902..a6348e13 100644 --- a/api/server-server/transactions.yaml +++ b/api/server-server/transactions.yaml @@ -77,43 +77,33 @@ paths: The result of processing the transaction. The server is to use this response even in the event of one or more PDUs failing to be processed. schema: - type: array - minItems: 2 - maxItems: 2 - items: - - type: integer - description: The value ``200``. - example: 200 - - type: object - title: PDU Processing Results - description: The results for the processing of each PDU in the transaction. - properties: - pdus: - type: object - description: |- - The PDUs from the original transaction. The string key represents the ID of the - PDU (event) that was processed. - additionalProperties: - type: object - title: PDU Processing Result - description: Information about how the PDU was handled. - properties: - error: - type: string - description: |- - A human readable description about what went wrong in processing this PDU. - If no error is present, the PDU can be considered successfully handled. - example: "You are not allowed to send a message to this room." - required: ['pdus'] + type: object + title: PDU Processing Results + description: The results for the processing of each PDU in the transaction. + properties: + pdus: + type: object + description: |- + The PDUs from the original transaction. The string key represents the ID of the + PDU (event) that was processed. + additionalProperties: + type: object + title: PDU Processing Result + description: Information about how the PDU was handled. + properties: + error: + type: string + description: |- + A human readable description about what went wrong in processing this PDU. + If no error is present, the PDU can be considered successfully handled. + example: "You are not allowed to send a message to this room." + required: ['pdus'] examples: - application/json: [ - 200, - { - "pdus": { - "$successful_event:example.org": {}, - "$failed_event:example.org": { - "error": "You are not allowed to send a message to this room." - } + application/json: { + "pdus": { + "$successful_event:example.org": {}, + "$failed_event:example.org": { + "error": "You are not allowed to send a message to this room." } } - ] + } diff --git a/changelogs/server_server/newsfragments/2560.clarification b/changelogs/server_server/newsfragments/2560.clarification new file mode 100644 index 00000000..c60bc6f3 --- /dev/null +++ b/changelogs/server_server/newsfragments/2560.clarification @@ -0,0 +1 @@ +Fix the response format of the ``/send`` endpoint. From 04e8244dbc3ffb7d805bd6c97bf7271c9a472162 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 17:44:40 -0600 Subject: [PATCH 110/162] Clarify various bits of wording Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- specification/proposals_intro.rst | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/specification/proposals_intro.rst b/specification/proposals_intro.rst index 7c8e4efd..2e6f4389 100644 --- a/specification/proposals_intro.rst +++ b/specification/proposals_intro.rst @@ -390,17 +390,15 @@ are required to use the following process to ensure that the official Matrix nam is not cluttered with development or testing data. .. Note:: - Proof of concept implementations that are solely left on a branch/unreleased for - demonstrating that a particular MSC works do not have to follow this. - -The process for an implementation willing to be ahead of a spec release is: + Unreleased implementations (including proofs-of-concept demonstrating that a + particular MSC works) do not have to follow this process. 1. Have an idea for a feature. 2. Implement the feature using unstable endpoints, vendor prefixes, and unstable feature flags as appropriate. * When using unstable endpoints, they MUST include a vendor prefix. For example: - `/_matrix/client/unstable/com.example/login`. Vendor prefixes throughout Matrix + ``/_matrix/client/unstable/com.example/login``. Vendor prefixes throughout Matrix always use the Java package naming convention. The MSC for the feature should identify which preferred vendor prefix is to be used by early adopters. * Note that unstable namespaces do not automatically inherit endpoints from stable From 8220de58bddeb7653dd917cf2831f543698dc50a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 17:48:13 -0600 Subject: [PATCH 111/162] Drop loudly and instead encourage --- specification/proposals_intro.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/specification/proposals_intro.rst b/specification/proposals_intro.rst index 2e6f4389..6d8dc8a9 100644 --- a/specification/proposals_intro.rst +++ b/specification/proposals_intro.rst @@ -436,7 +436,7 @@ is not cluttered with development or testing data. specification version, but does have the feature flag, unstable prefixes should still be used. 9. A transition period of about 2 months starts immediately after the spec release, - before implementations start to loudly encourage other implementations to switch + before implementations start to encourage other implementations to switch to stable endpoints. For example, a server implementation should start asking client implementations to support the stable endpoints 2 months after the spec release, if they haven't already. The same applies in the reverse: if clients @@ -444,9 +444,6 @@ is not cluttered with development or testing data. supporting the new spec release, some noise should be raised in the general direction of the implementation. - * Please be considerate when being noisy to implementations. A gentle reminder - in their issue tracker is generally good enough. - .. Note:: MSCs MUST still describe what the stable endpoints/feature looks like with a note towards the bottom for what the unstable feature flag/prefixes are. For example, From 1388742a52876f9b13d5f05631baf5ae3959825e Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 18:00:27 -0600 Subject: [PATCH 112/162] Mention that we use OpenAPI 3's response code format See comments of https://github.com/matrix-org/matrix-doc/pull/2238 --- api/openapi_extensions.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/api/openapi_extensions.md b/api/openapi_extensions.md index 339452ba..fced21fc 100644 --- a/api/openapi_extensions.md +++ b/api/openapi_extensions.md @@ -43,3 +43,11 @@ doesn't work, as in the following example: properties: ... ``` + +## OpenAPI 3's "2xx" format for response codes + + + +In some cases, the schema will have HTTP response code definitions like +`2xx`, `3xx`, and `4xx`. These indicate that a response code within those +ranges (`2xx` = `200` to `299`) is valid for the schema. From b11cf560b54ddc7a73c03443a05cde41f7a98302 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 May 2020 17:24:27 -0600 Subject: [PATCH 113/162] Add spec for new alias handling (client-server) MSC: https://github.com/matrix-org/matrix-doc/pull/2432 This commit does not deal with areas which will be covered by the room version specifications (namely the redaction algorithm). It feels a bit overly cruel to completely obliterate all mentions of `m.room.aliases` from the spec as client/server developers may encounter the event type in the wild. To ensure that CTRL+F still works, a brief mention that they do nothing has been put in place, leaving no other references (except the redaction algorithm - see previous paragraph). --- api/client-server/directory.yaml | 82 ++++++++++++++++++- api/client-server/room_state.yaml | 24 ++++++ event-schemas/examples/m.room.aliases | 8 -- event-schemas/examples/m.room.canonical_alias | 6 +- event-schemas/schema/m.room.aliases | 24 ------ event-schemas/schema/m.room.canonical_alias | 22 +++-- specification/client_server_api.rst | 31 ++++--- specification/modules/instant_messaging.rst | 10 +-- 8 files changed, 144 insertions(+), 63 deletions(-) delete mode 100644 event-schemas/examples/m.room.aliases delete mode 100644 event-schemas/schema/m.room.aliases diff --git a/api/client-server/directory.yaml b/api/client-server/directory.yaml index 2c7c8386..67bdf236 100644 --- a/api/client-server/directory.yaml +++ b/api/client-server/directory.yaml @@ -19,7 +19,7 @@ host: localhost:8008 schemes: - https - http -basePath: /_matrix/client/%CLIENT_MAJOR_VERSION%/directory +basePath: /_matrix/client/%CLIENT_MAJOR_VERSION% consumes: - application/json produces: @@ -27,7 +27,7 @@ produces: securityDefinitions: $ref: definitions/security.yaml paths: - "/room/{roomAlias}": + "/directory/room/{roomAlias}": put: summary: Create a new mapping from room alias to room ID. operationId: setRoomAlias @@ -129,7 +129,16 @@ paths: description: |- Remove a mapping of room alias to room ID. - Servers may choose to implement additional access control checks here, for instance that room aliases can only be deleted by their creator or a server administrator. + Servers may choose to implement additional access control checks here, for instance that + room aliases can only be deleted by their creator or a server administrator. + + .. Note:: + Servers may choose to update the ``alt_aliases`` for the ``m.room.canonical_alias`` + state event in the room when an alias is removed. Servers which choose to update the + canonical alias event are recommended to, in addition to their other relevant permission + checks, delete the alias and return a successful response even if the user does not + have permission to update the ``m.room.canonical_alias`` event. + operationId: deleteRoomAlias security: - accessToken: [] @@ -159,3 +168,70 @@ paths: "$ref": "definitions/errors/error.yaml" tags: - Room directory + "/rooms/{roomId}/aliases": + get: + summary: Get a list of local aliases on a given room. + description: |- + Get a list of aliases maintained by the local server for the + given room. + + This endpoint can be called by users who are in the room (external + users receive an ``M_FORBIDDEN`` error response). If the room's + ``m.room.history_visibility`` maps to ``world_readable``, any + user can call this endpoint. + + Servers may choose to implement additional access control checks here, + such as allowing server administrators to view aliases regardless of + membership. + + .. Note:: + Clients are not recommended to display this list of aliases prominently + as they are not curated, unlike those listed in the ``m.room.canonical_alias`` + state event. + + operationId: getLocalAliases + security: + - accessToken: [] + parameters: + - in: path + type: string + name: roomId + description: The room ID to find local aliases of. + required: true + x-example: "!abc123:example.org" + responses: + 200: + description: |- + The list of local aliases for the room. + examples: + application/json: { + "aliases": [ + "#somewhere:example.com", + "#another:example.com", + "#hat_trick:example.com" + ] + } + schema: + type: object + properties: + aliases: + type: array + description: The server's local aliases on the room. Can be empty. + items: + type: string + required: ['aliases'] + 403: + description: The user is not permitted to retrieve the list of local aliases for the room. + examples: + application/json: { + "errcode": "M_FORBIDDEN", + "error": "You are not a member of the room." + } + schema: + "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" + tags: + - Room directory diff --git a/api/client-server/room_state.yaml b/api/client-server/room_state.yaml index 640e2009..20b9c1fd 100644 --- a/api/client-server/room_state.yaml +++ b/api/client-server/room_state.yaml @@ -45,6 +45,12 @@ paths: The body of the request should be the content object of the event; the fields in this object will vary depending on the type of event. See `Room Events`_ for the ``m.`` event specification. + + If the event type being sent is ``m.room.canonical_alias`` servers + SHOULD ensure that any new aliases being listed in the event are valid + per their grammar/syntax and that they point to the room ID where the + state event is to be sent. Servers do not validate aliases which are + being removed or are already present in the state event. operationId: setRoomStateWithKey security: - accessToken: [] @@ -104,5 +110,23 @@ paths: "errcode": "M_FORBIDDEN", "error": "You do not have permission to send the event." } + 400: + description: |- + The sender's request is malformed. + + Some example error codes include: + + * ``M_INVALID_PARAMETER``: One or more aliases within the ``m.room.canonical_alias`` + event have invalid syntax. + + * ``M_BAD_ALIAS``: One or more aliases within the ``m.room.canonical_alias`` event + do not point to the room ID for which the state event is to be sent to. + schema: + $ref: "definitions/errors/error.yaml" + examples: + application/json: { + "errcode": "M_BAD_ALIAS", + "error": "The alias '#hello:example.org' does not point to this room." + } tags: - Room participation diff --git a/event-schemas/examples/m.room.aliases b/event-schemas/examples/m.room.aliases deleted file mode 100644 index 3e20332f..00000000 --- a/event-schemas/examples/m.room.aliases +++ /dev/null @@ -1,8 +0,0 @@ -{ - "$ref": "core/state_event.json", - "state_key": "example.org", - "type": "m.room.aliases", - "content": { - "aliases": ["#somewhere:example.org", "#another:example.org"] - } -} diff --git a/event-schemas/examples/m.room.canonical_alias b/event-schemas/examples/m.room.canonical_alias index 06c3226c..698c11b8 100644 --- a/event-schemas/examples/m.room.canonical_alias +++ b/event-schemas/examples/m.room.canonical_alias @@ -3,6 +3,10 @@ "type": "m.room.canonical_alias", "state_key": "", "content": { - "alias": "#somewhere:localhost" + "alias": "#somewhere:localhost", + "alt_aliases": [ + "#somewhere:example.org", + "#myroom:example.com" + ] } } diff --git a/event-schemas/schema/m.room.aliases b/event-schemas/schema/m.room.aliases deleted file mode 100644 index 348d490d..00000000 --- a/event-schemas/schema/m.room.aliases +++ /dev/null @@ -1,24 +0,0 @@ ---- -allOf: - - $ref: core-event-schema/state_event.yaml -description: 'This event is sent by a homeserver directly to inform of changes to the list of aliases it knows about for that room. The ``state_key`` for this event is set to the homeserver which owns the room alias. The entire set of known aliases for the room is the union of all the ``m.room.aliases`` events, one for each homeserver. Clients **should** check the validity of any room alias given in this list before presenting it to the user as trusted fact. The lists given by this event should be considered simply as advice on which aliases might exist, for which the client can perform the lookup to confirm whether it receives the correct room ID.' -properties: - content: - properties: - aliases: - description: A list of room aliases. - items: - type: string - type: array - required: - - aliases - type: object - state_key: - description: The homeserver domain which owns these room aliases. - type: string - type: - enum: - - m.room.aliases - type: string -title: Informs the room about what room aliases it has been given. -type: object diff --git a/event-schemas/schema/m.room.canonical_alias b/event-schemas/schema/m.room.canonical_alias index 5e8e0524..f227871a 100644 --- a/event-schemas/schema/m.room.canonical_alias +++ b/event-schemas/schema/m.room.canonical_alias @@ -3,21 +3,25 @@ allOf: - $ref: core-event-schema/state_event.yaml description: |- This event is used to inform the room about which alias should be - considered the canonical one. This could be for display purposes or as - suggestion to users which alias to use to advertise the room. - - A room with an ``m.room.canonical_alias`` event with an absent, null, or - empty ``alias`` field should be treated the same as a room with no - ``m.room.canonical_alias`` event. + considered the canonical one, and which other aliases point to the room. + This could be for display purposes or as suggestion to users which alias + to use to advertise and access the room. properties: content: properties: alias: - description: The canonical alias. + description: | + The canonical alias for the room. If not present, null, or empty the + room should be considered to have no canonical alias. type: string + alt_aliases: + description: | + Alternative aliases the room advertises. This list can have aliases + despite the ``alias`` field being null, empty, or otherwise not present. + type: array + items: + type: string type: object - required: - - alias state_key: description: A zero-length string. pattern: '^$' diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index fce879a2..fda2c106 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1594,8 +1594,6 @@ Room Events This specification outlines several standard event types, all of which are prefixed with ``m.`` -{{m_room_aliases_event}} - {{m_room_canonical_alias_event}} {{m_room_create_event}} @@ -1608,6 +1606,15 @@ prefixed with ``m.`` {{m_room_redaction_event}} +Historical events ++++++++++++++++++ + +Some events within the ``m.`` namespace might appear in rooms, however they +serve no significant meaning in this version of the specification. They are: + +* ``m.room.aliases`` + +Previous versions of the specification have more information on these events. Syncing ~~~~~~~ @@ -1813,6 +1820,8 @@ one of the following event types: - ``m.room.aliases`` allows key ``aliases``. - ``m.room.history_visibility`` allows key ``history_visibility``. +.. TODO: Room v6 changes how redaction works in MSC2432 (no special meaning for aliases) + The server should add the event causing the redaction to the ``unsigned`` property of the redacted event, under the ``redacted_because`` key. When a client receives a redaction event it should change the redacted event in the @@ -1871,15 +1880,15 @@ send update requests to other servers. However, homeservers MUST handle ``GET`` requests to resolve aliases on other servers; they should do this using the federation API if necessary. -Rooms store a *partial* list of room aliases via the ``m.room.aliases`` state -event. This alias list is partial because it cannot guarantee that the alias -list is in any way accurate or up-to-date, as room aliases can point to -different room IDs over time. Crucially, the aliases in this event are -**purely informational** and SHOULD NOT be treated as accurate. They SHOULD -be checked before they are used or shared with another user. If a room -appears to have a room alias of ``#alias:example.com``, this SHOULD be checked -to make sure that the room's ID matches the ``room_id`` returned from the -request. +Rooms do not store a list of all aliases present on a room, though members +of the room with relevant permissions may publish preferred aliases through +the ``m.room.canonical_alias`` state event. The aliases in the state event +should point to the room ID they are published within, however room aliases +can and do drift to other room IDs over time. Clients SHOULD NOT treat the +aliases as accurate. They SHOULD be checked before they are used or shared +with another user. If a room appears to have a room alias of ``#alias:example.com``, +this SHOULD be checked to make sure that the room's ID matches the ``room_id`` +returned from the request. {{directory_cs_http_api}} diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 679eabdc..705fd2d9 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -275,13 +275,9 @@ choose a name: 1. If the room has an `m.room.name`_ state event with a non-empty ``name`` field, use the name given by that field. -#. If the room has an `m.room.canonical_alias`_ state event with a non-empty - ``alias`` field, use the alias given by that field as the name. - -#. If neither of the above conditions are met, the client can optionally guess - an alias from the ``m.room.alias`` events in the room. This is a temporary - measure while clients do not promote canonical aliases as prominently. This - step may be removed in a future version of the specification. +#. If the room has an `m.room.canonical_alias`_ state event with a valid + ``alias`` field, use the alias given by that field as the name. Note that + clients should avoid using ``alt_aliases`` when calculating the room name. #. If none of the above conditions are met, a name should be composed based on the members of the room. Clients should consider `m.room.member`_ events From 2b6086d87c269f4d246b2a9dc22fd7de1f5e792e Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 20:15:09 -0600 Subject: [PATCH 114/162] Add changelog entries It's a bit awkward that this shows up as a backwards compatible change, but it's also not breaking. --- changelogs/client_server/newsfragments/2562.feature | 1 + changelogs/client_server/newsfragments/2562.new | 1 + 2 files changed, 2 insertions(+) create mode 100644 changelogs/client_server/newsfragments/2562.feature create mode 100644 changelogs/client_server/newsfragments/2562.new diff --git a/changelogs/client_server/newsfragments/2562.feature b/changelogs/client_server/newsfragments/2562.feature new file mode 100644 index 00000000..556018da --- /dev/null +++ b/changelogs/client_server/newsfragments/2562.feature @@ -0,0 +1 @@ +Replaced legacy room alias handling with a more sustainable solution per `MSC2432 `_. diff --git a/changelogs/client_server/newsfragments/2562.new b/changelogs/client_server/newsfragments/2562.new new file mode 100644 index 00000000..c7b3e4ca --- /dev/null +++ b/changelogs/client_server/newsfragments/2562.new @@ -0,0 +1 @@ +Added ``/rooms/{roomId}/aliases`` for retrieving local aliases for a room. From 0300a3cab4a77a5757c9e6acd065261f4eb03954 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 20:45:19 -0600 Subject: [PATCH 115/162] Move redaction algorithm into room version specification We stick it in a client section of v1 as the earliest version to define the algorithm is v1, and the client-server spec tells clients to use this algorithm. --- specification/client_server_api.rst | 34 +---------------- specification/rooms/v1.rst | 59 +++++++++++++++++++++++++---- 2 files changed, 53 insertions(+), 40 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index fce879a2..8a94b711 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1779,39 +1779,7 @@ redacted include a ``redacted_because`` key whose value is the event that caused it to be redacted, which may include a reason. -Upon receipt of a redaction event, the server should strip off any keys not in -the following list: - -- ``event_id`` -- ``type`` -- ``room_id`` -- ``sender`` -- ``state_key`` -- ``content`` -- ``hashes`` -- ``signatures`` -- ``depth`` -- ``prev_events`` -- ``prev_state`` -- ``auth_events`` -- ``origin`` -- ``origin_server_ts`` -- ``membership`` - -.. Note: - Some of the keys, such as ``hashes``, will appear on the federation-formatted - event and therefore the client may not be aware of them. - -The content object should also be stripped of all keys, unless it is one of -one of the following event types: - -- ``m.room.member`` allows key ``membership``. -- ``m.room.create`` allows key ``creator``. -- ``m.room.join_rules`` allows key ``join_rule``. -- ``m.room.power_levels`` allows keys ``ban``, ``events``, ``events_default``, - ``kick``, ``redact``, ``state_default``, ``users``, ``users_default``. -- ``m.room.aliases`` allows key ``aliases``. -- ``m.room.history_visibility`` allows key ``history_visibility``. +The exact algorithm to apply against an event is defined in the `room version specification`_. The server should add the event causing the redaction to the ``unsigned`` property of the redacted event, under the ``redacted_because`` key. When a diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 10eee54d..23d8a833 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -1,4 +1,5 @@ .. Copyright 2017,2019 New Vector Ltd +.. Copyright 2020 The Matrix.org Foundation C.I.C. .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -21,13 +22,57 @@ blocks for other room versions. .. contents:: Table of Contents .. sectnum:: +Client considerations +--------------------- + +Clients may need to consider some algorithms performed by the server for their own +implementation. + +Redactions +~~~~~~~~~~ + +Upon receipt of a redaction event, the server should strip off any keys not in +the following list: + +- ``event_id`` +- ``type`` +- ``room_id`` +- ``sender`` +- ``state_key`` +- ``content`` +- ``hashes`` +- ``signatures`` +- ``depth`` +- ``prev_events`` +- ``prev_state`` +- ``auth_events`` +- ``origin`` +- ``origin_server_ts`` +- ``membership`` + +.. Note: + Some of the keys, such as ``hashes``, will appear on the federation-formatted + event and therefore the client may not be aware of them. + +The content object should also be stripped of all keys, unless it is one of +one of the following event types: + +- ``m.room.member`` allows key ``membership``. +- ``m.room.create`` allows key ``creator``. +- ``m.room.join_rules`` allows key ``join_rule``. +- ``m.room.power_levels`` allows keys ``ban``, ``events``, ``events_default``, + ``kick``, ``redact``, ``state_default``, ``users``, ``users_default``. +- ``m.room.aliases`` allows key ``aliases``. +- ``m.room.history_visibility`` allows key ``history_visibility``. + Server implementation components -------------------------------- .. WARNING:: The information contained in this section is strictly for server implementors. Applications which use the Client-Server API are generally unaffected by the - details contained here, and can safely ignore their presence. + intricacies contained here. The section above regarding client considerations + is the resource that Client-Server API use cases should reference. The algorithms defined here should only apply to version 1 rooms. Other algorithms @@ -112,7 +157,7 @@ The types of state events that affect authorization are: .. NOTE:: Power levels are inferred from defaults when not explicitly supplied. - For example, mentions of the ``sender``'s power level can also refer + For example, mentions of the ``sender``'s power level can also refer to the default power level for users in the room. The rules are as follows: @@ -250,7 +295,7 @@ The rules are as follows: #. If there is no previous ``m.room.power_levels`` event in the room, allow. #. For the keys ``users_default``, ``events_default``, - ``state_default``, ``ban``, ``redact``, ``kick``, ``invite`` check if they + ``state_default``, ``ban``, ``redact``, ``kick``, ``invite`` check if they were added, changed or removed. For each found alteration: i. If the current value is higher than the ``sender``'s current power level, @@ -258,13 +303,13 @@ The rules are as follows: #. If the new value is higher than the ``sender``'s current power level, reject. - - #. For each entry being added, changed or removed in both the ``events`` and + + #. For each entry being added, changed or removed in both the ``events`` and ``users`` keys: - + i. If the current value is higher than the ``sender``'s current power level, reject. - + #. If the new value is higher than the ``sender``'s current power level, reject. From be353115594b3572fc07c8f8b5d40e57b6c161fb Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 20:28:27 -0600 Subject: [PATCH 116/162] s/should/must for redaction algorithm This feels like it was a mistake some time ago considering the redaction algorithm is used in very strict algorithms like event signing. --- specification/rooms/v1.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 23d8a833..9282c1f3 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -31,7 +31,7 @@ implementation. Redactions ~~~~~~~~~~ -Upon receipt of a redaction event, the server should strip off any keys not in +Upon receipt of a redaction event, the server must strip off any keys not in the following list: - ``event_id`` @@ -54,7 +54,7 @@ the following list: Some of the keys, such as ``hashes``, will appear on the federation-formatted event and therefore the client may not be aware of them. -The content object should also be stripped of all keys, unless it is one of +The content object must also be stripped of all keys, unless it is one of one of the following event types: - ``m.room.member`` allows key ``membership``. From a1324aa9dca7e9ca01acde43cc5db3c7f95ad718 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 20:44:08 -0600 Subject: [PATCH 117/162] Move MSC2432 (alias handling) to v6 --- specification/index.rst | 1 + specification/rooms/v6.rst | 59 ++++++++++++++++++++++++++++++++++++++ specification/targets.yaml | 4 +++ 3 files changed, 64 insertions(+) create mode 100644 specification/rooms/v6.rst diff --git a/specification/index.rst b/specification/index.rst index 728383ab..aa8a4641 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -550,6 +550,7 @@ The available room versions are: * `Version 3 `_ - **Stable**. Introduces events whose IDs are the event's hash. * `Version 4 `_ - **Stable**. Builds on v3 by using URL-safe base64 for event IDs. * `Version 5 `_ - **Stable**. Introduces enforcement of signing key validity periods. +* `Version 6 `_ - **Stable**. Alters several authorization rules for events. Specification Versions ---------------------- diff --git a/specification/rooms/v6.rst b/specification/rooms/v6.rst new file mode 100644 index 00000000..de0ab889 --- /dev/null +++ b/specification/rooms/v6.rst @@ -0,0 +1,59 @@ +.. Copyright 2020 The Matrix.org Foundation C.I.C. +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +Room Version 6 +============== + +This room version builds on `version 5 `_ while changing various +authorization rules performed on events. + +.. contents:: Table of Contents +.. sectnum:: + + +Client considerations +--------------------- + +The redaction algorithm has changed from `room version 1 `_ to remove +all rules against events of type ``m.room.aliases``. Room versions 2, 3, 4, and +5 all use v1's redaction algorithm. The algorithm is otherwise unchanged. + + +Server implementation components +-------------------------------- + +.. WARNING:: + The information contained in this section is strictly for server implementors. + Applications which use the Client-Server API are generally unaffected by the + intricacies contained here. The section above regarding client considerations + is the resource that Client-Server API use cases should reference. + + +Room version 6 makes the following alterations to algorithms described in `room version 5 `_. + +Redactions +~~~~~~~~~~ + +As mentioned in the client considerations portion of this specification, all +special meaning has been removed for events of type ``m.room.aliases``. The +algorithm is otherwise unchanged. + +Authorization rules for events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Like redactions, all rules relating specifically to events of type ``m.room.aliases`` +are removed. They must still pass authorization checks relating to state events. + +The remaining rules are the same as in `room version 3 `_ +(the last inherited room version to specify the authorization rules). diff --git a/specification/targets.yaml b/specification/targets.yaml index 4e0b068d..5f0e32cc 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -45,6 +45,10 @@ targets: files: - rooms/v5.rst version_label: v5 + rooms@v6: # this is translated to be rooms/v6.html + files: + - rooms/v6.rst + version_label: v6 appendices: files: - appendices.rst From 74c51b05a434ddcfb3b8dc63ebfeda56c29456ba Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 22:13:55 -0600 Subject: [PATCH 118/162] Incorporate MSC2209 (`notifications` auth rules) MSC: https://github.com/matrix-org/matrix-doc/pull/2209 The changes are slightly difficult to word without dumping the text in and playing a game of spot the difference, so we now use our pre-existing pygments support to render a representation of the difference. The difference is shown in markdown-like format instead of RST for ease of understanding. It's also not rendered HTML for largely complexity reasons. --- scripts/css/pygments.css | 83 ++++++++++++++++++++++++++++++++++++++ scripts/gendoc.py | 1 + specification/rooms/v6.rst | 32 +++++++++++++++ 3 files changed, 116 insertions(+) create mode 100644 scripts/css/pygments.css diff --git a/scripts/css/pygments.css b/scripts/css/pygments.css new file mode 100644 index 00000000..a212151a --- /dev/null +++ b/scripts/css/pygments.css @@ -0,0 +1,83 @@ +/* +Original styles generated from: + pygmentize -f html -S colorful -a pre.code > ./scripts/css/pygments.css + +Rules for which we don't want the syntax highlighter to kick in are commented +out at the bottom. + +Windows users: if you regenerate this file, you'll need to re-save it as utf-8 +to make docutils happy. +*/ + +/* DIFFS */ +pre.code .gd { color: #A00000 } /* Generic.Deleted */ +pre.code .gi { color: #00A000 } /* Generic.Inserted */ + +/* UNUSED */ +/*pre.code .hll { background-color: #ffffcc }*/ +/*pre.code { background: #ffffff; }*/ +/*pre.code .c { color: #888888 } !* Comment *!*/ +/*pre.code .err { color: #FF0000; background-color: #FFAAAA } !* Error *!*/ +/*pre.code .k { color: #008800; font-weight: bold } !* Keyword *!*/ +/*pre.code .o { color: #333333 } !* Operator *!*/ +/*pre.code .ch { color: #888888 } !* Comment.Hashbang *!*/ +/*pre.code .cm { color: #888888 } !* Comment.Multiline *!*/ +/*pre.code .cp { color: #557799 } !* Comment.Preproc *!*/ +/*pre.code .cpf { color: #888888 } !* Comment.PreprocFile *!*/ +/*pre.code .c1 { color: #888888 } !* Comment.Single *!*/ +/*pre.code .cs { color: #cc0000; font-weight: bold } !* Comment.Special *!*/ +/*pre.code .ge { font-style: italic } !* Generic.Emph *!*/ +/*pre.code .gr { color: #FF0000 } !* Generic.Error *!*/ +/*pre.code .gh { color: #000080; font-weight: bold } !* Generic.Heading *!*/ +/*pre.code .go { color: #888888 } !* Generic.Output *!*/ +/*pre.code .gp { color: #c65d09; font-weight: bold } !* Generic.Prompt *!*/ +/*pre.code .gs { font-weight: bold } !* Generic.Strong *!*/ +/*pre.code .gu { color: #800080; font-weight: bold } !* Generic.Subheading *!*/ +/*pre.code .gt { color: #0044DD } !* Generic.Traceback *!*/ +/*pre.code .kc { color: #008800; font-weight: bold } !* Keyword.Constant *!*/ +/*pre.code .kd { color: #008800; font-weight: bold } !* Keyword.Declaration *!*/ +/*pre.code .kn { color: #008800; font-weight: bold } !* Keyword.Namespace *!*/ +/*pre.code .kp { color: #003388; font-weight: bold } !* Keyword.Pseudo *!*/ +/*pre.code .kr { color: #008800; font-weight: bold } !* Keyword.Reserved *!*/ +/*pre.code .kt { color: #333399; font-weight: bold } !* Keyword.Type *!*/ +/*pre.code .m { color: #6600EE; font-weight: bold } !* Literal.Number *!*/ +/*pre.code .s { background-color: #fff0f0 } !* Literal.String *!*/ +/*pre.code .na { color: #0000CC } !* Name.Attribute *!*/ +/*pre.code .nb { color: #007020 } !* Name.Builtin *!*/ +/*pre.code .nc { color: #BB0066; font-weight: bold } !* Name.Class *!*/ +/*pre.code .no { color: #003366; font-weight: bold } !* Name.Constant *!*/ +/*pre.code .nd { color: #555555; font-weight: bold } !* Name.Decorator *!*/ +/*pre.code .ni { color: #880000; font-weight: bold } !* Name.Entity *!*/ +/*pre.code .ne { color: #FF0000; font-weight: bold } !* Name.Exception *!*/ +/*pre.code .nf { color: #0066BB; font-weight: bold } !* Name.Function *!*/ +/*pre.code .nl { color: #997700; font-weight: bold } !* Name.Label *!*/ +/*pre.code .nn { color: #0e84b5; font-weight: bold } !* Name.Namespace *!*/ +/*pre.code .nt { color: #007700 } !* Name.Tag *!*/ +/*pre.code .nv { color: #996633 } !* Name.Variable *!*/ +/*pre.code .ow { color: #000000; font-weight: bold } !* Operator.Word *!*/ +/*pre.code .w { color: #bbbbbb } !* Text.Whitespace *!*/ +/*pre.code .mb { color: #6600EE; font-weight: bold } !* Literal.Number.Bin *!*/ +/*pre.code .mf { color: #6600EE; font-weight: bold } !* Literal.Number.Float *!*/ +/*pre.code .mh { color: #005588; font-weight: bold } !* Literal.Number.Hex *!*/ +/*pre.code .mi { color: #0000DD; font-weight: bold } !* Literal.Number.Integer *!*/ +/*pre.code .mo { color: #4400EE; font-weight: bold } !* Literal.Number.Oct *!*/ +/*pre.code .sa { background-color: #fff0f0 } !* Literal.String.Affix *!*/ +/*pre.code .sb { background-color: #fff0f0 } !* Literal.String.Backtick *!*/ +/*pre.code .sc { color: #0044DD } !* Literal.String.Char *!*/ +/*pre.code .dl { background-color: #fff0f0 } !* Literal.String.Delimiter *!*/ +/*pre.code .sd { color: #DD4422 } !* Literal.String.Doc *!*/ +/*pre.code .s2 { background-color: #fff0f0 } !* Literal.String.Double *!*/ +/*pre.code .se { color: #666666; font-weight: bold; background-color: #fff0f0 } !* Literal.String.Escape *!*/ +/*pre.code .sh { background-color: #fff0f0 } !* Literal.String.Heredoc *!*/ +/*pre.code .si { background-color: #eeeeee } !* Literal.String.Interpol *!*/ +/*pre.code .sx { color: #DD2200; background-color: #fff0f0 } !* Literal.String.Other *!*/ +/*pre.code .sr { color: #000000; background-color: #fff0ff } !* Literal.String.Regex *!*/ +/*pre.code .s1 { background-color: #fff0f0 } !* Literal.String.Single *!*/ +/*pre.code .ss { color: #AA6600 } !* Literal.String.Symbol *!*/ +/*pre.code .bp { color: #007020 } !* Name.Builtin.Pseudo *!*/ +/*pre.code .fm { color: #0066BB; font-weight: bold } !* Name.Function.Magic *!*/ +/*pre.code .vc { color: #336699 } !* Name.Variable.Class *!*/ +/*pre.code .vg { color: #dd7700; font-weight: bold } !* Name.Variable.Global *!*/ +/*pre.code .vi { color: #3333BB } !* Name.Variable.Instance *!*/ +/*pre.code .vm { color: #996633 } !* Name.Variable.Magic *!*/ +/*pre.code .il { color: #0000DD; font-weight: bold } !* Literal.Number.Integer.Long *!*/ diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 72e3047c..7e68ccd7 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -273,6 +273,7 @@ def rst2html(i, o, stylesheets): writer_name="html", settings_overrides={ "stylesheet_path": stylesheets, + "syntax_highlight": "short", }, ) diff --git a/specification/rooms/v6.rst b/specification/rooms/v6.rst index de0ab889..e165f771 100644 --- a/specification/rooms/v6.rst +++ b/specification/rooms/v6.rst @@ -55,5 +55,37 @@ Authorization rules for events Like redactions, all rules relating specifically to events of type ``m.room.aliases`` are removed. They must still pass authorization checks relating to state events. +Additionally, the authorization rules for events of type ``m.room.power_levels`` +now include the content key ``notifications``. This new rule takes the place of the +rule which checks the ``events`` and ``users`` keys. + +For completeness, the changes to the auth rules can be represented as follows: + +.. code:: diff + + ... + + -If type is `m.room.aliases`: + - + - a. If event has no `state_key`, reject. + - b. If sender's domain doesn't matches `state_key`, reject. + - c. Otherwise, allow. + + ... + + If type is `m.room.power_levels`: + + ... + + - * For each entry being added, changed or removed in both the `events` and `users` keys: + + * For each entry being added, changed or removed in the `events`, `users`, and `notifications` keys: + + i. If the current value is higher than the `sender`'s current power level, reject. + + ii. If the new value is higher than the `sender`'s current power level, reject. + + ... + + The remaining rules are the same as in `room version 3 `_ (the last inherited room version to specify the authorization rules). From 66ab480967c7dc1b2ad4f3750b94f837d1eda9ef Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 21:58:58 -0600 Subject: [PATCH 119/162] Incorporate MSC2540 (Canonical JSON validation) MSC: https://github.com/matrix-org/matrix-doc/pull/2540 --- specification/appendices/signing_json.rst | 11 +++++++++++ specification/rooms/v1.rst | 6 ++++++ specification/rooms/v6.rst | 9 +++++++++ 3 files changed, 26 insertions(+) diff --git a/specification/appendices/signing_json.rst b/specification/appendices/signing_json.rst index 8036950e..fbeb0010 100644 --- a/specification/appendices/signing_json.rst +++ b/specification/appendices/signing_json.rst @@ -39,6 +39,17 @@ range where they can be accurately represented using IEEE double precision floating point numbers since a number of JSON libraries represent all numbers using this representation. +.. WARNING:: + Events in room versions 1, 2, 3, 4, and 5 might not be fully compliant with + these restrictions. Servers SHOULD be capable of handling JSON which is considered + invalid by these restrictions where possible. + + The most notable consideration is that integers might not be in the range + specified above. + +.. Note:: + Float values are not permitted by this encoding. + .. code:: python import json diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 9282c1f3..a71bdfb4 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -352,6 +352,12 @@ Events in version 1 rooms have the following structure: {{definition_ss_pdu}} +Canonical JSON +~~~~~~~~~~~~~~ + +Servers MUST NOT strictly enforce the JSON format specified in the +`appendices <../appendices.html#canonical-json>`_ for the reasons described there. + .. _`auth events selection`: ../server_server/%SERVER_RELEASE_LABEL%.html#auth-events-selection .. _`Signing Events`: ../server_server/%SERVER_RELEASE_LABEL%.html#signing-events diff --git a/specification/rooms/v6.rst b/specification/rooms/v6.rst index e165f771..b1acccd2 100644 --- a/specification/rooms/v6.rst +++ b/specification/rooms/v6.rst @@ -89,3 +89,12 @@ For completeness, the changes to the auth rules can be represented as follows: The remaining rules are the same as in `room version 3 `_ (the last inherited room version to specify the authorization rules). + +Canonical JSON +~~~~~~~~~~~~~~ + +Servers MUST strictly enforce the JSON format specified in the +`appendices <../appendices.html#canonical-json>`_. This translates to a 400 ``M_BAD_JSON`` error +on most endpoints, or discarding of events over federation. For example, the federation API's +``/send`` endpoint would discard the event whereas the Client Server API's ``/send/{eventType}`` +endpoint would return a ``M_BAD_JSON`` error. From 98416bf94898395ae5f1b81bfcf8f659a1e76d8b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 22:16:16 -0600 Subject: [PATCH 120/162] Add changelog for client-server API --- changelogs/client_server/newsfragments/2563.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2563.clarification diff --git a/changelogs/client_server/newsfragments/2563.clarification b/changelogs/client_server/newsfragments/2563.clarification new file mode 100644 index 00000000..a8805f4a --- /dev/null +++ b/changelogs/client_server/newsfragments/2563.clarification @@ -0,0 +1 @@ +Move redaction algorithm into the room version specifications. From 5b38c04e1921261dfe5eb7169068c1ef86a9cd33 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 22 May 2020 15:50:32 -0400 Subject: [PATCH 121/162] make definition of signatures fields more consistent --- api/client-server/definitions/device_keys.yaml | 5 +---- api/server-server/definitions/keys.yaml | 18 ++++++------------ api/server-server/third_party_invite.yaml | 12 ++++++------ api/server-server/user_keys.yaml | 8 +++++--- 4 files changed, 18 insertions(+), 25 deletions(-) diff --git a/api/client-server/definitions/device_keys.yaml b/api/client-server/definitions/device_keys.yaml index 41504aba..ce770c09 100644 --- a/api/client-server/definitions/device_keys.yaml +++ b/api/client-server/definitions/device_keys.yaml @@ -47,16 +47,13 @@ properties: "ed25519:JLAFKJWSCS": "lEuiRJBit0IG6nUf5pUzWTUEsRVVe/HJkoKuEww9ULI" signatures: type: object + title: Signatures description: |- Signatures for the device key object. A map from user ID, to a map from ``:`` to the signature. The signature is calculated using the process described at `Signing JSON`_. - additionalProperties: - type: object - additionalProperties: - type: string example: "@alice:example.com": "ed25519:JLAFKJWSCS": "dSO80A01XiigH3uBiDVx/EjzaoycHcjq9lfQX0uWsqxl2giMIiSPR8a4d291W1ihKJL/a+myXS367WT6NAIcBA" diff --git a/api/server-server/definitions/keys.yaml b/api/server-server/definitions/keys.yaml index 1e025a52..7e300402 100644 --- a/api/server-server/definitions/keys.yaml +++ b/api/server-server/definitions/keys.yaml @@ -75,18 +75,12 @@ properties: required: ["expired_ts", "key"] signatures: type: object - description: Digital signatures for this object signed using the ``verify_keys``. - additionalProperties: - type: object - title: Signed Server - example: { - "example.org": { - "ad25519:abc123": "VGhpcyBzaG91bGQgYWN0dWFsbHkgYmUgYSBzaWduYXR1cmU" - } - } - additionalProperties: - type: string - name: Encoded Signature Verification Key + description: |- + Digital signatures for this object signed using the ``verify_keys``. + + The signature is calculated using the process described at `Signing + JSON`_. + title: Signatures valid_until_ts: type: integer format: int64 diff --git a/api/server-server/third_party_invite.yaml b/api/server-server/third_party_invite.yaml index 0b7aac5b..8897bdcf 100644 --- a/api/server-server/third_party_invite.yaml +++ b/api/server-server/third_party_invite.yaml @@ -102,12 +102,12 @@ paths: properties: signatures: type: object - description: The server signatures for this event. - additionalProperties: - type: object - title: Server Signatures - additionalProperties: - type: string + title: Signatures + description: |- + The server signatures for this event. + + The signature is calculated using the process + described at `Signing JSON`_. example: { "magic.forest": { "ed25519:3": "fQpGIW1Snz+pwLZu6sTy2aHy/DYWWTspTJRPyNp0PKkymfIsNffysMl6ObMMFdIJhk6g6pwlIqZ54rxo8SLmAg" diff --git a/api/server-server/user_keys.yaml b/api/server-server/user_keys.yaml index 93237d80..9224d508 100644 --- a/api/server-server/user_keys.yaml +++ b/api/server-server/user_keys.yaml @@ -88,10 +88,12 @@ paths: description: The key, encoded using unpadded base64. signatures: type: object + title: Signatures description: |- - Signature for the device. Mapped from user ID to signature object. - additionalProperties: - type: string + Signature of the key object. + + The signature is calculated using the process described at `Signing + JSON`_. required: ['key', 'signatures'] example: { "@alice:example.com": { From 6a8ba9b1d6372f9a71d6b8554947548556e84c8e Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 22 May 2020 16:10:16 -0400 Subject: [PATCH 122/162] improve description of key object in CS and include a link in Fed --- api/server-server/user_keys.yaml | 2 +- .../modules/end_to_end_encryption.rst | 20 ++++++++++++++++--- specification/server_server_api.rst | 1 + 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/api/server-server/user_keys.yaml b/api/server-server/user_keys.yaml index 9224d508..166f1ff0 100644 --- a/api/server-server/user_keys.yaml +++ b/api/server-server/user_keys.yaml @@ -73,7 +73,7 @@ paths: One-time keys for the queried devices. A map from user ID, to a map from devices to a map from ``:`` to the key object. - See the Client-Server Key Algorithms section for more information on + See the `Client-Server Key Algorithms`_ section for more information on the Key Object format. additionalProperties: type: object diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 7411abe4..8bc46de8 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -83,9 +83,23 @@ Base64`_. Example: "JGLn/yafz74HB2AbPLYJWIVGnKAtqECOBf11yyXac2Y" The name ``signed_curve25519`` also corresponds to the Curve25519 algorithm, -but keys using this algorithm are objects with the properties ``key`` (giving -the Base64-encoded 32-byte Curve25519 public key), and ``signatures`` (giving a -signature for the key object, as described in `Signing JSON`_). Example: +but a key using this algorithm is represented by an object with a the following +properties: + +``KeyObject`` + +========== ================ ===================================================== +Parameter Type Description +========== ================ ===================================================== +key string **Required.** The unpadded Base64-encoded 32-byte + Curve25519 public key. +signatures Signatures **Required.** Signatures of the key object. + + The signature is calculated using the process described + at `Signing JSON`_. +========== ================ ===================================================== + +Example: .. code:: json diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index b0a2bc1a..00ddeffc 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -1257,3 +1257,4 @@ issue. .. _`Device Management module`: ../client_server/%CLIENT_RELEASE_LABEL%.html#device-management .. _`End-to-End Encryption module`: ../client_server/%CLIENT_RELEASE_LABEL%.html#end-to-end-encryption .. _`room version specification`: ../index.html#room-versions +.. _`Client-Server Key Algorithms`: ../client_server/%CLIENT_RELEASE_LABEL%.html#key-algorithms From 5270ed0c0ae6d1d35375c4aa2832cfa2391c2544 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Sun, 24 May 2020 14:24:49 +0300 Subject: [PATCH 123/162] Fix typo in read markers module Signed-off-by: Tulir Asokan --- specification/modules/read_markers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/modules/read_markers.rst b/specification/modules/read_markers.rst index a77f2013..be06d037 100644 --- a/specification/modules/read_markers.rst +++ b/specification/modules/read_markers.rst @@ -47,7 +47,7 @@ The read markers API can additionally update the user's read receipt (``m.read`` location in the same operation as setting the fully read marker location. This is because read receipts and read markers are commonly updated at the same time, and therefore the client might wish to save an extra HTTP call. Providing an -``m.read`` location performs the same task as a request to ``/receipts/m.read/$event:example.org``. +``m.read`` location performs the same task as a request to ``/receipt/m.read/$event:example.org``. {{read_markers_cs_http_api}} @@ -56,7 +56,7 @@ Server behaviour The server MUST prevent clients from setting ``m.fully_read`` directly in room account data. The server must additionally ensure that it treats the presence of ``m.read`` in the ``/read_markers`` request the same as how it -would for a request to ``/receipts/m.read/$event:example.org``. +would for a request to ``/receipt/m.read/$event:example.org``. Upon updating the ``m.fully_read`` event due to a request to ``/read_markers``, the server MUST send the updated account data event through to the client via From bdd8f03fef638cb07cf0d38a21ad211ecf0d7ce1 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 24 May 2020 11:13:58 -0600 Subject: [PATCH 124/162] Add changelog --- changelogs/client_server/newsfragments/2569.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2569.clarification diff --git a/changelogs/client_server/newsfragments/2569.clarification b/changelogs/client_server/newsfragments/2569.clarification new file mode 100644 index 00000000..902a9c3f --- /dev/null +++ b/changelogs/client_server/newsfragments/2569.clarification @@ -0,0 +1 @@ +Fix various spelling errors throughout the specification. From 93f68866c5635078e84eaf8bcf55c8eb274c57f8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 08:06:57 -0600 Subject: [PATCH 125/162] Fix spec generation not using titles on pseudo objects --- scripts/templating/matrix_templates/units.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 9d2c69f2..5a7ff94e 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -213,6 +213,7 @@ def get_json_schema_object_fields(obj, enforce_title=False): res = process_data_type(additionalProps) tables = res["tables"] val_title = res["title"] + gen_title = "{%s: %s}" % (key_type, val_title) if res.get("enum_desc") and val_title != "enum": # A map to enum needs another table with enum description tables.append(TypeTable( @@ -220,7 +221,7 @@ def get_json_schema_object_fields(obj, enforce_title=False): rows=[TypeTableRow(key="(mapped value)", title="enum", desc=res["desc"])] )) return { - "title": "{%s: %s}" % (key_type, val_title), + "title": obj_title if obj_title else gen_title, "tables": tables, } From 36ed6e94a2bd2c5e591fdd87cb15acdae53ca03a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 08:08:10 -0600 Subject: [PATCH 126/162] Re-introduce signature schemas --- api/client-server/definitions/device_keys.yaml | 4 ++++ api/client-server/definitions/third_party_signed.yaml | 4 ++++ api/server-server/definitions/keys.yaml | 4 ++++ api/server-server/third_party_invite.yaml | 4 ++++ api/server-server/user_keys.yaml | 4 ++++ event-schemas/schema/m.room.member | 4 ++++ 6 files changed, 24 insertions(+) diff --git a/api/client-server/definitions/device_keys.yaml b/api/client-server/definitions/device_keys.yaml index ce770c09..4f7cffe7 100644 --- a/api/client-server/definitions/device_keys.yaml +++ b/api/client-server/definitions/device_keys.yaml @@ -54,6 +54,10 @@ properties: The signature is calculated using the process described at `Signing JSON`_. + additionalProperties: + type: object + additionalProperties: + type: string example: "@alice:example.com": "ed25519:JLAFKJWSCS": "dSO80A01XiigH3uBiDVx/EjzaoycHcjq9lfQX0uWsqxl2giMIiSPR8a4d291W1ihKJL/a+myXS367WT6NAIcBA" diff --git a/api/client-server/definitions/third_party_signed.yaml b/api/client-server/definitions/third_party_signed.yaml index c9c761a1..7ce1a1d1 100644 --- a/api/client-server/definitions/third_party_signed.yaml +++ b/api/client-server/definitions/third_party_signed.yaml @@ -33,6 +33,10 @@ properties: type: object description: A signatures object containing a signature of the entire signed object. title: Signatures + additionalProperties: + type: object + additionalProperties: + type: string example: { "example.org": { "ed25519:0": "some9signature" diff --git a/api/server-server/definitions/keys.yaml b/api/server-server/definitions/keys.yaml index 7e300402..306d4d00 100644 --- a/api/server-server/definitions/keys.yaml +++ b/api/server-server/definitions/keys.yaml @@ -81,6 +81,10 @@ properties: The signature is calculated using the process described at `Signing JSON`_. title: Signatures + additionalProperties: + type: object + additionalProperties: + type: string valid_until_ts: type: integer format: int64 diff --git a/api/server-server/third_party_invite.yaml b/api/server-server/third_party_invite.yaml index 8897bdcf..fbd2a9d9 100644 --- a/api/server-server/third_party_invite.yaml +++ b/api/server-server/third_party_invite.yaml @@ -103,6 +103,10 @@ paths: signatures: type: object title: Signatures + additionalProperties: + type: object + additionalProperties: + type: string description: |- The server signatures for this event. diff --git a/api/server-server/user_keys.yaml b/api/server-server/user_keys.yaml index 166f1ff0..6ed51af0 100644 --- a/api/server-server/user_keys.yaml +++ b/api/server-server/user_keys.yaml @@ -89,6 +89,10 @@ paths: signatures: type: object title: Signatures + additionalProperties: + type: object + additionalProperties: + type: string description: |- Signature of the key object. diff --git a/event-schemas/schema/m.room.member b/event-schemas/schema/m.room.member index 1033c098..bff2002e 100644 --- a/event-schemas/schema/m.room.member +++ b/event-schemas/schema/m.room.member @@ -82,6 +82,10 @@ properties: description: 'A single signature from the verifying server, in the format specified by the Signing Events section of the server-server API.' title: Signatures type: object + additionalProperties: + type: object + additionalProperties: + type: string token: description: The token property of the containing third_party_invite object. type: string From 730d164df7fd23fb908f27bda4511f76b769cd36 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 08:15:56 -0600 Subject: [PATCH 127/162] Add changelogs --- changelogs/client_server/newsfragments/2566.clarification | 1 + changelogs/server_server/newsfragments/2566.clarification | 1 + 2 files changed, 2 insertions(+) create mode 100644 changelogs/client_server/newsfragments/2566.clarification create mode 100644 changelogs/server_server/newsfragments/2566.clarification diff --git a/changelogs/client_server/newsfragments/2566.clarification b/changelogs/client_server/newsfragments/2566.clarification new file mode 100644 index 00000000..15e29b2f --- /dev/null +++ b/changelogs/client_server/newsfragments/2566.clarification @@ -0,0 +1 @@ +Clarify signature object structures for encryption. diff --git a/changelogs/server_server/newsfragments/2566.clarification b/changelogs/server_server/newsfragments/2566.clarification new file mode 100644 index 00000000..15e29b2f --- /dev/null +++ b/changelogs/server_server/newsfragments/2566.clarification @@ -0,0 +1 @@ +Clarify signature object structures for encryption. From b876ac5893dddf4a6b6b4b8152ce0ea5c2d2f482 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 08:27:55 -0600 Subject: [PATCH 128/162] Clarify which server names to sign requests with Fixes https://github.com/matrix-org/matrix-doc/issues/2512 --- changelogs/server_server/newsfragments/2570.clarification | 1 + specification/server_server_api.rst | 5 +++++ 2 files changed, 6 insertions(+) create mode 100644 changelogs/server_server/newsfragments/2570.clarification diff --git a/changelogs/server_server/newsfragments/2570.clarification b/changelogs/server_server/newsfragments/2570.clarification new file mode 100644 index 00000000..3fac40bf --- /dev/null +++ b/changelogs/server_server/newsfragments/2570.clarification @@ -0,0 +1 @@ +Clarify the server names to use when signing requests. diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index b0a2bc1a..ca147665 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -291,6 +291,11 @@ Step 1 sign JSON: } } +The server names in the JSON above are the server names involved. Delegation from +the `server name resolution section <#resolving-server-names>`_ above do not affect +these - the server names from before delegation would take place are used. This +same condition applies throughout the request signing process. + Step 2 add Authorization header: .. code:: From 200212ffd741be53beec8fea1de1100826d44c66 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 08:33:08 -0600 Subject: [PATCH 129/162] Clarify that a creation event and member event are needed for createRoom Fixes https://github.com/matrix-org/matrix-doc/issues/2494 --- api/client-server/create_room.yaml | 16 +++++++++++----- .../newsfragments/2571.clarification | 1 + 2 files changed, 12 insertions(+), 5 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2571.clarification diff --git a/api/client-server/create_room.yaml b/api/client-server/create_room.yaml index 2487707e..446d8aef 100644 --- a/api/client-server/create_room.yaml +++ b/api/client-server/create_room.yaml @@ -38,20 +38,26 @@ paths: the new room, including checking power levels for each event. It MUST apply the events implied by the request in the following order: - 0. A default ``m.room.power_levels`` event, giving the room creator + 1. The ``m.room.create`` event itself. Must be the first event in the + room. + + 2. An ``m.room.member`` event for the creator to join the room. This is + needed so the remaining events can be sent. + + 3. A default ``m.room.power_levels`` event, giving the room creator (and not other members) permission to send state events. Overridden by the ``power_level_content_override`` parameter. - 1. Events set by the ``preset``. Currently these are the ``m.room.join_rules``, + 4. Events set by the ``preset``. Currently these are the ``m.room.join_rules``, ``m.room.history_visibility``, and ``m.room.guest_access`` state events. - 2. Events listed in ``initial_state``, in the order that they are + 5. Events listed in ``initial_state``, in the order that they are listed. - 3. Events implied by ``name`` and ``topic`` (``m.room.name`` and ``m.room.topic`` + 6. Events implied by ``name`` and ``topic`` (``m.room.name`` and ``m.room.topic`` state events). - 4. Invite events implied by ``invite`` and ``invite_3pid`` (``m.room.member`` with + 7. Invite events implied by ``invite`` and ``invite_3pid`` (``m.room.member`` with ``membership: invite`` and ``m.room.third_party_invite``). The available presets do the following with respect to room state: diff --git a/changelogs/client_server/newsfragments/2571.clarification b/changelogs/client_server/newsfragments/2571.clarification new file mode 100644 index 00000000..e9437f3c --- /dev/null +++ b/changelogs/client_server/newsfragments/2571.clarification @@ -0,0 +1 @@ +Clarify which events are created as part of ``/createRoom``. From ac187904bbf334ea2a6b31abf4059f5eb3d27637 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 08:35:26 -0600 Subject: [PATCH 130/162] Update MSC1779 to forward link to matrix.org/foundation --- proposals/1779-open-governance.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/proposals/1779-open-governance.md b/proposals/1779-open-governance.md index bfe86eb8..d9a53c51 100644 --- a/proposals/1779-open-governance.md +++ b/proposals/1779-open-governance.md @@ -1,4 +1,11 @@ -# Proposal for Open Governance of Matrix.org +# Proposal for Open Governance of Matrix.org + +---- + +**For updated information on the Foundation, please see https://matrix.org/foundation/**. +This MSC exists for historical reference. + +---- This whole document is the proposed constitution proposal for Matrix.org, and will form the basis of the first full Articles of Association (AoA) for [The @@ -446,7 +453,7 @@ a high-level mission lock aligned with the above: The foundation was then converted into a Community Interest Company, formalising its non-profit status under the approval of the independent [Community Interest -Companies Regulator](https://www.gov.uk/government/organisations/office-of-the-regulator-of-community-interest-companies), +Companies Regulator](https://www.gov.uk/government/organisations/office-of-the-regulator-of-community-interest-companies), which took effect Jan 2019. We are currently planning to release r0 of the Matrix Spec at the end of Jan 2019, and From e8548f3c3b92e92990657571af61dea1efc95b25 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 09:42:54 -0600 Subject: [PATCH 131/162] Remove claim that profile info is exclusively managed by the server Fixes https://github.com/matrix-org/matrix-doc/issues/2496 --- changelogs/client_server/newsfragments/2574.clarification | 1 + event-schemas/schema/m.room.member | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2574.clarification diff --git a/changelogs/client_server/newsfragments/2574.clarification b/changelogs/client_server/newsfragments/2574.clarification new file mode 100644 index 00000000..d771d374 --- /dev/null +++ b/changelogs/client_server/newsfragments/2574.clarification @@ -0,0 +1 @@ +Remove claims that the homeserver is exclusively responsible for profile information in membership events. diff --git a/event-schemas/schema/m.room.member b/event-schemas/schema/m.room.member index 1033c098..dabd1b08 100644 --- a/event-schemas/schema/m.room.member +++ b/event-schemas/schema/m.room.member @@ -48,10 +48,10 @@ properties: content: properties: avatar_url: - description: 'The avatar URL for this user, if any. This is added by the homeserver.' + description: 'The avatar URL for this user, if any.' type: string displayname: - description: 'The display name for this user, if any. This is added by the homeserver.' + description: 'The display name for this user, if any.' type: - "string" - "null" From 06b0c1762a879f4db86b09261c69cfc1066c4231 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 09:55:32 -0600 Subject: [PATCH 132/162] Move the send_join response to its own definition Both APIs need clarification, so dedupe the changes. We should also probably deprecate some of this stuff so we can eventually remove it. --- .../definitions/send_join_response.yaml | 55 +++++++++++++++++++ api/server-server/joins-v1.yaml | 42 +------------- api/server-server/joins-v2.yaml | 42 +------------- 3 files changed, 57 insertions(+), 82 deletions(-) create mode 100644 api/server-server/definitions/send_join_response.yaml diff --git a/api/server-server/definitions/send_join_response.yaml b/api/server-server/definitions/send_join_response.yaml new file mode 100644 index 00000000..96ceaefc --- /dev/null +++ b/api/server-server/definitions/send_join_response.yaml @@ -0,0 +1,55 @@ +# Copyright 2018 New Vector Ltd +# Copyright 2020 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +type: object +title: Room State +description: The state for the room. +properties: + origin: + type: string + description: The resident server's DNS name. + auth_chain: + type: array + description: |- + The auth chain. Note that events have a different format depending on + the room version - check the `room version specification`_ for precise + event formats. + items: + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ that make up the auth chain. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. + schema: + type: object + properties: [] + example: + $ref: "../examples/minimal_pdu.json" + state: + type: array + description: |- + The room state. The event format varies depending on the room version - + check the `room version specification`_ for precise event formats. + items: + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ for the fully resolved state of the room. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. + schema: + type: object + properties: [] + example: + $ref: "../examples/minimal_pdu.json" +required: ["auth_chain", "state", "origin"] diff --git a/api/server-server/joins-v1.yaml b/api/server-server/joins-v1.yaml index 39287264..18c88db0 100644 --- a/api/server-server/joins-v1.yaml +++ b/api/server-server/joins-v1.yaml @@ -262,47 +262,7 @@ paths: - type: integer description: The value ``200``. example: 200 - - type: object - title: Room State - description: The state for the room. - properties: - origin: - type: string - description: The resident server's DNS name. - auth_chain: - type: array - description: |- - The auth chain. Note that events have a different format depending on - the room version - check the `room version specification`_ for precise - event formats. - items: - type: object - title: PDU - description: |- - The `PDUs <#pdus>`_ that make up the auth chain. The event format varies depending - on the room version - check the `room version specification`_ for precise event formats. - schema: - type: object - properties: [] - example: - $ref: "examples/minimal_pdu.json" - state: - type: array - description: |- - The room state. The event format varies depending on the room version - - check the `room version specification`_ for precise event formats. - items: - type: object - title: PDU - description: |- - The `PDUs <#pdus>`_ for the fully resolved state of the room. The event format varies depending - on the room version - check the `room version specification`_ for precise event formats. - schema: - type: object - properties: [] - example: - $ref: "examples/minimal_pdu.json" - required: ["auth_chain", "state", "origin"] + - $ref: "./definitions/send_join_response.yaml" examples: application/json: [ 200, diff --git a/api/server-server/joins-v2.yaml b/api/server-server/joins-v2.yaml index a98d20fd..b5a1544c 100644 --- a/api/server-server/joins-v2.yaml +++ b/api/server-server/joins-v2.yaml @@ -127,47 +127,7 @@ paths: description: |- The full state for the room, having accepted the join event. schema: - type: object - title: Room State - description: The state for the room. - properties: - origin: - type: string - description: The resident server's DNS name. - auth_chain: - type: array - description: |- - The auth chain. Note that events have a different format depending on - the room version - check the `room version specification`_ for precise - event formats. - items: - type: object - title: PDU - description: |- - The `PDUs <#pdus>`_ that make up the auth chain. The event format varies depending - on the room version - check the `room version specification`_ for precise event formats. - schema: - type: object - properties: [] - example: - $ref: "examples/minimal_pdu.json" - state: - type: array - description: |- - The room state. The event format varies depending on the room version - - check the `room version specification`_ for precise event formats. - items: - type: object - title: PDU - description: |- - The `PDUs <#pdus>`_ for the fully resolved state of the room. The event format varies depending - on the room version - check the `room version specification`_ for precise event formats. - schema: - type: object - properties: [] - example: - $ref: "examples/minimal_pdu.json" - required: ["auth_chain", "state", "origin"] + $ref: "./definitions/send_join_response.yaml" examples: application/json: { "origin": "matrix.org", From 67d46fa881169ee6cea9d20028189d1869c5c83b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 10:59:41 -0600 Subject: [PATCH 133/162] Clarify the auth_chain and state for send_join Fixes https://github.com/matrix-org/matrix-doc/issues/2467 Fixes https://github.com/matrix-org/matrix-doc/issues/2464 This is based on looking at Synapse, not actually testing it. Within the send_join handling functions, an "event context" is acquired [here](https://github.com/matrix-org/synapse/blob/d14c4d6b6d02e8c55a6647ce0f0b3b44356d6173/synapse/handlers/federation.py#L1472) which then later uses the previous state (populated [here](https://github.com/matrix-org/synapse/blob/d14c4d6b6d02e8c55a6647ce0f0b3b44356d6173/synapse/state/__init__.py#L286-L289) in the event context) [here](https://github.com/matrix-org/synapse/blob/d14c4d6b6d02e8c55a6647ce0f0b3b44356d6173/synapse/handlers/federation.py#L1494-L1501) to return a response. Within the event context, the previous state IDs are populated [here](https://github.com/matrix-org/synapse/blob/d14c4d6b6d02e8c55a6647ce0f0b3b44356d6173/synapse/state/__init__.py#L351). --- .../definitions/send_join_response.yaml | 13 ++++++++----- .../server_server/newsfragments/2575.clarification | 1 + 2 files changed, 9 insertions(+), 5 deletions(-) create mode 100644 changelogs/server_server/newsfragments/2575.clarification diff --git a/api/server-server/definitions/send_join_response.yaml b/api/server-server/definitions/send_join_response.yaml index 96ceaefc..2837556b 100644 --- a/api/server-server/definitions/send_join_response.yaml +++ b/api/server-server/definitions/send_join_response.yaml @@ -22,9 +22,10 @@ properties: auth_chain: type: array description: |- - The auth chain. Note that events have a different format depending on - the room version - check the `room version specification`_ for precise - event formats. + The auth chain for the entire current room state prior to the join event. + + Note that events have a different format depending on the room version - check the + `room version specification`_ for precise event formats. items: type: object title: PDU @@ -39,8 +40,10 @@ properties: state: type: array description: |- - The room state. The event format varies depending on the room version - - check the `room version specification`_ for precise event formats. + The resolved current room state prior to the join event. + + The event format varies depending on the room version - check the `room version specification`_ + for precise event formats. items: type: object title: PDU diff --git a/changelogs/server_server/newsfragments/2575.clarification b/changelogs/server_server/newsfragments/2575.clarification new file mode 100644 index 00000000..5eb18394 --- /dev/null +++ b/changelogs/server_server/newsfragments/2575.clarification @@ -0,0 +1 @@ +Clarify the state/auth chain requirements for ``/send_join``. From e9998c53350442cc145dc7d05b7d77b18cd6abe7 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 12:21:27 -0600 Subject: [PATCH 134/162] Clarify that server names are at least 1 character Fixes https://github.com/matrix-org/matrix-doc/issues/2460 --- specification/appendices/identifier_grammar.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index 4cce1f90..95c22fb1 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -45,7 +45,7 @@ following grammar:: IPv6char = DIGIT / %x41-46 / %x61-66 / ":" / "." ; 0-9, A-F, a-f, :, . - dns-name = *255dns-char + dns-name = 1*255dns-char dns-char = DIGIT / ALPHA / "-" / "." From 669377888ee6cf831f9ce0de2d1fa1b1457a1f90 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 25 May 2020 12:24:29 -0600 Subject: [PATCH 135/162] Fix more typos throughout the specification Fixes https://github.com/matrix-org/matrix-doc/issues/2193 --- api/server-server/keys_server.yaml | 2 +- changelogs/server_server/newsfragments/2577.clarification | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 changelogs/server_server/newsfragments/2577.clarification diff --git a/api/server-server/keys_server.yaml b/api/server-server/keys_server.yaml index 465bb294..81b98b1e 100644 --- a/api/server-server/keys_server.yaml +++ b/api/server-server/keys_server.yaml @@ -34,7 +34,7 @@ paths: Intermediate notary servers should cache a response for half of its lifetime to avoid serving a stale response. Originating servers should avoid returning responses that expire in less than an hour to avoid - repeated reqests for a certificate that is about to expire. Requesting + repeated requests for a certificate that is about to expire. Requesting servers should limit how frequently they query for certificates to avoid flooding a server with requests. diff --git a/changelogs/server_server/newsfragments/2577.clarification b/changelogs/server_server/newsfragments/2577.clarification new file mode 100644 index 00000000..902a9c3f --- /dev/null +++ b/changelogs/server_server/newsfragments/2577.clarification @@ -0,0 +1 @@ +Fix various spelling errors throughout the specification. From d4c19a0e808e39dbcc92a7af1193cbb1daae803f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 26 May 2020 12:36:18 -0600 Subject: [PATCH 136/162] Federation Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- specification/rooms/v6.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/rooms/v6.rst b/specification/rooms/v6.rst index b1acccd2..e5378d0e 100644 --- a/specification/rooms/v6.rst +++ b/specification/rooms/v6.rst @@ -95,6 +95,6 @@ Canonical JSON Servers MUST strictly enforce the JSON format specified in the `appendices <../appendices.html#canonical-json>`_. This translates to a 400 ``M_BAD_JSON`` error -on most endpoints, or discarding of events over federation. For example, the federation API's +on most endpoints, or discarding of events over federation. For example, the Federation API's ``/send`` endpoint would discard the event whereas the Client Server API's ``/send/{eventType}`` endpoint would return a ``M_BAD_JSON`` error. From bea6b321a59fb06d255bb2d02862613ca54fe732 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 26 May 2020 12:37:52 -0600 Subject: [PATCH 137/162] Change the nots order Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- api/client-server/directory.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/client-server/directory.yaml b/api/client-server/directory.yaml index 67bdf236..7a08faf2 100644 --- a/api/client-server/directory.yaml +++ b/api/client-server/directory.yaml @@ -185,7 +185,7 @@ paths: membership. .. Note:: - Clients are not recommended to display this list of aliases prominently + Clients are recommended not to display this list of aliases prominently as they are not curated, unlike those listed in the ``m.room.canonical_alias`` state event. From a5cb9c655c083e38989215e6e405599def15f9f7 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 26 May 2020 12:38:40 -0600 Subject: [PATCH 138/162] It's important to say who Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- api/client-server/registration.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index a6d5d275..bab9139c 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -346,7 +346,7 @@ paths: logout_devices: type: boolean description: |- - Whether the other access tokens, and their associated devices, for the user should be + Whether the user's other access tokens, and their associated devices, should be revoked if the request succeeds. Defaults to true. When ``false``, the server can still take advantage of `the soft logout method <#soft-logout>`_ From 47c531adff5cea38dd4621a943257e6052a4997a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 26 May 2020 12:46:50 -0600 Subject: [PATCH 139/162] Clarify exactly which servers Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- specification/server_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index ca147665..3d591ca8 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -291,7 +291,7 @@ Step 1 sign JSON: } } -The server names in the JSON above are the server names involved. Delegation from +The server names in the JSON above are the server names for each homeserver involved. Delegation from the `server name resolution section <#resolving-server-names>`_ above do not affect these - the server names from before delegation would take place are used. This same condition applies throughout the request signing process. From 4b64d804374eea4d48185772a5599f9c0201dd8f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 26 May 2020 13:04:04 -0600 Subject: [PATCH 140/162] Remove v6 TODO To avoid the conflict, hopefully. --- specification/client_server_api.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index fda2c106..6060388b 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1820,8 +1820,6 @@ one of the following event types: - ``m.room.aliases`` allows key ``aliases``. - ``m.room.history_visibility`` allows key ``history_visibility``. -.. TODO: Room v6 changes how redaction works in MSC2432 (no special meaning for aliases) - The server should add the event causing the redaction to the ``unsigned`` property of the redacted event, under the ``redacted_because`` key. When a client receives a redaction event it should change the redacted event in the From 19a97c62e241347a3507d51c7bb617b6922aff5a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 27 May 2020 07:19:31 -0600 Subject: [PATCH 141/162] Client-Server r0.6.1 --- changelogs/client_server.rst | 37 +++++++++++++++++++ .../newsfragments/2245.clarification | 1 - .../newsfragments/2351.clarification | 1 - .../newsfragments/2369.clarification | 1 - .../newsfragments/2381.clarification | 1 - .../newsfragments/2415.clarification | 1 - .../newsfragments/2430.clarification | 1 - .../client_server/newsfragments/2434.feature | 1 - .../newsfragments/2453.clarification | 1 - .../newsfragments/2492.clarification | 1 - .../newsfragments/2519.clarification | 1 - .../client_server/newsfragments/2523.feature | 1 - .../newsfragments/2524.clarification | 1 - .../newsfragments/2525.clarification | 1 - .../client_server/newsfragments/2532.feature | 1 - .../client_server/newsfragments/2546.feature | 1 - .../newsfragments/2553.clarification | 1 - .../newsfragments/2554.clarification | 1 - .../client_server/newsfragments/2562.feature | 1 - .../client_server/newsfragments/2562.new | 1 - .../newsfragments/2563.clarification | 1 - .../newsfragments/2566.clarification | 1 - .../newsfragments/2569.clarification | 1 - .../newsfragments/2571.clarification | 1 - .../newsfragments/2574.clarification | 1 - specification/client_server_api.rst | 1 + 26 files changed, 38 insertions(+), 24 deletions(-) delete mode 100644 changelogs/client_server/newsfragments/2245.clarification delete mode 100644 changelogs/client_server/newsfragments/2351.clarification delete mode 100644 changelogs/client_server/newsfragments/2369.clarification delete mode 100644 changelogs/client_server/newsfragments/2381.clarification delete mode 100644 changelogs/client_server/newsfragments/2415.clarification delete mode 100644 changelogs/client_server/newsfragments/2430.clarification delete mode 100644 changelogs/client_server/newsfragments/2434.feature delete mode 100644 changelogs/client_server/newsfragments/2453.clarification delete mode 100644 changelogs/client_server/newsfragments/2492.clarification delete mode 100644 changelogs/client_server/newsfragments/2519.clarification delete mode 100644 changelogs/client_server/newsfragments/2523.feature delete mode 100644 changelogs/client_server/newsfragments/2524.clarification delete mode 100644 changelogs/client_server/newsfragments/2525.clarification delete mode 100644 changelogs/client_server/newsfragments/2532.feature delete mode 100644 changelogs/client_server/newsfragments/2546.feature delete mode 100644 changelogs/client_server/newsfragments/2553.clarification delete mode 100644 changelogs/client_server/newsfragments/2554.clarification delete mode 100644 changelogs/client_server/newsfragments/2562.feature delete mode 100644 changelogs/client_server/newsfragments/2562.new delete mode 100644 changelogs/client_server/newsfragments/2563.clarification delete mode 100644 changelogs/client_server/newsfragments/2566.clarification delete mode 100644 changelogs/client_server/newsfragments/2569.clarification delete mode 100644 changelogs/client_server/newsfragments/2571.clarification delete mode 100644 changelogs/client_server/newsfragments/2574.clarification diff --git a/changelogs/client_server.rst b/changelogs/client_server.rst index bc12da5c..6f8aee1c 100644 --- a/changelogs/client_server.rst +++ b/changelogs/client_server.rst @@ -1,3 +1,40 @@ +r0.6.1 +====== + +New Endpoints +------------- + +- Added ``/rooms/{roomId}/aliases`` for retrieving local aliases for a room. (`#2562 `_) + + +Backwards Compatible Changes +---------------------------- + +- Added data structures for defining moderation policies in rooms per `MSC2313 `_. (`#2434 `_) +- Optionally invalidate other access tokens during password modification per `MSC2457 `_. (`#2523 `_) +- Add User-Interactive Authentication for SSO-backed homeserver per `MSC2454 `_. (`#2532 `_) +- Add soft-logout support per `MSC1466 `_. (`#2546 `_) +- Replaced legacy room alias handling with a more sustainable solution per `MSC2432 `_. (`#2562 `_) + + +Spec Clarifications +------------------- + +- List available enum values for the room versions capability. (`#2245 `_) +- Fix various spelling errors throughout the specification. (`#2351 `_, `#2415 `_, `#2453 `_, `#2524 `_, `#2553 `_, `#2569 `_) +- Minor clarifications to token-based User-Interactive Authentication. (`#2369 `_) +- Minor clarification for what the user directory searches. (`#2381 `_) +- Fix key export format example to match the specification. (`#2430 `_) +- Clarify the IV data type for encrypted files. (`#2492 `_) +- Fix the ``.m.rule.contains_user_name`` default push rule to set the highlight tweak. (`#2519 `_) +- Clarify that an ``event_id`` is returned when sending events. (`#2525 `_) +- Fix some numbers in the specification to match their explanation text. (`#2554 `_) +- Move redaction algorithm into the room version specifications. (`#2563 `_) +- Clarify signature object structures for encryption. (`#2566 `_) +- Clarify which events are created as part of ``/createRoom``. (`#2571 `_) +- Remove claims that the homeserver is exclusively responsible for profile information in membership events. (`#2574 `_) + + r0.6.0 ====== diff --git a/changelogs/client_server/newsfragments/2245.clarification b/changelogs/client_server/newsfragments/2245.clarification deleted file mode 100644 index 31e5c2df..00000000 --- a/changelogs/client_server/newsfragments/2245.clarification +++ /dev/null @@ -1 +0,0 @@ -List available enum values for the room versions capability. diff --git a/changelogs/client_server/newsfragments/2351.clarification b/changelogs/client_server/newsfragments/2351.clarification deleted file mode 100644 index 902a9c3f..00000000 --- a/changelogs/client_server/newsfragments/2351.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix various spelling errors throughout the specification. diff --git a/changelogs/client_server/newsfragments/2369.clarification b/changelogs/client_server/newsfragments/2369.clarification deleted file mode 100644 index c7fd0fd9..00000000 --- a/changelogs/client_server/newsfragments/2369.clarification +++ /dev/null @@ -1 +0,0 @@ -Minor clarifications to token-based User-Interactive Authentication. diff --git a/changelogs/client_server/newsfragments/2381.clarification b/changelogs/client_server/newsfragments/2381.clarification deleted file mode 100644 index 70bd6d18..00000000 --- a/changelogs/client_server/newsfragments/2381.clarification +++ /dev/null @@ -1 +0,0 @@ -Minor clarification for what the user directory searches. diff --git a/changelogs/client_server/newsfragments/2415.clarification b/changelogs/client_server/newsfragments/2415.clarification deleted file mode 100644 index 902a9c3f..00000000 --- a/changelogs/client_server/newsfragments/2415.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix various spelling errors throughout the specification. diff --git a/changelogs/client_server/newsfragments/2430.clarification b/changelogs/client_server/newsfragments/2430.clarification deleted file mode 100644 index 56be0c1a..00000000 --- a/changelogs/client_server/newsfragments/2430.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix key export format example to match the specification. diff --git a/changelogs/client_server/newsfragments/2434.feature b/changelogs/client_server/newsfragments/2434.feature deleted file mode 100644 index 2b8f36f5..00000000 --- a/changelogs/client_server/newsfragments/2434.feature +++ /dev/null @@ -1 +0,0 @@ -Added data structures for defining moderation policies in rooms per `MSC2313 `_. diff --git a/changelogs/client_server/newsfragments/2453.clarification b/changelogs/client_server/newsfragments/2453.clarification deleted file mode 100644 index 902a9c3f..00000000 --- a/changelogs/client_server/newsfragments/2453.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix various spelling errors throughout the specification. diff --git a/changelogs/client_server/newsfragments/2492.clarification b/changelogs/client_server/newsfragments/2492.clarification deleted file mode 100644 index 88c478ec..00000000 --- a/changelogs/client_server/newsfragments/2492.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify the IV data type for encrypted files. diff --git a/changelogs/client_server/newsfragments/2519.clarification b/changelogs/client_server/newsfragments/2519.clarification deleted file mode 100644 index 4b32da1e..00000000 --- a/changelogs/client_server/newsfragments/2519.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix the ``.m.rule.contains_user_name`` default push rule to set the highlight tweak. diff --git a/changelogs/client_server/newsfragments/2523.feature b/changelogs/client_server/newsfragments/2523.feature deleted file mode 100644 index e45d1c2f..00000000 --- a/changelogs/client_server/newsfragments/2523.feature +++ /dev/null @@ -1 +0,0 @@ -Optionally invalidate other access tokens during password modification per `MSC2457 `_. diff --git a/changelogs/client_server/newsfragments/2524.clarification b/changelogs/client_server/newsfragments/2524.clarification deleted file mode 100644 index 902a9c3f..00000000 --- a/changelogs/client_server/newsfragments/2524.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix various spelling errors throughout the specification. diff --git a/changelogs/client_server/newsfragments/2525.clarification b/changelogs/client_server/newsfragments/2525.clarification deleted file mode 100644 index df1fc538..00000000 --- a/changelogs/client_server/newsfragments/2525.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify that an ``event_id`` is returned when sending events. diff --git a/changelogs/client_server/newsfragments/2532.feature b/changelogs/client_server/newsfragments/2532.feature deleted file mode 100644 index cf74a289..00000000 --- a/changelogs/client_server/newsfragments/2532.feature +++ /dev/null @@ -1 +0,0 @@ -Add User-Interactive Authentication for SSO-backed homeserver per `MSC2454 `_. diff --git a/changelogs/client_server/newsfragments/2546.feature b/changelogs/client_server/newsfragments/2546.feature deleted file mode 100644 index 40c066f4..00000000 --- a/changelogs/client_server/newsfragments/2546.feature +++ /dev/null @@ -1 +0,0 @@ -Add soft-logout support per `MSC1466 `_. diff --git a/changelogs/client_server/newsfragments/2553.clarification b/changelogs/client_server/newsfragments/2553.clarification deleted file mode 100644 index 902a9c3f..00000000 --- a/changelogs/client_server/newsfragments/2553.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix various spelling errors throughout the specification. diff --git a/changelogs/client_server/newsfragments/2554.clarification b/changelogs/client_server/newsfragments/2554.clarification deleted file mode 100644 index 826fafd8..00000000 --- a/changelogs/client_server/newsfragments/2554.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix some numbers in the specification to match their explanation text. diff --git a/changelogs/client_server/newsfragments/2562.feature b/changelogs/client_server/newsfragments/2562.feature deleted file mode 100644 index 556018da..00000000 --- a/changelogs/client_server/newsfragments/2562.feature +++ /dev/null @@ -1 +0,0 @@ -Replaced legacy room alias handling with a more sustainable solution per `MSC2432 `_. diff --git a/changelogs/client_server/newsfragments/2562.new b/changelogs/client_server/newsfragments/2562.new deleted file mode 100644 index c7b3e4ca..00000000 --- a/changelogs/client_server/newsfragments/2562.new +++ /dev/null @@ -1 +0,0 @@ -Added ``/rooms/{roomId}/aliases`` for retrieving local aliases for a room. diff --git a/changelogs/client_server/newsfragments/2563.clarification b/changelogs/client_server/newsfragments/2563.clarification deleted file mode 100644 index a8805f4a..00000000 --- a/changelogs/client_server/newsfragments/2563.clarification +++ /dev/null @@ -1 +0,0 @@ -Move redaction algorithm into the room version specifications. diff --git a/changelogs/client_server/newsfragments/2566.clarification b/changelogs/client_server/newsfragments/2566.clarification deleted file mode 100644 index 15e29b2f..00000000 --- a/changelogs/client_server/newsfragments/2566.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify signature object structures for encryption. diff --git a/changelogs/client_server/newsfragments/2569.clarification b/changelogs/client_server/newsfragments/2569.clarification deleted file mode 100644 index 902a9c3f..00000000 --- a/changelogs/client_server/newsfragments/2569.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix various spelling errors throughout the specification. diff --git a/changelogs/client_server/newsfragments/2571.clarification b/changelogs/client_server/newsfragments/2571.clarification deleted file mode 100644 index e9437f3c..00000000 --- a/changelogs/client_server/newsfragments/2571.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify which events are created as part of ``/createRoom``. diff --git a/changelogs/client_server/newsfragments/2574.clarification b/changelogs/client_server/newsfragments/2574.clarification deleted file mode 100644 index d771d374..00000000 --- a/changelogs/client_server/newsfragments/2574.clarification +++ /dev/null @@ -1 +0,0 @@ -Remove claims that the homeserver is exclusively responsible for profile information in membership events. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index ad8539cb..574999be 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -45,6 +45,7 @@ Other versions of this specification The following other versions are also available, in reverse chronological order: - `HEAD `_: Includes all changes since the latest versioned release. +- `r0.6.1 `_ - `r0.6.0 `_ - `r0.5.0 `_ - `r0.4.0 `_ From 09986af8fc2ccda4a4bbbfec3c0651663da2f84d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 27 May 2020 07:21:59 -0600 Subject: [PATCH 142/162] Server-Server r0.1.4 Includes a link to r0.1.3 because we forgot --- changelogs/server_server.rst | 31 +++++++++++++++++++ .../server_server/newsfragments/2035.new | 1 - .../newsfragments/2392.clarification | 1 - .../server_server/newsfragments/2470.removal | 1 - .../newsfragments/2510.clarification | 1 - .../newsfragments/2527.clarification | 1 - .../newsfragments/2538.clarification | 1 - .../server_server/newsfragments/2547.new | 1 - .../newsfragments/2559.clarification | 1 - .../newsfragments/2560.clarification | 1 - .../newsfragments/2566.clarification | 1 - .../newsfragments/2570.clarification | 1 - .../newsfragments/2575.clarification | 1 - .../newsfragments/2577.clarification | 1 - specification/server_server_api.rst | 2 ++ 15 files changed, 33 insertions(+), 13 deletions(-) delete mode 100644 changelogs/server_server/newsfragments/2035.new delete mode 100644 changelogs/server_server/newsfragments/2392.clarification delete mode 100644 changelogs/server_server/newsfragments/2470.removal delete mode 100644 changelogs/server_server/newsfragments/2510.clarification delete mode 100644 changelogs/server_server/newsfragments/2527.clarification delete mode 100644 changelogs/server_server/newsfragments/2538.clarification delete mode 100644 changelogs/server_server/newsfragments/2547.new delete mode 100644 changelogs/server_server/newsfragments/2559.clarification delete mode 100644 changelogs/server_server/newsfragments/2560.clarification delete mode 100644 changelogs/server_server/newsfragments/2566.clarification delete mode 100644 changelogs/server_server/newsfragments/2570.clarification delete mode 100644 changelogs/server_server/newsfragments/2575.clarification delete mode 100644 changelogs/server_server/newsfragments/2577.clarification diff --git a/changelogs/server_server.rst b/changelogs/server_server.rst index 22da543c..ee4a3c1c 100644 --- a/changelogs/server_server.rst +++ b/changelogs/server_server.rst @@ -1,3 +1,34 @@ +r0.1.4 +====== + +New Endpoints +------------- + +- Add new ``POST /publicRooms`` endpoint for filtering the room directory. (`#2035 `_) +- Add new v2 ``/send_join`` and ``/send_leave`` endpoints per `MSC1802 `_. (`#2547 `_) + + +Removed Endpoints +----------------- + +- Remove the unused ``query_auth`` API per `MSC2451 `_. (`#2470 `_) + + +Spec Clarifications +------------------- + +- Move auth event selection to a more obvious location. (`#2392 `_) +- Fix typo in Request Authentication python example. (`#2510 `_) +- Clarify which fields are required on the key server endpoints. (`#2527 `_) +- Clarify the limits of ``prev_events`` and ``auth_events`` for PDUs. (`#2538 `_) +- Clarify which events are targeted by backfill. (`#2559 `_) +- Fix the response format of the ``/send`` endpoint. (`#2560 `_) +- Clarify signature object structures for encryption. (`#2566 `_) +- Clarify the server names to use when signing requests. (`#2570 `_) +- Clarify the state/auth chain requirements for ``/send_join``. (`#2575 `_) +- Fix various spelling errors throughout the specification. (`#2577 `_) + + r0.1.3 ====== diff --git a/changelogs/server_server/newsfragments/2035.new b/changelogs/server_server/newsfragments/2035.new deleted file mode 100644 index 6794c4ea..00000000 --- a/changelogs/server_server/newsfragments/2035.new +++ /dev/null @@ -1 +0,0 @@ -Add new ``POST /publicRooms`` endpoint for filtering the room directory. diff --git a/changelogs/server_server/newsfragments/2392.clarification b/changelogs/server_server/newsfragments/2392.clarification deleted file mode 100644 index b09420e4..00000000 --- a/changelogs/server_server/newsfragments/2392.clarification +++ /dev/null @@ -1 +0,0 @@ -Move auth event selection to a more obvious location. diff --git a/changelogs/server_server/newsfragments/2470.removal b/changelogs/server_server/newsfragments/2470.removal deleted file mode 100644 index 51cdadd4..00000000 --- a/changelogs/server_server/newsfragments/2470.removal +++ /dev/null @@ -1 +0,0 @@ -Remove the unused ``query_auth`` API per `MSC2451 `_. diff --git a/changelogs/server_server/newsfragments/2510.clarification b/changelogs/server_server/newsfragments/2510.clarification deleted file mode 100644 index 9c96c5a5..00000000 --- a/changelogs/server_server/newsfragments/2510.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix typo in Request Authentication python example. diff --git a/changelogs/server_server/newsfragments/2527.clarification b/changelogs/server_server/newsfragments/2527.clarification deleted file mode 100644 index 329d5da2..00000000 --- a/changelogs/server_server/newsfragments/2527.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify which fields are required on the key server endpoints. diff --git a/changelogs/server_server/newsfragments/2538.clarification b/changelogs/server_server/newsfragments/2538.clarification deleted file mode 100644 index 4b709a8d..00000000 --- a/changelogs/server_server/newsfragments/2538.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify the limits of ``prev_events`` and ``auth_events`` for PDUs. diff --git a/changelogs/server_server/newsfragments/2547.new b/changelogs/server_server/newsfragments/2547.new deleted file mode 100644 index c558fb5b..00000000 --- a/changelogs/server_server/newsfragments/2547.new +++ /dev/null @@ -1 +0,0 @@ -Add new v2 ``/send_join`` and ``/send_leave`` endpoints per `MSC1802 `_. diff --git a/changelogs/server_server/newsfragments/2559.clarification b/changelogs/server_server/newsfragments/2559.clarification deleted file mode 100644 index cdb9cfe0..00000000 --- a/changelogs/server_server/newsfragments/2559.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify which events are targeted by backfill. diff --git a/changelogs/server_server/newsfragments/2560.clarification b/changelogs/server_server/newsfragments/2560.clarification deleted file mode 100644 index c60bc6f3..00000000 --- a/changelogs/server_server/newsfragments/2560.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix the response format of the ``/send`` endpoint. diff --git a/changelogs/server_server/newsfragments/2566.clarification b/changelogs/server_server/newsfragments/2566.clarification deleted file mode 100644 index 15e29b2f..00000000 --- a/changelogs/server_server/newsfragments/2566.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify signature object structures for encryption. diff --git a/changelogs/server_server/newsfragments/2570.clarification b/changelogs/server_server/newsfragments/2570.clarification deleted file mode 100644 index 3fac40bf..00000000 --- a/changelogs/server_server/newsfragments/2570.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify the server names to use when signing requests. diff --git a/changelogs/server_server/newsfragments/2575.clarification b/changelogs/server_server/newsfragments/2575.clarification deleted file mode 100644 index 5eb18394..00000000 --- a/changelogs/server_server/newsfragments/2575.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify the state/auth chain requirements for ``/send_join``. diff --git a/changelogs/server_server/newsfragments/2577.clarification b/changelogs/server_server/newsfragments/2577.clarification deleted file mode 100644 index 902a9c3f..00000000 --- a/changelogs/server_server/newsfragments/2577.clarification +++ /dev/null @@ -1 +0,0 @@ -Fix various spelling errors throughout the specification. diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 61d4375d..9deb267c 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -80,6 +80,8 @@ Other versions of this specification The following other versions are also available, in reverse chronological order: - `HEAD `_: Includes all changes since the latest versioned release. +- `r0.1.4 `_ +- `r0.1.3 `_ - `r0.1.2 `_ - `r0.1.1 `_ - `r0.1.0 `_ From 3fc5540755b21ff36499e74137cd31f1b9413e63 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 29 May 2020 07:00:38 -0600 Subject: [PATCH 143/162] Fix r0.1.4 changelog reference --- changelogs/server_server.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/server_server.rst b/changelogs/server_server.rst index ee4a3c1c..367fb092 100644 --- a/changelogs/server_server.rst +++ b/changelogs/server_server.rst @@ -4,7 +4,7 @@ r0.1.4 New Endpoints ------------- -- Add new ``POST /publicRooms`` endpoint for filtering the room directory. (`#2035 `_) +- Add new ``POST /publicRooms`` endpoint for filtering the room directory. (`#2305 `_) - Add new v2 ``/send_join`` and ``/send_leave`` endpoints per `MSC1802 `_. (`#2547 `_) From ffea5d5e610f429b558e07df3f3d00f1b74a9aeb Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 31 May 2020 21:36:47 -0600 Subject: [PATCH 144/162] Document top-level UnsignedData objects --- scripts/templating/matrix_templates/units.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 5a7ff94e..ff614e34 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -910,6 +910,13 @@ class MatrixUnits(Units): Units.prop(json_schema, "properties/content") ) + # Include UnsignedData if it is present on the object + unsigned = Units.prop(json_schema, "properties/unsigned") + if unsigned: + tbls = get_tables_for_schema(unsigned) + for tbl in tbls: + schema["content_fields"].append(tbl) + # grab msgtype if it is the right kind of event msgtype = Units.prop( json_schema, "properties/content/properties/msgtype/enum" From 1c1e3517782484df5947a4686974f5a5782883c9 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 31 May 2020 21:37:18 -0600 Subject: [PATCH 145/162] Move m.room.member's unsigned data to the top level --- event-schemas/schema/m.room.member | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/event-schemas/schema/m.room.member b/event-schemas/schema/m.room.member index 1ccf5ab3..7fe23f40 100644 --- a/event-schemas/schema/m.room.member +++ b/event-schemas/schema/m.room.member @@ -100,16 +100,6 @@ properties: - signed title: Invite type: object - unsigned: - type: object - title: UnsignedData - description: Contains optional extra information about the event. - properties: - invite_room_state: - description: 'A subset of the state of the room at the time of the invite, if ``membership`` is ``invite``. Note that this state is informational, and SHOULD NOT be trusted; once the client has joined the room, it SHOULD fetch the live state from the server and discard the invite_room_state. Also, clients must not rely on any particular state being present here; they SHOULD behave properly (with possibly a degraded but not a broken experience) in the absence of any particular events here. If they are set on the room, at least the state for ``m.room.avatar``, ``m.room.canonical_alias``, ``m.room.join_rules``, and ``m.room.name`` SHOULD be included.' - items: - $ref: "stripped_state.yaml" - type: array required: - membership title: EventContent @@ -124,5 +114,15 @@ properties: enum: - m.room.member type: string + unsigned: + type: object + title: UnsignedData + description: Contains optional extra information about the event. + properties: + invite_room_state: + description: 'A subset of the state of the room at the time of the invite, if ``membership`` is ``invite``. Note that this state is informational, and SHOULD NOT be trusted; once the client has joined the room, it SHOULD fetch the live state from the server and discard the invite_room_state. Also, clients must not rely on any particular state being present here; they SHOULD behave properly (with possibly a degraded but not a broken experience) in the absence of any particular events here. If they are set on the room, at least the state for ``m.room.avatar``, ``m.room.canonical_alias``, ``m.room.join_rules``, and ``m.room.name`` SHOULD be included.' + items: + $ref: "stripped_state.yaml" + type: array title: The current membership state of a user in the room. type: object From 33d738e6650037824e52ace02407dd120c0ef803 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 31 May 2020 21:41:29 -0600 Subject: [PATCH 146/162] Document the m.room.member unsigned object in full --- .../core-event-schema/sync_room_event.yaml | 19 +---------- .../core-event-schema/unsigned_prop.yaml | 32 +++++++++++++++++++ event-schemas/schema/m.room.member | 18 +++++------ 3 files changed, 42 insertions(+), 27 deletions(-) create mode 100644 event-schemas/schema/core-event-schema/unsigned_prop.yaml diff --git a/event-schemas/schema/core-event-schema/sync_room_event.yaml b/event-schemas/schema/core-event-schema/sync_room_event.yaml index 8006148f..fef13ad0 100644 --- a/event-schemas/schema/core-event-schema/sync_room_event.yaml +++ b/event-schemas/schema/core-event-schema/sync_room_event.yaml @@ -34,24 +34,7 @@ properties: type: integer format: int64 unsigned: - description: Contains optional extra information about the event. - properties: - age: - description: The time in milliseconds that has elapsed since the event was - sent. This field is generated by the local homeserver, and may be incorrect - if the local time on at least one of the two servers is out of sync, which can - cause the age to either be negative or greater than it actually is. - type: integer - redacted_because: - description: Optional. The event that redacted this event, if any. - title: Event - type: object - transaction_id: - description: The client-supplied transaction ID, if the client being given - the event is the same one which sent it. - type: string - title: UnsignedData - type: object + $ref: unsigned_prop.yaml required: - event_id - sender diff --git a/event-schemas/schema/core-event-schema/unsigned_prop.yaml b/event-schemas/schema/core-event-schema/unsigned_prop.yaml new file mode 100644 index 00000000..39cfb267 --- /dev/null +++ b/event-schemas/schema/core-event-schema/unsigned_prop.yaml @@ -0,0 +1,32 @@ +# Copyright 2020 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +title: UnsignedData +type: object +description: Contains optional extra information about the event. +properties: + age: + description: The time in milliseconds that has elapsed since the event was + sent. This field is generated by the local homeserver, and may be incorrect + if the local time on at least one of the two servers is out of sync, which can + cause the age to either be negative or greater than it actually is. + type: integer + redacted_because: + description: Optional. The event that redacted this event, if any. + title: Event + type: object + transaction_id: + description: The client-supplied transaction ID, if the client being given + the event is the same one which sent it. + type: string diff --git a/event-schemas/schema/m.room.member b/event-schemas/schema/m.room.member index 7fe23f40..36490c6b 100644 --- a/event-schemas/schema/m.room.member +++ b/event-schemas/schema/m.room.member @@ -115,14 +115,14 @@ properties: - m.room.member type: string unsigned: - type: object - title: UnsignedData - description: Contains optional extra information about the event. - properties: - invite_room_state: - description: 'A subset of the state of the room at the time of the invite, if ``membership`` is ``invite``. Note that this state is informational, and SHOULD NOT be trusted; once the client has joined the room, it SHOULD fetch the live state from the server and discard the invite_room_state. Also, clients must not rely on any particular state being present here; they SHOULD behave properly (with possibly a degraded but not a broken experience) in the absence of any particular events here. If they are set on the room, at least the state for ``m.room.avatar``, ``m.room.canonical_alias``, ``m.room.join_rules``, and ``m.room.name`` SHOULD be included.' - items: - $ref: "stripped_state.yaml" - type: array + allOf: + - $ref: "core-event-schema/unsigned_prop.yaml" + - type: object + properties: + invite_room_state: + description: 'A subset of the state of the room at the time of the invite, if ``membership`` is ``invite``. Note that this state is informational, and SHOULD NOT be trusted; once the client has joined the room, it SHOULD fetch the live state from the server and discard the invite_room_state. Also, clients must not rely on any particular state being present here; they SHOULD behave properly (with possibly a degraded but not a broken experience) in the absence of any particular events here. If they are set on the room, at least the state for ``m.room.avatar``, ``m.room.canonical_alias``, ``m.room.join_rules``, and ``m.room.name`` SHOULD be included.' + items: + $ref: "stripped_state.yaml" + type: array title: The current membership state of a user in the room. type: object From 0f9ffdf01ade1e680c503525cf0c0dbc04505f55 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 31 May 2020 21:43:27 -0600 Subject: [PATCH 147/162] Make server ACL example inherit from state events --- event-schemas/examples/m.room.server_acl | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/event-schemas/examples/m.room.server_acl b/event-schemas/examples/m.room.server_acl index 86da2093..06586f79 100644 --- a/event-schemas/examples/m.room.server_acl +++ b/event-schemas/examples/m.room.server_acl @@ -1,14 +1,10 @@ { - "age": 242352, + "$ref": "core/state_event.json", + "type": "m.room.server_acl", + "state_key": "", "content": { "allow_ip_literals": false, "allow": ["*"], "deny": ["*.evil.com", "evil.com"] - }, - "state_key": "", - "origin_server_ts": 1431961217939, - "event_id": "$WLGTSEFSEF:localhost", - "type": "m.room.server_acl", - "room_id": "!Cuyf34gef24t:localhost", - "sender": "@example:localhost" + } } From 33309fb68ece8264245ff2ab40bfd171b5610de0 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 31 May 2020 21:46:15 -0600 Subject: [PATCH 148/162] Add changelog --- changelogs/client_server/newsfragments/2591.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2591.clarification diff --git a/changelogs/client_server/newsfragments/2591.clarification b/changelogs/client_server/newsfragments/2591.clarification new file mode 100644 index 00000000..7473f453 --- /dev/null +++ b/changelogs/client_server/newsfragments/2591.clarification @@ -0,0 +1 @@ +Fix issues with ``age`` and ``unsigned`` being shown in the wrong places. From bda05a0d4450e4465e7e10f8e2c599f2e03ad24f Mon Sep 17 00:00:00 2001 From: Kitsune Ral Date: Mon, 1 Jun 2020 09:50:18 +0200 Subject: [PATCH 149/162] capabilities.yaml: drop an extraneous title AvailableRoomVersions sticks itself as a property type, preempting the mention of RoomVersionStability in the generated text. --- api/client-server/capabilities.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/api/client-server/capabilities.yaml b/api/client-server/capabilities.yaml index 0d6b0677..a50908f7 100644 --- a/api/client-server/capabilities.yaml +++ b/api/client-server/capabilities.yaml @@ -96,7 +96,6 @@ paths: example: "1" available: type: object - title: AvailableRoomVersions description: |- A detailed description of the room versions the server supports. additionalProperties: From c02ca21a94e93570dcfad0b50d25c6681a1013b3 Mon Sep 17 00:00:00 2001 From: Dominic Fischer Date: Mon, 1 Jun 2020 17:32:10 +0100 Subject: [PATCH 150/162] Fix Typo --- specification/modules/end_to_end_encryption.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 8bc46de8..a357944b 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -587,7 +587,7 @@ The process between Alice and Bob verifying each other would be: the same for Bob's device keys and key IDs concurrently with Alice. #. When the other device receives the ``m.key.verification.mac`` message, the device calculates the HMAC of its copies of the other device's keys given in the message, - as well as the HMAC of the comma-seperated, sorted, list of key IDs in the message. + as well as the HMAC of the comma-separated, sorted, list of key IDs in the message. The device compares these with the HMAC values given in the message, and if everything matches then the device keys are verified. From 9535540161c2d8c2caf91603716914faec2b8d87 Mon Sep 17 00:00:00 2001 From: Dominic Fischer Date: Mon, 1 Jun 2020 17:35:36 +0100 Subject: [PATCH 151/162] Add changelog --- changelogs/client_server/newsfragments/2594.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2594.clarification diff --git a/changelogs/client_server/newsfragments/2594.clarification b/changelogs/client_server/newsfragments/2594.clarification new file mode 100644 index 00000000..3ccb2333 --- /dev/null +++ b/changelogs/client_server/newsfragments/2594.clarification @@ -0,0 +1 @@ +Fix various typos throughout the specification. From 40c50c80ea4a8c26dfd20d766c995e11ee7837e1 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 1 Jun 2020 22:34:45 -0400 Subject: [PATCH 152/162] Apply suggestions from code review Co-authored-by: Travis Ralston --- api/client-server/key_backup.yaml | 2 +- specification/modules/end_to_end_encryption.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index 1a79db31..ffdafcf4 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -1,4 +1,4 @@ -# Copyright 2019 The Matrix.org Foundation C.I.C. +# Copyright 2019-2020 The Matrix.org Foundation C.I.C. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 8f26477a..79228596 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -785,7 +785,7 @@ To create a backup, a client will call `POST encrypted through the backup's ``auth_data``; other clients can discover the backup by calling `GET /_matrix/client/r0/room_keys/version/{version}`_, setting ``{version}`` to the empty string. Keys are encrypted according to the -backups ``auth_data`` and added to the backup by calling `PUT +backup's ``auth_data`` and added to the backup by calling `PUT /_matrix/client/r0/room_keys/keys`_ or one of its variants, and can be retrieved by calling `GET /_matrix/client/r0/room_keys/keys`_ or one of its variants. Keys can only be written to the most recently created version of the From 8532cfd706a199d44bdc7358b8c88c2c7d643aef Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 1 Jun 2020 20:35:22 -0600 Subject: [PATCH 153/162] Clarify which transaction ID --- event-schemas/schema/core-event-schema/unsigned_prop.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/event-schemas/schema/core-event-schema/unsigned_prop.yaml b/event-schemas/schema/core-event-schema/unsigned_prop.yaml index 39cfb267..9571c33b 100644 --- a/event-schemas/schema/core-event-schema/unsigned_prop.yaml +++ b/event-schemas/schema/core-event-schema/unsigned_prop.yaml @@ -27,6 +27,8 @@ properties: title: Event type: object transaction_id: - description: The client-supplied transaction ID, if the client being given - the event is the same one which sent it. + description: | + The client-supplied transaction ID, for example, provided via + ``PUT /_matrix/client/r0/rooms/{roomId}/send/{eventType}/{txnId}``, + if the client being given the event is the same one which sent it. type: string From d9a1de1d30e431ee96f98b92907410e0edaf9405 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 2 Jun 2020 08:13:07 -0600 Subject: [PATCH 154/162] Fix typos and clarify the user ID in login sections --- api/client-server/login.yaml | 2 +- api/client-server/logout.yaml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/api/client-server/login.yaml b/api/client-server/login.yaml index f6c92638..f122f209 100644 --- a/api/client-server/login.yaml +++ b/api/client-server/login.yaml @@ -154,7 +154,7 @@ paths: properties: user_id: type: string - description: The fully-qualified Matrix ID that has been registered. + description: The fully-qualified Matrix ID for the account. access_token: type: string description: |- diff --git a/api/client-server/logout.yaml b/api/client-server/logout.yaml index 75a3ec87..eb84a739 100644 --- a/api/client-server/logout.yaml +++ b/api/client-server/logout.yaml @@ -39,7 +39,7 @@ paths: - accessToken: [] responses: 200: - description: The access token used in the request was succesfully invalidated. + description: The access token used in the request was successfully invalidated. schema: type: object properties: {} @@ -64,7 +64,7 @@ paths: - accessToken: [] responses: 200: - description: The user's access tokens were succesfully invalidated. + description: The user's access tokens were successfully invalidated. schema: type: object properties: {} From bcce6b9a7c51ac7e5ad0bcf4178528e4158d7f1b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 2 Jun 2020 08:21:20 -0600 Subject: [PATCH 155/162] changelog --- changelogs/client_server/newsfragments/2599.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2599.clarification diff --git a/changelogs/client_server/newsfragments/2599.clarification b/changelogs/client_server/newsfragments/2599.clarification new file mode 100644 index 00000000..3ccb2333 --- /dev/null +++ b/changelogs/client_server/newsfragments/2599.clarification @@ -0,0 +1 @@ +Fix various typos throughout the specification. From a36284810d7243061099402bc1fd3fa4ca7c9170 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 2 Jun 2020 16:25:31 -0400 Subject: [PATCH 156/162] more clarifications and fixes --- api/client-server/key_backup.yaml | 56 +++++++++++++++++++ .../modules/end_to_end_encryption.rst | 30 +++++----- 2 files changed, 72 insertions(+), 14 deletions(-) diff --git a/api/client-server/key_backup.yaml b/api/client-server/key_backup.yaml index ffdafcf4..34dce118 100644 --- a/api/client-server/key_backup.yaml +++ b/api/client-server/key_backup.yaml @@ -77,6 +77,10 @@ paths: example: "1" required: - version + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" tags: - End-to-end encryption get: @@ -144,6 +148,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" tags: - End-to-end encryption "/room_keys/version/{version}": @@ -222,6 +230,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" tags: - End-to-end encryption put: @@ -305,6 +317,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" tags: - End-to-end encryption delete: @@ -345,6 +361,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" tags: - End-to-end encryption "/room_keys/keys/{roomId}/{sessionId}": @@ -412,6 +432,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" tags: - End-to-end encryption get: @@ -455,6 +479,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" delete: summary: Delete a key from the backup description: |- @@ -511,6 +539,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" "/room_keys/keys/{roomId}": put: summary: Store several keys in the backup for a given room. @@ -595,6 +627,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" tags: - End-to-end encryption get: @@ -650,6 +686,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" delete: summary: Delete a key from the backup description: |- @@ -700,6 +740,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" "/room_keys/keys": put: summary: Store several keys in the backup. @@ -784,6 +828,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" tags: - End-to-end encryption get: @@ -838,6 +886,10 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" delete: summary: Delete a key from the backup description: |- @@ -882,3 +934,7 @@ paths: } schema: "$ref": "definitions/errors/error.yaml" + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 79228596..eec40159 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -783,13 +783,12 @@ decrypt the session keys. To create a backup, a client will call `POST /_matrix/client/r0/room_keys/version`_ and define how the keys are to be encrypted through the backup's ``auth_data``; other clients can discover the -backup by calling `GET /_matrix/client/r0/room_keys/version/{version}`_, -setting ``{version}`` to the empty string. Keys are encrypted according to the -backup's ``auth_data`` and added to the backup by calling `PUT -/_matrix/client/r0/room_keys/keys`_ or one of its variants, and can be -retrieved by calling `GET /_matrix/client/r0/room_keys/keys`_ or one of its -variants. Keys can only be written to the most recently created version of the -backup. Backups can also be deleted using `DELETE +backup by calling `GET /_matrix/client/r0/room_keys/version`_. Keys are +encrypted according to the backup's ``auth_data`` and added to the backup by +calling `PUT /_matrix/client/r0/room_keys/keys`_ or one of its variants, and +can be retrieved by calling `GET /_matrix/client/r0/room_keys/keys`_ or one of +its variants. Keys can only be written to the most recently created version of +the backup. Backups can also be deleted using `DELETE /_matrix/client/r0/room_keys/version/{version}`_, or individual keys can be deleted using `DELETE /_matrix/client/r0/room_keys/keys`_ or one of its variants. @@ -823,7 +822,10 @@ follows: together to form a parity byte. This parity byte is appended to the byte string. 3. The byte string is encoded using base58, using the same `mapping as is used - for Bitcoin addresses `_. + for Bitcoin addresses + `_, + that is, using the alphabet + ``123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz``. 4. A space should be added after every 4th character. When reading in a recovery key, clients must disregard whitespace, and perform @@ -844,7 +846,7 @@ following format: ========== =========== ====================================================== Parameter Type Description ========== =========== ====================================================== - public_key string Required. The curve25519 public key used to encrypt + public_key string **Required.** The curve25519 public key used to encrypt the backups, encoded in unpadded base64. signatures Signatures Optional. Signatures of the ``auth_data``, as Signed JSON @@ -860,19 +862,19 @@ The ``session_data`` field in the backups is constructed as follows: =============================== ======== ========================================= Parameter Type Description =============================== ======== ========================================= - algorithm string Required. The end-to-end message + algorithm string **Required.** The end-to-end message encryption algorithm that the key is for. Must be ``m.megolm.v1.aes-sha2``. - forwarding_curve25519_key_chain [string] Required. Chain of Curve25519 keys + forwarding_curve25519_key_chain [string] **Required.** Chain of Curve25519 keys through which this session was forwarded, via `m.forwarded_room_key`_ events. - sender_key string Required. Unpadded base64-encoded + sender_key string **Required.** Unpadded base64-encoded device curve25519 key. - sender_claimed_keys {string: Required. A map from algorithm name + sender_claimed_keys {string: **Required.** A map from algorithm name string} (``ed25519``) to the identity key for the sending device. - session_key string Required. Unpadded base64-encoded + session_key string **Required.** Unpadded base64-encoded session key in `session-sharing format `_. =============================== ======== ========================================= From 7fc811d4bdf276d54c3571416628616466ae10ca Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 2 Jun 2020 15:10:18 -0600 Subject: [PATCH 157/162] Update event-schemas/schema/core-event-schema/unsigned_prop.yaml Co-authored-by: Hubert Chathi --- event-schemas/schema/core-event-schema/unsigned_prop.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/event-schemas/schema/core-event-schema/unsigned_prop.yaml b/event-schemas/schema/core-event-schema/unsigned_prop.yaml index 9571c33b..36c37230 100644 --- a/event-schemas/schema/core-event-schema/unsigned_prop.yaml +++ b/event-schemas/schema/core-event-schema/unsigned_prop.yaml @@ -23,7 +23,7 @@ properties: cause the age to either be negative or greater than it actually is. type: integer redacted_because: - description: Optional. The event that redacted this event, if any. + description: The event that redacted this event, if any. title: Event type: object transaction_id: From ea02029f89765e64e0032bbe29112dfc41c2ec4e Mon Sep 17 00:00:00 2001 From: Kitsune Ral Date: Thu, 4 Jun 2020 11:05:36 +0200 Subject: [PATCH 158/162] Add a changelog --- changelogs/client_server/newsfragments/2592.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2592.clarification diff --git a/changelogs/client_server/newsfragments/2592.clarification b/changelogs/client_server/newsfragments/2592.clarification new file mode 100644 index 00000000..fb770404 --- /dev/null +++ b/changelogs/client_server/newsfragments/2592.clarification @@ -0,0 +1 @@ +c2s: fix disconnected data structure in the documentation on server capabilities. From f632f4a20f524b6ca697164ab9acbd832d07d68f Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Thu, 4 Jun 2020 17:36:40 +0100 Subject: [PATCH 159/162] Clarify link to OpenID Connect specification (#2605) OpenID Connect and OpenID are different things. This seems to be talking about the former. --- api/client-server/openid.yaml | 2 +- changelogs/client_server/newsfragments/2605.clarification | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/2605.clarification diff --git a/api/client-server/openid.yaml b/api/client-server/openid.yaml index 01fdf68f..98a2449e 100644 --- a/api/client-server/openid.yaml +++ b/api/client-server/openid.yaml @@ -62,7 +62,7 @@ paths: 200: description: |- OpenID token information. This response is nearly compatible with the - response documented in the `OpenID 1.0 Specification `_ + response documented in the `OpenID Connect 1.0 Specification `_ with the only difference being the lack of an ``id_token``. Instead, the Matrix homeserver's name is provided. examples: diff --git a/changelogs/client_server/newsfragments/2605.clarification b/changelogs/client_server/newsfragments/2605.clarification new file mode 100644 index 00000000..e3d1fe98 --- /dev/null +++ b/changelogs/client_server/newsfragments/2605.clarification @@ -0,0 +1 @@ +Clarify link to OpenID Connect specification. From 97809c5eae6aa3d22ab4e1c56462d865fe1a0671 Mon Sep 17 00:00:00 2001 From: Kitsune Ral Date: Fri, 5 Jun 2020 12:38:10 +0200 Subject: [PATCH 160/162] Update changelogs/client_server/newsfragments/2592.clarification Co-authored-by: Travis Ralston --- changelogs/client_server/newsfragments/2592.clarification | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/client_server/newsfragments/2592.clarification b/changelogs/client_server/newsfragments/2592.clarification index fb770404..7414e00c 100644 --- a/changelogs/client_server/newsfragments/2592.clarification +++ b/changelogs/client_server/newsfragments/2592.clarification @@ -1 +1 @@ -c2s: fix disconnected data structure in the documentation on server capabilities. +Fix definitions for room version capabilities. From da740bfbca85920d7ddf93d319f3780f02cc9044 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 5 Jun 2020 17:32:13 +0100 Subject: [PATCH 161/162] Clarifications to SSO login/UIA (#2608) including a bunch of text about security --- CONTRIBUTING.rst | 2 +- .../newsfragments/2608.clarification | 1 + specification/client_server_api.rst | 32 +- specification/modules/sso_login.rst | 328 +++++++++++++++--- 4 files changed, 291 insertions(+), 72 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2608.clarification diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index fc5b146f..2d26e8a8 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -13,7 +13,7 @@ https://github.com/matrix-org/matrix-doc/blob/master/meta/documentation_style.rs Python code within the ``matrix-doc`` project should follow the same style as synapse, which is documented at -https://github.com/matrix-org/synapse/tree/master/docs/code_style.rst. +https://github.com/matrix-org/synapse/tree/master/docs/code_style.md. Matrix-doc workflows -------------------- diff --git a/changelogs/client_server/newsfragments/2608.clarification b/changelogs/client_server/newsfragments/2608.clarification new file mode 100644 index 00000000..da704fc5 --- /dev/null +++ b/changelogs/client_server/newsfragments/2608.clarification @@ -0,0 +1 @@ +Clarify the behaviour of SSO login and UI-Auth. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 574999be..27cbe8c5 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1,4 +1,4 @@ -.. Copyright 2016 OpenMarket Ltd +.. Copyright 2016-2020 The Matrix.org Foundation C.I.C. .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -425,6 +425,8 @@ on the server-side and the user simply needs to provide their credentials again. In either case, the client's previously known access token will no longer function. +.. _`user-interactive authentication`: + User-Interactive Authentication API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -813,28 +815,8 @@ Single Sign-On provider. A client wanting to complete authentication using SSO should use the -`Fallback`_ authentication flow by opening a browser window for -``/_matrix/client/r0/auth/m.login.sso/fallback/web?session=<...>`` with the -session parameter set to the session ID provided by the server. - -The homeserver should return a page which asks for the user's confirmation -before proceeding. For example, the page could say words to the effect of: - - A client is trying to remove a device/add an email address/take over your - account. To confirm this action, re-authenticate with single sign-on. If you - did not expect this, your account may be compromised! - -Once the user has confirmed they should be redirected to the single sign-on -provider's login page. Once the provider has validated the user, the browser is -redirected back to the homeserver. - -The homeserver then validates the response from the single sign-on provider and -updates the user-interactive authentication session to mark the single sign-on -stage has been completed. The browser is shown the fallback authentication -completion page. - -Once the flow has completed, the client retries the request with the session -only, as above. +`Fallback`_ mechanism. See `SSO during User-Interactive Authentication`_ for +more information. Email-based (identity / homeserver) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< @@ -940,6 +922,8 @@ should open is:: Where ``auth type`` is the type name of the stage it is attempting and ``session ID`` is the ID of the session given by the homeserver. +.. _`user-interactive authentication fallback completion`: + This MUST return an HTML page which can perform this authentication stage. This page must use the following JavaScript when the authentication has been completed: @@ -1157,7 +1141,7 @@ with ``403 Forbidden`` and an error code of ``M_FORBIDDEN``. If the homeserver advertises ``m.login.sso`` as a viable flow, and the client supports it, the client should redirect the user to the ``/redirect`` endpoint -for `Single Sign-On <#sso-client-login>`_. After authentication is complete, the +for `client login via SSO`_. After authentication is complete, the client will need to submit a ``/login`` request matching ``m.login.token``. {{login_cs_http_api}} diff --git a/specification/modules/sso_login.rst b/specification/modules/sso_login.rst index 6dd4a332..986bed94 100644 --- a/specification/modules/sso_login.rst +++ b/specification/modules/sso_login.rst @@ -1,4 +1,4 @@ -.. Copyright 2019 New Vector Ltd +.. Copyright 2019-2020 The Matrix.org Foundation C.I.C. .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -12,34 +12,98 @@ .. See the License for the specific language governing permissions and .. limitations under the License. -SSO client login -================ +SSO client login/authentication +=============================== .. _module:sso_login: Single Sign-On (SSO) is a generic term which refers to protocols which allow users to log into applications via a single web-based authentication portal. -Examples include "Central Authentication Service" (CAS) and SAML. +Examples include OpenID Connect, "Central Authentication Service" (CAS) and SAML. -An overview of the process, as used in Matrix, is as follows: +This module allows a Matrix homeserver to delegate user authentication to an +external authentication server supporting one of these protocols. In this +process, there are three systems involved: -1. The Matrix client instructs the user's browser to navigate to the - |/login/sso/redirect|_ endpoint on the user's homeserver. + * A Matrix client, using the APIs defined this specification, which is seeking + to authenticate a user to a Matrix homeserver. + + * A Matrix homeserver, implementing the APIs defined in this specification, but + which is delegating user authentication to the authentication server. + + * An "authentication server", which is responsible for authenticating the + user. + +This specification is concerned only with communication between the Matrix +client and the homeserver, and is independent of the SSO protocol used to +communicate with the authentication server. Different Matrix homeserver +implementations might support different SSO protocols. + +Clients and homeservers implementing the SSO flow will need to consider both login_ +and `user-interactive authentication`_. The flow is +similar in both cases, but there are slight differences. + +Typically, SSO systems require a single "callback" URI to be configured at the +authentication server. Once the user is authenticated, their browser is +redirected to that URI. It is up to the Matrix homeserver implementation to +implement a suitable endpoint. For example, for CAS authentication the +homeserver should provide a means for the administrator to configure where the +CAS server is and the REST endpoints which consume the ticket. + +Client login via SSO +--------------------- + +An overview of the process is as follows: + +0. The Matrix client calls |GET /login|_ to find the supported login + types, and the homeserver includes a flow with ``"type": "m.login.sso"`` in the + response. + +1. To initiate the ``m.login.sso`` login type, the Matrix client instructs the + user's browser to navigate to the |/login/sso/redirect|_ endpoint on the + user's homeserver. 2. The homeserver responds with an HTTP redirect to the SSO user interface, which the browser follows. -3. The SSO system authenticates the user. +3. The authentication server and the homeserver interact to verify the user's + identity and other authentication information, potentially using a number of + redirects. -4. The SSO server and the homeserver interact to verify the user's identity - and other authentication information, potentially using a number of redirects. - -5. The browser is directed to the ``redirectUrl`` provided by the client with +4. The browser is directed to the ``redirectUrl`` provided by the client with a ``loginToken`` query parameter for the client to log in with. +5. The client exchanges the login token for an access token by calling the + |/login|_ endpoint with a ``type`` of ``m.login.token``. + +For native applications, typically steps 1 to 4 are carried out by opening an +embedded web view. + +These steps are illustrated as follows:: + + Matrix Client Matrix Homeserver Auth Server + | | | + |-------------(0) GET /login----------->| | + |<-------------login types--------------| | + | | | + | Webview | | + | | | | + |----->| | | + | |--(1) GET /login/sso/redirect-->| | + | |<---------(2) 302---------------| | + | | | | + | |<========(3) Authentication process================>| + | | | | + | |<--(4) redirect to redirectUrl--| | + |<-----| | | + | | | + |---(5) POST /login with login token--->| | + |<-------------access token-------------| | + + .. Note:: In the older `r0.4.0 version `_ - of this specification it was possible to authenticate via CAS when the server + of this specification it was possible to authenticate via CAS when the homeserver provides a ``m.login.cas`` login flow. This specification deprecates the use of ``m.login.cas`` to instead prefer ``m.login.sso``, which is the same process with the only change being which redirect endpoint to use: for ``m.login.cas``, use @@ -47,40 +111,65 @@ An overview of the process, as used in Matrix, is as follows: The endpoints are otherwise the same. Client behaviour ----------------- +~~~~~~~~~~~~~~~~ The client starts the process by instructing the browser to navigate to |/login/sso/redirect|_ with an appropriate ``redirectUrl``. Once authentication is successful, the browser will be redirected to that ``redirectUrl``. -.. TODO-spec +{{sso_login_redirect_cs_http_api}} - Should we recommend some sort of CSRF protection here (specifically, we - should guard against people accidentally logging in by sending them a link - to ``/login/sso/redirect``. +Security considerations ++++++++++++++++++++++++ - Maybe we should recommend that the ``redirectUrl`` should contain a CSRF - token which the client should then check before sending the login token to - ``/login``? +1. CSRF attacks via manipulation of parameters on the ``redirectUrl`` -{{sso_login_redirect_cs_http_api}} + Clients should validate any requests to the ``redirectUrl``. In particular, it + may be possible for attackers to falsify any query parameters, leading to + cross-site request forgery (CSRF) attacks. + + For example, consider a web-based client at ``https://client.example.com``, + which wants to initiate SSO login on the homeserver at ``server.example.org``. + It does this by storing the homeserver name in a query parameter for the + ``redirectUrl``: it redirects to + ``https://server.example.org/login/sso/redirect?redirectUrl=https://client.example.com?hs=server.example.org``. + + An attacker could trick a victim into following a link to + ``https://server.example.org/login/sso/redirect?redirectUrl=https://client.example.com?hs=evil.com``, + which would result in the client sending a login token for the victim's + account to the attacker-controlled site ``evil.com``. + + To guard against this, clients MUST NOT store state (such as the address of + the homeserver being logged into) anywhere it can be modified by external + processes. + + Instead, the state could be stored in `localStorage + `_ or + in a cookie. + +2. For added security, clients SHOULD include a unique identifier in the + ``redirectUrl`` and reject any callbacks that do not contain a recognised + identifier, to guard against unsolicited login attempts and replay attacks. Server behaviour ----------------- +~~~~~~~~~~~~~~~~ + +Redirecting to the Authentication server +++++++++++++++++++++++++++++++++++++++++ -The URI for the SSO system to be used should be configured on the server by the -server administrator. The server is expected to set up any endpoints required to -interact with that SSO system. For example, for CAS authentication the homeserver -should provide a means for the administrator to configure where the CAS server is -and the REST endpoints which consume the ticket. A good reference for how CAS could -be implemented is available in the older `r0.4.0 version `_ -of this specification. +The server should handle +``/_matrix/client/%CLIENT_MAJOR_VERSION%/login/sso/redirect`` as follows: -Handling the redirect endpoint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +#. It should build a suitable request for the SSO system. -When responding to the ``/login/sso/redirect`` endpoint, the server must -generate a URI for the SSO login page with any appropriate parameters. +#. It should store enough state that the flow can be securely resumed after the + SSO process completes. One way to do this is by storing a cookie which is + stored in the user's browser, by adding a ``Set-Cookie`` header to the response. + +#. It should redirect the user's browser to the SSO login page with the + appropriate parameters. + +See also the "Security considerations" below. .. TODO-spec: @@ -89,24 +178,169 @@ generate a URI for the SSO login page with any appropriate parameters. endpoint, and to give more meaningful errors in the case of faulty/poorly-configured clients. -Handling the authentication endpoint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Handling the callback from the Authentication server +++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Note that there will normally be a single callback URI which is used for both login +and user-interactive authentication: it is up to the homeserver implementation +to distinguish which is taking place. + +The homeserver should validate the response from the SSO system: this may +require additional calls to the authentication server, and/or may require +checking a signature on the response. + +The homeserver then proceeds as follows: + +#. The homeserver MUST map the user details received from the authentication + server to a valid `Matrix user identifier <../appendices.html#user-identifiers>`_. + The guidance in `Mapping from other character sets + <../appendices.html#mapping-from-other-character-sets>`_ may be useful. + +#. If the generated user identifier represents a new user, it should be + registered as a new user. + +#. The homeserver should generate a short-term login token. This is an opaque + token, suitable for use with the ``m.login.token`` type of the |/login|_ + API. The lifetime of this token SHOULD be limited to around five + seconds. + +#. The homeserver adds a query parameter of ``loginToken``, with the value of + the generated login token, to the ``redirectUrl`` given in the + ``/_matrix/client/%CLIENT_MAJOR_VERSION%/login/sso/redirect`` + request. (Note: ``redirectURL`` may or may not include existing query + parameters. If it already includes one or more ``loginToken`` parameters, + they should be removed before adding the new one.) + +#. The homeserver redirects the user's browser to the URI thus built. + +Security considerations +~~~~~~~~~~~~~~~~~~~~~~~ + +1. Homeservers should ensure that login tokens are not sent to malicious + clients. + + For example, consider a homeserver at ``server.example.org``. An attacker tricks + a victim into following a link to + ``https://server.example.org/login/sso/redirect?redirectUrl=https://evil.com``, + resulting in a login token being sent to the attacker-controlled site + ``evil.com``. This is a form of cross-site request forgery (CSRF). + + To mitigate this, Homeservers SHOULD confirm with the user that they are + happy to grant access to their matrix account to the site named in the + ``redirectUrl``. This can be done either *before* redirecting to the SSO + login page when handling the + ``/_matrix/client/%CLIENT_MAJOR_VERSION%/login/sso/redirect`` endpoint, or + *after* login when handling the callback from the authentication server. (If + the check is performed before redirecting, it is particularly important that + the homeserver guards against unsolicited authentication attempts as below). + + It may be appropriate to whitelist a set of known-trusted client URLs in + this process. In particular, the homeserver's own `login fallback`_ + implementation could be excluded. + +2. For added security, homeservers SHOULD guard against unsolicited + authentication attempts by tracking pending requests. One way to do this is + to set a cookie when handling + ``/_matrix/client/%CLIENT_MAJOR_VERSION%/login/sso/redirect``, which is + checked and cleared when handling the callback from the authentication + server. + +SSO during User-Interactive Authentication +------------------------------------------ + +`User-interactive authentication`_ is used by client-server +endpoints which require additional confirmation of the user's identity (beyond +holding an access token). Typically this means that the user must re-enter +their password, but for homeservers which delegate to an SSO server, this means +redirecting to the authentication server during user-interactive auth. + +The implemementation of this is based on the `Fallback`_ mechanism for +user-interactive auth. + +Client behaviour +---------------- + +Clients do not need to take any particular additional steps beyond ensuring +that the fallback mechanism has been implemented, and treating the +``m.login.sso`` authentication type the same as any other unknown type +(i.e. they should open a browser window for +``/_matrix/client/%CLIENT_MAJOR_VERSION%/auth/m.login.sso/fallback/web?session=``. +Once the flow has completed, the client retries the request with the session +only.) + +Server behaviour +---------------- + +Redirecting to the Authentication server +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The server should handle +``/_matrix/client/%CLIENT_MAJOR_VERSION%/auth/m.login.sso/fallback/web`` in +much the same way as +``/_matrix/client/%CLIENT_MAJOR_VERSION%/login/sso/redirect``, which is to say: + +#. It should build a suitable request for the SSO system. + +#. It should store enough state that the flow can be securely resumed after the + SSO process completes. One way to do this is by storing a cookie which is + stored in the user's browser, by adding a ``Set-Cookie`` header to the response. + +#. It should redirect the user's browser to the SSO login page with the + appropriate parameters. + +See also the "Security considerations" below. + +Handling the callback from the Authentication server +++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Note that there will normally be a single callback URI which is used for both login +and user-interactive authentication: it is up to the homeserver implementation +to distinguish which is taking place. + +The homeserver should validate the response from the SSO system: this may +require additional calls to the authentication server, and/or may require +checking a signature on the response. + +The homeserver then returns the `user-interactive authentication fallback +completion`_ page to the user's browser. + +Security considerations ++++++++++++++++++++++++ + +1. Confirming the operation + + The homeserver SHOULD confirm that the user is happy for the operation to go + ahead. The goal of the user-interactive authentication operation is to guard + against a compromised ``access_token`` being used to take over the user's + account. Simply redirecting the user to the SSO system is insufficient, + since they may not realise what is being asked of them, or the SSO system + may even confirm the authentication automatically. + + For example, the homeserver might serve a page with words to the effect of: + + A client is trying to remove a device from your account. To confirm this + action, re-authenticate with single sign-on. If you did not expect this, your + account may be compromised! -Once the homeserver has verified the user's identity with the SSO system, it -MUST map the user ID to a valid `Matrix user identifier <../index.html#user-identifiers>`_. -The guidance in `Mapping from other character sets -<../index.html#mapping-from-other-character-sets>`_ may be useful. + This confirmation could take place before redirecting to the SSO + authentication page (when handling the + ``/_matrix/client/%CLIENT_MAJOR_VERSION%/auth/m.login.sso/fallback/web`` + endpoint), or *after* authentication when handling the callback from the + authentication server. (If the check is performed before redirecting, it is + particularly important that the homeserver guards against unsolicited + authentication attempts as below). -If the generated user identifier represents a new user, it should be registered -as a new user. +2. For added security, homeservers SHOULD guard against unsolicited + authentication attempts by tracking pending requests. One way to do this is + to set a cookie when handling + ``/_matrix/client/%CLIENT_MAJOR_VERSION%/auth/m.login.sso/fallback/web``, + which is checked and cleared when handling the callback from the + authentication server. -Finally, the server should generate a short-term login token. The generated -token should be a macaroon, suitable for use with the ``m.login.token`` type of -the |/login|_ API, and `token-based interactive login <#token-based>`_. The -lifetime of this token SHOULD be limited to around five seconds. This token is -given to the client via the ``loginToken`` query parameter previously mentioned. +.. |GET /login| replace:: ``GET /login`` +.. _GET /login: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login .. |/login| replace:: ``/login`` .. _/login: #post-matrix-client-%CLIENT_MAJOR_VERSION%-login .. |/login/sso/redirect| replace:: ``/login/sso/redirect`` From 4aa778cdee0527fd1d7ec362762c2399617d76a6 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Wed, 10 Jun 2020 13:40:12 +0100 Subject: [PATCH 162/162] Put the MSC number in the proposal template (#2622) this seems to be our convention. --- proposals/0000-proposal-template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-proposal-template.md b/proposals/0000-proposal-template.md index e50aed8b..8acbef85 100644 --- a/proposals/0000-proposal-template.md +++ b/proposals/0000-proposal-template.md @@ -1,4 +1,4 @@ -# Example: Proposal to adopt a template for MSCs +# MSC0000: Template for new MSCs *Note: Text written in italics represents notes about the section or proposal process. This document serves as an example of what a proposal could look like (in this case, a proposal to have a template)