From 0300a3cab4a77a5757c9e6acd065261f4eb03954 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 20:45:19 -0600 Subject: [PATCH 1/7] Move redaction algorithm into room version specification We stick it in a client section of v1 as the earliest version to define the algorithm is v1, and the client-server spec tells clients to use this algorithm. --- specification/client_server_api.rst | 34 +---------------- specification/rooms/v1.rst | 59 +++++++++++++++++++++++++---- 2 files changed, 53 insertions(+), 40 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index fce879a2..8a94b711 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1779,39 +1779,7 @@ redacted include a ``redacted_because`` key whose value is the event that caused it to be redacted, which may include a reason. -Upon receipt of a redaction event, the server should strip off any keys not in -the following list: - -- ``event_id`` -- ``type`` -- ``room_id`` -- ``sender`` -- ``state_key`` -- ``content`` -- ``hashes`` -- ``signatures`` -- ``depth`` -- ``prev_events`` -- ``prev_state`` -- ``auth_events`` -- ``origin`` -- ``origin_server_ts`` -- ``membership`` - -.. Note: - Some of the keys, such as ``hashes``, will appear on the federation-formatted - event and therefore the client may not be aware of them. - -The content object should also be stripped of all keys, unless it is one of -one of the following event types: - -- ``m.room.member`` allows key ``membership``. -- ``m.room.create`` allows key ``creator``. -- ``m.room.join_rules`` allows key ``join_rule``. -- ``m.room.power_levels`` allows keys ``ban``, ``events``, ``events_default``, - ``kick``, ``redact``, ``state_default``, ``users``, ``users_default``. -- ``m.room.aliases`` allows key ``aliases``. -- ``m.room.history_visibility`` allows key ``history_visibility``. +The exact algorithm to apply against an event is defined in the `room version specification`_. The server should add the event causing the redaction to the ``unsigned`` property of the redacted event, under the ``redacted_because`` key. When a diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 10eee54d..23d8a833 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -1,4 +1,5 @@ .. Copyright 2017,2019 New Vector Ltd +.. Copyright 2020 The Matrix.org Foundation C.I.C. .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -21,13 +22,57 @@ blocks for other room versions. .. contents:: Table of Contents .. sectnum:: +Client considerations +--------------------- + +Clients may need to consider some algorithms performed by the server for their own +implementation. + +Redactions +~~~~~~~~~~ + +Upon receipt of a redaction event, the server should strip off any keys not in +the following list: + +- ``event_id`` +- ``type`` +- ``room_id`` +- ``sender`` +- ``state_key`` +- ``content`` +- ``hashes`` +- ``signatures`` +- ``depth`` +- ``prev_events`` +- ``prev_state`` +- ``auth_events`` +- ``origin`` +- ``origin_server_ts`` +- ``membership`` + +.. Note: + Some of the keys, such as ``hashes``, will appear on the federation-formatted + event and therefore the client may not be aware of them. + +The content object should also be stripped of all keys, unless it is one of +one of the following event types: + +- ``m.room.member`` allows key ``membership``. +- ``m.room.create`` allows key ``creator``. +- ``m.room.join_rules`` allows key ``join_rule``. +- ``m.room.power_levels`` allows keys ``ban``, ``events``, ``events_default``, + ``kick``, ``redact``, ``state_default``, ``users``, ``users_default``. +- ``m.room.aliases`` allows key ``aliases``. +- ``m.room.history_visibility`` allows key ``history_visibility``. + Server implementation components -------------------------------- .. WARNING:: The information contained in this section is strictly for server implementors. Applications which use the Client-Server API are generally unaffected by the - details contained here, and can safely ignore their presence. + intricacies contained here. The section above regarding client considerations + is the resource that Client-Server API use cases should reference. The algorithms defined here should only apply to version 1 rooms. Other algorithms @@ -112,7 +157,7 @@ The types of state events that affect authorization are: .. NOTE:: Power levels are inferred from defaults when not explicitly supplied. - For example, mentions of the ``sender``'s power level can also refer + For example, mentions of the ``sender``'s power level can also refer to the default power level for users in the room. The rules are as follows: @@ -250,7 +295,7 @@ The rules are as follows: #. If there is no previous ``m.room.power_levels`` event in the room, allow. #. For the keys ``users_default``, ``events_default``, - ``state_default``, ``ban``, ``redact``, ``kick``, ``invite`` check if they + ``state_default``, ``ban``, ``redact``, ``kick``, ``invite`` check if they were added, changed or removed. For each found alteration: i. If the current value is higher than the ``sender``'s current power level, @@ -258,13 +303,13 @@ The rules are as follows: #. If the new value is higher than the ``sender``'s current power level, reject. - - #. For each entry being added, changed or removed in both the ``events`` and + + #. For each entry being added, changed or removed in both the ``events`` and ``users`` keys: - + i. If the current value is higher than the ``sender``'s current power level, reject. - + #. If the new value is higher than the ``sender``'s current power level, reject. From be353115594b3572fc07c8f8b5d40e57b6c161fb Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 20:28:27 -0600 Subject: [PATCH 2/7] s/should/must for redaction algorithm This feels like it was a mistake some time ago considering the redaction algorithm is used in very strict algorithms like event signing. --- specification/rooms/v1.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 23d8a833..9282c1f3 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -31,7 +31,7 @@ implementation. Redactions ~~~~~~~~~~ -Upon receipt of a redaction event, the server should strip off any keys not in +Upon receipt of a redaction event, the server must strip off any keys not in the following list: - ``event_id`` @@ -54,7 +54,7 @@ the following list: Some of the keys, such as ``hashes``, will appear on the federation-formatted event and therefore the client may not be aware of them. -The content object should also be stripped of all keys, unless it is one of +The content object must also be stripped of all keys, unless it is one of one of the following event types: - ``m.room.member`` allows key ``membership``. From a1324aa9dca7e9ca01acde43cc5db3c7f95ad718 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 20:44:08 -0600 Subject: [PATCH 3/7] Move MSC2432 (alias handling) to v6 --- specification/index.rst | 1 + specification/rooms/v6.rst | 59 ++++++++++++++++++++++++++++++++++++++ specification/targets.yaml | 4 +++ 3 files changed, 64 insertions(+) create mode 100644 specification/rooms/v6.rst diff --git a/specification/index.rst b/specification/index.rst index 728383ab..aa8a4641 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -550,6 +550,7 @@ The available room versions are: * `Version 3 `_ - **Stable**. Introduces events whose IDs are the event's hash. * `Version 4 `_ - **Stable**. Builds on v3 by using URL-safe base64 for event IDs. * `Version 5 `_ - **Stable**. Introduces enforcement of signing key validity periods. +* `Version 6 `_ - **Stable**. Alters several authorization rules for events. Specification Versions ---------------------- diff --git a/specification/rooms/v6.rst b/specification/rooms/v6.rst new file mode 100644 index 00000000..de0ab889 --- /dev/null +++ b/specification/rooms/v6.rst @@ -0,0 +1,59 @@ +.. Copyright 2020 The Matrix.org Foundation C.I.C. +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +Room Version 6 +============== + +This room version builds on `version 5 `_ while changing various +authorization rules performed on events. + +.. contents:: Table of Contents +.. sectnum:: + + +Client considerations +--------------------- + +The redaction algorithm has changed from `room version 1 `_ to remove +all rules against events of type ``m.room.aliases``. Room versions 2, 3, 4, and +5 all use v1's redaction algorithm. The algorithm is otherwise unchanged. + + +Server implementation components +-------------------------------- + +.. WARNING:: + The information contained in this section is strictly for server implementors. + Applications which use the Client-Server API are generally unaffected by the + intricacies contained here. The section above regarding client considerations + is the resource that Client-Server API use cases should reference. + + +Room version 6 makes the following alterations to algorithms described in `room version 5 `_. + +Redactions +~~~~~~~~~~ + +As mentioned in the client considerations portion of this specification, all +special meaning has been removed for events of type ``m.room.aliases``. The +algorithm is otherwise unchanged. + +Authorization rules for events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Like redactions, all rules relating specifically to events of type ``m.room.aliases`` +are removed. They must still pass authorization checks relating to state events. + +The remaining rules are the same as in `room version 3 `_ +(the last inherited room version to specify the authorization rules). diff --git a/specification/targets.yaml b/specification/targets.yaml index 4e0b068d..5f0e32cc 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -45,6 +45,10 @@ targets: files: - rooms/v5.rst version_label: v5 + rooms@v6: # this is translated to be rooms/v6.html + files: + - rooms/v6.rst + version_label: v6 appendices: files: - appendices.rst From 74c51b05a434ddcfb3b8dc63ebfeda56c29456ba Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 22:13:55 -0600 Subject: [PATCH 4/7] Incorporate MSC2209 (`notifications` auth rules) MSC: https://github.com/matrix-org/matrix-doc/pull/2209 The changes are slightly difficult to word without dumping the text in and playing a game of spot the difference, so we now use our pre-existing pygments support to render a representation of the difference. The difference is shown in markdown-like format instead of RST for ease of understanding. It's also not rendered HTML for largely complexity reasons. --- scripts/css/pygments.css | 83 ++++++++++++++++++++++++++++++++++++++ scripts/gendoc.py | 1 + specification/rooms/v6.rst | 32 +++++++++++++++ 3 files changed, 116 insertions(+) create mode 100644 scripts/css/pygments.css diff --git a/scripts/css/pygments.css b/scripts/css/pygments.css new file mode 100644 index 00000000..a212151a --- /dev/null +++ b/scripts/css/pygments.css @@ -0,0 +1,83 @@ +/* +Original styles generated from: + pygmentize -f html -S colorful -a pre.code > ./scripts/css/pygments.css + +Rules for which we don't want the syntax highlighter to kick in are commented +out at the bottom. + +Windows users: if you regenerate this file, you'll need to re-save it as utf-8 +to make docutils happy. +*/ + +/* DIFFS */ +pre.code .gd { color: #A00000 } /* Generic.Deleted */ +pre.code .gi { color: #00A000 } /* Generic.Inserted */ + +/* UNUSED */ +/*pre.code .hll { background-color: #ffffcc }*/ +/*pre.code { background: #ffffff; }*/ +/*pre.code .c { color: #888888 } !* Comment *!*/ +/*pre.code .err { color: #FF0000; background-color: #FFAAAA } !* Error *!*/ +/*pre.code .k { color: #008800; font-weight: bold } !* Keyword *!*/ +/*pre.code .o { color: #333333 } !* Operator *!*/ +/*pre.code .ch { color: #888888 } !* Comment.Hashbang *!*/ +/*pre.code .cm { color: #888888 } !* Comment.Multiline *!*/ +/*pre.code .cp { color: #557799 } !* Comment.Preproc *!*/ +/*pre.code .cpf { color: #888888 } !* Comment.PreprocFile *!*/ +/*pre.code .c1 { color: #888888 } !* Comment.Single *!*/ +/*pre.code .cs { color: #cc0000; font-weight: bold } !* Comment.Special *!*/ +/*pre.code .ge { font-style: italic } !* Generic.Emph *!*/ +/*pre.code .gr { color: #FF0000 } !* Generic.Error *!*/ +/*pre.code .gh { color: #000080; font-weight: bold } !* Generic.Heading *!*/ +/*pre.code .go { color: #888888 } !* Generic.Output *!*/ +/*pre.code .gp { color: #c65d09; font-weight: bold } !* Generic.Prompt *!*/ +/*pre.code .gs { font-weight: bold } !* Generic.Strong *!*/ +/*pre.code .gu { color: #800080; font-weight: bold } !* Generic.Subheading *!*/ +/*pre.code .gt { color: #0044DD } !* Generic.Traceback *!*/ +/*pre.code .kc { color: #008800; font-weight: bold } !* Keyword.Constant *!*/ +/*pre.code .kd { color: #008800; font-weight: bold } !* Keyword.Declaration *!*/ +/*pre.code .kn { color: #008800; font-weight: bold } !* Keyword.Namespace *!*/ +/*pre.code .kp { color: #003388; font-weight: bold } !* Keyword.Pseudo *!*/ +/*pre.code .kr { color: #008800; font-weight: bold } !* Keyword.Reserved *!*/ +/*pre.code .kt { color: #333399; font-weight: bold } !* Keyword.Type *!*/ +/*pre.code .m { color: #6600EE; font-weight: bold } !* Literal.Number *!*/ +/*pre.code .s { background-color: #fff0f0 } !* Literal.String *!*/ +/*pre.code .na { color: #0000CC } !* Name.Attribute *!*/ +/*pre.code .nb { color: #007020 } !* Name.Builtin *!*/ +/*pre.code .nc { color: #BB0066; font-weight: bold } !* Name.Class *!*/ +/*pre.code .no { color: #003366; font-weight: bold } !* Name.Constant *!*/ +/*pre.code .nd { color: #555555; font-weight: bold } !* Name.Decorator *!*/ +/*pre.code .ni { color: #880000; font-weight: bold } !* Name.Entity *!*/ +/*pre.code .ne { color: #FF0000; font-weight: bold } !* Name.Exception *!*/ +/*pre.code .nf { color: #0066BB; font-weight: bold } !* Name.Function *!*/ +/*pre.code .nl { color: #997700; font-weight: bold } !* Name.Label *!*/ +/*pre.code .nn { color: #0e84b5; font-weight: bold } !* Name.Namespace *!*/ +/*pre.code .nt { color: #007700 } !* Name.Tag *!*/ +/*pre.code .nv { color: #996633 } !* Name.Variable *!*/ +/*pre.code .ow { color: #000000; font-weight: bold } !* Operator.Word *!*/ +/*pre.code .w { color: #bbbbbb } !* Text.Whitespace *!*/ +/*pre.code .mb { color: #6600EE; font-weight: bold } !* Literal.Number.Bin *!*/ +/*pre.code .mf { color: #6600EE; font-weight: bold } !* Literal.Number.Float *!*/ +/*pre.code .mh { color: #005588; font-weight: bold } !* Literal.Number.Hex *!*/ +/*pre.code .mi { color: #0000DD; font-weight: bold } !* Literal.Number.Integer *!*/ +/*pre.code .mo { color: #4400EE; font-weight: bold } !* Literal.Number.Oct *!*/ +/*pre.code .sa { background-color: #fff0f0 } !* Literal.String.Affix *!*/ +/*pre.code .sb { background-color: #fff0f0 } !* Literal.String.Backtick *!*/ +/*pre.code .sc { color: #0044DD } !* Literal.String.Char *!*/ +/*pre.code .dl { background-color: #fff0f0 } !* Literal.String.Delimiter *!*/ +/*pre.code .sd { color: #DD4422 } !* Literal.String.Doc *!*/ +/*pre.code .s2 { background-color: #fff0f0 } !* Literal.String.Double *!*/ +/*pre.code .se { color: #666666; font-weight: bold; background-color: #fff0f0 } !* Literal.String.Escape *!*/ +/*pre.code .sh { background-color: #fff0f0 } !* Literal.String.Heredoc *!*/ +/*pre.code .si { background-color: #eeeeee } !* Literal.String.Interpol *!*/ +/*pre.code .sx { color: #DD2200; background-color: #fff0f0 } !* Literal.String.Other *!*/ +/*pre.code .sr { color: #000000; background-color: #fff0ff } !* Literal.String.Regex *!*/ +/*pre.code .s1 { background-color: #fff0f0 } !* Literal.String.Single *!*/ +/*pre.code .ss { color: #AA6600 } !* Literal.String.Symbol *!*/ +/*pre.code .bp { color: #007020 } !* Name.Builtin.Pseudo *!*/ +/*pre.code .fm { color: #0066BB; font-weight: bold } !* Name.Function.Magic *!*/ +/*pre.code .vc { color: #336699 } !* Name.Variable.Class *!*/ +/*pre.code .vg { color: #dd7700; font-weight: bold } !* Name.Variable.Global *!*/ +/*pre.code .vi { color: #3333BB } !* Name.Variable.Instance *!*/ +/*pre.code .vm { color: #996633 } !* Name.Variable.Magic *!*/ +/*pre.code .il { color: #0000DD; font-weight: bold } !* Literal.Number.Integer.Long *!*/ diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 72e3047c..7e68ccd7 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -273,6 +273,7 @@ def rst2html(i, o, stylesheets): writer_name="html", settings_overrides={ "stylesheet_path": stylesheets, + "syntax_highlight": "short", }, ) diff --git a/specification/rooms/v6.rst b/specification/rooms/v6.rst index de0ab889..e165f771 100644 --- a/specification/rooms/v6.rst +++ b/specification/rooms/v6.rst @@ -55,5 +55,37 @@ Authorization rules for events Like redactions, all rules relating specifically to events of type ``m.room.aliases`` are removed. They must still pass authorization checks relating to state events. +Additionally, the authorization rules for events of type ``m.room.power_levels`` +now include the content key ``notifications``. This new rule takes the place of the +rule which checks the ``events`` and ``users`` keys. + +For completeness, the changes to the auth rules can be represented as follows: + +.. code:: diff + + ... + + -If type is `m.room.aliases`: + - + - a. If event has no `state_key`, reject. + - b. If sender's domain doesn't matches `state_key`, reject. + - c. Otherwise, allow. + + ... + + If type is `m.room.power_levels`: + + ... + + - * For each entry being added, changed or removed in both the `events` and `users` keys: + + * For each entry being added, changed or removed in the `events`, `users`, and `notifications` keys: + + i. If the current value is higher than the `sender`'s current power level, reject. + + ii. If the new value is higher than the `sender`'s current power level, reject. + + ... + + The remaining rules are the same as in `room version 3 `_ (the last inherited room version to specify the authorization rules). From 66ab480967c7dc1b2ad4f3750b94f837d1eda9ef Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 21:58:58 -0600 Subject: [PATCH 5/7] Incorporate MSC2540 (Canonical JSON validation) MSC: https://github.com/matrix-org/matrix-doc/pull/2540 --- specification/appendices/signing_json.rst | 11 +++++++++++ specification/rooms/v1.rst | 6 ++++++ specification/rooms/v6.rst | 9 +++++++++ 3 files changed, 26 insertions(+) diff --git a/specification/appendices/signing_json.rst b/specification/appendices/signing_json.rst index 8036950e..fbeb0010 100644 --- a/specification/appendices/signing_json.rst +++ b/specification/appendices/signing_json.rst @@ -39,6 +39,17 @@ range where they can be accurately represented using IEEE double precision floating point numbers since a number of JSON libraries represent all numbers using this representation. +.. WARNING:: + Events in room versions 1, 2, 3, 4, and 5 might not be fully compliant with + these restrictions. Servers SHOULD be capable of handling JSON which is considered + invalid by these restrictions where possible. + + The most notable consideration is that integers might not be in the range + specified above. + +.. Note:: + Float values are not permitted by this encoding. + .. code:: python import json diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 9282c1f3..a71bdfb4 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -352,6 +352,12 @@ Events in version 1 rooms have the following structure: {{definition_ss_pdu}} +Canonical JSON +~~~~~~~~~~~~~~ + +Servers MUST NOT strictly enforce the JSON format specified in the +`appendices <../appendices.html#canonical-json>`_ for the reasons described there. + .. _`auth events selection`: ../server_server/%SERVER_RELEASE_LABEL%.html#auth-events-selection .. _`Signing Events`: ../server_server/%SERVER_RELEASE_LABEL%.html#signing-events diff --git a/specification/rooms/v6.rst b/specification/rooms/v6.rst index e165f771..b1acccd2 100644 --- a/specification/rooms/v6.rst +++ b/specification/rooms/v6.rst @@ -89,3 +89,12 @@ For completeness, the changes to the auth rules can be represented as follows: The remaining rules are the same as in `room version 3 `_ (the last inherited room version to specify the authorization rules). + +Canonical JSON +~~~~~~~~~~~~~~ + +Servers MUST strictly enforce the JSON format specified in the +`appendices <../appendices.html#canonical-json>`_. This translates to a 400 ``M_BAD_JSON`` error +on most endpoints, or discarding of events over federation. For example, the federation API's +``/send`` endpoint would discard the event whereas the Client Server API's ``/send/{eventType}`` +endpoint would return a ``M_BAD_JSON`` error. From 98416bf94898395ae5f1b81bfcf8f659a1e76d8b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 May 2020 22:16:16 -0600 Subject: [PATCH 6/7] Add changelog for client-server API --- changelogs/client_server/newsfragments/2563.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2563.clarification diff --git a/changelogs/client_server/newsfragments/2563.clarification b/changelogs/client_server/newsfragments/2563.clarification new file mode 100644 index 00000000..a8805f4a --- /dev/null +++ b/changelogs/client_server/newsfragments/2563.clarification @@ -0,0 +1 @@ +Move redaction algorithm into the room version specifications. From d4c19a0e808e39dbcc92a7af1193cbb1daae803f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 26 May 2020 12:36:18 -0600 Subject: [PATCH 7/7] Federation Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- specification/rooms/v6.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/rooms/v6.rst b/specification/rooms/v6.rst index b1acccd2..e5378d0e 100644 --- a/specification/rooms/v6.rst +++ b/specification/rooms/v6.rst @@ -95,6 +95,6 @@ Canonical JSON Servers MUST strictly enforce the JSON format specified in the `appendices <../appendices.html#canonical-json>`_. This translates to a 400 ``M_BAD_JSON`` error -on most endpoints, or discarding of events over federation. For example, the federation API's +on most endpoints, or discarding of events over federation. For example, the Federation API's ``/send`` endpoint would discard the event whereas the Client Server API's ``/send/{eventType}`` endpoint would return a ``M_BAD_JSON`` error.