From 87330b9b9b01dc089a8781d9775e9252c207b4cb Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Mon, 5 Nov 2018 18:23:23 +0000 Subject: [PATCH 001/170] Proposal for .well-known for server discovery --- proposals/1708-well-known-for-federation.md | 126 ++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 proposals/1708-well-known-for-federation.md diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md new file mode 100644 index 000000000..2b8a3711d --- /dev/null +++ b/proposals/1708-well-known-for-federation.md @@ -0,0 +1,126 @@ +# .well-known support for server name resolution + +Currently, mapping from a server name to a hostname for federation is done via +`SRV` records. This presents two principal difficulties: + + * SRV records are not widely used, and administrators may be unfamiliar with + them, and there may be other practical difficulties in their deployment such + as poor support from hosting providers. [^1] + + * It is likely that we will soon require valid X.509 certificates on the + federation endpoint. It will then be necessary for the homeserver to present + a certificate which is valid for the server name. This presents difficulties + for hosted server offerings: BigCorp may be reluctant to hand over the + keys for `bigcorp.com` to the administrators of the `bigcorp.com` matrix + homeserver. + +Here we propose to solve these problems by augmenting the current `SRV` record +with a `.well-known` lookup. + +## Proposal + +For reference, the current [specification for resolving server +names](https://matrix.org/docs/spec/server_server/unstable.html#resolving-server-names) +is as follows: + +* If the hostname is an IP literal, then that IP address should be used, + together with the given port number, or 8448 if no port is given. + +* Otherwise, if the port is present, then an IP address is discovered by + looking up an AAAA or A record for the hostname, and the specified port is + used. + +* If the hostname is not an IP literal and no port is given, the server is + discovered by first looking up a `_matrix._tcp` SRV record for the + hostname, which may give a hostname (to be looked up using AAAA or A queries) + and port. If the SRV record does not exist, then the server is discovered by + looking up an AAAA or A record on the hostname and taking the default + fallback port number of 8448. + + Homeservers may use SRV records to load balance requests between multiple TLS + endpoints or to failover to another endpoint if an endpoint fails. + +The first two points remain unchanged: if the server name is an IP literal, or +contains a port, then requests will be made directly as before. + +If the hostname is neither an IP literal, nor does it have an explicit port, +then the requesting server should continue to make an SRV lookup as before, and +use the result if one is found. + +If *no* result is found, the requesting server should make a `GET` request to +`https://\/.well-known/matrix/server`, with normal X.509 +certificate validation. If the request fails in any way, then we fall back as +before to using using port 8448 on the hostname. + +Rationale: Falling back to port 8448 (rather than aborting the request) is +necessary to maintain compatibility with existing deployments, which may not +present valid certificates on port 443, or may return 4xx or 5xx errors. + +If the GET request succeeds, it should result in a JSON response, with contents +structured as shown: + +```json +{ + "server": "[:]" +} +``` + +The `server` property has the same format as a [server +name](https://matrix.org/docs/spec/appendices.html#server-name): a hostname +followed by an optional port. + +If the response cannot be parsed as JSON, or lacks a valid `server` property, +the request is considered to have failed, and no fallback to port 8448 takes +place. + +Otherwise, the requesting server performs an `AAAA/A` lookup on the hostname, +and connects to the resultant address and the specifed port. The port defaults +to 8448, if unspecified. + +### Caching + +Servers should not look up the `.well-known` file for every request, as this +would impose an unacceptable overhead on both sides. Instead, the results of +the `.well-known` request should be cached according to the HTTP response +headers, as per [RFC7234](https://tools.ietf.org/html/rfc7234). If the response +does not include an explicit expiry time, the requesting server should use a +sensible default: 24 hours is suggested. + +Because there is no way to request a revalidation, it is also recommended that +requesting servers cap the expiry time. 48 hours is suggested. + +Similarly, a failure to retrieve the `.well-known` file should be cached for +a reasonable period. 24 hours is suggested again. + +### The future of SRV records + +It's worth noting that this proposal is very clear in that we will maintain +support for SRV records for the immediate future; there are no current plans to +deprecate them. + +However, clearly a `.well-known` file can provide much of the functionality of +an SRV record, and having to support both may be undesirable. Accordingly, we +may consider sunsetting SRV record support at some point in the future. + +### Outstanding questions + +Should we follow 30x redirects for the .well-known file? On the one hand, there +is no obvious usecase and they add complexity (for example: how do they +interact with caches?). On the other hand, we'll presumably be using an HTTP +client library to handle some of the caching stuff, and they might be useful +for something? + +## Security considerations + +The `.well-known` file potentially broadens the attack surface for an attacker +wishing to intercept federation traffic to a particular server. + +## Conclusion + +This proposal adds a new mechanism, alongside the existing `SRV` record lookup +for finding the server responsible for a particular matrix server_name, which +will allow greater flexibility in deploying homeservers. + + +[^1] For example, Cloudflare automatically "flattens" SRV record responses. + From e3f10a4fd2049bcd71f4e77fe4192946498059f2 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Mon, 5 Nov 2018 18:37:25 +0000 Subject: [PATCH 002/170] Update 1708-well-known-for-federation.md fix title --- proposals/1708-well-known-for-federation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 2b8a3711d..beec384f9 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -1,4 +1,4 @@ -# .well-known support for server name resolution +# MSC1708: .well-known support for server name resolution Currently, mapping from a server name to a hostname for federation is done via `SRV` records. This presents two principal difficulties: From c4e1949cf8f81b137d9fedebb4d5bf2c5116b112 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 6 Nov 2018 11:38:23 +0000 Subject: [PATCH 003/170] Clarifications about what `server` means --- proposals/1708-well-known-for-federation.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index beec384f9..8854eb9f9 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -47,8 +47,8 @@ If the hostname is neither an IP literal, nor does it have an explicit port, then the requesting server should continue to make an SRV lookup as before, and use the result if one is found. -If *no* result is found, the requesting server should make a `GET` request to -`https://\/.well-known/matrix/server`, with normal X.509 +If *no* SRV result is found, the requesting server should make a `GET` request +to `https://\/.well-known/matrix/server`, with normal X.509 certificate validation. If the request fails in any way, then we fall back as before to using using port 8448 on the hostname. @@ -65,17 +65,19 @@ structured as shown: } ``` -The `server` property has the same format as a [server -name](https://matrix.org/docs/spec/appendices.html#server-name): a hostname -followed by an optional port. +The `server` property should be a hostname or IP address, followed by an +optional port. If the response cannot be parsed as JSON, or lacks a valid `server` property, the request is considered to have failed, and no fallback to port 8448 takes place. -Otherwise, the requesting server performs an `AAAA/A` lookup on the hostname, -and connects to the resultant address and the specifed port. The port defaults -to 8448, if unspecified. +Otherwise, the requesting server performs an `AAAA/A` lookup on the hostname +(if necessary), and connects to the resultant address and the specifed +port. The port defaults to 8448, if unspecified. + +(The formal grammar for the `server` property is identical to that of a [server +name](https://matrix.org/docs/spec/appendices.html#server-name).) ### Caching From 09d41464e7b3d03eb07fe066b4fae4153f79b3c2 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 6 Nov 2018 11:38:46 +0000 Subject: [PATCH 004/170] Add problems section xs --- proposals/1708-well-known-for-federation.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 8854eb9f9..21365be18 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -112,6 +112,21 @@ interact with caches?). On the other hand, we'll presumably be using an HTTP client library to handle some of the caching stuff, and they might be useful for something? +## Problems + +It will take a while for `.well-known` to be supported across the ecosystem; +until it is, it will be difficult to deploy homeservers which rely on it for +their routing: if Alice is using a current homeserver implementation, and Bob +deploys a new implementation which relies on `.well-known` for routing, then +Alice will be unable to send messages to Bob. (This is the same problem we have with +[SNI](https://github.com/matrix-org/synapse/issues/1491#issuecomment-415153428).) + +The main defence against this seems to be to release support for `.well-known` +as soom as possible, to maximise uptake in the ecosystem. It is likely that, as +we approach Matrix 1.0, there will be sufficient other new features (such as +new Room versions) that upgading will be necessary anyway. + + ## Security considerations The `.well-known` file potentially broadens the attack surface for an attacker From b1e79ac7ab37302c2c90f2f031a8c6f8d4d2d34b Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Fri, 9 Nov 2018 11:22:22 +0000 Subject: [PATCH 005/170] Update 1708-well-known-for-federation.md --- proposals/1708-well-known-for-federation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 21365be18..bd2f32b89 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -122,9 +122,9 @@ Alice will be unable to send messages to Bob. (This is the same problem we have [SNI](https://github.com/matrix-org/synapse/issues/1491#issuecomment-415153428).) The main defence against this seems to be to release support for `.well-known` -as soom as possible, to maximise uptake in the ecosystem. It is likely that, as +as soon as possible, to maximise uptake in the ecosystem. It is likely that, as we approach Matrix 1.0, there will be sufficient other new features (such as -new Room versions) that upgading will be necessary anyway. +new Room versions) that upgrading will be necessary anyway. ## Security considerations From e789eb186a8f12fe2517d1b30c8e38989d3c3b5f Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Mon, 12 Nov 2018 14:02:44 +0000 Subject: [PATCH 006/170] link to MSC1711 --- proposals/1708-well-known-for-federation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index bd2f32b89..8fa66c7a9 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -7,7 +7,8 @@ Currently, mapping from a server name to a hostname for federation is done via them, and there may be other practical difficulties in their deployment such as poor support from hosting providers. [^1] - * It is likely that we will soon require valid X.509 certificates on the + * [MSC1711](https://github.com/matrix-org/matrix-doc/pull/1711) proposes + requiring valid X.509 certificates on the federation endpoint. It will then be necessary for the homeserver to present a certificate which is valid for the server name. This presents difficulties for hosted server offerings: BigCorp may be reluctant to hand over the @@ -140,4 +141,3 @@ will allow greater flexibility in deploying homeservers. [^1] For example, Cloudflare automatically "flattens" SRV record responses. - From ffe577371db7482a3df92a4f9dee6f984922119a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 10 Sep 2018 16:29:30 -0600 Subject: [PATCH 007/170] Add a room version specification The "Room Specification" (or "Room Version Specification") is the specification that defines which room versions do what and are intended to be documents which speak the truth about how rooms operate under the hood. The approach taken here is a bit different than other specifications. For starters, the specification is versioned in this project instead of relying on the matrix.org repository to track compiled HTML. This is done for a couple reasons, the first being we're still developing the v1 specification while concurrently making a v2 spec and the second being trying to reduce the reliance on matrix.org's repository for specifications. Because the room spec is built into versions, some changes needed to be made. The `targets.yaml` now has a special syntax for indicating what version something is at, and the changelog generator can handle rendering different versions of the same changelog (as parsed from the RST). Some additional work has been put in to the changelog parsing to allow us to reference the v1 room spec as "v1" without having to sacrifice clarity in the changelog headings. Finally, this moves the state resolution algorithms into the versioned spec as a result of MSC1759 (https://github.com/matrix-org/matrix-doc/pull/1759). Note: this does not introduce the concept of versioned schemas (tabs) that I was previously working with. There's currently no use for them, so they are shelved elsewhere. --- changelogs/rooms.rst | 11 + changelogs/rooms/newsfragments/.gitignore | 1 + changelogs/rooms/pyproject.toml | 30 +++ meta/releasing-rooms-v2.md | 38 ++++ scripts/gendoc.py | 7 +- .../templating/matrix_templates/sections.py | 34 ++-- scripts/templating/matrix_templates/units.py | 56 +++++- .../appendices/identifier_grammar.rst | 42 +--- specification/rooms/intro.rst | 70 +++++++ specification/rooms/unstable.rst | 54 +++++ specification/rooms/v1.rst | 112 +++++++++++ specification/rooms/v2.rst | 181 +++++++++++++++++ specification/server_server_api.rst | 189 +----------------- specification/targets.yaml | 29 +++ 14 files changed, 609 insertions(+), 245 deletions(-) create mode 100644 changelogs/rooms.rst create mode 100644 changelogs/rooms/newsfragments/.gitignore create mode 100644 changelogs/rooms/pyproject.toml create mode 100644 meta/releasing-rooms-v2.md create mode 100644 specification/rooms/intro.rst create mode 100644 specification/rooms/unstable.rst create mode 100644 specification/rooms/v1.rst create mode 100644 specification/rooms/v2.rst diff --git a/changelogs/rooms.rst b/changelogs/rooms.rst new file mode 100644 index 000000000..55a167901 --- /dev/null +++ b/changelogs/rooms.rst @@ -0,0 +1,11 @@ +.. version: v2 + +.. Note: We set the version as "version 2" so that we can maintain a specific version +.. variable in the changelog. We already know the next version is going to be v2, so +.. this makes it easier to copy/paste unstable.rst to v2.rst + +Room version 1 +============== +.. version: v1 + +This is the first iteration of rooms in Matrix. diff --git a/changelogs/rooms/newsfragments/.gitignore b/changelogs/rooms/newsfragments/.gitignore new file mode 100644 index 000000000..b722e9e13 --- /dev/null +++ b/changelogs/rooms/newsfragments/.gitignore @@ -0,0 +1 @@ +!.gitignore \ No newline at end of file diff --git a/changelogs/rooms/pyproject.toml b/changelogs/rooms/pyproject.toml new file mode 100644 index 000000000..b56e19a96 --- /dev/null +++ b/changelogs/rooms/pyproject.toml @@ -0,0 +1,30 @@ +[tool.towncrier] + filename = "../rooms.rst" + directory = "newsfragments" + issue_format = "`#{issue} `_" + title_format = "{version}" + + [[tool.towncrier.type]] + directory = "breaking" + name = "Breaking Changes" + showcontent = true + + [[tool.towncrier.type]] + directory = "deprecation" + name = "Deprecations" + showcontent = true + + [[tool.towncrier.type]] + directory = "new" + name = "New Endpoints" + showcontent = true + + [[tool.towncrier.type]] + directory = "feature" + name = "Backwards Compatible Changes" + showcontent = true + + [[tool.towncrier.type]] + directory = "clarification" + name = "Spec Clarifications" + showcontent = true diff --git a/meta/releasing-rooms-v2.md b/meta/releasing-rooms-v2.md new file mode 100644 index 000000000..802777493 --- /dev/null +++ b/meta/releasing-rooms-v2.md @@ -0,0 +1,38 @@ +# How to release Room Version 2 + +Room versions are a bit special for the release given they have been +introduced at v2 rather than ever getting a v1 release. Additionally, +room versions have different release requirements due to the in-project +versioning of documents rather than relying on matrix.org to maintain +old generated output of specifications. + +As of writing, the project is structured to support 3 logical versions +for rooms: v1, v2, and unstable. Unstable is currently pointed at v2 +in order to aid development. After v2 is released, unstable may wish +to be left as an independent version or similarly be pointed at a "v3" +document. + +Due to room versions being versioned in-project, the generated output +from a release is not to be sent off to matrix-doc/matrix.org. Instead, +in `gendoc.py` the default value for `--room_version` should be set to +the current release (`v2`, for example) so the index renders the right +edition in the table. + +After editing `gendoc.py`, the changelog should be generated according +to the towncrier instructions. You may need to fix the `.. version: v2` +comment located in the `rooms.rst` changelog to be just underneath the +title instead of at the end of the section. + +The `targets.yaml` file needs to be set up to point unstable to the +right set of files. Ensure that `unstable.rst` is referenced instead +of `v2.rst`, and that `unstable.rst` has appropriate contents. + +Finally, in the `intro.rst` for room versions, re-add unstable to the +list of available versions. It is currently commented out to avoid +confusing people, so it should be possible to uncomment it and put it +back into the list. + +From there, the standard process of using a release branch, tagging it, +and announcing it to the world should be followed. If required, the +various other APIs should be updated to better support the new room +version. diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 8310ad58f..0455c90ab 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -457,7 +457,7 @@ def main(targets, dest_dir, keep_intermediates, substitutions): rst_file = os.path.join(tmp_dir, "spec_%s.rst" % (target_name,)) if version_label: - d = os.path.join(dest_dir, target_name) + d = os.path.join(dest_dir, target_name.split('@')[0]) if not os.path.exists(d): os.mkdir(d) html_file = os.path.join(d, "%s.html" % version_label) @@ -529,6 +529,10 @@ if __name__ == '__main__': "--identity_release", "-i", action="store", default="unstable", help="The identity service release tag to generate, e.g. r1.2" ) + parser.add_argument( + "--room_version", "-r", action="store", default="unstable", + help="The current room version to advertise, e.g. v2" + ) parser.add_argument( "--list_targets", action="store_true", help="Do not update the specification. Instead print a list of targets.", @@ -555,6 +559,7 @@ if __name__ == '__main__': "%APPSERVICE_RELEASE_LABEL%": args.appservice_release, "%IDENTITY_RELEASE_LABEL%": args.identity_release, "%PUSH_GATEWAY_RELEASE_LABEL%": args.push_gateway_release, + "%CURRENT_ROOM_VERSION%": args.room_version, } exit (main(args.target or ["all"], args.dest, args.nodelete, substitutions)) diff --git a/scripts/templating/matrix_templates/sections.py b/scripts/templating/matrix_templates/sections.py index 185970c9f..af4976744 100644 --- a/scripts/templating/matrix_templates/sections.py +++ b/scripts/templating/matrix_templates/sections.py @@ -17,8 +17,11 @@ from batesian.sections import Sections import inspect import json import os +import logging +logger = logging.getLogger(__name__) + class MatrixSections(Sections): # pass through git ver so it'll be dropped in the input file @@ -28,26 +31,19 @@ class MatrixSections(Sections): def render_git_rev(self): return self.units.get("git_version")["revision"] - def render_client_server_changelog(self): - changelogs = self.units.get("changelogs") - return changelogs["client_server"] - - # TODO: We should make this a generic variable instead of having to add functions all the time. - def render_push_gateway_changelog(self): - changelogs = self.units.get("changelogs") - return changelogs["push_gateway"] - - def render_identity_service_changelog(self): - changelogs = self.units.get("changelogs") - return changelogs["identity_service"] - - def render_server_server_changelog(self): - changelogs = self.units.get("changelogs") - return changelogs["server_server"] - - def render_application_service_changelog(self): + def render_changelogs(self): + rendered = {} changelogs = self.units.get("changelogs") - return changelogs["application_service"] + for spec, versioned in changelogs.items(): + spec_var = "%s_changelog" % spec + logger.info("Rendering changelog for spec: %s" % spec) + for version, changelog in versioned.items(): + version_var = "%s_%s" % (spec_var, version) + logger.info("Rendering changelog for %s" % version_var) + rendered[version_var] = changelog + if version == "unstable": + rendered[spec_var] = changelog + return rendered def _render_events(self, filterFn, sortFn): template = self.env.get_template("events.tmpl") diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 11a9d441b..666434a53 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -757,6 +757,7 @@ class MatrixUnits(Units): is_ver = substitutions.get("%IDENTITY_RELEASE_LABEL%", "unstable") as_ver = substitutions.get("%APPSERVICE_RELEASE_LABEL%", "unstable") push_gw_ver = substitutions.get("%PUSH_GATEWAY_RELEASE_LABEL%", "unstable") + room_ver = substitutions.get("%CURRENT_ROOM_VERSION%", "unstable") # we abuse the typetable to return this info to the templates return TypeTable(rows=[ @@ -780,6 +781,10 @@ class MatrixUnits(Units): "`Push Gateway API `_", push_gw_ver, "Push notifications for Matrix events", + ), TypeTableRow( + "`Rooms `_", + room_ver, + "Specification for behaviour of rooms, such as event formats", ), ]) @@ -906,11 +911,26 @@ class MatrixUnits(Units): def load_changelogs(self): changelogs = {} + # Changelog generation is a bit complicated. We rely on towncrier to + # generate the unstable/current changelog, but otherwise use the RST + # edition to record historical changelogs. This is done by prepending + # the towncrier output to the RST in memory, then parsing the RST by + # hand. We parse the entire changelog to create a changelog for each + # version which may be of use in some APIs. + + # Map specific headers to specific keys that'll be used eventually + # in variables. Things not listed here will get lowercased and formatted + # such that characters not [a-z0-9] will be replaced with an underscore. + keyword_versions = { + "Unreleased Changes": "unstable" + } + + # Only generate changelogs for things that have an RST document for f in os.listdir(CHANGELOG_DIR): if not f.endswith(".rst"): continue path = os.path.join(CHANGELOG_DIR, f) - name = f[:-4] + name = f[:-4] # take off ".rst" # If there's a directory with the same name, we'll try to generate # a towncrier changelog and prepend it to the general changelog. @@ -959,15 +979,39 @@ class MatrixUnits(Units): prev_line = line else: # have title, get body (stop on next title or EOF) if re.match("^[=]{3,}$", line.strip()): - # we added the title in the previous iteration, pop it - # then bail out. - changelog_lines.pop() - break + # we hit another title, so pop the last line of + # the changelog and record the changelog + new_title = changelog_lines.pop() + if name not in changelogs: + changelogs[name] = {} + if title_part in keyword_versions: + title_part = keyword_versions[title_part] + title_part = title_part.strip().replace("^[a-zA-Z0-9]", "_").lower() + changelog = "".join(changelog_lines) + changelogs[name][title_part] = changelog + + # reset for the next version + changelog_lines = [] + title_part = new_title.strip() + continue # Don't generate subheadings (we'll keep the title though) if re.match("^[-]{3,}$", line.strip()): continue + if line.strip().startswith(".. version: "): + # The changelog is directing us to use a different title + # for the changelog. + title_part = line.strip()[len(".. version: "):] + continue + if line.strip().startswith(".. "): + continue # skip comments changelog_lines.append(" " + line + '\n') - changelogs[name] = "".join(changelog_lines) + if len(changelog_lines) > 0 and title_part is not None: + if name not in changelogs: + changelogs[name] = {} + if title_part in keyword_versions: + title_part = keyword_versions[title_part] + changelog = "".join(changelog_lines) + changelogs[name][title_part.replace("^[a-zA-Z0-9]", "_").lower()] = changelog return changelogs diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index bb5f9297c..496aba315 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -16,6 +16,12 @@ Identifier Grammar ------------------ +Some identifiers are specific to given room versions, please see the +`room specification`_ for more information. + +.. _`room specification`: ../rooms/latest.html + + Server Name ~~~~~~~~~~~ @@ -78,38 +84,6 @@ Some recommendations for a choice of server name follow: * The length of the complete server name should not exceed 230 characters. * Server names should not use upper-case characters. - -Room Versions -~~~~~~~~~~~~~ - -Room versions are used to change properties of rooms that may not be compatible -with other servers. For example, changing the rules for event authorization would -cause older servers to potentially end up in a split-brain situation due to them -not understanding the new rules. - -A room version is defined as a string of characters which MUST NOT exceed 32 -codepoints in length. Room versions MUST NOT be empty and SHOULD contain only -the characters ``a-z``, ``0-9``, ``.``, and ``-``. - -Room versions are not intended to be parsed and should be treated as opaque -identifiers. Room versions consisting only of the characters ``0-9`` and ``.`` -are reserved for future versions of the Matrix protocol. - -The complete grammar for a legal room version is:: - - room_version = 1*room_version_char - room_version_char = DIGIT - / %x61-7A ; a-z - / "-" / "." - -Examples of valid room versions are: - -* ``1`` (would be reserved by the Matrix protocol) -* ``1.2`` (would be reserved by the Matrix protocol) -* ``1.2-beta`` -* ``com.example.version`` - - Common Identifier Format ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -327,7 +301,7 @@ matrix.to navigation .. NOTE:: This namespacing is in place pending a ``matrix://`` (or similar) URI scheme. - This is **not** meant to be interpreted as an available web service - see + This is **not** meant to be interpreted as an available web service - see below for more details. Rooms, users, aliases, and groups may be represented as a "matrix.to" URI. @@ -343,7 +317,7 @@ in RFC 3986: The identifier may be a room ID, room alias, user ID, or group ID. The extra parameter is only used in the case of permalinks where an event ID is referenced. The matrix.to URI, when referenced, must always start with ``https://matrix.to/#/`` -followed by the identifier. +followed by the identifier. Clients should not rely on matrix.to URIs falling back to a web server if accessed and instead should perform some sort of action within the client. For example, if diff --git a/specification/rooms/intro.rst b/specification/rooms/intro.rst new file mode 100644 index 000000000..6231d066b --- /dev/null +++ b/specification/rooms/intro.rst @@ -0,0 +1,70 @@ +.. Copyright 2018 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +Room Specification +================== + +.. contents:: Table of Contents +.. sectnum:: + +Rooms are central to how Matrix operates, and have strict rules for what +is allowed to be contained within them. Rooms can also have various +algorithms that handle different tasks, such as what to do when two or +more events collide in the underlying DAG. To allow rooms to be improved +upon through new algorithms or rules, "room versions" are employed to +manage a set of expectations for each room. + + +Room version grammar +-------------------- + +Room versions are used to change properties of rooms that may not be compatible +with other servers. For example, changing the rules for event authorization would +cause older servers to potentially end up in a split-brain situation due to them +not understanding the new rules. + +A room version is defined as a string of characters which MUST NOT exceed 32 +codepoints in length. Room versions MUST NOT be empty and SHOULD contain only +the characters ``a-z``, ``0-9``, ``.``, and ``-``. + +Room versions are not intended to be parsed and should be treated as opaque +identifiers. Room versions consisting only of the characters ``0-9`` and ``.`` +are reserved for future versions of the Matrix protocol. + +The complete grammar for a legal room version is:: + + room_version = 1*room_version_char + room_version_char = DIGIT + / %x61-7A ; a-z + / "-" / "." + +Examples of valid room versions are: + +* ``1`` (would be reserved by the Matrix protocol) +* ``1.2`` (would be reserved by the Matrix protocol) +* ``1.2-beta`` +* ``com.example.version`` + + +Other room versions +------------------- + +The available room versions are: + +* `Version 1 `_ - The current version of most rooms. +* `Version 2 `_ - Currently in development. + +.. Note: the 'unstable' version is commented out pending a real release of rooms v2 +.. See meta/releasing-rooms-v2.md +.. * `Unstable `_ - The upcoming version of the room specification. diff --git a/specification/rooms/unstable.rst b/specification/rooms/unstable.rst new file mode 100644 index 000000000..44261814f --- /dev/null +++ b/specification/rooms/unstable.rst @@ -0,0 +1,54 @@ +.. Copyright 2018 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + + +.. DEV NOTE: This is stubbed as a template and not actually used anywhere. +.. See v2.rst for the "unstable" room version, which is currently under +.. development. +.. +.. See meta/releasing-rooms-v2.md + + +.. Note: This document appended to the end of the intro, so this next line +.. appears under "Other Room Versions". + +.. Warning:: + + This is the specification for unreleased changes to rooms. The stability + of rooms using this specification cannot be guaranteed. + + +Changelog +--------- + +.. topic:: unstable +{{rooms_changelog_unstable}} + +This version of the specification is generated from +`matrix-doc `_ as of Git commit +`{{git_version}} `_. + +For the full historical changelog, see +https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst + + +Some Module +----------- + +Lorem ipsum dolor sit amet, consectetur adipiscing elit. In sit amet +eros turpis. Quisque commodo diam vel massa ultrices, vel egestas eros +dignissim. Sed sit amet lacus eget metus auctor malesuada at ut odio. +In turpis leo, viverra et mi porttitor, condimentum bibendum dolor. + +.. {-{versioned_test_definition}-} diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst new file mode 100644 index 000000000..6c6795a4f --- /dev/null +++ b/specification/rooms/v1.rst @@ -0,0 +1,112 @@ +.. Copyright 2018 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + + +.. Note: This document appended to the end of the intro, so this next line +.. appears under "Other Room Versions". + +This is the specification for **room version 1** (``"1"``). + +Changelog +--------- + +.. topic:: Room version 1 +{{rooms_changelog_v1}} + +This version of the specification is generated from +`matrix-doc `_ as of Git commit +`{{git_version}} `_. + +For the full historical changelog, see +https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst + + +Server implementation components +-------------------------------- + +.. WARNING:: + The information contained in this section is strictly for server implementors. + Applications which use the Client-Server API are generally unaffected by the + details contained here, and can safely ignore their presence. + + +The algorithms defined here should only apply to version 1 rooms. Other algorithms +may be used by other room versions, and as such servers should be aware of which +version room they are dealing with prior to executing a given algorithm. + +.. WARNING:: + Although room version 1 is the most popular room version, it is known to have + undesirable effects. Servers implementing support for room version 1 should be + aware that restrictions should be generally relaxed and be aware that inconsistencies + may occur until room version 2 is ready and adopted. + +State resolution +~~~~~~~~~~~~~~~~ + +.. WARNING:: + This section documents the state resolution algorithm as implemented by + Synapse as of December 2017 (and therefore the de-facto Matrix protocol). + However, this algorithm is known to have some problems. + +The room state :math:`S'(E)` after an event :math:`E` is defined in terms of +the room state :math:`S(E)` before :math:`E`, and depends on whether +:math:`E` is a state event or a message event: + +* If :math:`E` is a message event, then :math:`S'(E) = S(E)`. + +* If :math:`E` is a state event, then :math:`S'(E)` is :math:`S(E)`, except + that its entry corresponding to :math:`E`'s ``event_type`` and ``state_key`` + is replaced by :math:`E`'s ``event_id``. + +The room state :math:`S(E)` before :math:`E` is the *resolution* of the set of +states :math:`\{ S'(E'), S'(E''), … \}` consisting of the states after each of +:math:`E`'s ``prev_event``\s :math:`\{ E', E'', … \}`. + +The *resolution* of a set of states is defined as follows. The resolved state +is built up in a number of passes; here we use :math:`R` to refer to the +results of the resolution so far. + +* Start by setting :math:`R` to the union of the states to be resolved, + excluding any *conflicting* events. + +* First we resolve conflicts between ``m.room.power_levels`` events. If there + is no conflict, this step is skipped, otherwise: + + * Assemble all the ``m.room.power_levels`` events from the states to + be resolved into a list. + + * Sort the list by ascending ``depth`` then descending ``sha1(event_id)``. + + * Add the first event in the list to :math:`R`. + + * For each subsequent event in the list, check that the event would be + allowed by the `authorization rules`_ for a room in state :math:`R`. If the + event would be allowed, then update :math:`R` with the event and continue + with the next event in the list. If it would not be allowed, stop and + continue below with ``m.room.join_rules`` events. + +* Repeat the above process for conflicts between ``m.room.join_rules`` events. + +* Repeat the above process for conflicts between ``m.room.member`` events. + +* No other events affect the authorization rules, so for all other conflicts, + just pick the event with the highest depth and lowest ``sha1(event_id)`` that + passes authentication in :math:`R` and add it to :math:`R`. + +A *conflict* occurs between states where those states have different +``event_ids`` for the same ``(state_type, state_key)``. The events thus +affected are said to be *conflicting* events. + + +.. _`authorization rules`: ../server_server/unstable.html#authorization-rules diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst new file mode 100644 index 000000000..a000f0568 --- /dev/null +++ b/specification/rooms/v2.rst @@ -0,0 +1,181 @@ +.. Copyright 2018 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + + +.. Note: This document appended to the end of the intro, so this next line +.. appears under "Other Room Versions". + +This is the specification for **room version 2** (``"2"``). + +.. Warning:: + + Room version 2 is under development and cannot be relied on in production + environments. + + +Changelog +--------- + +.. topic:: Room version 2 +{{rooms_changelog_v2}} + +This version of the specification is generated from +`matrix-doc `_ as of Git commit +`{{git_version}} `_. + +For the full historical changelog, see +https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst + + +Server implementation components +-------------------------------- + +.. WARNING:: + The information contained in this section is strictly for server implementors. + Applications which use the Client-Server API are generally unaffected by the + details contained here, and can safely ignore their presence. + + +The algorithms defined here should only apply to version 2 rooms. Other algorithms +may be used by other room versions, and as such servers should be aware of which +version room they are dealing with prior to executing a given algorithm. + + +State resolution +~~~~~~~~~~~~~~~~ + +The room state :math:`S'(E)` after an event :math:`E` is defined in terms of +the room state :math:`S(E)` before :math:`E`, and depends on whether +:math:`E` is a state event or a message event: + +* If :math:`E` is a message event, then :math:`S'(E) = S(E)`. + +* If :math:`E` is a state event, then :math:`S'(E)` is :math:`S(E)`, except + that its entry corresponding to :math:`E`'s ``event_type`` and ``state_key`` + is replaced by :math:`E`'s ``event_id``. + +The room state :math:`S(E)` before :math:`E` is the *resolution* of the set of +states :math:`\{ S'(E_1), S'(E_2), … \}` consisting of the states after each of +:math:`E`'s ``prev_event``\s :math:`\{ E_1, E_2, … \}`, where the resolution of +a set of states is given in the algorithm below. + +Definitions ++++++++++++ + +The state resolution algorithm for version 2 rooms uses the following +definitions, given the set of room states :math:`\{ S_1, S_2, \ldots \}`: + +Power events + A *power event* is a state event with type ``m.room.power_levels`` or + ``m.room.join_rules``, or a state event with type ``m.room.member`` where the + ``membership`` is ``leave`` or ``ban`` and the ``sender`` does not match the + ``state_key``. The idea behind this is that power events are events that have + may remove someone's ability to do something in the room. + +Unconflicted state map and conflicted state set + The *unconflicted state map* is the state where the value of each key exists + and is the same in each state :math:`S_i`. The *conflicted state set* is the + set of all other state events. Note that the unconflicted state map only has + one event per ``(event_type, state_key)``, whereas the conflicted state set + may have multiple events. + +Auth difference + The *auth difference* is calculated by first calculating the full auth chain + for each state :math:`S_i`, that is the union of the auth chains for each + event in :math:`S_i`, and then taking every event that doesn't appear in + every auth chain. If :math:`C_i` is the full auth chain of :math:`S_i`, then + the auth difference is :math:`\cup C_i - \cap C_i`. + +Full conflicted set + The *full conflicted set* is the union of the conflicted state set and the + auth difference. + +Reverse topological power ordering + The *reverse topological power ordering* of a set of events is the + lexicographically smallest topological ordering based on the DAG formed by + auth events. The reverse topological power ordering is ordered from earliest + event to latest. For comparing two topological orderings to determine which + is the lexicographically smallest, the following comparison relation on + events is used: for events :math:`x` and :math:`y`, :math:`x Date: Mon, 7 Jan 2019 13:09:21 -0700 Subject: [PATCH 008/170] Incorporate MSC1693 This is largely blatant copy/paste from the MSC with some formatting done to tidy it up a bit. --- specification/rooms/v2.rst | 43 +++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index a000f0568..b04f94b4a 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -155,7 +155,8 @@ Iterative auth checks state event is not allowed by the authorization rules, then the event is ignored. If a ``(event_type, state_key)`` key that is required for checking the authorization rules is not present in the state, then the appropriate - state event from the event's ``auth_events`` is used. + state event from the event's ``auth_events`` is used if the auth event is + not rejected. Algorithm +++++++++ @@ -179,3 +180,43 @@ The *resolution* of a set of states is obtained as follows: .. _`authorization rules`: ../server_server/unstable.html#authorization-rules + +Rejected events ++++++++++++++++ + +Events that have been rejected due to failing auth based on the state at the +event (rather than based on their auth chain) are handled as usual by the +algorithm, unless otherwise specified. + +Note that no events rejected due to failure to auth against their auth chain +should appear in the process, as they should not appear in state (the algorithm +only uses events that appear in either the state sets or in the auth chain of +the events in the state sets). + +.. admonition:: Rationale + + This helps ensure that different servers' view of state is more likely to + converge, since rejection state of an event may be different. This can happen if + a third server gives an incorrect version of the state when a server joins a + room via it (either due to being faulty or malicious). Convergence of state is a + desirable property as it ensures that all users in the room have a (mostly) + consistent view of the state of the room. If the view of the state on different + servers diverges it can lead to bifurcation of the room due to e.g. servers + disagreeing on who is in the room. + + Intuitively, using rejected events feels dangerous, however: + + 1. Servers cannot arbitrarily make up state, since they still need to pass the + auth checks based on the event's auth chain (e.g. they can't grant themselves + power levels if they didn't have them before). + 2. For a previously rejected event to pass auth there must be a set of state + that allows said event. A malicious server could therefore produce a + fork where it claims the state is that particular set of state, duplicate the + rejected event to point to that fork, and send the event. The + duplicated event would then pass the auth checks. Ignoring rejected events + would therefore not eliminate any potential attack vectors. + + +Rejected auth events are deliberately excluded from use in the iterative auth +checks, as auth events aren't re-authed (although non-auth events are) during +the iterative auth checks. From ccc1cdaeadd5a207cc3f23c938e46512e994204b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 7 Jan 2019 14:38:21 -0700 Subject: [PATCH 009/170] Add support for unstable feature advertising via /versions Incorporates https://github.com/matrix-org/matrix-doc/issues/1497 --- api/client-server/versions.yaml | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/api/client-server/versions.yaml b/api/client-server/versions.yaml index d3aa03594..4d0fd49e8 100644 --- a/api/client-server/versions.yaml +++ b/api/client-server/versions.yaml @@ -33,13 +33,25 @@ paths: Only the latest ``Z`` value will be reported for each supported ``X.Y`` value. i.e. if the server implements ``r0.0.0``, ``r0.0.1``, and ``r1.2.0``, it will report ``r0.0.1`` and ``r1.2.0``. + + The server may additionally advertise experimental features it supports + through ``unstable_features``. These features should be namespaced and + may optionally include version information within their name if desired. + Features listed here are not for optionally toggling parts of the Matrix + specification and should only be used to advertise support for a feature + which has not yet landed in the spec. For example, an accepted proposal + for a feature needing implementation would advertise itself here with + the intention of being removed from this list once the spec changes land. operationId: getVersions responses: 200: description: The versions supported by the server. examples: application/json: { - "versions": ["r0.0.1"] + "versions": ["r0.0.1"], + "unstable_features": { + "org.example.my_feature": true + } } schema: type: object @@ -50,5 +62,15 @@ paths: items: type: string description: The supported versions + unstable_features: + type: object + description: |- + Experimental features the server supports. Features not listed here, + or the lack of this property all together, indicate that a feature is + not supported. + additionalProperties: + type: boolean + description: Whether or not the namespaced feature is supported. + required: ['versions'] tags: - Server administration From 75c084e987d2418f0ca6b0a3a0d321cd49720ecb Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 7 Jan 2019 14:50:55 -0700 Subject: [PATCH 010/170] changelog --- changelogs/client_server/newsfragments/1786.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1786.feature diff --git a/changelogs/client_server/newsfragments/1786.feature b/changelogs/client_server/newsfragments/1786.feature new file mode 100644 index 000000000..6f21778c0 --- /dev/null +++ b/changelogs/client_server/newsfragments/1786.feature @@ -0,0 +1 @@ +Add support for advertising experimental features to clients. From f33a540e6dbc6287b03c3633f09144a3266c26b4 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 8 Jan 2019 00:25:06 +0000 Subject: [PATCH 011/170] Do a SRV lookup before .well-known lookup also other clarifications and corrections. --- proposals/1708-well-known-for-federation.md | 133 ++++++++++---------- 1 file changed, 69 insertions(+), 64 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 8fa66c7a9..6e857b854 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -1,21 +1,17 @@ # MSC1708: .well-known support for server name resolution Currently, mapping from a server name to a hostname for federation is done via -`SRV` records. This presents two principal difficulties: - - * SRV records are not widely used, and administrators may be unfamiliar with - them, and there may be other practical difficulties in their deployment such - as poor support from hosting providers. [^1] - - * [MSC1711](https://github.com/matrix-org/matrix-doc/pull/1711) proposes - requiring valid X.509 certificates on the - federation endpoint. It will then be necessary for the homeserver to present - a certificate which is valid for the server name. This presents difficulties - for hosted server offerings: BigCorp may be reluctant to hand over the - keys for `bigcorp.com` to the administrators of the `bigcorp.com` matrix - homeserver. - -Here we propose to solve these problems by augmenting the current `SRV` record +`SRV` records. However, +[MSC1711](https://github.com/matrix-org/matrix-doc/pull/1711) proposes +requiring valid X.509 certificates on the federation endpoint. It will then be +necessary for the homeserver to present a certificate which is valid for the +server name. This presents difficulties for hosted server offerings: BigCorp +may want to delegate responsibility for running its Matrix homeserver to an +outside supplier, but it may be difficult for that supplier to obtain a TLS +certificate for `bigcorp.com` (and BigCorp may be reluctant to let them have +one). + +This MSC proposes to solve this problem by augmenting the current `SRV` record with a `.well-known` lookup. ## Proposal @@ -24,59 +20,80 @@ For reference, the current [specification for resolving server names](https://matrix.org/docs/spec/server_server/unstable.html#resolving-server-names) is as follows: -* If the hostname is an IP literal, then that IP address should be used, - together with the given port number, or 8448 if no port is given. +1. If the hostname is an IP literal, then that IP address should be used, + together with the given port number, or 8448 if no port is given. + +2. Otherwise, if the port is present, then an IP address is discovered by + looking up an AAAA or A record for the hostname, and the specified port is + used. + +3. If the hostname is not an IP literal and no port is given, the server is + discovered by first looking up a `_matrix._tcp` SRV record for the + hostname, which may give a hostname (to be looked up using AAAA or A queries) + and port. + +4. Finally, the server is discovered by looking up an AAAA or A record on the + hostname, and taking the default fallback port number of 8448. + +We insert the following between Steps 3 and 4: + +If the SRV record does not exist, the requesting server should make a `GET` +request to `https:///.well-known/matrix/server`, with normal +X.509 certificate validation. If the request does not return a 200, continue +to step 4, otherwise: + +XXX: should we follow redirects? -* Otherwise, if the port is present, then an IP address is discovered by - looking up an AAAA or A record for the hostname, and the specified port is - used. +The response must have a `Content-Type` of `application/json`, and must be +valid JSON which follows the structure documented below. Otherwise, the +request is aborted. -* If the hostname is not an IP literal and no port is given, the server is - discovered by first looking up a `_matrix._tcp` SRV record for the - hostname, which may give a hostname (to be looked up using AAAA or A queries) - and port. If the SRV record does not exist, then the server is discovered by - looking up an AAAA or A record on the hostname and taking the default - fallback port number of 8448. +If the response is valid, the `m.server` property is parsed as +`[:]`, and processed as follows: - Homeservers may use SRV records to load balance requests between multiple TLS - endpoints or to failover to another endpoint if an endpoint fails. + a. If `` is an IP literal, then that IP address should + be used, together with ``, or 8448 if no port is + given. The server should present a valid TLS certificate for + ``. -The first two points remain unchanged: if the server name is an IP literal, or -contains a port, then requests will be made directly as before. + b. Otherwise, if the port is present, then an IP address is discovered by + looking up an AAAA or A record for ``, and the + specified port is used. The server should present a valid TLS certificate + for ``. -If the hostname is neither an IP literal, nor does it have an explicit port, -then the requesting server should continue to make an SRV lookup as before, and -use the result if one is found. + (In other words, the federation connection is made to + `https://:`). -If *no* SRV result is found, the requesting server should make a `GET` request -to `https://\/.well-known/matrix/server`, with normal X.509 -certificate validation. If the request fails in any way, then we fall back as -before to using using port 8448 on the hostname. + c. If the hostname is not an IP literal and no port is given, a second SRV + record is looked up; this time for `_matrix._tcp.`, + which may give yet another hostname (to be looked up using A/AAAA queries) + and port. The server must present a TLS cert for the + `` from the .well-known. -Rationale: Falling back to port 8448 (rather than aborting the request) is -necessary to maintain compatibility with existing deployments, which may not -present valid certificates on port 443, or may return 4xx or 5xx errors. + d. If no SRV record is found, the server is discovered by looking up an AAAA + or A record on ``, and taking the default fallback + port number of 8448. -If the GET request succeeds, it should result in a JSON response, with contents -structured as shown: + (In other words, the federation connection is made to + `https://:8448`). + +### Structure of the `.well-known` response + +The contents of the `.well-known` response should be structured as shown: ```json { - "server": "[:]" + "m.server": "[:]" } ``` -The `server` property should be a hostname or IP address, followed by an +The `m.server` property should be a hostname or IP address, followed by an optional port. If the response cannot be parsed as JSON, or lacks a valid `server` property, the request is considered to have failed, and no fallback to port 8448 takes place. -Otherwise, the requesting server performs an `AAAA/A` lookup on the hostname -(if necessary), and connects to the resultant address and the specifed -port. The port defaults to 8448, if unspecified. - (The formal grammar for the `server` property is identical to that of a [server name](https://matrix.org/docs/spec/appendices.html#server-name).) @@ -92,18 +109,10 @@ sensible default: 24 hours is suggested. Because there is no way to request a revalidation, it is also recommended that requesting servers cap the expiry time. 48 hours is suggested. -Similarly, a failure to retrieve the `.well-known` file should be cached for -a reasonable period. 24 hours is suggested again. - -### The future of SRV records - -It's worth noting that this proposal is very clear in that we will maintain -support for SRV records for the immediate future; there are no current plans to -deprecate them. - -However, clearly a `.well-known` file can provide much of the functionality of -an SRV record, and having to support both may be undesirable. Accordingly, we -may consider sunsetting SRV record support at some point in the future. +A failure to retrieve the `.well-known` file should also be cached, though care +must be taken that a single 500 error or connection failure should not break +federation for an extended period. A short cache time of about an hour might be +appropriate; alternatively, servers might use an exponential backoff. ### Outstanding questions @@ -127,7 +136,6 @@ as soon as possible, to maximise uptake in the ecosystem. It is likely that, as we approach Matrix 1.0, there will be sufficient other new features (such as new Room versions) that upgrading will be necessary anyway. - ## Security considerations The `.well-known` file potentially broadens the attack surface for an attacker @@ -138,6 +146,3 @@ wishing to intercept federation traffic to a particular server. This proposal adds a new mechanism, alongside the existing `SRV` record lookup for finding the server responsible for a particular matrix server_name, which will allow greater flexibility in deploying homeservers. - - -[^1] For example, Cloudflare automatically "flattens" SRV record responses. From fb171cadf48def72258433d2f19fd7d17df03c9d Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 8 Jan 2019 12:51:02 +0000 Subject: [PATCH 012/170] formatting fix --- proposals/1708-well-known-for-federation.md | 50 ++++++++++----------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 6e857b854..14ba3bf6a 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -51,31 +51,31 @@ request is aborted. If the response is valid, the `m.server` property is parsed as `[:]`, and processed as follows: - a. If `` is an IP literal, then that IP address should - be used, together with ``, or 8448 if no port is - given. The server should present a valid TLS certificate for - ``. - - b. Otherwise, if the port is present, then an IP address is discovered by - looking up an AAAA or A record for ``, and the - specified port is used. The server should present a valid TLS certificate - for ``. - - (In other words, the federation connection is made to - `https://:`). - - c. If the hostname is not an IP literal and no port is given, a second SRV - record is looked up; this time for `_matrix._tcp.`, - which may give yet another hostname (to be looked up using A/AAAA queries) - and port. The server must present a TLS cert for the - `` from the .well-known. - - d. If no SRV record is found, the server is discovered by looking up an AAAA - or A record on ``, and taking the default fallback - port number of 8448. - - (In other words, the federation connection is made to - `https://:8448`). +a. If `` is an IP literal, then that IP address should + be used, together with ``, or 8448 if no port is + given. The server should present a valid TLS certificate for + ``. + +b. Otherwise, if the port is present, then an IP address is discovered by + looking up an AAAA or A record for ``, and the + specified port is used. The server should present a valid TLS certificate + for ``. + + (In other words, the federation connection is made to + `https://:`). + +c. If the hostname is not an IP literal and no port is given, a second SRV + record is looked up; this time for `_matrix._tcp.`, + which may give yet another hostname (to be looked up using A/AAAA queries) + and port. The server must present a TLS cert for the + `` from the .well-known. + +d. If no SRV record is found, the server is discovered by looking up an AAAA + or A record on ``, and taking the default fallback + port number of 8448. + + (In other words, the federation connection is made to + `https://:8448`). ### Structure of the `.well-known` response From f1ebbc358bd873988b2e987e32c53105a0e55293 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 8 Jan 2019 12:44:22 +0000 Subject: [PATCH 013/170] document dismissed options --- proposals/1708-well-known-for-federation.md | 67 +++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 14ba3bf6a..b100ef38c 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -141,6 +141,73 @@ new Room versions) that upgrading will be necessary anyway. The `.well-known` file potentially broadens the attack surface for an attacker wishing to intercept federation traffic to a particular server. +## Dismissed alternatives + +For future reference, here are the alternative solutions which have been +considered and dismissed. + +### Look up the `.well-known` file before the SRV record + +We could make the request for `.well-known` before looking up the `SRV` +record. On the one hand this is maybe marginally simpler (and avoids the +overhead of having to make *two* `SRV` lookups in the case that a `.well-known` +is found. It might also open a future path for using `.well-known` for +information other than delegation. + +Ultimately we decided to include the initial `SRV` lookup so that deployments +have a mechanism to avoid the `.well-known` overhead in the common case that it +is not required. + +### Subdomain hack + +As well as accepting TLS certs for `example.com`, we could also accept them for +`delegated--matrix.example.com`. This would allow `example.com` to delegate its +matrix hosting by (a) setting up the SRV record at `_matrix._tcp.example.com` +and (b) setting up a CNAME at `delegated--matrix.example.com`. The latter would +enable the delegatee to obtain an acceptable TLS certificate. + +This was certainly an interesting idea, but we dismissed it for the following +reasons: + +* There's a security trap for anybody who lets people sign up for subdomains + (which is certainly not an uncommon business model): if you can register for + delegated--matrix.example.com, you get to intercept all the matrix traffic + for example.com. + +* Generally it feels quite unintuitive and violates the principle of least + surprise. + +* The fact that we can't find any prior art for this sets off alarm bells too. + +### Rely on DNS/DNSSEC + +If we could trust SRV records, we would be able to accept TLS certs for the +*target* of the SRV record, which avoids this whole problem. + +Such trust could come from assuming that plain DNS is "good enough". However, +DNS cache poisoning attacks are a real thing, and the fact that the designers +of TLS chose to implement a server-name check specifically to deal with this +case suggests we would be foolish to make this assumption. + +The alternative is to rely on DNSSEC to provide security for SRV records. The +problem here is simply that DNSSEC is not that widely deployed currently. A +number of large organisations are actively avoiding enabling it on their +domains, so requiring DNSSEC would be a direct impediment to the uptake of +Matrix. Furthermore, if we required DNSSEC-authenticated SRV records for +domains doing delegation, we would end up with a significant number of +homeservers unable to talk to such domains, because their local DNS +infrastructure may not implement DNSSEC. + +Finally, if we're expecting servers to present the cert for the *target* of the +SRV record, then we'll have to change the Host and SNI fields, and that will +break backwards compat everywhere (and it's hard to see how to mitigate that). + +### Stick with perspectives + +The final option is to double-down on the Perspectives approach, ie to skip +[MSC1711](https://github.com/matrix-org/matrix-doc/pull/1711). MSC1711 +discusses the reasons we do not believe this to be a viable option. + ## Conclusion This proposal adds a new mechanism, alongside the existing `SRV` record lookup From 5812450299fc88c58acf284e9f911152e2484812 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 8 Jan 2019 12:50:42 +0000 Subject: [PATCH 014/170] spec that we follow redirects --- proposals/1708-well-known-for-federation.md | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index b100ef38c..5981ab9f7 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -35,14 +35,12 @@ is as follows: 4. Finally, the server is discovered by looking up an AAAA or A record on the hostname, and taking the default fallback port number of 8448. -We insert the following between Steps 3 and 4: +We insert the following between Steps 3 and 4. If the SRV record does not exist, the requesting server should make a `GET` -request to `https:///.well-known/matrix/server`, with normal -X.509 certificate validation. If the request does not return a 200, continue -to step 4, otherwise: - -XXX: should we follow redirects? +request to `https:///.well-known/matrix/server`, with normal X.509 +certificate validation, and following 30x redirects. If the request does not +return a 200, continue to step 4, otherwise: The response must have a `Content-Type` of `application/json`, and must be valid JSON which follows the structure documented below. Otherwise, the @@ -114,14 +112,6 @@ must be taken that a single 500 error or connection failure should not break federation for an extended period. A short cache time of about an hour might be appropriate; alternatively, servers might use an exponential backoff. -### Outstanding questions - -Should we follow 30x redirects for the .well-known file? On the one hand, there -is no obvious usecase and they add complexity (for example: how do they -interact with caches?). On the other hand, we'll presumably be using an HTTP -client library to handle some of the caching stuff, and they might be useful -for something? - ## Problems It will take a while for `.well-known` to be supported across the ecosystem; From b541c2a247d6b849031574e3151ca44427c4cdb0 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 8 Jan 2019 12:54:52 +0000 Subject: [PATCH 015/170] more formatting --- proposals/1708-well-known-for-federation.md | 49 ++++++++++----------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 5981ab9f7..98cfe8d82 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -49,31 +49,30 @@ request is aborted. If the response is valid, the `m.server` property is parsed as `[:]`, and processed as follows: -a. If `` is an IP literal, then that IP address should - be used, together with ``, or 8448 if no port is - given. The server should present a valid TLS certificate for - ``. - -b. Otherwise, if the port is present, then an IP address is discovered by - looking up an AAAA or A record for ``, and the - specified port is used. The server should present a valid TLS certificate - for ``. - - (In other words, the federation connection is made to - `https://:`). - -c. If the hostname is not an IP literal and no port is given, a second SRV - record is looked up; this time for `_matrix._tcp.`, - which may give yet another hostname (to be looked up using A/AAAA queries) - and port. The server must present a TLS cert for the - `` from the .well-known. - -d. If no SRV record is found, the server is discovered by looking up an AAAA - or A record on ``, and taking the default fallback - port number of 8448. - - (In other words, the federation connection is made to - `https://:8448`). +* If `` is an IP literal, then that IP address should be + used, together with ``, or 8448 if no port is given. The + server should present a valid TLS certificate for ``. + +* Otherwise, if the port is present, then an IP address is discovered by + looking up an AAAA or A record for ``, and the + specified port is used. The server should present a valid TLS certificate + for ``. + + (In other words, the federation connection is made to + `https://:`). + +* If the hostname is not an IP literal and no port is given, a second SRV + record is looked up; this time for `_matrix._tcp.`, + which may give yet another hostname (to be looked up using A/AAAA queries) + and port. The server must present a TLS cert for the + `` from the .well-known. + +* If no SRV record is found, the server is discovered by looking up an AAAA + or A record on ``, and taking the default fallback + port number of 8448. + + (In other words, the federation connection is made to + `https://:8448`). ### Structure of the `.well-known` response From 3e7a5f5ea4841e96508f6eed7e950736a714abc9 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 9 Jan 2019 00:09:38 -0700 Subject: [PATCH 016/170] Initial draft for SSO support --- api/client-server/sso_login_redirect.yaml | 46 ++++++++++ specification/client_server_api.rst | 12 ++- specification/modules/sso_login.rst | 100 ++++++++++++++++++++++ specification/targets.yaml | 1 + 4 files changed, 158 insertions(+), 1 deletion(-) create mode 100644 api/client-server/sso_login_redirect.yaml create mode 100644 specification/modules/sso_login.rst diff --git a/api/client-server/sso_login_redirect.yaml b/api/client-server/sso_login_redirect.yaml new file mode 100644 index 000000000..acbafc578 --- /dev/null +++ b/api/client-server/sso_login_redirect.yaml @@ -0,0 +1,46 @@ +# Copyright 2019 New Vector Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +swagger: '2.0' +info: + title: "Matrix Client-Server SSO Login API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/%CLIENT_MAJOR_VERSION% +paths: + "/login/sso/redirect": + get: + summary: Redirect the user's browser to the SSO interface. + description: |- + A web-based Matrix client should instruct the user's browser to + navigate to this endpoint in order to log in via SSO. + + The server MUST respond with an HTTP redirect to the SSO interface. + operationId: redirectToSSO + parameters: + - in: query + type: string + name: redirectUrl + description: |- + URI to which the user will be redirected after the homeserver has + authenticated the user with SSO. + required: true + responses: + 302: + description: A redirect to the SSO interface. + headers: + Location: + type: "string" diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 82a576f14..973d3b551 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1016,9 +1016,19 @@ follows: } As with `token-based`_ interactive login, the ``token`` must encode the -user id. In the case that the token is not valid, the homeserver must respond +user ID. In the case that the token is not valid, the homeserver must respond with ``403 Forbidden`` and an error code of ``M_FORBIDDEN``. +To log in with through a Central Authentication Service (CAS) or via Single +Sign-On (SSO), clients should first make a request to ``GET /login`` to ensure +the homeserver supports the appropriate login type. Clients should use the +`CAS endpoints`_ to complete logins for ``m.login.cas`` and the `SSO endpoints`_ +for ``m.login.sso``. In either case, the client is expected to redirect the user +to the appropriate ``/redirect`` endpoint. + +.. _`CAS endpoints`: #cas-based-client-login +.. _`SSO endpoints`: #sso-based-client-login + {{login_cs_http_api}} {{logout_cs_http_api}} diff --git a/specification/modules/sso_login.rst b/specification/modules/sso_login.rst new file mode 100644 index 000000000..a493c851a --- /dev/null +++ b/specification/modules/sso_login.rst @@ -0,0 +1,100 @@ +.. Copyright 2019 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +SSO-based client login +====================== + +.. _module:sso_login: + +Single Sign-On (SSO) is a generic web-based protocol for logging in users. +This section is the more generic version of `CAS-based client login`_ and +is applicable to a wider range of SSO protocols. + +An overview of the process, as used in Matrix, is as follows: + +1. The Matrix client instructs the user's browser to navigate to the + |/login/sso/redirect|_ endpoint on the user's homeserver. + +2. The homeserver responds with an HTTP redirect to the SSO user interface, + which the browser follows. + +3. The SSO system authenticates the user. + +4. The SSO server and the homeserver interact to verify the user's identity + and other authentication information, potentially using a number of redirects. + +7. The Matrix client receives the login token and passes it to the |/login|_ + API. + +Client behaviour +---------------- + +The client starts the process by instructing the browser to navigate to +|/login/sso/redirect|_ with an appropriate ``redirectUrl``. Once authentication +is successful, the browser will be redirected to that ``redirectUrl``. + +.. TODO-spec + + Should we recommend some sort of CSRF protection here (specifically, we + should guard against people accidentally logging in by sending them a link + to ``/login/sso/redirect``. + + Maybe we should recommend that the ``redirectUrl`` should contain a CSRF + token which the client should then check before sending the login token to + ``/login``? + +{{sso_login_redirect_cs_http_api}} + +Server behaviour +---------------- + +The URI for the SSO system to be used should be configured on the server by the +server administrator. The server is expected to set up any endpoints required to +interact with that SSO system. + +Handling the redirect endpoint +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When responding to the ``/login/sso/redirect`` endpoint, the server must +generate a URI for the SSO login page with any appropriate parameters. + +.. TODO-spec: + + It might be nice if the server did some validation of the ``redirectUrl`` + parameter, so that we could check that aren't going to redirect to a non-TLS + endpoint, and to give more meaningful errors in the case of + faulty/poorly-configured clients. + +Handling the authentication endpoint +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once the homeserver has verified the user's identity with the SSO system, it +MUST map the user ID to a valid `Matrix user identifier <../index.html#user-identifiers>`_. +The guidance in `Mapping from other character sets +<../index.html#mapping-from-other-character-sets>`_ may be useful. + +If the generated user identifier represents a new user, it should be registered +as a new user. + +Finally, the server should generate a short-term login token. The generated +token should be a macaroon, suitable for use with the ``m.login.token`` type of +the |/login|_ API, and `token-based interactive login <#token-based>`_. The +lifetime of this token SHOULD be limited to around five seconds. + + +.. |/login| replace:: ``/login`` +.. _/login: #post-matrix-client-%CLIENT_MAJOR_VERSION%-login +.. |/login/sso/redirect| replace:: ``/login/sso/redirect`` +.. _/login/sso/redirect: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-sso-redirect +.. _`CAS-based client login`: #cas-based-client-login diff --git a/specification/targets.yaml b/specification/targets.yaml index 93e1b8a6b..493814acf 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -62,6 +62,7 @@ groups: # reusable blobs of files when prefixed with 'group:' - modules/admin.rst - modules/event_context.rst - modules/cas_login.rst + - modules/sso_login.rst - modules/dm.rst - modules/ignore_users.rst - modules/stickers.rst From c10394d03f9f82533c8db0a823052df6ba23453d Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 9 Jan 2019 11:26:14 +0000 Subject: [PATCH 017/170] Clarifications thanks to @uhoreg --- proposals/1708-well-known-for-federation.md | 23 ++++++++++----------- 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 98cfe8d82..8105a6381 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -39,8 +39,9 @@ We insert the following between Steps 3 and 4. If the SRV record does not exist, the requesting server should make a `GET` request to `https:///.well-known/matrix/server`, with normal X.509 -certificate validation, and following 30x redirects. If the request does not -return a 200, continue to step 4, otherwise: +certificate validation, and following 30x redirects (being careful to avoid +redirect loops). If the request does not return a 200, continue to step 4, +otherwise: The response must have a `Content-Type` of `application/json`, and must be valid JSON which follows the structure documented below. Otherwise, the @@ -53,10 +54,10 @@ If the response is valid, the `m.server` property is parsed as used, together with ``, or 8448 if no port is given. The server should present a valid TLS certificate for ``. -* Otherwise, if the port is present, then an IP address is discovered by - looking up an AAAA or A record for ``, and the - specified port is used. The server should present a valid TLS certificate - for ``. +* If `` is not an IP literal, and `` is + present, then an IP address is discovered by looking up an AAAA or A record + for ``, and the specified port is used. The server + should present a valid TLS certificate for ``. (In other words, the federation connection is made to `https://:`). @@ -84,15 +85,13 @@ The contents of the `.well-known` response should be structured as shown: } ``` -The `m.server` property should be a hostname or IP address, followed by an -optional port. - -If the response cannot be parsed as JSON, or lacks a valid `server` property, +If the response cannot be parsed as JSON, or lacks a valid `m.server` property, the request is considered to have failed, and no fallback to port 8448 takes place. -(The formal grammar for the `server` property is identical to that of a [server -name](https://matrix.org/docs/spec/appendices.html#server-name).) +The formal grammar for the `m.server` property is the same as that of a [server +name](https://matrix.org/docs/spec/appendices.html#server-name): it is a +hostname or IP address, followed by an optional port. ### Caching From d6c33ea0a5d5ea17cfe54a2aa4828831a5025042 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 9 Jan 2019 14:41:46 -0700 Subject: [PATCH 018/170] Make CAS a subset of SSO --- specification/client_server_api.rst | 19 ++--- specification/modules/cas_login.rst | 119 ---------------------------- specification/modules/sso_login.rst | 97 +++++++++++++++++++++-- specification/targets.yaml | 1 - 4 files changed, 102 insertions(+), 134 deletions(-) delete mode 100644 specification/modules/cas_login.rst diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 973d3b551..03d712460 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1019,15 +1019,16 @@ As with `token-based`_ interactive login, the ``token`` must encode the user ID. In the case that the token is not valid, the homeserver must respond with ``403 Forbidden`` and an error code of ``M_FORBIDDEN``. -To log in with through a Central Authentication Service (CAS) or via Single -Sign-On (SSO), clients should first make a request to ``GET /login`` to ensure -the homeserver supports the appropriate login type. Clients should use the -`CAS endpoints`_ to complete logins for ``m.login.cas`` and the `SSO endpoints`_ -for ``m.login.sso``. In either case, the client is expected to redirect the user -to the appropriate ``/redirect`` endpoint. - -.. _`CAS endpoints`: #cas-based-client-login -.. _`SSO endpoints`: #sso-based-client-login +If the homeserver advertises ``m.login.sso`` as a viable flow, and the client +supports it, the client should redirect the user to the ``/redirect`` endpoint +for `Single Sign-On <#sso-client-login>`_. After authentication is complete, the +client will need to submit a ``/login`` request matching ``m.login.token``. + +If the homeserver advertises ``m.login.cas`` as a viable flow, and the client +supports it, the client should redirect the user to the ``/redirect`` endpoint +for `CAS <#cas-based-client-login>`_. Just like SSO authentication, the client +is expected to submit a ``/login`` request matching ``m.login.token`` upon +successful authentication. {{login_cs_http_api}} diff --git a/specification/modules/cas_login.rst b/specification/modules/cas_login.rst deleted file mode 100644 index 5de980575..000000000 --- a/specification/modules/cas_login.rst +++ /dev/null @@ -1,119 +0,0 @@ -.. Copyright 2016 OpenMarket Ltd -.. -.. Licensed under the Apache License, Version 2.0 (the "License"); -.. you may not use this file except in compliance with the License. -.. You may obtain a copy of the License at -.. -.. http://www.apache.org/licenses/LICENSE-2.0 -.. -.. Unless required by applicable law or agreed to in writing, software -.. distributed under the License is distributed on an "AS IS" BASIS, -.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -.. See the License for the specific language governing permissions and -.. limitations under the License. - -CAS-based client login -====================== - -.. _module:cas_login: - -`Central Authentication Service -`_ -(CAS) is a web-based single sign-on protocol. - -An overview of the process, as used in Matrix, is as follows: - -1. The Matrix client instructs the user's browser to navigate to the - |/login/cas/redirect|_ endpoint on the user's homeserver. - -2. The homeserver responds with an HTTP redirect to the CAS user interface, - which the browser follows. - -3. The CAS system authenticates the user. - -4. The CAS server responds to the user's browser with a redirect back to the - |/login/cas/ticket|_ endpoint on the homeserver, which the browser - follows. A 'ticket' identifier is passed as a query parameter in the - redirect. - -5. The homeserver receives the ticket ID from the user's browser, and makes a - request to the CAS server to validate the ticket. - -6. Having validated the ticket, the homeserver responds to the browser with a - third HTTP redirect, back to the Matrix client application. A login token - is passed as a query parameter in the redirect. - -7. The Matrix client receives the login token and passes it to the |/login|_ - API. - -Client behaviour ----------------- - -The client starts the process by instructing the browser to navigate to -|/login/cas/redirect|_ with an appropriate ``redirectUrl``. Once authentication -is successful, the browser will be redirected to that ``redirectUrl``. - -.. TODO-spec - - Should we recommend some sort of CSRF protection here (specifically, we - should guard against people accidentally logging in by sending them a link - to ``/login/cas/redirect``. - - Maybe we should recommend that the ``redirectUrl`` should contain a CSRF - token which the client should then check before sending the login token to - ``/login``? - -{{cas_login_redirect_cs_http_api}} -{{cas_login_ticket_cs_http_api}} - -Server behaviour ----------------- - -The URI for the CAS system to be used should be configured on the server by the -server administrator. - -Handling the redirect endpoint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When responding to the ``/login/cas/redirect`` endpoint, the server must -generate a URI for the CAS login page. The server should take the base CAS URI -described above, and add a ``service`` query parameter. This parameter MUST be -the URI of the ``/login/cas/ticket`` endpoint, including the ``redirectUrl`` -query parameter. Because the homeserver may not know its base URI, this may -also require manual configuration. - -.. TODO-spec: - - It might be nice if the server did some validation of the ``redirectUrl`` - parameter, so that we could check that aren't going to redirect to a non-TLS - endpoint, and to give more meaningful errors in the case of - faulty/poorly-configured clients. - -Handling the authentication endpoint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When responding to the ``/login/cas/ticket`` endpoint, the server MUST make a -request to the CAS server to validate the provided ticket. The server MAY also -check for certain user attributes in the response. Any required attributes -should be configured by the server administrator. - -Once the ticket has been validated, the server MUST map the CAS ``user_id`` -to a valid `Matrix user identifier <../index.html#user-identifiers>`_. The -guidance in `Mapping from other character sets -<../index.html#mapping-from-other-character-sets>`_ may be useful. - -If the generated user identifier represents a new user, it should be registered -as a new user. - -Finally, the server should generate a short-term login token. The generated -token should be a macaroon, suitable for use with the ``m.login.token`` type of -the |/login|_ API, and `token-based interactive login <#token-based>`_. The -lifetime of this token SHOULD be limited to around five seconds. - - -.. |/login| replace:: ``/login`` -.. _/login: #post-matrix-client-%CLIENT_MAJOR_VERSION%-login -.. |/login/cas/redirect| replace:: ``/login/cas/redirect`` -.. _/login/cas/redirect: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-cas-redirect -.. |/login/cas/ticket| replace:: ``/login/cas/ticket`` -.. _/login/cas/ticket: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-cas-ticket diff --git a/specification/modules/sso_login.rst b/specification/modules/sso_login.rst index a493c851a..977278a7e 100644 --- a/specification/modules/sso_login.rst +++ b/specification/modules/sso_login.rst @@ -12,14 +12,14 @@ .. See the License for the specific language governing permissions and .. limitations under the License. -SSO-based client login -====================== +SSO client login +================ .. _module:sso_login: Single Sign-On (SSO) is a generic web-based protocol for logging in users. -This section is the more generic version of `CAS-based client login`_ and -is applicable to a wider range of SSO protocols. +As a special case, the Matrix specification supports `CAS-based login <#cas-based-client-login>`_ +as well as a subset of SSO login. An overview of the process, as used in Matrix, is as follows: @@ -93,8 +93,95 @@ the |/login|_ API, and `token-based interactive login <#token-based>`_. The lifetime of this token SHOULD be limited to around five seconds. +CAS-based client login +---------------------- + +.. _module:cas_login: + +`Central Authentication Service +`_ +(CAS) is a web-based single sign-on protocol. The protocol defined here is an +extension of the SSO protocol defined in this module: it is largely the same, +but has some specific details which make it different. + +An overview of the process, as used in Matrix, is as follows: + +1. The Matrix client instructs the user's browser to navigate to the + |/login/cas/redirect|_ endpoint on the user's homeserver. + +2. The homeserver responds with an HTTP redirect to the CAS user interface, + which the browser follows. + +3. The CAS system authenticates the user. + +4. The CAS server responds to the user's browser with a redirect back to the + |/login/cas/ticket|_ endpoint on the homeserver, which the browser + follows. A 'ticket' identifier is passed as a query parameter in the + redirect. + +5. The homeserver receives the ticket ID from the user's browser, and makes a + request to the CAS server to validate the ticket. + +6. Having validated the ticket, the homeserver responds to the browser with a + third HTTP redirect, back to the Matrix client application. A login token + is passed as a query parameter in the redirect. + +7. The Matrix client receives the login token and passes it to the |/login|_ + API. + +Client behaviour +~~~~~~~~~~~~~~~~ + +The client starts the process by instructing the browser to navigate to +|/login/cas/redirect|_ with an appropriate ``redirectUrl``. Once authentication +is successful, the browser will be redirected to that ``redirectUrl``. + +{{cas_login_redirect_cs_http_api}} +{{cas_login_ticket_cs_http_api}} + +Server behaviour +~~~~~~~~~~~~~~~~ + +The URI for the CAS system to be used should be configured on the server by the +server administrator. + +Handling the redirect endpoint +++++++++++++++++++++++++++++++ + +When responding to the ``/login/cas/redirect`` endpoint, the server must +generate a URI for the CAS login page. The server should take the base CAS URI +described above, and add a ``service`` query parameter. This parameter MUST be +the URI of the ``/login/cas/ticket`` endpoint, including the ``redirectUrl`` +query parameter. Because the homeserver may not know its base URI, this may +also require manual configuration. + +Handling the authentication endpoint +++++++++++++++++++++++++++++++++++++ + +When responding to the ``/login/cas/ticket`` endpoint, the server MUST make a +request to the CAS server to validate the provided ticket. The server MAY also +check for certain user attributes in the response. Any required attributes +should be configured by the server administrator. + +Once the ticket has been validated, the server MUST map the CAS ``user_id`` +to a valid `Matrix user identifier <../index.html#user-identifiers>`_. The +guidance in `Mapping from other character sets +<../index.html#mapping-from-other-character-sets>`_ may be useful. + +If the generated user identifier represents a new user, it should be registered +as a new user. + +Finally, the server should generate a short-term login token. The generated +token should be a macaroon, suitable for use with the ``m.login.token`` type of +the |/login|_ API, and `token-based interactive login <#token-based>`_. The +lifetime of this token SHOULD be limited to around five seconds. + + .. |/login| replace:: ``/login`` .. _/login: #post-matrix-client-%CLIENT_MAJOR_VERSION%-login +.. |/login/cas/redirect| replace:: ``/login/cas/redirect`` +.. _/login/cas/redirect: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-cas-redirect +.. |/login/cas/ticket| replace:: ``/login/cas/ticket`` +.. _/login/cas/ticket: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-cas-ticket .. |/login/sso/redirect| replace:: ``/login/sso/redirect`` .. _/login/sso/redirect: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-sso-redirect -.. _`CAS-based client login`: #cas-based-client-login diff --git a/specification/targets.yaml b/specification/targets.yaml index 493814acf..4172e617f 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -61,7 +61,6 @@ groups: # reusable blobs of files when prefixed with 'group:' - modules/account_data.rst - modules/admin.rst - modules/event_context.rst - - modules/cas_login.rst - modules/sso_login.rst - modules/dm.rst - modules/ignore_users.rst From 0eabf108d9a8765213f6f80d27c5dcb6a5c2acd3 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 9 Jan 2019 15:05:27 -0700 Subject: [PATCH 019/170] Add a mechanism for redirecting clients after login Implements https://github.com/matrix-org/matrix-doc/pull/1730 --- .../definitions/wellknown/full.yaml | 39 +++++++++++++++++++ api/client-server/login.yaml | 18 ++++++++- api/client-server/wellknown.yaml | 21 +--------- 3 files changed, 57 insertions(+), 21 deletions(-) create mode 100644 api/client-server/definitions/wellknown/full.yaml diff --git a/api/client-server/definitions/wellknown/full.yaml b/api/client-server/definitions/wellknown/full.yaml new file mode 100644 index 000000000..8d8f40387 --- /dev/null +++ b/api/client-server/definitions/wellknown/full.yaml @@ -0,0 +1,39 @@ +# Copyright 2019 New Vector Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +title: Discovery Information +description: |- + Used by clients to determine the homeserver, identity server, and other + optional components they should be interacting with. +type: object +properties: + "m.homeserver": + $ref: "homeserver.yaml" + "m.identity_server": + $ref: "identity_server.yaml" +additionalProperties: + type: object + description: Application-dependent keys using Java package naming convention. +required: + - m.homeserver +example: { + "m.homeserver": { + "base_url": "https://matrix.example.com" + }, + "m.identity_server": { + "base_url": "https://identity.example.com" + }, + "org.example.custom.property": { + "app_url": "https://custom.app.example.org" + } +} diff --git a/api/client-server/login.yaml b/api/client-server/login.yaml index 43aae5dfd..cef3d26f5 100644 --- a/api/client-server/login.yaml +++ b/api/client-server/login.yaml @@ -139,7 +139,15 @@ paths: application/json: { "user_id": "@cheeky_monkey:matrix.org", "access_token": "abc123", - "device_id": "GHTYAJCE" + "device_id": "GHTYAJCE", + "well_known": { + "m.homeserver": { + "base_url": "https://example.org" + }, + "m.identity_server": { + "base_url": "https://id.example.org" + } + } } schema: type: object @@ -166,6 +174,14 @@ paths: description: |- ID of the logged-in device. Will be the same as the corresponding parameter in the request, if one was specified. + well_known: + type: object + description: |- + Optional client configuration provided by the server. If present, + clients SHOULD ise the provided object to reconfigure themselves, + optionally validating the URLs within. This object takes the same + form as the one returned from .well-known autodiscovery. + "$ref": "definitions/wellknown/full.yaml" 400: description: |- Part of the request was invalid. For example, the login type may not be recognised. diff --git a/api/client-server/wellknown.yaml b/api/client-server/wellknown.yaml index 24e190f96..b63bd0410 100644 --- a/api/client-server/wellknown.yaml +++ b/api/client-server/wellknown.yaml @@ -38,28 +38,9 @@ paths: responses: 200: description: Server discovery information. - examples: - application/json: { - "m.homeserver": { - "base_url": "https://matrix.example.com" - }, - "m.identity_server": { - "base_url": "https://identity.example.com" - } - } schema: type: object - properties: - m.homeserver: - description: Information about the homeserver to connect to. - "$ref": "definitions/wellknown/homeserver.yaml" - m.identity_server: - description: Optional. Information about the identity server to connect to. - "$ref": "definitions/wellknown/identity_server.yaml" - additionalProperties: - description: Application-dependent keys using Java package naming convention. - required: - - m.homeserver + "$ref": "definitions/wellknown/full.yaml" 404: description: No server discovery information available. tags: From 510468a3b1736d55a4823596bc6359ed2fccb0c8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 9 Jan 2019 15:31:04 -0700 Subject: [PATCH 020/170] Changelog --- changelogs/client_server/newsfragments/1789.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1789.feature diff --git a/changelogs/client_server/newsfragments/1789.feature b/changelogs/client_server/newsfragments/1789.feature new file mode 100644 index 000000000..97c1e5ca9 --- /dev/null +++ b/changelogs/client_server/newsfragments/1789.feature @@ -0,0 +1 @@ +Add a generic SSO login API. From fbd88611803ced1fcb5900f3b91fab4b95bd7555 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 9 Jan 2019 15:31:43 -0700 Subject: [PATCH 021/170] Changelog --- changelogs/client_server/newsfragments/1790.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1790.feature diff --git a/changelogs/client_server/newsfragments/1790.feature b/changelogs/client_server/newsfragments/1790.feature new file mode 100644 index 000000000..26dccd056 --- /dev/null +++ b/changelogs/client_server/newsfragments/1790.feature @@ -0,0 +1 @@ +Add a mechanism for servers to redirect clients to an alternative homeserver after logging in. From b85f7bb24827b19c5f38e2c6021dc126767e16e5 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 9 Jan 2019 17:02:09 -0700 Subject: [PATCH 022/170] Add room version upgrades Implements https://github.com/matrix-org/matrix-doc/issues/1501 --- api/client-server/room_upgrades.yaml | 95 +++++++++++++++++++++++++ event-schemas/examples/m.room.create | 6 +- event-schemas/examples/m.room.tombstone | 9 +++ event-schemas/schema/m.room.create | 12 ++++ event-schemas/schema/m.room.tombstone | 27 +++++++ proposals/1501-room-version-upgrades.md | 2 +- specification/modules/room_upgrades.rst | 76 ++++++++++++++++++++ specification/targets.yaml | 1 + 8 files changed, 226 insertions(+), 2 deletions(-) create mode 100644 api/client-server/room_upgrades.yaml create mode 100644 event-schemas/examples/m.room.tombstone create mode 100644 event-schemas/schema/m.room.tombstone create mode 100644 specification/modules/room_upgrades.rst diff --git a/api/client-server/room_upgrades.yaml b/api/client-server/room_upgrades.yaml new file mode 100644 index 000000000..4d6b86b42 --- /dev/null +++ b/api/client-server/room_upgrades.yaml @@ -0,0 +1,95 @@ +# Copyright 2019 New Vector Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +swagger: '2.0' +info: + title: "Matrix Client-Server Room Upgrades API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/%CLIENT_MAJOR_VERSION% +consumes: + - application/json +produces: + - application/json +securityDefinitions: + $ref: definitions/security.yaml +paths: + "/rooms/{roomId}/upgrade": + post: + summary: Upgrades a room to a new room version. + description: |- + Upgrades the given room to a particular room version, migrating as much + data as possible over to the new room. See the `room_upgrades`_ module + for more information on what this entails. + operationId: upgradeRoom + security: + - accessToken: [] + parameters: + - in: path + type: string + name: roomId + required: true + description: The ID of the room to upgrade. + x-example: "!oldroom:example.org" + - in: body + name: body + required: true + description: The request body + schema: + type: object + properties: + new_version: + type: string + description: The new version for the room. + example: {"new_version": "2"} + required: [new_version] + responses: + 200: + description: The room was successfully upgraded. + examples: + application/json: { + "replacement_room": "!newroom:example.org" + } + schema: + type: object + properties: + replacement_room: + type: string + description: The ID of the new room. + required: [replacement_room] + 400: + description: |- + The request was invalid. One way this can happen is if the room version + requested is not supported by the homeserver. + examples: + application/json: { + "errcode": "M_UNSUPPORTED_ROOM_VERSION", + "error": "This server does not support that room version" + } + schema: + "$ref": "definitions/errors/error.yaml" + 403: + description: |- + The user is not permitted to upgrade the room. + examples: + application/json: { + "errcode": "M_FORBIDDEN", + "error": "You cannot upgrade this room" + } + schema: + "$ref": "definitions/errors/error.yaml" + tags: + - Room ugprades diff --git a/event-schemas/examples/m.room.create b/event-schemas/examples/m.room.create index 29b6237ac..e33dbc3ba 100644 --- a/event-schemas/examples/m.room.create +++ b/event-schemas/examples/m.room.create @@ -5,6 +5,10 @@ "content": { "creator": "@example:example.org", "room_version": "1", - "m.federate": true + "m.federate": true, + "predecessor": { + "event_id": "$something:example.org", + "room_id": "!oldroom:example.org" + } } } diff --git a/event-schemas/examples/m.room.tombstone b/event-schemas/examples/m.room.tombstone new file mode 100644 index 000000000..b62244764 --- /dev/null +++ b/event-schemas/examples/m.room.tombstone @@ -0,0 +1,9 @@ +{ + "$ref": "core/state_event.json", + "type": "m.room.tombstone", + "state_key": "", + "content": { + "body": "This room has been replaced", + "replacement_room": "!newroom:example.org" + } +} diff --git a/event-schemas/schema/m.room.create b/event-schemas/schema/m.room.create index d12b9ccd7..a6fe331e4 100644 --- a/event-schemas/schema/m.room.create +++ b/event-schemas/schema/m.room.create @@ -14,6 +14,18 @@ properties: room_version: description: The version of the room. Defaults to ``"1"`` if the key does not exist. type: string + predecessor: + description: A reference to the room this room replaces, if the previous room was upgraded. + type: object + title: Previous Room + properties: + room_id: + type: string + description: The ID of the old room. + event_id: + type: string + description: The event ID of the last known event in the old room. + required: [room_id, event_id] required: - creator type: object diff --git a/event-schemas/schema/m.room.tombstone b/event-schemas/schema/m.room.tombstone new file mode 100644 index 000000000..0fd8ba452 --- /dev/null +++ b/event-schemas/schema/m.room.tombstone @@ -0,0 +1,27 @@ +--- +allOf: + - $ref: core-event-schema/state_event.yaml +description: 'A state event signifying that a room has been upgraded to a different room version, and that clients should go there.' +properties: + content: + properties: + body: + type: string + description: A server-defined message. + replacement_room: + type: string + description: The new room the client should be visiting. + required: + - replacement_room + - body + type: object + state_key: + description: A zero-length string. + pattern: '^$' + type: string + type: + enum: + - m.room.tombstone + type: string +title: Indication that the room has been upgraded. +type: object diff --git a/proposals/1501-room-version-upgrades.md b/proposals/1501-room-version-upgrades.md index 3a7262506..7bd310ea6 100644 --- a/proposals/1501-room-version-upgrades.md +++ b/proposals/1501-room-version-upgrades.md @@ -48,7 +48,7 @@ When this is called, the server: "state_key": "", "room_id": "!QtykxKocfsgujksjgd:matrix.org", "content": { - "version": "2", + "room_version": "2", "predecessor": { "room_id": "!cURbaf:matrix.org", "event_id": "$1235135aksjgdkg:matrix.org" diff --git a/specification/modules/room_upgrades.rst b/specification/modules/room_upgrades.rst new file mode 100644 index 000000000..81da8aaf4 --- /dev/null +++ b/specification/modules/room_upgrades.rst @@ -0,0 +1,76 @@ +.. Copyright 2019 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +Room Upgrades +============= + +.. _module:room-upgrades: + +From time to time, a room may need to be upgraded to a different room version for a +variety for reasons. This module defines a way for rooms to upgrade to a different +room version when needed. + +Events +------ + +{{m_room_tombstone_event}} + +Client behaviour +---------------- + +Clients which understand ``m.room.tombstone`` events MUST: + +* Hide the old room from the user's list of rooms. Permalinks, backlinks, etc should + still be accessible to this room, however. +* At the very bottom of the old room's timeline/message view, display a message which + indicates the room has been upgraded with a permalink to the new room. When the user + accesses the permalink, they are to be joined to the new room. + + * Note that if the homeserver doesn't support the room version that the user may + receive an error upon trying to join. + * If the replacement room has a tombstone event, the client may automatically follow + the chain until it finds a room which does not have a tombstone in it. + +* Optionally, the client may wish to improve the presentation of unread messages when + the user is in both the old and new rooms. For example, if the user was not active + during the upgrade and had unread messages in the old room, the new room may have an + indicator which shows the sum of unread notifications between the rooms. + +Clients which understand ``m.room.tombstone`` events must also understand the ``predecessor`` +field on ``m.room.create`` events such that: + +* At the top of the scrollback/message view for the new room a message is displayed + indicating that the room is a continuation of a previous room, with a permalink to + the old room. +* Optionally supporting search and other functions of the room to span across the old + and new rooms. + +{{room_upgrades_cs_http_api}} + +Server behaviour +---------------- + +When the client requests to upgrade a known room to a known version, the server: + +1. Checks that the user has permission to send ``m.room.tombstone`` events in the room. +2. Creates a replacement room with a ``m.room.create`` event containing a ``predecessor`` + field and the applicable ``room_version``. +3. Replicates the power levels, privacy, topic, and other transferable state events to + the new room. This generally excludes membership events. +4. Moves any local aliases to the new room. +5. Sends a ``m.room.tombstone`` event to the old room to indicate that it is not intended + to be used any further. +6. If possible, the power levels in the old room should also be modified to prevent sending + of events and inviting new users. For example, setting ``events_default`` and ``invite`` + to the greater of ``50`` and ``users_default + 1``. diff --git a/specification/targets.yaml b/specification/targets.yaml index 93e1b8a6b..60da93cdc 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -70,6 +70,7 @@ groups: # reusable blobs of files when prefixed with 'group:' - modules/openid.rst - modules/server_acls.rst - modules/mentions.rst + - modules/room_upgrades.rst title_styles: ["=", "-", "~", "+", "^", "`", "@", ":"] From 56bfa7676530d4d52cf26b4acde923315e99d6af Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 9 Jan 2019 17:03:05 -0700 Subject: [PATCH 023/170] changelog --- changelogs/client_server/newsfragments/1791.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1791.feature diff --git a/changelogs/client_server/newsfragments/1791.feature b/changelogs/client_server/newsfragments/1791.feature new file mode 100644 index 000000000..ae961ad3f --- /dev/null +++ b/changelogs/client_server/newsfragments/1791.feature @@ -0,0 +1 @@ +Add room version upgrades From 5cbfafaab73c2585c57a1703f39fa6cb8206dcee Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 9 Jan 2019 17:17:50 -0700 Subject: [PATCH 024/170] Fix link to module --- api/client-server/room_upgrades.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/client-server/room_upgrades.yaml b/api/client-server/room_upgrades.yaml index 4d6b86b42..6511d9fcf 100644 --- a/api/client-server/room_upgrades.yaml +++ b/api/client-server/room_upgrades.yaml @@ -32,8 +32,8 @@ paths: summary: Upgrades a room to a new room version. description: |- Upgrades the given room to a particular room version, migrating as much - data as possible over to the new room. See the `room_upgrades`_ module - for more information on what this entails. + data as possible over to the new room. See the `room_upgrades <#room-upgrades>`_ + module for more information on what this entails. operationId: upgradeRoom security: - accessToken: [] From b0adfc67d8e07ec031762601dac4c84be36ac01b Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 10 Jan 2019 12:17:48 +0000 Subject: [PATCH 025/170] MSC 1794 - Federation v2 Invite API --- proposals/1794-federation-v2-invites.md | 79 +++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 proposals/1794-federation-v2-invites.md diff --git a/proposals/1794-federation-v2-invites.md b/proposals/1794-federation-v2-invites.md new file mode 100644 index 000000000..3879683b9 --- /dev/null +++ b/proposals/1794-federation-v2-invites.md @@ -0,0 +1,79 @@ +# MSC 1794 - Federation v2 Invite API + +This proposal adds a new `/invite` API to federation that supports different +room versions. + +## Motivation + +It is planned for future room versions to be able to change the format of events +in various ways, and so to support this all servers must know the room version +whenever they need to parse an event. Currently the `/invite` API does not +include the room version, and so the event included in the payload would not be +able to be parsed. + +## Proposal + +Add a new version of the invite API under the prefix `/_matrix/federation/v2`, +which has a payload of: + +``` +PUT /_matrix/federation/v2/invite// + +{ + "room_version": , + "event": { ... }, + "invite_room_state": { ... } +} +``` + +The differences between this and `v1` are: + +1. The payload in `v1` is the event, while in `v2` the event is instead placed + under an `"event"` key. This is for consistency with other APIs, and to allow + extra data to be added to the request payload separately from the event. +2. A required field called `"room_version"` is added that specifies the room + version. +3. The `"invite_room_state"` is moved from the `unsigned` section of the event + to a top level key. The key `invite_room_state` being in the `event.unsigned` + was a hack due to point 1. above. + + +The response is identical to `v1`, except that: + +1. If the receiving server does not support the given room version the + appropriate incompatible-room-version exception is returned, in the same way + as e.g. for `/make_join` APIs. +2. The response payload is no longer in the format of `[200, { ... }]`, and is + instead simply the `{ ... }` portion. This fixes a historical accident to + bring the invite API into line with the rest of the federation API. + + +If a call to `v2` `/invite` results in a unrecognised request exception **AND** +the room version is `1` or `2` then the sending server should retry the request +with the `v1` API. + + +## Alternatives + + +### Reusing V1 API + +One alternative is to add a `room_version` query string parameter to the `v1` +`/invite` API in a similar way as for the `/make_join` APIs. However, older +servers would ignore the query string parameter while processing an incoming +`/invite` request, resulting in the server attempting to parse the event in the +old `v1` format. This would likely result in either a `400` or `500` response, +which the sending server could interpret as the receiving server not supporting +the room version. + +This method, however, is fragile and could easily mask legitimate `400` and +`500` errors that are not due to not supporting the room version. + + +### Using V1 API for V1 room versions + +Instead of all servers attempting to use the new API and falling back if the API +is not found, servers could instead always use the current API for V1 and V2 +rooms. + +However, this would not allow us to deprecate the `v1` API. From 2109314c524fbc63952f96452e4989cd2786a863 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 11 Jan 2019 14:34:31 +0000 Subject: [PATCH 026/170] Apply suggestions from code review Co-Authored-By: erikjohnston --- proposals/1794-federation-v2-invites.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/1794-federation-v2-invites.md b/proposals/1794-federation-v2-invites.md index 3879683b9..8402c447d 100644 --- a/proposals/1794-federation-v2-invites.md +++ b/proposals/1794-federation-v2-invites.md @@ -6,9 +6,9 @@ room versions. ## Motivation It is planned for future room versions to be able to change the format of events -in various ways, and so to support this all servers must know the room version +in various ways. To support this, all servers must know the room version whenever they need to parse an event. Currently the `/invite` API does not -include the room version, and so the event included in the payload would not be +include the room version, so the target server would be unable to parse the event included in the payload. able to be parsed. ## Proposal @@ -41,7 +41,7 @@ The differences between this and `v1` are: The response is identical to `v1`, except that: 1. If the receiving server does not support the given room version the - appropriate incompatible-room-version exception is returned, in the same way + appropriate incompatible-room-version error is returned, in the same way as e.g. for `/make_join` APIs. 2. The response payload is no longer in the format of `[200, { ... }]`, and is instead simply the `{ ... }` portion. This fixes a historical accident to From 6e8739c989172c351a6ead66f073b413f6340326 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 11 Jan 2019 08:15:08 -0700 Subject: [PATCH 027/170] Fix typo Co-Authored-By: turt2live --- api/client-server/login.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/client-server/login.yaml b/api/client-server/login.yaml index cef3d26f5..98914a060 100644 --- a/api/client-server/login.yaml +++ b/api/client-server/login.yaml @@ -178,7 +178,7 @@ paths: type: object description: |- Optional client configuration provided by the server. If present, - clients SHOULD ise the provided object to reconfigure themselves, + clients SHOULD use the provided object to reconfigure themselves, optionally validating the URLs within. This object takes the same form as the one returned from .well-known autodiscovery. "$ref": "definitions/wellknown/full.yaml" From c88c9c29413c2da12b2d0b9fc9a1b698ddd7cdba Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 14 Jan 2019 14:19:58 +0000 Subject: [PATCH 028/170] Update proposals/1794-federation-v2-invites.md Co-Authored-By: erikjohnston --- proposals/1794-federation-v2-invites.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1794-federation-v2-invites.md b/proposals/1794-federation-v2-invites.md index 8402c447d..de5c70f6e 100644 --- a/proposals/1794-federation-v2-invites.md +++ b/proposals/1794-federation-v2-invites.md @@ -48,7 +48,7 @@ The response is identical to `v1`, except that: bring the invite API into line with the rest of the federation API. -If a call to `v2` `/invite` results in a unrecognised request exception **AND** +If a call to `v2` `/invite` results in an unrecognised request exception **AND** the room version is `1` or `2` then the sending server should retry the request with the `v1` API. From b90ee6baaba293681e24ce503465626363d87c30 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 14 Jan 2019 14:20:57 +0000 Subject: [PATCH 029/170] 'invite_room_state' should be an array --- proposals/1794-federation-v2-invites.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1794-federation-v2-invites.md b/proposals/1794-federation-v2-invites.md index de5c70f6e..65e638251 100644 --- a/proposals/1794-federation-v2-invites.md +++ b/proposals/1794-federation-v2-invites.md @@ -22,7 +22,7 @@ PUT /_matrix/federation/v2/invite// { "room_version": , "event": { ... }, - "invite_room_state": { ... } + "invite_room_state": [ ... ] } ``` From ebf37178b5acb9f3ba376fa429fb56263a87e4a3 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Mon, 14 Jan 2019 15:53:23 +0000 Subject: [PATCH 030/170] Update proposals/1794-federation-v2-invites.md Co-Authored-By: erikjohnston --- proposals/1794-federation-v2-invites.md | 1 - 1 file changed, 1 deletion(-) diff --git a/proposals/1794-federation-v2-invites.md b/proposals/1794-federation-v2-invites.md index 65e638251..a2f2bb35c 100644 --- a/proposals/1794-federation-v2-invites.md +++ b/proposals/1794-federation-v2-invites.md @@ -9,7 +9,6 @@ It is planned for future room versions to be able to change the format of events in various ways. To support this, all servers must know the room version whenever they need to parse an event. Currently the `/invite` API does not include the room version, so the target server would be unable to parse the event included in the payload. -able to be parsed. ## Proposal From aeb524ef89f2ef6751dd0b7067c225b0c8a6a7c2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 16 Jan 2019 16:13:53 -0700 Subject: [PATCH 031/170] Remove CAS login and reference it against r0.4.0 The SSO module should cover what CAS provides, and r0.4.0 is good as a reference for how CAS could be implemented without us repeating it here. --- api/client-server/cas_login_redirect.yaml | 54 ----------- api/client-server/cas_login_ticket.yaml | 66 ------------- specification/client_server_api.rst | 6 -- specification/modules/sso_login.rst | 110 ++++------------------ 4 files changed, 18 insertions(+), 218 deletions(-) delete mode 100644 api/client-server/cas_login_redirect.yaml delete mode 100644 api/client-server/cas_login_ticket.yaml diff --git a/api/client-server/cas_login_redirect.yaml b/api/client-server/cas_login_redirect.yaml deleted file mode 100644 index abe9069b5..000000000 --- a/api/client-server/cas_login_redirect.yaml +++ /dev/null @@ -1,54 +0,0 @@ -# Copyright 2016 OpenMarket Ltd -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -swagger: '2.0' -info: - title: "Matrix Client-Server CAS Login API" - version: "1.0.0" -host: localhost:8008 -schemes: - - https - - http -basePath: /_matrix/client/%CLIENT_MAJOR_VERSION% -paths: - "/login/cas/redirect": - get: - summary: Redirect the user's browser to the CAS interface. - description: |- - A web-based Matrix client should instruct the user's browser to - navigate to this endpoint in order to log in via CAS. - - The server MUST respond with an HTTP redirect to the CAS interface. The - URI MUST include a ``service`` parameter giving the path of the - |/login/cas/ticket|_ endpoint (including the ``redirectUrl`` query - parameter). - - For example, if the endpoint is called with - ``redirectUrl=https://client.example.com/?q=p``, it might redirect to - ``https://cas.example.com/?service=https%3A%2F%2Fserver.example.com%2F_matrix%2Fclient%2F%CLIENT_MAJOR_VERSION%%2Flogin%2Fcas%2Fticket%3FredirectUrl%3Dhttps%253A%252F%252Fclient.example.com%252F%253Fq%253Dp``. - - operationId: redirectToCAS - parameters: - - in: query - type: string - name: redirectUrl - description: |- - URI to which the user will be redirected after the homeserver has - authenticated the user with CAS. - required: true - responses: - 302: - description: A redirect to the CAS interface. - headers: - Location: - type: "string" diff --git a/api/client-server/cas_login_ticket.yaml b/api/client-server/cas_login_ticket.yaml deleted file mode 100644 index a08565a0d..000000000 --- a/api/client-server/cas_login_ticket.yaml +++ /dev/null @@ -1,66 +0,0 @@ -# Copyright 2016 OpenMarket Ltd -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -swagger: '2.0' -info: - title: "Matrix Client-Server CAS Login API" - version: "1.0.0" -host: localhost:8008 -schemes: - - https - - http -basePath: /_matrix/client/%CLIENT_MAJOR_VERSION% -paths: - "/login/cas/ticket": - get: - summary: Receive and validate a CAS login ticket. - description: |- - Once the CAS server has authenticated the user, it will redirect the - browser to this endpoint (assuming |/login/cas/redirect|_ gave it the - correct ``service`` parameter). - - The server MUST call ``/proxyValidate`` on the CAS server, to validate - the ticket supplied by the browser. - - If validation is successful, the server must generate a Matrix login - token. It must then respond with an HTTP redirect to the URI given in - the ``redirectUrl`` parameter, adding a ``loginToken`` query parameter - giving the generated token. - - If validation is unsuccessful, the server should respond with a ``401 - Unauthorized`` error, the body of which will be displayed to the user. - operationId: loginByCASTicket - parameters: - - in: query - type: string - name: redirectUrl - description: |- - The ``redirectUrl`` originally provided by the client to - |/login/cas/redirect|_. - required: true - - in: query - type: string - name: ticket - description: |- - CAS authentication ticket. - required: true - responses: - 302: - description: A redirect to the Matrix client. - headers: - Location: - type: "string" - x-example: |- - https://client.example.com/?q=p&loginToken=secrettoken - 401: - description: The server was unable to validate the CAS ticket. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 03d712460..4df838142 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1024,12 +1024,6 @@ supports it, the client should redirect the user to the ``/redirect`` endpoint for `Single Sign-On <#sso-client-login>`_. After authentication is complete, the client will need to submit a ``/login`` request matching ``m.login.token``. -If the homeserver advertises ``m.login.cas`` as a viable flow, and the client -supports it, the client should redirect the user to the ``/redirect`` endpoint -for `CAS <#cas-based-client-login>`_. Just like SSO authentication, the client -is expected to submit a ``/login`` request matching ``m.login.token`` upon -successful authentication. - {{login_cs_http_api}} {{logout_cs_http_api}} diff --git a/specification/modules/sso_login.rst b/specification/modules/sso_login.rst index 977278a7e..57188500c 100644 --- a/specification/modules/sso_login.rst +++ b/specification/modules/sso_login.rst @@ -34,8 +34,17 @@ An overview of the process, as used in Matrix, is as follows: 4. The SSO server and the homeserver interact to verify the user's identity and other authentication information, potentially using a number of redirects. -7. The Matrix client receives the login token and passes it to the |/login|_ - API. +5. The browser is directed to the ``redirectUrl`` provided by the client with + a ``loginToken`` query parameter for the client to log in with. + +.. Note:: + In the older `r0.4.0 version `_ + of this specification it was possible to authenticate via CAS when the server + provides a ``m.login.cas`` login flow. This specification deprecates the use + of ``m.login.cas`` to instead prefer ``m.login.sso``, which is the same process + with the only change being which redirect endpoint to use: for ``m.login.cas``, use + ``/cas/redirect`` and for ``m.login.sso`` use ``/sso/redirect`` (described below). + The endpoints are otherwise the same. Client behaviour ---------------- @@ -61,7 +70,11 @@ Server behaviour The URI for the SSO system to be used should be configured on the server by the server administrator. The server is expected to set up any endpoints required to -interact with that SSO system. +interact with that SSO system. For example, for CAS authentication the homeserver +should provide a means for the administrator to configure where the CAS server is +and the REST endpoints which consume the ticket. A good reference for how CAS could +be implemented is available in the older `r0.4.0 version `_ +of this specification. Handling the redirect endpoint ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -90,98 +103,11 @@ as a new user. Finally, the server should generate a short-term login token. The generated token should be a macaroon, suitable for use with the ``m.login.token`` type of the |/login|_ API, and `token-based interactive login <#token-based>`_. The -lifetime of this token SHOULD be limited to around five seconds. - - -CAS-based client login ----------------------- - -.. _module:cas_login: - -`Central Authentication Service -`_ -(CAS) is a web-based single sign-on protocol. The protocol defined here is an -extension of the SSO protocol defined in this module: it is largely the same, -but has some specific details which make it different. - -An overview of the process, as used in Matrix, is as follows: - -1. The Matrix client instructs the user's browser to navigate to the - |/login/cas/redirect|_ endpoint on the user's homeserver. - -2. The homeserver responds with an HTTP redirect to the CAS user interface, - which the browser follows. - -3. The CAS system authenticates the user. - -4. The CAS server responds to the user's browser with a redirect back to the - |/login/cas/ticket|_ endpoint on the homeserver, which the browser - follows. A 'ticket' identifier is passed as a query parameter in the - redirect. - -5. The homeserver receives the ticket ID from the user's browser, and makes a - request to the CAS server to validate the ticket. - -6. Having validated the ticket, the homeserver responds to the browser with a - third HTTP redirect, back to the Matrix client application. A login token - is passed as a query parameter in the redirect. - -7. The Matrix client receives the login token and passes it to the |/login|_ - API. - -Client behaviour -~~~~~~~~~~~~~~~~ - -The client starts the process by instructing the browser to navigate to -|/login/cas/redirect|_ with an appropriate ``redirectUrl``. Once authentication -is successful, the browser will be redirected to that ``redirectUrl``. - -{{cas_login_redirect_cs_http_api}} -{{cas_login_ticket_cs_http_api}} - -Server behaviour -~~~~~~~~~~~~~~~~ - -The URI for the CAS system to be used should be configured on the server by the -server administrator. - -Handling the redirect endpoint -++++++++++++++++++++++++++++++ - -When responding to the ``/login/cas/redirect`` endpoint, the server must -generate a URI for the CAS login page. The server should take the base CAS URI -described above, and add a ``service`` query parameter. This parameter MUST be -the URI of the ``/login/cas/ticket`` endpoint, including the ``redirectUrl`` -query parameter. Because the homeserver may not know its base URI, this may -also require manual configuration. - -Handling the authentication endpoint -++++++++++++++++++++++++++++++++++++ - -When responding to the ``/login/cas/ticket`` endpoint, the server MUST make a -request to the CAS server to validate the provided ticket. The server MAY also -check for certain user attributes in the response. Any required attributes -should be configured by the server administrator. - -Once the ticket has been validated, the server MUST map the CAS ``user_id`` -to a valid `Matrix user identifier <../index.html#user-identifiers>`_. The -guidance in `Mapping from other character sets -<../index.html#mapping-from-other-character-sets>`_ may be useful. - -If the generated user identifier represents a new user, it should be registered -as a new user. - -Finally, the server should generate a short-term login token. The generated -token should be a macaroon, suitable for use with the ``m.login.token`` type of -the |/login|_ API, and `token-based interactive login <#token-based>`_. The -lifetime of this token SHOULD be limited to around five seconds. +lifetime of this token SHOULD be limited to around five seconds. This token is +given to the client via the ``loginToken`` query parameter previously mentioned. .. |/login| replace:: ``/login`` .. _/login: #post-matrix-client-%CLIENT_MAJOR_VERSION%-login -.. |/login/cas/redirect| replace:: ``/login/cas/redirect`` -.. _/login/cas/redirect: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-cas-redirect -.. |/login/cas/ticket| replace:: ``/login/cas/ticket`` -.. _/login/cas/ticket: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-cas-ticket .. |/login/sso/redirect| replace:: ``/login/sso/redirect`` .. _/login/sso/redirect: #get-matrix-client-%CLIENT_MAJOR_VERSION%-login-sso-redirect From 71e6321f4d448d840bfb6634d3ee381876d35a97 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 16 Jan 2019 16:57:45 -0700 Subject: [PATCH 032/170] Rework how room versions are represented Versions are actually on a scale of recommendations, and are expected to be created as needed. The scale presented here (develop/beta/default/recommended/mandatory) is a more wordy version of what was previously discussed/intended for room versions - the labels aren't final and may be changed. --- changelogs/rooms/newsfragments/.gitignore | 1 - changelogs/rooms/pyproject.toml | 30 -------- meta/releasing-rooms-v2.md | 38 --------- scripts/gendoc.py | 5 -- scripts/templating/matrix_templates/units.py | 5 -- specification/index.rst | 81 ++++++++++++++++++++ specification/rooms/intro.rst | 70 ----------------- specification/rooms/unstable.rst | 54 ------------- specification/rooms/v1.rst | 24 ++---- specification/rooms/v2.rst | 30 ++------ specification/targets.yaml | 21 ----- 11 files changed, 91 insertions(+), 268 deletions(-) delete mode 100644 changelogs/rooms/newsfragments/.gitignore delete mode 100644 changelogs/rooms/pyproject.toml delete mode 100644 meta/releasing-rooms-v2.md delete mode 100644 specification/rooms/intro.rst delete mode 100644 specification/rooms/unstable.rst diff --git a/changelogs/rooms/newsfragments/.gitignore b/changelogs/rooms/newsfragments/.gitignore deleted file mode 100644 index b722e9e13..000000000 --- a/changelogs/rooms/newsfragments/.gitignore +++ /dev/null @@ -1 +0,0 @@ -!.gitignore \ No newline at end of file diff --git a/changelogs/rooms/pyproject.toml b/changelogs/rooms/pyproject.toml deleted file mode 100644 index b56e19a96..000000000 --- a/changelogs/rooms/pyproject.toml +++ /dev/null @@ -1,30 +0,0 @@ -[tool.towncrier] - filename = "../rooms.rst" - directory = "newsfragments" - issue_format = "`#{issue} `_" - title_format = "{version}" - - [[tool.towncrier.type]] - directory = "breaking" - name = "Breaking Changes" - showcontent = true - - [[tool.towncrier.type]] - directory = "deprecation" - name = "Deprecations" - showcontent = true - - [[tool.towncrier.type]] - directory = "new" - name = "New Endpoints" - showcontent = true - - [[tool.towncrier.type]] - directory = "feature" - name = "Backwards Compatible Changes" - showcontent = true - - [[tool.towncrier.type]] - directory = "clarification" - name = "Spec Clarifications" - showcontent = true diff --git a/meta/releasing-rooms-v2.md b/meta/releasing-rooms-v2.md deleted file mode 100644 index 802777493..000000000 --- a/meta/releasing-rooms-v2.md +++ /dev/null @@ -1,38 +0,0 @@ -# How to release Room Version 2 - -Room versions are a bit special for the release given they have been -introduced at v2 rather than ever getting a v1 release. Additionally, -room versions have different release requirements due to the in-project -versioning of documents rather than relying on matrix.org to maintain -old generated output of specifications. - -As of writing, the project is structured to support 3 logical versions -for rooms: v1, v2, and unstable. Unstable is currently pointed at v2 -in order to aid development. After v2 is released, unstable may wish -to be left as an independent version or similarly be pointed at a "v3" -document. - -Due to room versions being versioned in-project, the generated output -from a release is not to be sent off to matrix-doc/matrix.org. Instead, -in `gendoc.py` the default value for `--room_version` should be set to -the current release (`v2`, for example) so the index renders the right -edition in the table. - -After editing `gendoc.py`, the changelog should be generated according -to the towncrier instructions. You may need to fix the `.. version: v2` -comment located in the `rooms.rst` changelog to be just underneath the -title instead of at the end of the section. - -The `targets.yaml` file needs to be set up to point unstable to the -right set of files. Ensure that `unstable.rst` is referenced instead -of `v2.rst`, and that `unstable.rst` has appropriate contents. - -Finally, in the `intro.rst` for room versions, re-add unstable to the -list of available versions. It is currently commented out to avoid -confusing people, so it should be possible to uncomment it and put it -back into the list. - -From there, the standard process of using a release branch, tagging it, -and announcing it to the world should be followed. If required, the -various other APIs should be updated to better support the new room -version. diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 0455c90ab..72e3047cf 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -529,10 +529,6 @@ if __name__ == '__main__': "--identity_release", "-i", action="store", default="unstable", help="The identity service release tag to generate, e.g. r1.2" ) - parser.add_argument( - "--room_version", "-r", action="store", default="unstable", - help="The current room version to advertise, e.g. v2" - ) parser.add_argument( "--list_targets", action="store_true", help="Do not update the specification. Instead print a list of targets.", @@ -559,7 +555,6 @@ if __name__ == '__main__': "%APPSERVICE_RELEASE_LABEL%": args.appservice_release, "%IDENTITY_RELEASE_LABEL%": args.identity_release, "%PUSH_GATEWAY_RELEASE_LABEL%": args.push_gateway_release, - "%CURRENT_ROOM_VERSION%": args.room_version, } exit (main(args.target or ["all"], args.dest, args.nodelete, substitutions)) diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 666434a53..721501ff5 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -757,7 +757,6 @@ class MatrixUnits(Units): is_ver = substitutions.get("%IDENTITY_RELEASE_LABEL%", "unstable") as_ver = substitutions.get("%APPSERVICE_RELEASE_LABEL%", "unstable") push_gw_ver = substitutions.get("%PUSH_GATEWAY_RELEASE_LABEL%", "unstable") - room_ver = substitutions.get("%CURRENT_ROOM_VERSION%", "unstable") # we abuse the typetable to return this info to the templates return TypeTable(rows=[ @@ -781,10 +780,6 @@ class MatrixUnits(Units): "`Push Gateway API `_", push_gw_ver, "Push notifications for Matrix events", - ), TypeTableRow( - "`Rooms `_", - room_ver, - "Specification for behaviour of rooms, such as event formats", ), ]) diff --git a/specification/index.rst b/specification/index.rst index bb9187393..2f17ec298 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -418,6 +418,87 @@ dedicated API. The API is symmetrical to managing Profile data. Would it really be overengineered to use the same API for both profile & private user data, but with different ACLs? + +Room Versions +------------- + +Rooms are central to how Matrix operates, and have strict rules for what +is allowed to be contained within them. Rooms can also have various +algorithms that handle different tasks, such as what to do when two or +more events collide in the underlying DAG. To allow rooms to be improved +upon through new algorithms or rules, "room versions" are employed to +manage a set of expectations for each room. New room versions are assigned +as needed. + +Room version grammar +~~~~~~~~~~~~~~~~~~~~ + +Room versions are used to change properties of rooms that may not be compatible +with other servers. For example, changing the rules for event authorization would +cause older servers to potentially end up in a split-brain situation due to them +not understanding the new rules. + +A room version is defined as a string of characters which MUST NOT exceed 32 +codepoints in length. Room versions MUST NOT be empty and SHOULD contain only +the characters ``a-z``, ``0-9``, ``.``, and ``-``. + +Room versions are not intended to be parsed and should be treated as opaque +identifiers. Room versions consisting only of the characters ``0-9`` and ``.`` +are reserved for future versions of the Matrix protocol. + +The complete grammar for a legal room version is:: + + room_version = 1*room_version_char + room_version_char = DIGIT + / %x61-7A ; a-z + / "-" / "." + +Examples of valid room versions are: + +* ``1`` (would be reserved by the Matrix protocol) +* ``1.2`` (would be reserved by the Matrix protocol) +* ``1.2-beta`` +* ``com.example.version`` + +Recommended room versions +~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are varying degrees of what "recommended" means for a given room version. +Currently, this is split into 5 categories: + +* **Development**: This is the default state for all room versions. When in this + state, a room version is documented but not recommended for use outside of a + development environment. These versions are not production-ready. +* **Beta**: Versions in this state are not intended for wide-spread usage but + should be stable enough if a room requires the feature(s) introduced within. + Rooms will need to opt-in to these versions and should not be promoted to + upgrade. +* **Default**: Exactly 1 room version will be in this category. The version under + this category should be used when creating rooms (unless another version is + requested by the user). Servers may wish to promote rooms to opt-in to this + version. +* **Recommended**: Exactly 1 room version will be in this category as well. The + version here should be heavily promoted by servers for rooms to opt-in to using. + This version is often going to be the same as the Default version. +* **Mandatory**: Servers are required to implement versions in this category. When + a version is flagged as mandatory, additional rules may apply to rooms. For example, + servers may be required to stop supporting another room version and automatically + upgrade all affected rooms. + +With the above categories, the following applies: + +* Servers MUST have Room Version 1 as the Default room version. +* Servers MUST have Room Version 1 as the Recommended room version. +* Servers MUST implement Room Version 1 as a Mandatory room version. + +Complete list of room versions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The available room versions are: + +* `Version 1 `_ - The current version of most rooms. +* `Version 2 `_ - **Beta**. Implements State Resolution Version 2. + Specification Versions ---------------------- diff --git a/specification/rooms/intro.rst b/specification/rooms/intro.rst deleted file mode 100644 index 6231d066b..000000000 --- a/specification/rooms/intro.rst +++ /dev/null @@ -1,70 +0,0 @@ -.. Copyright 2018 New Vector Ltd -.. -.. Licensed under the Apache License, Version 2.0 (the "License"); -.. you may not use this file except in compliance with the License. -.. You may obtain a copy of the License at -.. -.. http://www.apache.org/licenses/LICENSE-2.0 -.. -.. Unless required by applicable law or agreed to in writing, software -.. distributed under the License is distributed on an "AS IS" BASIS, -.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -.. See the License for the specific language governing permissions and -.. limitations under the License. - -Room Specification -================== - -.. contents:: Table of Contents -.. sectnum:: - -Rooms are central to how Matrix operates, and have strict rules for what -is allowed to be contained within them. Rooms can also have various -algorithms that handle different tasks, such as what to do when two or -more events collide in the underlying DAG. To allow rooms to be improved -upon through new algorithms or rules, "room versions" are employed to -manage a set of expectations for each room. - - -Room version grammar --------------------- - -Room versions are used to change properties of rooms that may not be compatible -with other servers. For example, changing the rules for event authorization would -cause older servers to potentially end up in a split-brain situation due to them -not understanding the new rules. - -A room version is defined as a string of characters which MUST NOT exceed 32 -codepoints in length. Room versions MUST NOT be empty and SHOULD contain only -the characters ``a-z``, ``0-9``, ``.``, and ``-``. - -Room versions are not intended to be parsed and should be treated as opaque -identifiers. Room versions consisting only of the characters ``0-9`` and ``.`` -are reserved for future versions of the Matrix protocol. - -The complete grammar for a legal room version is:: - - room_version = 1*room_version_char - room_version_char = DIGIT - / %x61-7A ; a-z - / "-" / "." - -Examples of valid room versions are: - -* ``1`` (would be reserved by the Matrix protocol) -* ``1.2`` (would be reserved by the Matrix protocol) -* ``1.2-beta`` -* ``com.example.version`` - - -Other room versions -------------------- - -The available room versions are: - -* `Version 1 `_ - The current version of most rooms. -* `Version 2 `_ - Currently in development. - -.. Note: the 'unstable' version is commented out pending a real release of rooms v2 -.. See meta/releasing-rooms-v2.md -.. * `Unstable `_ - The upcoming version of the room specification. diff --git a/specification/rooms/unstable.rst b/specification/rooms/unstable.rst deleted file mode 100644 index 44261814f..000000000 --- a/specification/rooms/unstable.rst +++ /dev/null @@ -1,54 +0,0 @@ -.. Copyright 2018 New Vector Ltd -.. -.. Licensed under the Apache License, Version 2.0 (the "License"); -.. you may not use this file except in compliance with the License. -.. You may obtain a copy of the License at -.. -.. http://www.apache.org/licenses/LICENSE-2.0 -.. -.. Unless required by applicable law or agreed to in writing, software -.. distributed under the License is distributed on an "AS IS" BASIS, -.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -.. See the License for the specific language governing permissions and -.. limitations under the License. - - -.. DEV NOTE: This is stubbed as a template and not actually used anywhere. -.. See v2.rst for the "unstable" room version, which is currently under -.. development. -.. -.. See meta/releasing-rooms-v2.md - - -.. Note: This document appended to the end of the intro, so this next line -.. appears under "Other Room Versions". - -.. Warning:: - - This is the specification for unreleased changes to rooms. The stability - of rooms using this specification cannot be guaranteed. - - -Changelog ---------- - -.. topic:: unstable -{{rooms_changelog_unstable}} - -This version of the specification is generated from -`matrix-doc `_ as of Git commit -`{{git_version}} `_. - -For the full historical changelog, see -https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst - - -Some Module ------------ - -Lorem ipsum dolor sit amet, consectetur adipiscing elit. In sit amet -eros turpis. Quisque commodo diam vel massa ultrices, vel egestas eros -dignissim. Sed sit amet lacus eget metus auctor malesuada at ut odio. -In turpis leo, viverra et mi porttitor, condimentum bibendum dolor. - -.. {-{versioned_test_definition}-} diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 6c6795a4f..c263a1080 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -1,4 +1,4 @@ -.. Copyright 2018 New Vector Ltd +.. Copyright 2018-2019 New Vector Ltd .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -12,25 +12,11 @@ .. See the License for the specific language governing permissions and .. limitations under the License. +Room Version 1 +============== -.. Note: This document appended to the end of the intro, so this next line -.. appears under "Other Room Versions". - -This is the specification for **room version 1** (``"1"``). - -Changelog ---------- - -.. topic:: Room version 1 -{{rooms_changelog_v1}} - -This version of the specification is generated from -`matrix-doc `_ as of Git commit -`{{git_version}} `_. - -For the full historical changelog, see -https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst - +This room version is the first ever version for rooms, and contains the building +blocks for other room versions. Server implementation components -------------------------------- diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index b04f94b4a..eef034977 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -1,4 +1,4 @@ -.. Copyright 2018 New Vector Ltd +.. Copyright 2018-2019 New Vector Ltd .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -12,31 +12,11 @@ .. See the License for the specific language governing permissions and .. limitations under the License. +Room Version 2 +============== -.. Note: This document appended to the end of the intro, so this next line -.. appears under "Other Room Versions". - -This is the specification for **room version 2** (``"2"``). - -.. Warning:: - - Room version 2 is under development and cannot be relied on in production - environments. - - -Changelog ---------- - -.. topic:: Room version 2 -{{rooms_changelog_v2}} - -This version of the specification is generated from -`matrix-doc `_ as of Git commit -`{{git_version}} `_. - -For the full historical changelog, see -https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst - +This room version builds off of `version 1 `_ with an improved state +resolution algorithm. Server implementation components -------------------------------- diff --git a/specification/targets.yaml b/specification/targets.yaml index b940148da..e966f66b0 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -26,35 +26,14 @@ targets: files: - push_gateway.rst version_label: "%PUSH_GATEWAY_RELEASE_LABEL%" - rooms: - files: - - rooms/intro.rst - - # TODO: Switch this back to unstable.rst after releasing rooms v2 - # We temporarily do this so that "unstable" points to the in-dev - # version, however we may also want to hardlink to v2 in places and - # thus we maintain a v2 version of the same doc. - # - # See meta/releasing-rooms-v2.md - - #- rooms/unstable.rst - - rooms/v2.rst - version_label: unstable rooms@v1: # this is translated to be rooms/v1.html files: - - rooms/intro.rst - rooms/v1.rst version_label: v1 rooms@v2: # this is translated to be rooms/v2.html files: - - rooms/intro.rst - rooms/v2.rst version_label: v2 - rooms@latest: # this is translated to be rooms/latest.html - files: - - rooms/intro.rst - - rooms/v1.rst # TODO: Find a better way to link to the v2 spec - version_label: latest appendices: files: - appendices.rst From 3b47a5924b2a67f817aa14ae8ec03e797f09e819 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 16 Jan 2019 17:02:41 -0700 Subject: [PATCH 033/170] Remove extraneous changelog --- changelogs/rooms.rst | 11 ----------- 1 file changed, 11 deletions(-) delete mode 100644 changelogs/rooms.rst diff --git a/changelogs/rooms.rst b/changelogs/rooms.rst deleted file mode 100644 index 55a167901..000000000 --- a/changelogs/rooms.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. version: v2 - -.. Note: We set the version as "version 2" so that we can maintain a specific version -.. variable in the changelog. We already know the next version is going to be v2, so -.. this makes it easier to copy/paste unstable.rst to v2.rst - -Room version 1 -============== -.. version: v1 - -This is the first iteration of rooms in Matrix. From 166d4ada8613a9e786c6d1b0b124b5bd6a02732f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 16 Jan 2019 17:05:57 -0700 Subject: [PATCH 034/170] Fix room versions reference in appendices & s2s spec --- specification/appendices/identifier_grammar.rst | 6 +++--- specification/server_server_api.rst | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index 496aba315..b46789886 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -16,10 +16,10 @@ Identifier Grammar ------------------ -Some identifiers are specific to given room versions, please see the -`room specification`_ for more information. +Some identifiers are specific to given room versions, please refer to the +`room versions specification`_ for more information. -.. _`room specification`: ../rooms/latest.html +.. _`room versions specification`: ../index.html#room-versions Server Name diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 05e6617b8..8645888b7 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -1318,4 +1318,4 @@ Example code .. _`Checking for a signature`: ../appendices.html#checking-for-a-signature .. _`Device Management module`: ../client_server/%CLIENT_RELEASE_LABEL%.html#device-management .. _`End-to-End Encryption module`: ../client_server/%CLIENT_RELEASE_LABEL%.html#end-to-end-encryption -.. _`room version specification`: ../rooms/latest.html +.. _`room version specification`: ../index.html#room-versions From 7fee7373ea619b05c9dc325a07bc8a9d7adcb912 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 16 Jan 2019 17:32:07 -0700 Subject: [PATCH 035/170] Proposal for advertising capable room versions to clients --- .../1804-advertising-capable-room-versions.md | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 proposals/1804-advertising-capable-room-versions.md diff --git a/proposals/1804-advertising-capable-room-versions.md b/proposals/1804-advertising-capable-room-versions.md new file mode 100644 index 000000000..fca224096 --- /dev/null +++ b/proposals/1804-advertising-capable-room-versions.md @@ -0,0 +1,49 @@ +# Proposal for advertising capable room versions to clients + +Currently clients need to guess at which room versions the server supports, if any. This is particularly +difficult to do as it generally requires knowing the state of the ecosystem and what versions are +available and how keen users are to upgrade their servers and clients. The impossible judgement call +for when to prompt people to upgrade shouldn't be impossible, or even a judgement call. + + +## Proposal + +Building off of [MSC1753](https://github.com/matrix-org/matrix-doc/pull/1753) (capabilities API) and +the [recommendations laid out for room versions](https://github.com/matrix-org/matrix-doc/pull/1773/files#diff-1436075794bb304492ca6953a6692cd0R463), +this proposal suggests a `m.room_versions` capability be introduced like the following: + +```json +{ + "capabilities": { + "m.room_versions": { + "m.development": ["state-res-v2-test", "event-ids-as-hashes-test", "3"], + "m.beta": ["2"], + "m.default": "1", + "m.recommended": "1", + "m.mandatory": ["1"] + } + } +} +``` + +The labels picked (`m.development`, etc) are based upon the categories for different room versions. +Note that `m.default` and `m.recommended` reinforce that there is exactly 1 version in each category. + +Clients are encouraged to make use of this capability to determine if the server supports a given +version, and at what state. Clients should cross-reference the versions with the room version spec +where possible. For example, if a slightly older server hasn't yet put version `2` into `m.default`, +but the spec says `2` is the new default and the client knows this, the client should treat `2` as +the new default. + +Similarly, clients should prompt room administrators (or those with enough permission) to upgrade +their rooms where possible. + + +## Potential issues + +Changes aren't pushed to the client, which means clients may want to poll this endpoint on some +heuristic instead. For example, clients may want to poll the endpoint weekly or when the user relaunches +the client. Clients may also wish to provide users a way to upgrade without considering the capabilities +of the server, expecting that the server may not support the user-provided version - the intention +being such a feature would be used by eager room administrators which do not want to relaunch their +client, for example. From 5da17d0114979e08fbbf5ab6855e1fb811266f5f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 09:41:11 -0700 Subject: [PATCH 036/170] Remove recommendation for clients to respect the spec over the server --- proposals/1804-advertising-capable-room-versions.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/1804-advertising-capable-room-versions.md b/proposals/1804-advertising-capable-room-versions.md index fca224096..924cd77f1 100644 --- a/proposals/1804-advertising-capable-room-versions.md +++ b/proposals/1804-advertising-capable-room-versions.md @@ -30,10 +30,10 @@ The labels picked (`m.development`, etc) are based upon the categories for diffe Note that `m.default` and `m.recommended` reinforce that there is exactly 1 version in each category. Clients are encouraged to make use of this capability to determine if the server supports a given -version, and at what state. Clients should cross-reference the versions with the room version spec -where possible. For example, if a slightly older server hasn't yet put version `2` into `m.default`, -but the spec says `2` is the new default and the client knows this, the client should treat `2` as -the new default. +version, and at what state. + +Clients should prompt people with sufficient permissions to perform an upgrade to upgrade their rooms +to the `m.recommended` room version. Similarly, clients should prompt room administrators (or those with enough permission) to upgrade their rooms where possible. From 853d7ede30e00a52b2ed516a956769b84f197f15 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 09:41:24 -0700 Subject: [PATCH 037/170] Clarify the categories and the interaction between them --- .../1804-advertising-capable-room-versions.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/proposals/1804-advertising-capable-room-versions.md b/proposals/1804-advertising-capable-room-versions.md index 924cd77f1..7686f1b2b 100644 --- a/proposals/1804-advertising-capable-room-versions.md +++ b/proposals/1804-advertising-capable-room-versions.md @@ -35,8 +35,18 @@ version, and at what state. Clients should prompt people with sufficient permissions to perform an upgrade to upgrade their rooms to the `m.recommended` room version. -Similarly, clients should prompt room administrators (or those with enough permission) to upgrade -their rooms where possible. +Room versions might appear under multiple categories under some circumstances. In particular, it is +expected that anything in `m.development` or `m.beta` appears exactly once in the whole capability +whereas `m.default`, `m.recommended`, and `m.mandatory` may duplicate a room version. The duplication +is possible due to the definitions of each category: + +* `m.default` - This is the room version that the server is going to apply to all new rooms by default. +* `m.recommended` - The version clients should be prompting people to upgrade to. +* `m.mandatory` - The version the server is going to enforce on all pre-existing rooms. + +With these definitions, it is possible that a room version fits multiple criteria (ie: "please upgrade +your rooms to version X which is also the default for new rooms"). Clients will generally only be +interested in the `m.recommended` room version, leaving the rest as informational for users. ## Potential issues From 82ee3a6035bb56240898f23a40fc83c2fd92c7e3 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 11:18:48 -0700 Subject: [PATCH 038/170] Adjust wording for SSO introduction --- specification/modules/sso_login.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/modules/sso_login.rst b/specification/modules/sso_login.rst index 57188500c..6dd4a3329 100644 --- a/specification/modules/sso_login.rst +++ b/specification/modules/sso_login.rst @@ -17,9 +17,9 @@ SSO client login .. _module:sso_login: -Single Sign-On (SSO) is a generic web-based protocol for logging in users. -As a special case, the Matrix specification supports `CAS-based login <#cas-based-client-login>`_ -as well as a subset of SSO login. +Single Sign-On (SSO) is a generic term which refers to protocols which allow +users to log into applications via a single web-based authentication portal. +Examples include "Central Authentication Service" (CAS) and SAML. An overview of the process, as used in Matrix, is as follows: From 5f12419afb108fab117840dfbed517165ef69049 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 11:41:50 -0700 Subject: [PATCH 039/170] Clarify that changelog entries end with a full stop --- CONTRIBUTING.rst | 6 +++--- changelogs/client_server/newsfragments/1744.clarification | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index c592cf02b..0666c5797 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -76,11 +76,11 @@ Adding to the changelog Currently only changes to the client-server API need to end up in a changelog. The other APIs are not yet stable and therefore do not have a changelog. Adding to the changelog can only be done after you've opened your pull request, so be sure to do -that first. +that first. The changelog is managed by Towncrier (https://github.com/hawkowl/towncrier) in the form of "news fragments". The news fragments for the client-server API are stored -under ``changelogs/client_server/newsfragments``. +under ``changelogs/client_server/newsfragments``. To create a changelog entry, create a file named in the format ``prNumber.type`` in the ``newsfragments`` directory. The ``type`` can be one of the following: @@ -99,7 +99,7 @@ the ``newsfragments`` directory. The ``type`` can be one of the following: * ``deprecation`` - Used when deprecating something All news fragments must have a brief summary explaining the change in the contents -of the file. +of the file. The summary must end in a full stop to be in line with the style guide. Changes that do not change the spec, such as changes to the build script, formatting, CSS, etc should not get a news fragment. diff --git a/changelogs/client_server/newsfragments/1744.clarification b/changelogs/client_server/newsfragments/1744.clarification index 6ed000674..dc1039200 100644 --- a/changelogs/client_server/newsfragments/1744.clarification +++ b/changelogs/client_server/newsfragments/1744.clarification @@ -1 +1 @@ -Add missing status_msg to m.presence schema +Add missing status_msg to m.presence schema. From 4e0533a5f34022d780bc43271d694d90f495e5a9 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 13:56:48 -0700 Subject: [PATCH 040/170] Soften UX requirements --- specification/modules/room_upgrades.rst | 33 ++++++------------------- 1 file changed, 7 insertions(+), 26 deletions(-) diff --git a/specification/modules/room_upgrades.rst b/specification/modules/room_upgrades.rst index 81da8aaf4..2015a4b11 100644 --- a/specification/modules/room_upgrades.rst +++ b/specification/modules/room_upgrades.rst @@ -29,32 +29,13 @@ Events Client behaviour ---------------- -Clients which understand ``m.room.tombstone`` events MUST: - -* Hide the old room from the user's list of rooms. Permalinks, backlinks, etc should - still be accessible to this room, however. -* At the very bottom of the old room's timeline/message view, display a message which - indicates the room has been upgraded with a permalink to the new room. When the user - accesses the permalink, they are to be joined to the new room. - - * Note that if the homeserver doesn't support the room version that the user may - receive an error upon trying to join. - * If the replacement room has a tombstone event, the client may automatically follow - the chain until it finds a room which does not have a tombstone in it. - -* Optionally, the client may wish to improve the presentation of unread messages when - the user is in both the old and new rooms. For example, if the user was not active - during the upgrade and had unread messages in the old room, the new room may have an - indicator which shows the sum of unread notifications between the rooms. - -Clients which understand ``m.room.tombstone`` events must also understand the ``predecessor`` -field on ``m.room.create`` events such that: - -* At the top of the scrollback/message view for the new room a message is displayed - indicating that the room is a continuation of a previous room, with a permalink to - the old room. -* Optionally supporting search and other functions of the room to span across the old - and new rooms. +Clients which understand ``m.room.tombstone`` events and the ``predecessor`` field on +``m.room.create`` events should communicate to the user that the room was upgraded. +One way of accomplishing this would be to hide the old room from the user's room list +and showing banners linking between the old and new room - ensuring that permalinks +work when referencing the old room. Another approach may be to virtually merge the +rooms such that the old room's timeline seamlessly continues into the new timeline +without the user having to jump between the rooms. {{room_upgrades_cs_http_api}} From 2457438f1eacb0cf26636a02df9a7e2c913586d2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 13:57:02 -0700 Subject: [PATCH 041/170] Encourage servers to transfer whatever they can while being open-ended --- specification/modules/room_upgrades.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specification/modules/room_upgrades.rst b/specification/modules/room_upgrades.rst index 2015a4b11..21a1ed2db 100644 --- a/specification/modules/room_upgrades.rst +++ b/specification/modules/room_upgrades.rst @@ -48,10 +48,14 @@ When the client requests to upgrade a known room to a known version, the server: 2. Creates a replacement room with a ``m.room.create`` event containing a ``predecessor`` field and the applicable ``room_version``. 3. Replicates the power levels, privacy, topic, and other transferable state events to - the new room. This generally excludes membership events. + the new room. This generally excludes membership events but may include client-specified + events and other presentation details. 4. Moves any local aliases to the new room. 5. Sends a ``m.room.tombstone`` event to the old room to indicate that it is not intended to be used any further. 6. If possible, the power levels in the old room should also be modified to prevent sending of events and inviting new users. For example, setting ``events_default`` and ``invite`` to the greater of ``50`` and ``users_default + 1``. + +When a user joins the new room, the server may wish to automatically transfer/replicate +some of the user's personalized settings such as notifications, tags, etc. From f5fa7e59241cc7d1907a11e0d194aa597f0c3439 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 14:05:30 -0700 Subject: [PATCH 042/170] Change wording to encourage servers to make reasonable decisions --- api/client-server/versions.yaml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/api/client-server/versions.yaml b/api/client-server/versions.yaml index 4d0fd49e8..109e852ca 100644 --- a/api/client-server/versions.yaml +++ b/api/client-server/versions.yaml @@ -39,9 +39,12 @@ paths: may optionally include version information within their name if desired. Features listed here are not for optionally toggling parts of the Matrix specification and should only be used to advertise support for a feature - which has not yet landed in the spec. For example, an accepted proposal - for a feature needing implementation would advertise itself here with - the intention of being removed from this list once the spec changes land. + which has not yet landed in the spec. For example, a feature currently + undergoing the proposal process may appear here and eventually be taken + off this list once the feature lands in the spec and the server deems it + reasonable to do so. Servers may wish to keep advertising features here + after they've been released into the spec to give clients a chance to + upgrade appropriately. operationId: getVersions responses: 200: From a6f5d015867eff953e8f6690c78d7ce1c9a0a729 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 14:39:06 -0700 Subject: [PATCH 043/170] Clarify that servers don't have to implement development/beta versions --- specification/index.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index 2f17ec298..ff51f7c02 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -468,11 +468,12 @@ Currently, this is split into 5 categories: * **Development**: This is the default state for all room versions. When in this state, a room version is documented but not recommended for use outside of a - development environment. These versions are not production-ready. + development environment. These versions are not production-ready and servers + are not required to implement these versions. * **Beta**: Versions in this state are not intended for wide-spread usage but should be stable enough if a room requires the feature(s) introduced within. Rooms will need to opt-in to these versions and should not be promoted to - upgrade. + upgrade. Servers additionally do not have to implement these versions. * **Default**: Exactly 1 room version will be in this category. The version under this category should be used when creating rooms (unless another version is requested by the user). Servers may wish to promote rooms to opt-in to this From 96d754f42904b62da0eb6bba17263e563b346e51 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 14:43:11 -0700 Subject: [PATCH 044/170] promote -> prompt --- specification/index.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index ff51f7c02..f4abf2921 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -472,14 +472,14 @@ Currently, this is split into 5 categories: are not required to implement these versions. * **Beta**: Versions in this state are not intended for wide-spread usage but should be stable enough if a room requires the feature(s) introduced within. - Rooms will need to opt-in to these versions and should not be promoted to + Rooms will need to opt-in to these versions and should not be prompted to upgrade. Servers additionally do not have to implement these versions. * **Default**: Exactly 1 room version will be in this category. The version under this category should be used when creating rooms (unless another version is - requested by the user). Servers may wish to promote rooms to opt-in to this + requested by the user). Servers may wish to prompt rooms to opt-in to this version. -* **Recommended**: Exactly 1 room version will be in this category as well. The - version here should be heavily promoted by servers for rooms to opt-in to using. +* **Recommended**: Exactly 1 room version will be in this category as well. Servers + should prompt as strongly as possible for rooms to opt-in to upgrade to this version. This version is often going to be the same as the Default version. * **Mandatory**: Servers are required to implement versions in this category. When a version is flagged as mandatory, additional rules may apply to rooms. For example, From 0dde2489b666be2c6448a785e27acb15e13f8581 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 14:48:42 -0700 Subject: [PATCH 045/170] Clarify what a Mandatory room version is --- specification/index.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index f4abf2921..cdd56cb2f 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -481,10 +481,13 @@ Currently, this is split into 5 categories: * **Recommended**: Exactly 1 room version will be in this category as well. Servers should prompt as strongly as possible for rooms to opt-in to upgrade to this version. This version is often going to be the same as the Default version. -* **Mandatory**: Servers are required to implement versions in this category. When - a version is flagged as mandatory, additional rules may apply to rooms. For example, - servers may be required to stop supporting another room version and automatically - upgrade all affected rooms. +* **Mandatory**: Servers are required to implement versions in this category, + regardless as to how they are otherwise categorized. This allows for a Beta room + version to be mandatorily implemented by all servers in extremely rare circumstances, + as an example. Being a Mandatory room version does not imply that it is Recommended + or a Default version, just that the server needs to support it. Additional rules + may apply to room versions which are Mandatory, such as forcing servers to upgrade + all known rooms to a particular version where possible. With the above categories, the following applies: From 19e94815f9bc65bacca00e9c3c078d983a941b91 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 15:13:01 -0700 Subject: [PATCH 046/170] Try and improve the understanding of room versions --- specification/index.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/specification/index.rst b/specification/index.rst index cdd56cb2f..ea4026f0b 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -430,6 +430,13 @@ upon through new algorithms or rules, "room versions" are employed to manage a set of expectations for each room. New room versions are assigned as needed. +There is no implicit ordering or hierarchy to room versions, and their principles +are immutable once placed in the specification. Although there is a recommended +set of versions, some rooms may benefit from features introduced by other versions. +Rooms move between different versions by "upgrading" to the desired version. Due +to versions not being ordered or hierarchical, this means a room can "upgrade" to +version 1 from version 2, if it so desired. + Room version grammar ~~~~~~~~~~~~~~~~~~~~ From ba37f2d311aaba01dfa395195f14394e1c91f633 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 16:19:25 -0700 Subject: [PATCH 047/170] prompt->advertise --- specification/index.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index ea4026f0b..67a0fa962 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -479,15 +479,15 @@ Currently, this is split into 5 categories: are not required to implement these versions. * **Beta**: Versions in this state are not intended for wide-spread usage but should be stable enough if a room requires the feature(s) introduced within. - Rooms will need to opt-in to these versions and should not be prompted to - upgrade. Servers additionally do not have to implement these versions. + Rooms may opt-in to these versions on their own, but should not be asked to + upgrade automatically. Servers do not have to implement these versions. * **Default**: Exactly 1 room version will be in this category. The version under this category should be used when creating rooms (unless another version is - requested by the user). Servers may wish to prompt rooms to opt-in to this + requested by the user). Servers may wish to advertise that rooms opt-in to this version. * **Recommended**: Exactly 1 room version will be in this category as well. Servers - should prompt as strongly as possible for rooms to opt-in to upgrade to this version. - This version is often going to be the same as the Default version. + should advertise as strongly as possible for rooms to opt-in to upgrade to this + version. This version is often going to be the same as the Default version. * **Mandatory**: Servers are required to implement versions in this category, regardless as to how they are otherwise categorized. This allows for a Beta room version to be mandatorily implemented by all servers in extremely rare circumstances, From ebe887d9314b3533b80956b0ef803e3b7e319be2 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 18 Jan 2019 09:56:04 -0700 Subject: [PATCH 048/170] Grammar Co-Authored-By: turt2live --- specification/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/index.rst b/specification/index.rst index 67a0fa962..0b024c9e3 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -435,7 +435,7 @@ are immutable once placed in the specification. Although there is a recommended set of versions, some rooms may benefit from features introduced by other versions. Rooms move between different versions by "upgrading" to the desired version. Due to versions not being ordered or hierarchical, this means a room can "upgrade" to -version 1 from version 2, if it so desired. +version 1 from version 2, if it is so desired. Room version grammar ~~~~~~~~~~~~~~~~~~~~ From 3ec3adbc24f287d8d38b837bbd5d1eb1a01fcd44 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 22 Jan 2019 10:51:22 +0000 Subject: [PATCH 049/170] Proposal for add room_version to make_* fed APIs --- proposals/1812-federation-make-membership.md | 21 ++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 proposals/1812-federation-make-membership.md diff --git a/proposals/1812-federation-make-membership.md b/proposals/1812-federation-make-membership.md new file mode 100644 index 000000000..7960bf42d --- /dev/null +++ b/proposals/1812-federation-make-membership.md @@ -0,0 +1,21 @@ +# MSC 1813 - Federation Make Membership Room Version + +This proposal adds a new `room_version` field to the responses of `/make_leave` +and `/make_join` APIs. + +## Motivation + +It is planned for future room versions to be able to change the format of events +in various ways. To support this, all servers must know the room version +whenever they need to parse an event. Currently the `/make_*` APIs do not +include the room version in the response, so the requesting server is unable to +parse the event included in the response body. + +## Proposal + +Add a new `room_version` field to the response body of the `v1` `/make_join` and +`/make_leave` APIs, which describes the room version. + +For backwards compatibility servers must correctly handle responses that do not +include the new field. In which case the room version is assumed to be one of +either `1` or `2`. From bd5e760a0d1a8fe3498fd77e27b948504762bef6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 22 Jan 2019 18:02:21 -0700 Subject: [PATCH 050/170] Simplify the description for room versions Instead of trying to describe maturity, stability, and recommendedness in one list we should describe what is "safe" and "unsafe" to use. The default version is just something that servers should use, and is normally going to be stable. --- specification/index.rst | 49 ++++++++++------------------------------- 1 file changed, 12 insertions(+), 37 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index 0b024c9e3..e5f01abf9 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -467,48 +467,23 @@ Examples of valid room versions are: * ``1.2-beta`` * ``com.example.version`` -Recommended room versions -~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are varying degrees of what "recommended" means for a given room version. -Currently, this is split into 5 categories: - -* **Development**: This is the default state for all room versions. When in this - state, a room version is documented but not recommended for use outside of a - development environment. These versions are not production-ready and servers - are not required to implement these versions. -* **Beta**: Versions in this state are not intended for wide-spread usage but - should be stable enough if a room requires the feature(s) introduced within. - Rooms may opt-in to these versions on their own, but should not be asked to - upgrade automatically. Servers do not have to implement these versions. -* **Default**: Exactly 1 room version will be in this category. The version under - this category should be used when creating rooms (unless another version is - requested by the user). Servers may wish to advertise that rooms opt-in to this - version. -* **Recommended**: Exactly 1 room version will be in this category as well. Servers - should advertise as strongly as possible for rooms to opt-in to upgrade to this - version. This version is often going to be the same as the Default version. -* **Mandatory**: Servers are required to implement versions in this category, - regardless as to how they are otherwise categorized. This allows for a Beta room - version to be mandatorily implemented by all servers in extremely rare circumstances, - as an example. Being a Mandatory room version does not imply that it is Recommended - or a Default version, just that the server needs to support it. Additional rules - may apply to room versions which are Mandatory, such as forcing servers to upgrade - all known rooms to a particular version where possible. - -With the above categories, the following applies: - -* Servers MUST have Room Version 1 as the Default room version. -* Servers MUST have Room Version 1 as the Recommended room version. -* Servers MUST implement Room Version 1 as a Mandatory room version. - Complete list of room versions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Room versions are divided into two distinct groups: stable and unstable. Stable +room versions may be used by rooms safely. Unstable room versions are everything +else which is either not listed in the specification or flagged as unstable for +some other reason. Versions can switch between stable and unstable periodically +for a variety of reasons, including discovered security vulnerabilites and age. + +Clients should not ask room administrators to upgrade their rooms if the room is +running a stable version. Servers SHOULD use room version 1 as the default room +version when creating new rooms. + The available room versions are: -* `Version 1 `_ - The current version of most rooms. -* `Version 2 `_ - **Beta**. Implements State Resolution Version 2. +* `Version 1 `_ - **Stable**. The current version of most rooms. +* `Version 2 `_ - **Stable**. Implements State Resolution Version 2. Specification Versions ---------------------- From 0dfc64a9f4f1772c5839c60fac63f15a8cc7603e Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 22 Jan 2019 18:11:36 -0700 Subject: [PATCH 051/170] Improve wording --- specification/modules/room_upgrades.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/room_upgrades.rst b/specification/modules/room_upgrades.rst index 21a1ed2db..49ff4414d 100644 --- a/specification/modules/room_upgrades.rst +++ b/specification/modules/room_upgrades.rst @@ -31,7 +31,7 @@ Client behaviour Clients which understand ``m.room.tombstone`` events and the ``predecessor`` field on ``m.room.create`` events should communicate to the user that the room was upgraded. -One way of accomplishing this would be to hide the old room from the user's room list +One way of accomplishing this would be hiding the old room from the user's room list and showing banners linking between the old and new room - ensuring that permalinks work when referencing the old room. Another approach may be to virtually merge the rooms such that the old room's timeline seamlessly continues into the new timeline From 50eba23669bf10d7b7b3f707ed11d70d51aa650a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 22 Jan 2019 18:24:04 -0700 Subject: [PATCH 052/170] Update MSC to match how room versions work As per 1773. --- .../1804-advertising-capable-room-versions.md | 38 +++++++------------ 1 file changed, 14 insertions(+), 24 deletions(-) diff --git a/proposals/1804-advertising-capable-room-versions.md b/proposals/1804-advertising-capable-room-versions.md index 7686f1b2b..abcf56e99 100644 --- a/proposals/1804-advertising-capable-room-versions.md +++ b/proposals/1804-advertising-capable-room-versions.md @@ -3,7 +3,7 @@ Currently clients need to guess at which room versions the server supports, if any. This is particularly difficult to do as it generally requires knowing the state of the ecosystem and what versions are available and how keen users are to upgrade their servers and clients. The impossible judgement call -for when to prompt people to upgrade shouldn't be impossible, or even a judgement call. +for when to encourage people to upgrade shouldn't be impossible, or even a judgement call. ## Proposal @@ -16,37 +16,27 @@ this proposal suggests a `m.room_versions` capability be introduced like the fol { "capabilities": { "m.room_versions": { - "m.development": ["state-res-v2-test", "event-ids-as-hashes-test", "3"], - "m.beta": ["2"], - "m.default": "1", - "m.recommended": "1", - "m.mandatory": ["1"] + "default": "1", + "available": { + "1": "stable", + "2": "stable", + "state-res-v2-test": "unstable", + "event-ids-as-hashes": "unstable", + "3": "unstable" + } } } } ``` -The labels picked (`m.development`, etc) are based upon the categories for different room versions. -Note that `m.default` and `m.recommended` reinforce that there is exactly 1 version in each category. - Clients are encouraged to make use of this capability to determine if the server supports a given -version, and at what state. - -Clients should prompt people with sufficient permissions to perform an upgrade to upgrade their rooms -to the `m.recommended` room version. - -Room versions might appear under multiple categories under some circumstances. In particular, it is -expected that anything in `m.development` or `m.beta` appears exactly once in the whole capability -whereas `m.default`, `m.recommended`, and `m.mandatory` may duplicate a room version. The duplication -is possible due to the definitions of each category: +version, and at what level of stability. -* `m.default` - This is the room version that the server is going to apply to all new rooms by default. -* `m.recommended` - The version clients should be prompting people to upgrade to. -* `m.mandatory` - The version the server is going to enforce on all pre-existing rooms. +The default version is the version that the server is using to create new rooms with. Clients can +make the assumption that the default version is a stable version. -With these definitions, it is possible that a room version fits multiple criteria (ie: "please upgrade -your rooms to version X which is also the default for new rooms"). Clients will generally only be -interested in the `m.recommended` room version, leaving the rest as informational for users. +Clients should encourage people with sufficient permissions to perform an upgrade to upgrade their +rooms to the `default` room version when the room is using an `unstable` version. ## Potential issues From 5cafcd103fbdbce8122fbe8f67c0d65d08ac28ac Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 22 Jan 2019 21:43:32 -0700 Subject: [PATCH 053/170] Fix copyright > Since this is a copy-and-paste of old text, I think the copyright year should match when the original text was written, which according to git was 2017. Co-Authored-By: turt2live --- specification/rooms/v1.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index c263a1080..207fe8bcd 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -1,4 +1,4 @@ -.. Copyright 2018-2019 New Vector Ltd +.. Copyright 2017,2019 New Vector Ltd .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. From 061f59547a393ab2d1d52eacf76368bc8c3800e6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 23 Jan 2019 09:10:14 -0700 Subject: [PATCH 054/170] Minor wording changes --- specification/index.rst | 8 ++++---- specification/rooms/v1.rst | 4 ++-- specification/rooms/v2.rst | 4 ++-- specification/server_server_api.rst | 5 ++--- 4 files changed, 10 insertions(+), 11 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index e5f01abf9..0a2224d22 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -434,16 +434,16 @@ There is no implicit ordering or hierarchy to room versions, and their principle are immutable once placed in the specification. Although there is a recommended set of versions, some rooms may benefit from features introduced by other versions. Rooms move between different versions by "upgrading" to the desired version. Due -to versions not being ordered or hierarchical, this means a room can "upgrade" to -version 1 from version 2, if it is so desired. +to versions not being ordered or hierarchical, this means a room can "upgrade" +from version 2 to version 1, if it is so desired. Room version grammar ~~~~~~~~~~~~~~~~~~~~ Room versions are used to change properties of rooms that may not be compatible with other servers. For example, changing the rules for event authorization would -cause older servers to potentially end up in a split-brain situation due to them -not understanding the new rules. +cause older servers to potentially end up in a split-brain situation due to not +understanding the new rules. A room version is defined as a string of characters which MUST NOT exceed 32 codepoints in length. Room versions MUST NOT be empty and SHOULD contain only diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 207fe8bcd..37b1b7ced 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -34,8 +34,8 @@ version room they are dealing with prior to executing a given algorithm. .. WARNING:: Although room version 1 is the most popular room version, it is known to have undesirable effects. Servers implementing support for room version 1 should be - aware that restrictions should be generally relaxed and be aware that inconsistencies - may occur until room version 2 is ready and adopted. + aware that restrictions should be generally relaxed and that inconsistencies + may occur until room version 2 (or later) is ready and adopted. State resolution ~~~~~~~~~~~~~~~~ diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index eef034977..d39a7caa3 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -60,8 +60,8 @@ Power events A *power event* is a state event with type ``m.room.power_levels`` or ``m.room.join_rules``, or a state event with type ``m.room.member`` where the ``membership`` is ``leave`` or ``ban`` and the ``sender`` does not match the - ``state_key``. The idea behind this is that power events are events that have - may remove someone's ability to do something in the room. + ``state_key``. The idea behind this is that power events are events that might + remove someone's ability to do something in the room. Unconflicted state map and conflicted state set The *unconflicted state map* is the state where the value of each key exists diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 8645888b7..09ad333fb 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -752,9 +752,8 @@ is at the top):: Suppose E3 and E4 are both ``m.room.name`` events which set the name of the room. What should the name of the room be at E5? -Servers must use the appropriate recursively-defined algorithm as required -by the room version. For a description of each room version's algorithm, please -see the `room version specification`_ . +The algorithm to be used for state resolution depends on the room version. For +a description of each room version's algorithm, please see the `room version specification`_. Backfilling and retrieving missing events From 413bfaeb68f3e9c072adf539aa54c2b6c7ee218f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 23 Jan 2019 19:03:14 -0700 Subject: [PATCH 055/170] Add clarification that clients shouldn't use unstable things as stable --- api/client-server/versions.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/api/client-server/versions.yaml b/api/client-server/versions.yaml index 109e852ca..2464af1dc 100644 --- a/api/client-server/versions.yaml +++ b/api/client-server/versions.yaml @@ -44,7 +44,8 @@ paths: off this list once the feature lands in the spec and the server deems it reasonable to do so. Servers may wish to keep advertising features here after they've been released into the spec to give clients a chance to - upgrade appropriately. + upgrade appropriately. Additionally, clients should avoid using unstable + features in their stable releases. operationId: getVersions responses: 200: From 577edeb53b7cf4dafa37b88f9d04cb90d91c6b3f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 25 Jan 2019 09:27:23 -0700 Subject: [PATCH 056/170] Say that !stable == unstable --- proposals/1804-advertising-capable-room-versions.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/proposals/1804-advertising-capable-room-versions.md b/proposals/1804-advertising-capable-room-versions.md index abcf56e99..684bd5243 100644 --- a/proposals/1804-advertising-capable-room-versions.md +++ b/proposals/1804-advertising-capable-room-versions.md @@ -22,7 +22,7 @@ this proposal suggests a `m.room_versions` capability be introduced like the fol "2": "stable", "state-res-v2-test": "unstable", "event-ids-as-hashes": "unstable", - "3": "unstable" + "3": "future-scifi-label" } } } @@ -30,7 +30,8 @@ this proposal suggests a `m.room_versions` capability be introduced like the fol ``` Clients are encouraged to make use of this capability to determine if the server supports a given -version, and at what level of stability. +version, and at what level of stability. Anything not flagged explicitly as `stable` should be treated +as `unstable` (ie: `future-scifi-label` is the same as `unstable`). The default version is the version that the server is using to create new rooms with. Clients can make the assumption that the default version is a stable version. From 9f443225ac377afb1cf2c0e7227ba394058b093d Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Sat, 26 Jan 2019 20:10:32 +0000 Subject: [PATCH 057/170] fix incorrect spelling of homeserver --- specification/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index 0a2224d22..3893ac4cc 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -315,8 +315,8 @@ The following conceptual diagram shows an | Content: { JSON object } | |....................................| -Federation maintains *shared data structures* per-room between multiple home -servers. The data is split into ``message events`` and ``state events``. +Federation maintains *shared data structures* per-room between multiple +homeservers. The data is split into ``message events`` and ``state events``. Message events: These describe transient 'once-off' activity in a room such as an From bf2b6e6daac2fc7cd7240050737cf372ed6ac60b Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Mon, 28 Jan 2019 15:05:10 +0000 Subject: [PATCH 058/170] remove references to presence lists --- api/client-server/presence.yaml | 100 ------------------ .../event-schemas/m.device_list_update.yaml | 2 +- specification/modules/presence.rst | 19 +--- specification/server_server_api.rst | 20 +--- 4 files changed, 7 insertions(+), 134 deletions(-) diff --git a/api/client-server/presence.yaml b/api/client-server/presence.yaml index 91b75c6a8..874ac59b9 100644 --- a/api/client-server/presence.yaml +++ b/api/client-server/presence.yaml @@ -136,103 +136,3 @@ paths: "$ref": "definitions/errors/error.yaml" tags: - Presence - "/presence/list/{userId}": - post: - summary: Add or remove users from this presence list. - description: |- - Adds or removes users from this presence list. - operationId: modifyPresenceList - security: - - accessToken: [] - parameters: - - in: path - type: string - name: userId - description: The user whose presence list is being modified. - required: true - x-example: "@alice:example.com" - - in: body - name: presence_diff - description: The modifications to make to this presence list. - required: true - schema: - type: object - example: { - "invite": [ - "@bob:matrix.org" - ], - "drop": [ - "@alice:matrix.org" - ] - } - properties: - invite: - type: array - description: A list of user IDs to add to the list. - items: - type: string - description: A list of user IDs. - drop: - type: array - description: A list of user IDs to remove from the list. - items: - type: string - description: A list of user IDs. - responses: - 200: - description: The list was updated. - examples: - application/json: { - } - schema: - type: object # empty json object - 429: - description: This request was rate-limited. - schema: - "$ref": "definitions/errors/rate_limited.yaml" - tags: - - Presence - get: - summary: Get presence events for this presence list. - description: |- - Retrieve a list of presence events for every user on this list. - operationId: getPresenceForList - parameters: - - in: path - type: string - name: userId - description: The user whose presence list should be retrieved. - required: true - x-example: "@alice:example.com" - responses: - 200: - description: A list of presence events for this list. - examples: - application/json: [ - { - "content": { - "last_active_ago": 395, - "presence": "offline", - "user_id": "@alice:matrix.org" - }, - "type": "m.presence" - }, - { - "content": { - "last_active_ago": 16874, - "presence": "online", - "user_id": "@marisa:matrix.org", - "currently_active": true - }, - "type": "m.presence" - } - ] - schema: - type: array - items: - type: object - title: PresenceEvent - allOf: - - "$ref": "definitions/event-schemas/schema/core-event-schema/event.yaml" - tags: - - Presence diff --git a/api/server-server/definitions/event-schemas/m.device_list_update.yaml b/api/server-server/definitions/event-schemas/m.device_list_update.yaml index f11e957e2..d511bfae7 100644 --- a/api/server-server/definitions/event-schemas/m.device_list_update.yaml +++ b/api/server-server/definitions/event-schemas/m.device_list_update.yaml @@ -21,7 +21,7 @@ description: |- # FIXME: It's very unclear why we have this API surface for synchronising # device lists, rather than just using a room (which could also be used for -# presence lists, profile info, etc). But documenting what we have for now... +# profile info, etc). But documenting what we have for now... allOf: - $ref: ../edu.yaml diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst index 822b298d4..4873776a3 100644 --- a/specification/modules/presence.rst +++ b/specification/modules/presence.rst @@ -26,14 +26,8 @@ Each user has the concept of presence information. This encodes: This information is collated from both per-device (``online``, ``idle``, ``last_active``) and per-user (status) data, aggregated by the user's homeserver -and transmitted as an ``m.presence`` event. This is one of the few events which -are sent *outside the context of a room*. Presence events are sent to all users -who subscribe to this user's presence through a presence list or by sharing -membership of a room. - -A presence list is a list of user IDs whose presence the user wants to follow. -To be added to this list, the user being added must be invited by the list owner -who must accept the invitation. +and transmitted as an ``m.presence`` event. Presence event are sent to +interested parties where users share a room membership. User's presence state is represented by the ``presence`` key, which is an enum of one of the following: @@ -53,17 +47,10 @@ Events Client behaviour ---------------- -Clients can manually set/get their presence/presence list using the HTTP APIs -listed below. +Clients can manually set/get their presence using the HTTP APIs listed below. {{presence_cs_http_api}} -Server behaviour ----------------- - -Each user's homeserver stores a "presence list" per user. Once a user accepts -a presence list, both user's HSes must track the subscription. - Last active ago ~~~~~~~~~~~~~~~ The server maintains a timestamp of the last time it saw a pro-active event from diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 09ad333fb..7457cfd4e 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -1012,15 +1012,8 @@ The server API for presence is based entirely on exchange of the following EDUs. There are no PDUs or Federation Queries involved. Servers should only send presence updates for users that the receiving server -would be interested in. This can include the receiving server sharing a room -with a given user, or a user on the receiving server has added one of the -sending server's users to their presence list. - -Clients may define lists of users that they are interested in via "Presence -Lists" through the `Client-Server API`_. When users are added to a presence -list, a ``m.presence_invite`` EDU is sent to them. The user may then accept -or deny their involvement in the list by sending either an ``m.presence_accept`` -or ``m.presence_deny`` EDU back. +would be interested in. Such as the receiving server sharing a room +with a given user. .. TODO-doc - Explain the timing-based round-trip reduction mechanism for presence @@ -1030,13 +1023,6 @@ or ``m.presence_deny`` EDU back. {{definition_ss_event_schemas_m_presence}} -{{definition_ss_event_schemas_m_presence_invite}} - -{{definition_ss_event_schemas_m_presence_accept}} - -{{definition_ss_event_schemas_m_presence_deny}} - - Receipts -------- @@ -1114,7 +1100,7 @@ which should be used to correlate with subsequent ``m.device_list_update`` EDUs. .. TODO: this whole thing desperately feels like it should just be state in a room, rather than inventing a whole different DAG. The same room could be used for - profiles, presence lists, etc. + profiles etc. {{user_devices_ss_http_api}} From ed68f940cd899183d36a19ed39a165a2fdaf4f8f Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Mon, 28 Jan 2019 17:49:20 +0000 Subject: [PATCH 059/170] towncrier --- changelogs/client_server/newsfragments/1817.deprecation | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1817.deprecation diff --git a/changelogs/client_server/newsfragments/1817.deprecation b/changelogs/client_server/newsfragments/1817.deprecation new file mode 100644 index 000000000..9a888e85d --- /dev/null +++ b/changelogs/client_server/newsfragments/1817.deprecation @@ -0,0 +1 @@ +Remove references to presence lists From 9f517f33cee2a555fa635b9291a2422b27ff7802 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Mon, 28 Jan 2019 17:59:17 +0000 Subject: [PATCH 060/170] Create 1819-remove-presence-lists.md --- proposals/1819-remove-presence-lists.md | 26 +++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 proposals/1819-remove-presence-lists.md diff --git a/proposals/1819-remove-presence-lists.md b/proposals/1819-remove-presence-lists.md new file mode 100644 index 000000000..7595f5699 --- /dev/null +++ b/proposals/1819-remove-presence-lists.md @@ -0,0 +1,26 @@ +# Remove references to presence lists + +[Presence](https://matrix.org/docs/spec/client_server/r0.4.0.html#id388) lists allow a given user the ability to subscribe to other users and be alerted to their current presence status. + +While spec'd, no established client has implemented support and the only server side implementation raises privacy concerns. + +The proposal is to simply remove references to presence lists with a view to revisiting the same ideas in the future. + +This MSC addresses [#1810](https://github.com/matrix-org/matrix-doc/issues/1810) + +## Proposal + +Presence lists seem like a good fit for ['MSC1769: Extensible profiles as rooms'](https://github.com/matrix-org/matrix-doc/pull/1769) proposal, meaning that the current design will most likely be superceded. + +Additionally, no major client has implemented the behaviour to date and the only server implementation of presence lists (Synapse) auto-accepts invites leading to privacy concerns in the wild. + +With this in mind the most pragmatic solution is to remove presence lists ahead of the r0 release. + +## Tradeoffs + +Ideally this proposal would also come with an alternative design for this functionality. Out of pragmatism the proposal only covers removal of what is there today. + + +## Conclusions + +This is a common sense attempt to remove unused portions of the spec ahead of an r0 release. It does not suggest that the ability to subscribe to the presence of others is undesirable and assumes that this behvaiour will return again in some form. From aca9437bf37fef38986a52a312bf7f4c58a4f5e7 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 28 Jan 2019 21:05:33 -0700 Subject: [PATCH 061/170] Specification for v2 s2s invite API Original proposal: https://github.com/matrix-org/matrix-doc/pull/1794 Implementation proofs: * https://github.com/matrix-org/synapse/pull/4402 * https://github.com/matrix-org/synapse/pull/4496 There are no changes from the original proposal. --- .../{invites.yaml => invites-v1.yaml} | 12 +- api/server-server/invites-v2.yaml | 234 ++++++++++++++++++ specification/server_server_api.rst | 7 +- 3 files changed, 246 insertions(+), 7 deletions(-) rename api/server-server/{invites.yaml => invites-v1.yaml} (96%) create mode 100644 api/server-server/invites-v2.yaml diff --git a/api/server-server/invites.yaml b/api/server-server/invites-v1.yaml similarity index 96% rename from api/server-server/invites.yaml rename to api/server-server/invites-v1.yaml index 6d905e178..44db1f8da 100644 --- a/api/server-server/invites.yaml +++ b/api/server-server/invites-v1.yaml @@ -1,4 +1,4 @@ -# Copyright 2018 New Vector Ltd +# Copyright 2018-2019 New Vector Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -20,7 +20,7 @@ host: localhost:8448 schemes: - https basePath: /_matrix/federation/v1 -consumes: +consumes: - application/json produces: - application/json @@ -32,9 +32,13 @@ paths: summary: Invites a remote user to a room description: |- Invites a remote user to a room. Once the event has been signed by both the inviting - homeserver and the invited homeserver, it can be sent to all of the servers in the + homeserver and the invited homeserver, it can be sent to all of the servers in the room by the inviting homeserver. - operationId: sendInvite + + Servers should prefer to use the v2 API for invites instead of the v1 API. Servers + which receive a v1 invite request must assume that the room version is either ``"1"`` + or ``"2"``. + operationId: sendInviteV1 security: - signedRequest: [] parameters: diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml new file mode 100644 index 000000000..4ebbd2189 --- /dev/null +++ b/api/server-server/invites-v2.yaml @@ -0,0 +1,234 @@ +# Copyright 2018-2019 New Vector Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +swagger: '2.0' +info: + title: "Matrix Federation Invite User To Room API" + version: "1.0.0" +host: localhost:8448 +schemes: + - https +basePath: /_matrix/federation/v2 +consumes: + - application/json +produces: + - application/json +securityDefinitions: + $ref: definitions/security.yaml +paths: + "/invite/{roomId}/{eventId}": + put: + summary: Invites a remote user to a room + description: |- + .. Note:: + This API is nearly identical to the v1 API with the exception of the request + body being different, and the response format fixed. + + Invites a remote user to a room. Once the event has been signed by both the inviting + homeserver and the invited homeserver, it can be sent to all of the servers in the + room by the inviting homeserver. + + This endpoint is preferred over the v1 API as it is more useful for servers. Senders + which receive a 400 or 404 response to this endpoint may wish to retry using the v1 + API as the server may be older, if the room version supports doing so. + operationId: sendInviteV2 + security: + - signedRequest: [] + parameters: + - in: path + name: roomId + type: string + description: The room ID that the user is being invited to. + required: true + x-example: "!abc123:matrix.org" + - in: path + name: eventId + type: string + description: The event ID for the invite event, generated by the inviting server. + required: true + x-example: "$abc123:example.org" + - in: body + name: body + type: object + required: true + schema: + type: object + properties: + room_version: + type: string + description: The version of the room where the user is being invited to. + example: "2" + event: + $ref: "definitions/invite_event.yaml" + invite_room_state: + type: array + description: |- + An optional list of simplified events to help the receiver of the invite + identify the room. The recommended events to include are the join rules, + canonical alias, avatar, and name of the room. + items: + type: object + title: Invite Room State Event + properties: + type: + type: string + description: The type of event. + example: "m.room.join_rules" + state_key: + type: string + description: The state key for the event. May be an empty string. + example: "" + content: + type: object + description: The content for the event. + sender: + type: string + description: The sender of the event. + example: "@someone:matrix.org" + required: ['type', 'state_key', 'content', 'sender'] + example: [ + { + "type": "m.room.join_rules", + "sender": "@someone:matrix.org", + "state_key": "", + "content": { + "join_rule": "public" + } + } + ] + required: ['room_version', 'event'] + example: { + "room_version": "2", + "event": { + "$ref": "examples/pdu.json", + "type": "m.room.member", + "state_key": "@joe:elsewhere.com", + "content": { + "membership": "invite" + }, + "signatures": { + "example.com": { + "ed25519:key_version": "SomeSignatureHere" + }, + } + }, + "invite_room_state": [ + { + "type": "m.room.join_rules", + "sender": "@someone:matrix.org", + "state_key": "", + "content": { + "join_rule": "public" + } + }, + { + "type": "m.room.name", + "sender": "@someone:matrix.org", + "state_key": "", + "content": { + "name": "Cool New Room" + } + } + ] + } + responses: + 200: + description: |- + The event with the invited server's signature added. All other fields of the events + should remain untouched. + schema: + type: object + description: An object containing the signed invite event. + title: Event Container + properties: + event: + $ref: "definitions/invite_event.yaml" + required: ['event'] + examples: + application/json: { + "event": { + "$ref": "examples/pdu.json", + "type": "m.room.member", + "state_key": "@someone:example.org", + "unsigned": { + "invite_room_state": [ + { + "type": "m.room.join_rules", + "sender": "@someone:matrix.org", + "state_key": "", + "content": { + "join_rule": "public" + } + }, + { + "type": "m.room.name", + "sender": "@someone:matrix.org", + "state_key": "", + "content": { + "name": "Cool New Room" + } + } + ] + }, + "content": { + "membership": "invite" + }, + "signatures": { + "example.com": { + "ed25519:key_version": "SomeSignatureHere" + }, + "elsewhere.com": { + "ed25519:k3y_versi0n": "SomeOtherSignatureHere" + } + } + } + } + 403: + description: |- + The invite is not allowed. This could be for a number of reasons, including: + + * The sender is not allowed to send invites to the target user/homeserver. + * The homeserver does not permit anyone to invite its users. + * The homeserver refuses to participate in the room. + schema: + $ref: "../client-server/definitions/errors/error.yaml" + examples: + application/json: { + "errcode": "M_FORBIDDEN", + "error": "User cannot invite the target user." + } + 400: + description: |- + The request is invalid or the room the server is attempting + to join has a version that is not listed in the ``ver`` + parameters. + + The error should be passed through to clients so that they + may give better feedback to users. + schema: + allOf: + - $ref: "../client-server/definitions/errors/error.yaml" + - type: object + properties: + room_version: + type: string + description: |- + The version of the room. Required if the ``errcode`` + is ``M_INCOMPATIBLE_ROOM_VERSION``. + examples: + application/json: { + "errcode": "M_INCOMPATIBLE_ROOM_VERSION", + "error": "Your homeserver does not support the features required to join this room", + "room_version": "3" + } diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 09ad333fb..a63bf0a6b 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -1,6 +1,5 @@ .. Copyright 2016 OpenMarket Ltd -.. Copyright 2017 New Vector Ltd -.. Copyright 2018 New Vector Ltd +.. Copyright 2017-2019 New Vector Ltd .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -889,7 +888,9 @@ the homeserver may sign the membership event itself and skip the process defined here. However, when a user invites another user on a different homeserver, a request to that homeserver to have the event signed and verified must be made. -{{invites_ss_http_api}} +{{invites_v1_ss_http_api}} + +{{invites_v2_ss_http_api}} Leaving Rooms (Rejecting Invites) --------------------------------- From d12593feeac9d3370afd8020e82eebab197fb967 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 28 Jan 2019 22:15:59 -0700 Subject: [PATCH 062/170] Specify the room_version response property on /make_{leave|join} Original proposal: https://github.com/matrix-org/matrix-doc/pull/1813 Implementation proof: https://github.com/matrix-org/synapse/pull/4447 There are no changes from the original proposal. --- api/server-server/joins.yaml | 15 +++++++++++---- api/server-server/leaving.yaml | 11 +++++++++-- 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/api/server-server/joins.yaml b/api/server-server/joins.yaml index 6d97e3aab..3c0ec48f0 100644 --- a/api/server-server/joins.yaml +++ b/api/server-server/joins.yaml @@ -65,6 +65,12 @@ paths: schema: type: object properties: + room_version: + type: string + description: |- + The version of the room where the server is trying to join. If not provided, + the room version is assumed to be either "1" or "2". + example: "2" event: allOf: - $ref: "definitions/unsigned_pdu.yaml" @@ -80,7 +86,7 @@ paths: origin: type: string description: The name of the resident homeserver. - example: "matrix.org" + example: "matrix.org" origin_server_ts: type: integer format: int64 @@ -113,7 +119,7 @@ paths: type: array description: |- An event reference list containing the authorization events that would - allow the member to join the room. This should normally be the + allow the member to join the room. This should normally be the ``m.room.create``, ``m.room.power_levels``, and ``m.room.join_rules`` events. items: @@ -143,7 +149,8 @@ paths: - state_key examples: application/json: { - event: { + "room_version": "2", + "event": { "$ref": "examples/unsigned_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", @@ -220,7 +227,7 @@ paths: origin: type: string description: The name of the joining homeserver. - example: "matrix.org" + example: "matrix.org" origin_server_ts: type: integer format: int64 diff --git a/api/server-server/leaving.yaml b/api/server-server/leaving.yaml index bb4bbe3d6..68ada9d70 100644 --- a/api/server-server/leaving.yaml +++ b/api/server-server/leaving.yaml @@ -57,6 +57,12 @@ paths: schema: type: object properties: + room_version: + type: string + description: |- + The version of the room where the server is trying to leave. If not provided, + the room version is assumed to be either "1" or "2". + example: "2" event: allOf: - $ref: "definitions/unsigned_pdu.yaml" @@ -101,7 +107,7 @@ paths: type: array description: |- An event reference list containing the authorization events that would - allow the member to leave the room. This should normally be the + allow the member to leave the room. This should normally be the ``m.room.create``, ``m.room.power_levels``, and ``m.room.join_rules`` events. items: @@ -131,6 +137,7 @@ paths: - state_key examples: application/json: { + "room_version": "2", "event": { "$ref": "examples/unsigned_pdu.json", "type": "m.room.member", @@ -194,7 +201,7 @@ paths: origin: type: string description: The name of the leaving homeserver. - example: "matrix.org" + example: "matrix.org" origin_server_ts: type: integer format: int64 From 0712fd36a2ed96e1fcf507832320e493358d8ea2 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 29 Jan 2019 14:36:26 +0000 Subject: [PATCH 063/170] 75% majority now required for FCP --- specification/proposals_intro.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/proposals_intro.rst b/specification/proposals_intro.rst index 96ffaafc0..d95128b70 100644 --- a/specification/proposals_intro.rst +++ b/specification/proposals_intro.rst @@ -141,7 +141,7 @@ follows: given, an FCP can be cancelled. It is often preceded by a comment summarising the current state of the discussion, along with reasoning for its occurrence. - A concern can be raised by a Core Team member at any time, which will block - an FCP from beginning. An FCP will only begin when a **majority** of core + an FCP from beginning. An FCP will only begin when a **75% majority** of core team members agree on its outcome, and all existing concerns have been resolved. - The FCP will then begin and last for 5 days, giving anyone else some time to @@ -233,7 +233,7 @@ Name GitHub Label Description =============================== ============================= ==================================== Proposal Drafting and Feedback N/A A proposal document which is still work-in-progress but is being shared to incorporate feedback. Please prefix your proposal's title with ``[WIP]`` to make it easier for reviewers to skim their notifications list. Proposal In Review proposal-in-review A proposal document which is now ready and waiting for review by the Core Team and community -Proposed Final Comment Period proposed-final-comment-period Currently awaiting signoff of a majority of team members in order to enter the final comment period +Proposed Final Comment Period proposed-final-comment-period Currently awaiting signoff of a 75% majority of team members in order to enter the final comment period Final Comment Period final-comment-period A proposal document which has reached final comment period either for merge, closure or postponement Final Commment Period Complete finished-final-comment-period The final comment period has been completed. Waiting for a demonstration implementation Spec PR Missing spec-pr-missing The proposal has been agreed, and proven with a demonstration implementation. Waiting for a PR against the Spec From de57d3950f0fb1fe7b0a9d21877d5e699709df36 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Tue, 29 Jan 2019 14:46:27 +0000 Subject: [PATCH 064/170] Relax the requirement for a content-type on .well-known (#1824) The main reason for this is that Apache etc won't stick a content-type on by default, because they don't know it's JSON, so requiring it seems like it will require unnecessary hoop-humping for everyone who wants to use a .well-known. --- proposals/1708-well-known-for-federation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index 8105a6381..d23c0e2ff 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -43,9 +43,9 @@ certificate validation, and following 30x redirects (being careful to avoid redirect loops). If the request does not return a 200, continue to step 4, otherwise: -The response must have a `Content-Type` of `application/json`, and must be -valid JSON which follows the structure documented below. Otherwise, the -request is aborted. +The response must be valid JSON which follows the structure documented +below. Otherwise, the request is aborted. It is NOT necessary for the response +to have a `Content-Type` of `application/json`. If the response is valid, the `m.server` property is parsed as `[:]`, and processed as follows: From c0039c30f28a546dab63de54a0ce00294460e707 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 29 Jan 2019 09:05:20 -0700 Subject: [PATCH 065/170] Minor wording changes from code review Co-Authored-By: turt2live --- api/server-server/invites-v2.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml index 4ebbd2189..f86951958 100644 --- a/api/server-server/invites-v2.yaml +++ b/api/server-server/invites-v2.yaml @@ -40,8 +40,8 @@ paths: room by the inviting homeserver. This endpoint is preferred over the v1 API as it is more useful for servers. Senders - which receive a 400 or 404 response to this endpoint may wish to retry using the v1 - API as the server may be older, if the room version supports doing so. + which receive a 400 or 404 response to this endpoint should retry using the v1 + API as the server may be older, if the room version is "1" or "2". operationId: sendInviteV2 security: - signedRequest: [] From c09fa6845ea6394588a10ee31115533c1620b65c Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Tue, 29 Jan 2019 18:23:30 +0000 Subject: [PATCH 066/170] Update presence.rst --- specification/modules/presence.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst index 4873776a3..53a335509 100644 --- a/specification/modules/presence.rst +++ b/specification/modules/presence.rst @@ -26,7 +26,7 @@ Each user has the concept of presence information. This encodes: This information is collated from both per-device (``online``, ``idle``, ``last_active``) and per-user (status) data, aggregated by the user's homeserver -and transmitted as an ``m.presence`` event. Presence event are sent to +and transmitted as an ``m.presence`` event. Presence events are sent to interested parties where users share a room membership. User's presence state is represented by the ``presence`` key, which is an enum From 5151aa0aa7bfa6ee3b8ce9c2d43deb2d6305aec4 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Wed, 30 Jan 2019 10:44:15 +0000 Subject: [PATCH 067/170] Update proposals/1819-remove-presence-lists.md Co-Authored-By: neilisfragile --- proposals/1819-remove-presence-lists.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1819-remove-presence-lists.md b/proposals/1819-remove-presence-lists.md index 7595f5699..e082f5fae 100644 --- a/proposals/1819-remove-presence-lists.md +++ b/proposals/1819-remove-presence-lists.md @@ -1,6 +1,6 @@ # Remove references to presence lists -[Presence](https://matrix.org/docs/spec/client_server/r0.4.0.html#id388) lists allow a given user the ability to subscribe to other users and be alerted to their current presence status. +[Presence](https://matrix.org/docs/spec/client_server/r0.4.0.html#id107) lists allow a given user the ability to subscribe to other users and be alerted to their current presence status. While spec'd, no established client has implemented support and the only server side implementation raises privacy concerns. From 4f13f5289e656813a43083bc84d81533241a5c8d Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 30 Jan 2019 11:20:38 +0000 Subject: [PATCH 068/170] Update 1819-remove-presence-lists.md Add references to exactly what this proposal would remove --- proposals/1819-remove-presence-lists.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/proposals/1819-remove-presence-lists.md b/proposals/1819-remove-presence-lists.md index e082f5fae..38c318835 100644 --- a/proposals/1819-remove-presence-lists.md +++ b/proposals/1819-remove-presence-lists.md @@ -16,6 +16,18 @@ Additionally, no major client has implemented the behaviour to date and the only With this in mind the most pragmatic solution is to remove presence lists ahead of the r0 release. +Specifically:- + +CS API: Remove +* [POST /_matrix/client/r0/presence/list/{userId}](https://matrix.org/docs/spec/client_server/r0.4.0.html#post-matrix-client-r0-presence-list-userid) +* [GET /_matrix/client/r0/presence/list/{userId}](https://matrix.org/docs/spec/client_server/r0.4.0.html#get-matrix-client-r0-presence-list-userid) + +SS API: Remove + * [m.presence_invite](https://matrix.org/docs/spec/server_server/unstable.html#m-presence-invite-schema) + * [m.presence_accept](https://matrix.org/docs/spec/server_server/unstable.html#m-presence-accept-schema) + * [m.presence_deny](https://matrix.org/docs/spec/server_server/unstable.html#m-presence-deny-schema) + + ## Tradeoffs Ideally this proposal would also come with an alternative design for this functionality. Out of pragmatism the proposal only covers removal of what is there today. From 2eae933ed6e03247ee3387ad0db267c19bff9679 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 30 Jan 2019 11:28:53 +0000 Subject: [PATCH 069/170] hard wrap to 80 chars --- proposals/1819-remove-presence-lists.md | 41 +++++++++++++++++-------- 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/proposals/1819-remove-presence-lists.md b/proposals/1819-remove-presence-lists.md index 38c318835..7c5425097 100644 --- a/proposals/1819-remove-presence-lists.md +++ b/proposals/1819-remove-presence-lists.md @@ -1,38 +1,55 @@ # Remove references to presence lists -[Presence](https://matrix.org/docs/spec/client_server/r0.4.0.html#id107) lists allow a given user the ability to subscribe to other users and be alerted to their current presence status. +[Presence](https://matrix.org/docs/spec/client_server/r0.4.0.html#id107) lists +allow a given user the ability to subscribe to other users and be alerted to +their current presence status. -While spec'd, no established client has implemented support and the only server side implementation raises privacy concerns. +While spec'd, no established client has implemented support and the only server +side implementation raises privacy concerns. -The proposal is to simply remove references to presence lists with a view to revisiting the same ideas in the future. +The proposal is to simply remove references to presence lists with a view to +revisiting the same ideas in the future. -This MSC addresses [#1810](https://github.com/matrix-org/matrix-doc/issues/1810) +This MSC addresses +[#1810](https://github.com/matrix-org/matrix-doc/issues/1810) ## Proposal -Presence lists seem like a good fit for ['MSC1769: Extensible profiles as rooms'](https://github.com/matrix-org/matrix-doc/pull/1769) proposal, meaning that the current design will most likely be superceded. +Presence lists seem like a good fit for ['MSC1769: Extensible profiles as +rooms'](https://github.com/matrix-org/matrix-doc/pull/1769) proposal, meaning +that the current design will most likely be superceded. -Additionally, no major client has implemented the behaviour to date and the only server implementation of presence lists (Synapse) auto-accepts invites leading to privacy concerns in the wild. +Additionally, no major client has implemented the behaviour to date and the +only server implementation of presence lists (Synapse) auto-accepts invites +leading to privacy concerns in the wild. -With this in mind the most pragmatic solution is to remove presence lists ahead of the r0 release. +With this in mind the most pragmatic solution is to remove presence lists ahead +of the r0 release. Specifically:- CS API: Remove -* [POST /_matrix/client/r0/presence/list/{userId}](https://matrix.org/docs/spec/client_server/r0.4.0.html#post-matrix-client-r0-presence-list-userid) -* [GET /_matrix/client/r0/presence/list/{userId}](https://matrix.org/docs/spec/client_server/r0.4.0.html#get-matrix-client-r0-presence-list-userid) +* [POST + /_matrix/client/r0/presence/list/{userId}](https://matrix.org/docs/spec/client_server/r0.4.0.html#post-matrix-client-r0-presence-list-userid) +* [GET + /_matrix/client/r0/presence/list/{userId}](https://matrix.org/docs/spec/client_server/r0.4.0.html#get-matrix-client-r0-presence-list-userid) SS API: Remove * [m.presence_invite](https://matrix.org/docs/spec/server_server/unstable.html#m-presence-invite-schema) * [m.presence_accept](https://matrix.org/docs/spec/server_server/unstable.html#m-presence-accept-schema) * [m.presence_deny](https://matrix.org/docs/spec/server_server/unstable.html#m-presence-deny-schema) - + ## Tradeoffs -Ideally this proposal would also come with an alternative design for this functionality. Out of pragmatism the proposal only covers removal of what is there today. +Ideally this proposal would also come with an alternative design for this +functionality. Out of pragmatism the proposal only covers removal of what is +there today. ## Conclusions -This is a common sense attempt to remove unused portions of the spec ahead of an r0 release. It does not suggest that the ability to subscribe to the presence of others is undesirable and assumes that this behvaiour will return again in some form. +This is a common sense attempt to remove unused portions of the spec ahead of +an r0 release. It does not suggest that the ability to subscribe to the +presence of others is undesirable and assumes that this behvaiour will return +again in some form. From afd399d7d4c75c921252bbb6111436138e1ea27a Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 30 Jan 2019 11:43:30 +0000 Subject: [PATCH 070/170] remove unused schema defs --- .../event-schemas/m.presence_accept.yaml | 46 ---------------- .../event-schemas/m.presence_deny.yaml | 55 ------------------- .../event-schemas/m.presence_invite.yaml | 45 --------------- 3 files changed, 146 deletions(-) delete mode 100644 api/server-server/definitions/event-schemas/m.presence_accept.yaml delete mode 100644 api/server-server/definitions/event-schemas/m.presence_deny.yaml delete mode 100644 api/server-server/definitions/event-schemas/m.presence_invite.yaml diff --git a/api/server-server/definitions/event-schemas/m.presence_accept.yaml b/api/server-server/definitions/event-schemas/m.presence_accept.yaml deleted file mode 100644 index 3ba78b477..000000000 --- a/api/server-server/definitions/event-schemas/m.presence_accept.yaml +++ /dev/null @@ -1,46 +0,0 @@ -# Copyright 2018 New Vector Ltd -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -type: object -title: m.presence_accept -description: |- - An EDU representing approval for the observing user to subscribe to the - presence of the the observed user. -allOf: - - $ref: ../edu.yaml - - type: object - properties: - edu_type: - type: enum - enum: ['m.presence_accept'] - description: The string ``m.presence_accept`` - example: "m.presence_accept" - content: - type: object - description: The invite information. - title: Invite Information - properties: - observed_user: - type: string - description: |- - The user ID that has approved the ``observer_user`` to - subscribe to their presence. - example: "@alice:elsewhere.com" - observer_user: - type: string - description: |- - The user ID that requested to subscribe to the presence of - the ``observed_user``. - example: "@john:matrix.org" - required: ['observer_user', 'observed_user'] diff --git a/api/server-server/definitions/event-schemas/m.presence_deny.yaml b/api/server-server/definitions/event-schemas/m.presence_deny.yaml deleted file mode 100644 index 811c296a3..000000000 --- a/api/server-server/definitions/event-schemas/m.presence_deny.yaml +++ /dev/null @@ -1,55 +0,0 @@ -# Copyright 2018 New Vector Ltd -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -type: object -title: m.presence_deny -description: |- - An EDU representing a declination or revocation for the observing user - to subscribe to the presence of the observed user. -example: { - "origin": "example.org", - "destination": "elsewhere.org", - "edu_type": "m.presence_deny", - "content": { - "observed_user": "@alice:elsewhere.org", - "observer_user": "@john:example.org" - } -} -allOf: - - $ref: ../edu.yaml - - type: object - properties: - edu_type: - type: enum - enum: ['m.presence_deny'] - description: The string ``m.presence_deny`` - example: "m.presence_deny" - content: - type: object - description: The invite information. - title: Invite Information - properties: - observed_user: - type: string - description: |- - The user ID that has declined or revoked the ``observer_user`` from - subscribing to their presence. - example: "@alice:elsewhere.com" - observer_user: - type: string - description: |- - The user ID that requested to subscribe to the presence of - the ``observed_user``. - example: "@john:matrix.org" - required: ['observer_user', 'observed_user'] diff --git a/api/server-server/definitions/event-schemas/m.presence_invite.yaml b/api/server-server/definitions/event-schemas/m.presence_invite.yaml deleted file mode 100644 index a584897b7..000000000 --- a/api/server-server/definitions/event-schemas/m.presence_invite.yaml +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright 2018 New Vector Ltd -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -type: object -title: m.presence_invite -description: |- - An EDU representing a request to subscribe to a user's presence. -allOf: - - $ref: ../edu.yaml - - type: object - properties: - edu_type: - type: enum - enum: ['m.presence_invite'] - description: The string ``m.presence_invite`` - example: "m.presence_invite" - content: - type: object - description: The invite information. - title: Invite Information - properties: - observed_user: - type: string - description: |- - The user ID the ``observer_user`` would like to subscribe - to the presence of. - example: "@alice:elsewhere.com" - observer_user: - type: string - description: |- - The user ID that is wishing to subscribe to the presence of - the ``observed_user``. - example: "@john:matrix.org" - required: ['observer_user', 'observed_user'] From 1c0742ed6a5febdaf1ec26d0d0a925af57c25b87 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 30 Jan 2019 17:08:21 +0000 Subject: [PATCH 071/170] MSC 1659 Proposal: Change Event IDs to Hashes (#1659) --- proposals/1659-event-id-as-hashes.md | 130 +++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 proposals/1659-event-id-as-hashes.md diff --git a/proposals/1659-event-id-as-hashes.md b/proposals/1659-event-id-as-hashes.md new file mode 100644 index 000000000..1187d95e6 --- /dev/null +++ b/proposals/1659-event-id-as-hashes.md @@ -0,0 +1,130 @@ +# Changing Event IDs to be Hashes + +## Motivation + +Having event IDs separate from the hashes leads to issues when a server receives +multiple events with the same event ID but different reference hashes. While +APIs could be changed to better support dealing with this situation, it is +easier and nicer to simply drop the idea of a separate event ID entirely, and +instead use the reference hash of an event as its ID. + +## Identifier Format + +Currently hashes in our event format include the hash name, allowing servers to +choose which hash functions to use. The idea here was to allow a gradual change +between hash functions without the need to globally coordinate shifting from one +hash function to another. + +However now that room versions exist, changing hash functions can be achieved by +bumping the room version. Using this method would allow using a simple string as +the event ID rather than a full structure, significantly easing their usage. + +One side effect of this would be that there would be no indication about which +hash function was actually used, and it would need to be inferred from the room +version. To aid debuggability it may be worth encoding the hash function into +the ID format. + +**Conclusion:** Don't encode the hash function, since the hash will depend on +the version specific redaction algorithm anyway. + +The proposal is therefore that the event IDs are a sha256 hash, encoded using +[unpadded +Base64](https://matrix.org/docs/spec/appendices.html#unpadded-base64), and +prefixed with `$` (to aid distinguishing different types of identifiers). For +example, an event ID might be: `$CD66HAED5npg6074c6pDtLKalHjVfYb2q4Q3LZgrW6o`. + +The hash is calculated in the same way as previous event reference hashes were, +which is: + +1. Redact the event +2. Remove `signatures` field from the event +3. Serialize the event to canonical JSON +4. Compute the hash of the JSON bytes + +Event IDs will no longer be included as part of the event, and so must be +calculated by servers receiving the event. + + +## Changes to Event Formats + +As well as changing the format of event IDs, we also change the format of the +`auth_events` and `prev_events` keys in events to simply be lists of event IDs +(rather than being lists of tuples). + +A full event would therefore look something like (note that this is just an +illustrative example, and that the hashes are not correct): + +```json +{ + "auth_events": [ + "$5hdALbO+xIhzcLTxCkspx5uqry9wO8322h/OI9ApnHE", + "$Ga0DBIICBsWIZbN292ATv8fTHIGGimwjb++w+zcHLRo", + "$zc4ip/DpPI9FZVLM1wN9RLqN19vuVBURmIqAohZ1HXg", + ], + "content": { + "body": "Here is the message content", + "msgtype": "m.message" + }, + "depth": 6, + "hashes": { + "sha256": "M6/LmcMMJKc1AZnNHsuzmf0PfwladVGK2Xbz+sUTN9k" + }, + "origin": "localhost:8800", + "origin_server_ts": 1548094046693, + "prev_events": [ + "$MoOzCuB/sacqHAvgBNOLICiGLZqGT4zB16MSFOuiO0s", + ], + "room_id": "!eBrhCHJWOgqrOizwwW:localhost:8800", + "sender": "@anon-20190121_180719-33:localhost:8800", + "signatures": { + "localhost:8800": { + "ed25519:a_iIHH": "N7hwZjvHyH6r811ebZ4wwLzofKhJuIAtrQzaD3NZbf4WQNijXl5Z2BNB047aWIQCS1JyFOQKPVom4et0q9UOAA" + } + }, + "type": "m.room.message" +} +``` + +## Changes to existing APIs + +All APIs that accept event IDs must accept event IDs in the new format. + +For S2S API, whenever a server needs to parse an event from a request or +response they must either already know the room version *or* be told the room +version in the request/response. There are separate MSCs to update APIs where +necessary. + +For C2S API, the only change clients will see is that the event IDs have changed +format. Clients should already be treating event IDs as opaque strings, so no +changes should be required. Servers must add the `event_id` when sending the +event to clients, however. + +Note that the `auth_events` and `prev_events` fields aren't sent to clients, and +so the changes proposed above won't affect clients. + + +## Protocol Changes + +The `auth_events` and `prev_events` fields on an event need to be changed from a +list of tuples to a list of strings, i.e. remove the old event ID and simply +have the list of hashes. + +The auth rules also need to change: + +- The event no longer needs to be signed by the domain of the event ID (but + still needs to be signed by the sender’s domain) + +- We currently allow redactions if the domain of the redaction event ID + matches the domain of the event ID it is redacting; which allows self + redaction. This check is removed and redaction events are always accepted. + Instead, the redaction event only takes effect and is sent down to clients + if/when the original event is received, and the domain of the events' + senders match. (While this is clearly suboptimal, it is the only practical + suggestion) + + +## Room Version + +There will be a new room version v3 that is the same as v2 except uses the new +event format proposed above. v3 will be marked as 'stable' as defined in [MSC1804](https://github.com/matrix-org/matrix-doc/blob/travis/msc/room-version-client-advertising/proposals/1804-advertising-capable-room-versions.md) + From ff75996524c573f7ca64e617732b94b3017a3d78 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 30 Jan 2019 17:03:03 -0700 Subject: [PATCH 072/170] Add specification for room version 3: Event IDs as hashes Original proposal: https://github.com/matrix-org/matrix-doc/pull/1659 Implementation proofs (some traversing of the PR tree may be required to get all of them): * https://github.com/matrix-org/synapse/pull/4483 * https://github.com/matrix-org/synapse/pull/4499 This doesn't intentionally change anything from the proposal. **Implementation details**: The simple part of this is the introduction of a rooms/v3.html document. The somewhat unclear part is the stuff done to the s2s definitions. This pulls `unsigned_pdu` out to `unsigned_pdu_base` (all fields except `event_id`) where it can be reused in `pdu` and `pdu_v3` (for rooms v3). These definitions are further moved into the room version specifications where they can highlight the exact schemas in detail. Version 1 has been updated to include the pre-existing event format, however the core principles of the room have not been changed. The same applies to room version 2. Room versions have immutable core principles once in the spec, otherwise these format changes would land in a pre-existing version. The client-server API event formats will need updating, however that is being punted to a different commit to try and keep these changes reviewable. --- api/server-server/definitions/pdu_v3.yaml | 73 +++++++ .../definitions/unsigned_pdu.yaml | 145 +------------ .../definitions/unsigned_pdu_base.yaml | 151 +++++++++++++ api/server-server/examples/pdu_v3.json | 19 ++ api/server-server/examples/unsigned_pdu.json | 29 +-- .../examples/unsigned_pdu_base.json | 26 +++ specification/index.rst | 1 + specification/rooms/v1.rst | 200 +++++++++++++++++- specification/rooms/v3.rst | 122 +++++++++++ specification/server_server_api.rst | 188 +--------------- specification/targets.yaml | 4 + 11 files changed, 610 insertions(+), 348 deletions(-) create mode 100644 api/server-server/definitions/pdu_v3.yaml create mode 100644 api/server-server/definitions/unsigned_pdu_base.yaml create mode 100644 api/server-server/examples/pdu_v3.json create mode 100644 api/server-server/examples/unsigned_pdu_base.json create mode 100644 specification/rooms/v3.rst diff --git a/api/server-server/definitions/pdu_v3.yaml b/api/server-server/definitions/pdu_v3.yaml new file mode 100644 index 000000000..97747ca8c --- /dev/null +++ b/api/server-server/definitions/pdu_v3.yaml @@ -0,0 +1,73 @@ +# Copyright 2019 New Vector Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +type: object +title: Persistent Data Unit +description: A persistent data unit (event) for room version 3. +example: + $ref: "../examples/pdu_v3.json" +allOf: + - $ref: "unsigned_pdu_base.yaml" + - type: object + properties: + auth_events: + type: array + items: + type: string + description: Event ID. + description: |- + Event IDs and reference hashes for the authorization events that would + allow this event to be in the room. + example: ["$abase64encodedhash", "$anotherevent"] + prev_events: + type: array + items: + type: string + description: Event ID. + description: |- + Event IDs and reference hashes for the most recent events in the room + that the homeserver was aware of when it made this event. + example: ["$abase64encodedhash", "$anotherevent"] + hashes: + type: object + title: Event Hash + description: |- + Content hashes of the PDU, following the algorithm specified in `Signing Events`_. + example: { + "sha256": "thishashcoversallfieldsincasethisisredacted" + } + properties: + sha256: + type: string + description: The hash. + example: thishashcoversallfieldsincasthisisredacted + required: ['sha256'] + signatures: + type: object + description: |- + Signatures for the PDU, following the algorithm specified in `Signing Events`_. + example: { + "example.com": { + "ed25519:key_version:": "these86bytesofbase64signaturecoveressentialfieldsincludinghashessocancheckredactedpdus" + } + } + additionalProperties: + type: object + title: Server Signatures + additionalProperties: + type: string + required: + - auth_events + - prev_events + - hashes + - signatures diff --git a/api/server-server/definitions/unsigned_pdu.yaml b/api/server-server/definitions/unsigned_pdu.yaml index c5ad801db..a27a21cde 100644 --- a/api/server-server/definitions/unsigned_pdu.yaml +++ b/api/server-server/definitions/unsigned_pdu.yaml @@ -1,4 +1,4 @@ -# Copyright 2018 New Vector Ltd +# Copyright 2018-2019 New Vector Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -16,140 +16,13 @@ title: Unsigned Persistent Data Unit description: An unsigned persistent data unit (event) example: $ref: "../examples/unsigned_pdu.json" -properties: - event_id: - type: string - description: The event ID for the PDU. - example: "$a4ecee13e2accdadf56c1025:example.com" - room_id: - type: string - description: Room identifier. - example: "!abc123:matrix.org" - sender: - type: string - description: The ID of the user sending the event. - example: "@someone:matrix.org" - origin: - type: string - description: The ``server_name`` of the homeserver that created this event. - example: "matrix.org" - origin_server_ts: - type: integer - format: int64 - description: Timestamp in milliseconds on origin homeserver when this event was created. - example: 1234567890 - type: - type: string - description: Event type - example: "m.room.message" - state_key: - type: string - description: |- - If this key is present, the event is a state event, and it will replace previous events - with the same ``type`` and ``state_key`` in the room state. - example: "my_key" - content: - type: object - description: The content of the event. - example: {"key": "value"} - prev_events: - type: array - description: |- - Event IDs and reference hashes for the most recent events in the room - that the homeserver was aware of when it made this event. - items: - type: array - maxItems: 2 - minItems: 2 - items: - - type: string - title: Event ID - example: "$abc123:matrix.org" - - type: object - title: Event Hash - example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" - } - properties: - sha256: - type: string - description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong - required: ['sha256'] - depth: - type: integer - description: |- - The maximum depth of the ``prev_events``, plus one. Must be less than the - maximum value for an integer (2^63 - 1). If the room's depth is already at - the limit, the depth must be set to the limit. - example: 12 - auth_events: - type: array - description: |- - Event IDs and reference hashes for the authorization events that would - allow this event to be in the room. - items: - type: array - maxItems: 2 - minItems: 2 - items: - - type: string - title: Event ID - example: "$abc123:matrix.org" - - type: object - title: Event Hash - example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" - } - properties: - sha256: - type: string - description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong - required: ['sha256'] - redacts: - type: string - description: For redaction events, the ID of the event being redacted. - example: "$def456:matrix.org" - unsigned: - type: object - title: Example Unsigned Data - description: |- - Additional data added by the origin server but not covered by the ``signatures``. More - keys than those defined here may be used. - example: {"key": "value"} +allOf: + - $ref: "unsigned_pdu_base.yaml" + - type: object properties: - age: - type: integer - description: The number of milliseconds that have passed since this message was sent. - example: 4612 - replaces_state: + event_id: type: string - description: The event ID of the state event this event replaces. - example: "$state_event:example.org" - prev_sender: - type: string - description: The sender of the replaced state event. - example: "@someone:example.org" - prev_content: - type: object - description: The content of the replaced state event. - example: { - "membership": "join", - "displayname": "Bob" - } - redacted_because: - type: string - description: A reason for why the event was redacted. - example: "Inappropriate content" -required: - - event_id - - room_id - - sender - - origin - - origin_server_ts - - type - - content - - prev_events - - depth - - auth_events + description: The event ID for the PDU. + example: "$a4ecee13e2accdadf56c1025:example.com" + required: + - event_id diff --git a/api/server-server/definitions/unsigned_pdu_base.yaml b/api/server-server/definitions/unsigned_pdu_base.yaml new file mode 100644 index 000000000..d447b3a3d --- /dev/null +++ b/api/server-server/definitions/unsigned_pdu_base.yaml @@ -0,0 +1,151 @@ +# Copyright 2018 New Vector Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +type: object +title: Unsigned Persistent Data Unit +description: An unsigned persistent data unit (event) +example: + $ref: "../examples/unsigned_pdu_base.json" +properties: + room_id: + type: string + description: Room identifier. + example: "!abc123:matrix.org" + sender: + type: string + description: The ID of the user sending the event. + example: "@someone:matrix.org" + origin: + type: string + description: The ``server_name`` of the homeserver that created this event. + example: "matrix.org" + origin_server_ts: + type: integer + format: int64 + description: Timestamp in milliseconds on origin homeserver when this event was created. + example: 1234567890 + type: + type: string + description: Event type + example: "m.room.message" + state_key: + type: string + description: |- + If this key is present, the event is a state event, and it will replace previous events + with the same ``type`` and ``state_key`` in the room state. + example: "my_key" + content: + type: object + description: The content of the event. + example: {"key": "value"} + prev_events: + type: array + description: |- + Event IDs and reference hashes for the most recent events in the room + that the homeserver was aware of when it made this event. + items: + type: array + maxItems: 2 + minItems: 2 + items: + - type: string + title: Event ID + example: "$abc123:matrix.org" + - type: object + title: Event Hash + example: { + "sha256": "abase64encodedsha256hashshouldbe43byteslong" + } + properties: + sha256: + type: string + description: The event hash. + example: abase64encodedsha256hashshouldbe43byteslong + required: ['sha256'] + depth: + type: integer + description: |- + The maximum depth of the ``prev_events``, plus one. Must be less than the + maximum value for an integer (2^63 - 1). If the room's depth is already at + the limit, the depth must be set to the limit. + example: 12 + auth_events: + type: array + description: |- + Event IDs and reference hashes for the authorization events that would + allow this event to be in the room. + items: + type: array + maxItems: 2 + minItems: 2 + items: + - type: string + title: Event ID + example: "$abc123:matrix.org" + - type: object + title: Event Hash + example: { + "sha256": "abase64encodedsha256hashshouldbe43byteslong" + } + properties: + sha256: + type: string + description: The event hash. + example: abase64encodedsha256hashshouldbe43byteslong + required: ['sha256'] + redacts: + type: string + description: For redaction events, the ID of the event being redacted. + example: "$def456:matrix.org" + unsigned: + type: object + title: Example Unsigned Data + description: |- + Additional data added by the origin server but not covered by the ``signatures``. More + keys than those defined here may be used. + example: {"key": "value"} + properties: + age: + type: integer + description: The number of milliseconds that have passed since this message was sent. + example: 4612 + replaces_state: + type: string + description: The event ID of the state event this event replaces. + example: "$state_event:example.org" + prev_sender: + type: string + description: The sender of the replaced state event. + example: "@someone:example.org" + prev_content: + type: object + description: The content of the replaced state event. + example: { + "membership": "join", + "displayname": "Bob" + } + redacted_because: + type: string + description: A reason for why the event was redacted. + example: "Inappropriate content" +required: + - event_id + - room_id + - sender + - origin + - origin_server_ts + - type + - content + - prev_events + - depth + - auth_events diff --git a/api/server-server/examples/pdu_v3.json b/api/server-server/examples/pdu_v3.json new file mode 100644 index 000000000..6a454b4ea --- /dev/null +++ b/api/server-server/examples/pdu_v3.json @@ -0,0 +1,19 @@ +{ + "$ref": "unsigned_pdu_base.json", + "hashes": { + "sha256": "thishashcoversallfieldsincasethisisredacted" + }, + "signatures": { + "example.com": { + "ed25519:key_version:": "these86bytesofbase64signaturecoveressentialfieldsincludinghashessocancheckredactedpdus" + } + }, + "auth_events": [ + "$base64encodedeventid", + "$adifferenteventid" + ], + "prev_events": [ + "$base64encodedeventid", + "$adifferenteventid" + ] +} diff --git a/api/server-server/examples/unsigned_pdu.json b/api/server-server/examples/unsigned_pdu.json index f4d2e749f..b886a3658 100644 --- a/api/server-server/examples/unsigned_pdu.json +++ b/api/server-server/examples/unsigned_pdu.json @@ -1,27 +1,4 @@ { - "room_id": "!UcYsUzyxTGDxLBEvLy:example.org", - "sender": "@alice:example.com", - "origin": "example.com", - "event_id": "$a4ecee13e2accdadf56c1025:example.com", - "origin_server_ts": 1404838188000, - "depth": 12, - "auth_events": [ - [ - "$af232176:example.org", - {"sha256": "abase64encodedsha256hashshouldbe43byteslong"} - ] - ], - "type": "m.room.message", - "prev_events": [ - [ - "$af232176:example.org", - {"sha256": "abase64encodedsha256hashshouldbe43byteslong"} - ] - ], - "content": { - "key": "value" - }, - "unsigned": { - "age": 4612 - } -} \ No newline at end of file + "$ref": "unsigned_pdu_base.json", + "event_id": "$a4ecee13e2accdadf56c1025:example.com" +} diff --git a/api/server-server/examples/unsigned_pdu_base.json b/api/server-server/examples/unsigned_pdu_base.json new file mode 100644 index 000000000..4826ccefc --- /dev/null +++ b/api/server-server/examples/unsigned_pdu_base.json @@ -0,0 +1,26 @@ +{ + "room_id": "!UcYsUzyxTGDxLBEvLy:example.org", + "sender": "@alice:example.com", + "origin": "example.com", + "origin_server_ts": 1404838188000, + "depth": 12, + "auth_events": [ + [ + "$af232176:example.org", + {"sha256": "abase64encodedsha256hashshouldbe43byteslong"} + ] + ], + "type": "m.room.message", + "prev_events": [ + [ + "$af232176:example.org", + {"sha256": "abase64encodedsha256hashshouldbe43byteslong"} + ] + ], + "content": { + "key": "value" + }, + "unsigned": { + "age": 4612 + } +} diff --git a/specification/index.rst b/specification/index.rst index 3893ac4cc..fa9ca3d65 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -484,6 +484,7 @@ The available room versions are: * `Version 1 `_ - **Stable**. The current version of most rooms. * `Version 2 `_ - **Stable**. Implements State Resolution Version 2. +* `Version 3 `_ - **Stable**. Introduces a new event format. Specification Versions ---------------------- diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 37b1b7ced..e09420e47 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -77,7 +77,7 @@ results of the resolution so far. * Add the first event in the list to :math:`R`. * For each subsequent event in the list, check that the event would be - allowed by the `authorization rules`_ for a room in state :math:`R`. If the + allowed by the authorization rules for a room in state :math:`R`. If the event would be allowed, then update :math:`R` with the event and continue with the next event in the list. If it would not be allowed, stop and continue below with ``m.room.join_rules`` events. @@ -95,4 +95,200 @@ A *conflict* occurs between states where those states have different affected are said to be *conflicting* events. -.. _`authorization rules`: ../server_server/unstable.html#authorization-rules +Authorization rules +~~~~~~~~~~~~~~~~~~~ + +The types of state events that affect authorization are: + +- ``m.room.create`` +- ``m.room.member`` +- ``m.room.join_rules`` +- ``m.room.power_levels`` +- ``m.room.third_party_invite`` + +The rules are as follows: + +1. If type is ``m.room.create``: + + a. If it has any previous events, reject. + b. If the domain of the ``room_id`` does not match the domain of the + ``sender``, reject. + c. If ``content.room_version`` is present and is not a recognised version, + reject. + d. If ``content`` has no ``creator`` field, reject. + e. Otherwise, allow. + +#. Reject if event has ``auth_events`` that: + + a. have duplicate entries for a given ``type`` and ``state_key`` pair + #. have entries whose ``type`` and ``state_key`` don't match those + specified by the `auth events selection`_ algorithm described in the + server specification. + +#. If event does not have a ``m.room.create`` in its ``auth_events``, reject. + +#. If type is ``m.room.aliases``: + + a. If event has no ``state_key``, reject. + b. If sender's domain doesn't matches ``state_key``, reject. + c. Otherwise, allow. + +#. If type is ``m.room.member``: + + a. If no ``state_key`` key or ``membership`` key in ``content``, reject. + + #. If ``membership`` is ``join``: + + i. If the only previous event is an ``m.room.create`` + and the ``state_key`` is the creator, allow. + + #. If the ``sender`` does not match ``state_key``, reject. + + #. If the ``sender`` is banned, reject. + + #. If the ``join_rule`` is ``invite`` then allow if membership state + is ``invite`` or ``join``. + + #. If the ``join_rule`` is ``public``, allow. + + #. Otherwise, reject. + + #. If ``membership`` is ``invite``: + + i. If ``content`` has ``third_party_invite`` key: + + #. If *target user* is banned, reject. + + #. If ``content.third_party_invite`` does not have a + ``signed`` key, reject. + + #. If ``signed`` does not have ``mxid`` and ``token`` keys, reject. + + #. If ``mxid`` does not match ``state_key``, reject. + + #. If there is no ``m.room.third_party_invite`` event in the + current room state with ``state_key`` matching ``token``, reject. + + #. If ``sender`` does not match ``sender`` of the + ``m.room.third_party_invite``, reject. + + #. If any signature in ``signed`` matches any public key in the + ``m.room.third_party_invite`` event, allow. The public keys are + in ``content`` of ``m.room.third_party_invite`` as: + + #. A single public key in the ``public_key`` field. + #. A list of public keys in the ``public_keys`` field. + + #. Otherwise, reject. + + #. If the ``sender``'s current membership state is not ``join``, reject. + + #. If *target user*'s current membership state is ``join`` or ``ban``, + reject. + + #. If the ``sender``'s power level is greater than or equal to the *invite + level*, allow. + + #. Otherwise, reject. + + #. If ``membership`` is ``leave``: + + i. If the ``sender`` matches ``state_key``, allow if and only if that user's + current membership state is ``invite`` or ``join``. + + #. If the ``sender``'s current membership state is not ``join``, reject. + + #. If the *target user*'s current membership state is ``ban``, and the + ``sender``'s power level is less than the *ban level*, reject. + + #. If the ``sender``'s power level is greater than or equal to the *kick + level*, and the *target user*'s power level is less than the + ``sender``'s power level, allow. + + #. Otherwise, reject. + + #. If ``membership`` is ``ban``: + + i. If the ``sender``'s current membership state is not ``join``, reject. + + #. If the ``sender``'s power level is greater than or equal to the *ban + level*, and the *target user*'s power level is less than the + ``sender``'s power level, allow. + + #. Otherwise, reject. + + #. Otherwise, the membership is unknown. Reject. + +#. If the ``sender``'s current membership state is not ``join``, reject. + +#. If type is ``m.room.third_party_invite``: + + a. Allow if and only if ``sender``'s current power level is greater than + or equal to the *invite level*. + +#. If the event type's *required power level* is greater than the ``sender``'s power + level, reject. + +#. If the event has a ``state_key`` that starts with an ``@`` and does not match + the ``sender``, reject. + +#. If type is ``m.room.power_levels``: + + a. If ``users`` key in ``content`` is not a dictionary with keys that are + valid user IDs with values that are integers (or a string that is an + integer), reject. + + #. If there is no previous ``m.room.power_levels`` event in the room, allow. + + #. For each of the keys ``users_default``, ``events_default``, + ``state_default``, ``ban``, ``redact``, ``kick``, ``invite``, as well as + each entry being changed under the ``events`` or ``users`` keys: + + i. If the current value is higher than the ``sender``'s current power level, + reject. + + #. If the new value is higher than the ``sender``'s current power level, + reject. + + #. For each entry being changed under the ``users`` key, other than the + ``sender``'s own entry: + + i. If the current value is equal to the ``sender``'s current power level, + reject. + + #. Otherwise, allow. + +#. If type is ``m.room.redaction``: + + a. If the ``sender``'s power level is greater than or equal to the *redact + level*, allow. + + #. If the domain of the ``event_id`` of the event being redacted is the same + as the domain of the ``event_id`` of the ``m.room.redaction``, allow. + + #. Otherwise, reject. + +#. Otherwise, allow. + +.. NOTE:: + + Some consequences of these rules: + + * Unless you are a member of the room, the only permitted operations (apart + from the intial create/join) are: joining a public room; accepting or + rejecting an invitation to a room. + + * To unban somebody, you must have power level greater than or equal to both + the kick *and* ban levels, *and* greater than the target user's power + level. + +Event format +~~~~~~~~~~~~ + +Events in version 1 rooms have the following structure: + +{{definition_ss_pdu}} + + +.. _`auth events selection`: ../../server_server/unstable.html#auth-events-selection +.. _`Signing Events`: ../../server_server/unstable.html#signing-events diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst new file mode 100644 index 000000000..22276ab32 --- /dev/null +++ b/specification/rooms/v3.rst @@ -0,0 +1,122 @@ +.. Copyright 2018-2019 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +Room Version 3 +============== + +This room version builds off of `version 2 `_ with an improved event format. + +.. note: + All requirements listed in this room version specification are scoped to rooms + which actually use this room version. For example, a requirement of "all APIs must + accept the new event format" does in fact apply to all APIs, but only so much as + where the contextual room of the request is using this room version. Rooms using + other room versions should not be affected by these sweeping requirements. + + +Client considerations +--------------------- + +This room version changes the format for event IDs sent to clients. Clients should be +aware that these event IDs may contain slashes and other potentially problematic +characters. Clients should be treating event IDs as opaque identifiers and should not +be attempting to parse them into a usable form, just like with other room versions. + +Clients should expect to see event IDs changed from the format of ``$randomstring:example.org`` +to something like ``$acR1l0raoZnm60CBwAVgqbZqoO/mYU81xysh1u7XcJk`` (note the lack of +domain and the potentially problematic slash). + + +Server implementation components +-------------------------------- + +.. WARNING:: + The information contained in this section is strictly for server implementors. + Applications which use the Client-Server API are generally unaffected by the + intricacies contained here. The section above regarding client considerations + is the resource that Client-Server API use cases should reference. + + +Room version 3 uses the state resolution algorithm defined in `room version 2 `_, +and the event format defined here. Other event formats and applicable algorithms +may be used in other room versions. + +Event IDs +~~~~~~~~~ + +In other room versions (namely version 1 and 2) the event ID is a distinct field +from the remainder of the event, which must be tracked as such. This leads to +complications where servers receive multiple events with the same ID in either the +same or different rooms where the server cannot easily keep track of which event it +should be using. By removing the use of a dedicated event ID, servers are required +to track the hashes on an event to determine its ID. + +The event ID must be a sha256 hash of the event, encoded using `Unpadded Base64`_ +and prefixed with `$`. For example, an event ID might be +``$CD66HAED5npg6074c6pDtLKalHjVfYb2q4Q3LZgrW6o``. + +The hash itself is calculated the same as previous reference hashes are: + +1. Redact the event. +2. Remove the `signatures` field from the event. +3. Serialize the event into `Canonical JSON`_. +4. Compute the hash of the JSON bytes. + +Event IDs should not be sent over federation to servers when the room uses +this room version. On the receiving end of an event, the server should compute +the relevant event ID for itself. + +Additionally, the ``auth_events`` and ``prev_events`` have had a format change +compared to other room versions to make it easier to handle. Instead of a tuple +of values, they are now plain lists of events. + +{{definition_ss_pdu_v3}} + +Changes to APIs +~~~~~~~~~~~~~~~ + +Due to the event ID being removed from the event, some APIs need to change. All +APIs which currently accept an event ID must do so with the new format. Servers +must append the calculated event ID to all events sent to clients where an event +ID would normally be expected. + +Because servers are not communicating the event ID over the wire to each other, +servers must be aware of the room version where the event resides so that the +server may parse and handle the event. The federation API has taken this concern +into consideration by ensuring that servers are aware of (or can find) the room +version during a request. + +Authorization rules for events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The authorization rules for a given event have changed in this room version due +to the change in event format: + +* The event no longer needs to be signed by the domain of the event ID (as there + is no domain in the event ID), but still needs to be signed by the sender's + domain. + +* Previously, redactions were allowed if the sender's domain matched the domain + in the event ID it was redacting, allowing self redaction. This check is removed + and redaction events are always accepted. Redaction events only take effect + when the original event is received, and the domain of the each event matches. + Servers should not send redactions down to clients until the redaction has + taken effect. + +The remaining rules are the same as room version 1. + + +.. _`Unpadded Base64`: ../../appendices.html#unpadded-base64 +.. _`Canonical JSON`: ../../appendices.html#canonical-json +.. _`Signing Events`: ../../server_server/unstable.html#signing-events diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index a63bf0a6b..8f021e620 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -327,7 +327,7 @@ following subset of the room state: ``m.room.third_party_invite`` event with ``state_key`` matching ``content.third_party_invite.signed.token``, if any. -{{definition_ss_pdu}} +For a full schema of what a PDU looks like, see the `room version specification`_. Checks performed on receipt of a PDU ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -379,189 +379,9 @@ Authorization rules The rules governing whether an event is authorized depends on a set of state. A given event is checked multiple times against different sets of state, as -specified above. The types of state events that affect authorization are: - -- ``m.room.create`` -- ``m.room.member`` -- ``m.room.join_rules`` -- ``m.room.power_levels`` -- ``m.room.third_party_invite`` - -The rules are as follows: - -1. If type is ``m.room.create``: - - a. If it has any previous events, reject. - b. If the domain of the ``room_id`` does not match the domain of the - ``sender``, reject. - c. If ``content.room_version`` is present and is not a recognised version, - reject. - d. If ``content`` has no ``creator`` field, reject. - e. Otherwise, allow. - -#. Reject if event has ``auth_events`` that: - - a. have duplicate entries for a given ``type`` and ``state_key`` pair - #. have entries whose ``type`` and ``state_key`` don't match those - specified by the `auth events selection`_ algorithm described above. - -#. If event does not have a ``m.room.create`` in its ``auth_events``, reject. - -#. If type is ``m.room.aliases``: - - a. If event has no ``state_key``, reject. - b. If sender's domain doesn't matches ``state_key``, reject. - c. Otherwise, allow. - -#. If type is ``m.room.member``: - - a. If no ``state_key`` key or ``membership`` key in ``content``, reject. - - #. If ``membership`` is ``join``: - - i. If the only previous event is an ``m.room.create`` - and the ``state_key`` is the creator, allow. - - #. If the ``sender`` does not match ``state_key``, reject. - - #. If the ``sender`` is banned, reject. - - #. If the ``join_rule`` is ``invite`` then allow if membership state - is ``invite`` or ``join``. - - #. If the ``join_rule`` is ``public``, allow. - - #. Otherwise, reject. - - #. If ``membership`` is ``invite``: - - i. If ``content`` has ``third_party_invite`` key: - - #. If *target user* is banned, reject. - - #. If ``content.third_party_invite`` does not have a - ``signed`` key, reject. - - #. If ``signed`` does not have ``mxid`` and ``token`` keys, reject. - - #. If ``mxid`` does not match ``state_key``, reject. - - #. If there is no ``m.room.third_party_invite`` event in the - current room state with ``state_key`` matching ``token``, reject. - - #. If ``sender`` does not match ``sender`` of the - ``m.room.third_party_invite``, reject. - - #. If any signature in ``signed`` matches any public key in the - ``m.room.third_party_invite`` event, allow. The public keys are - in ``content`` of ``m.room.third_party_invite`` as: - - #. A single public key in the ``public_key`` field. - #. A list of public keys in the ``public_keys`` field. - - #. Otherwise, reject. - - #. If the ``sender``'s current membership state is not ``join``, reject. - - #. If *target user*'s current membership state is ``join`` or ``ban``, - reject. - - #. If the ``sender``'s power level is greater than or equal to the *invite - level*, allow. - - #. Otherwise, reject. - - #. If ``membership`` is ``leave``: - - i. If the ``sender`` matches ``state_key``, allow if and only if that user's - current membership state is ``invite`` or ``join``. - - #. If the ``sender``'s current membership state is not ``join``, reject. - - #. If the *target user*'s current membership state is ``ban``, and the - ``sender``'s power level is less than the *ban level*, reject. - - #. If the ``sender``'s power level is greater than or equal to the *kick - level*, and the *target user*'s power level is less than the - ``sender``'s power level, allow. - - #. Otherwise, reject. - - #. If ``membership`` is ``ban``: - - i. If the ``sender``'s current membership state is not ``join``, reject. - - #. If the ``sender``'s power level is greater than or equal to the *ban - level*, and the *target user*'s power level is less than the - ``sender``'s power level, allow. - - #. Otherwise, reject. - - #. Otherwise, the membership is unknown. Reject. - -#. If the ``sender``'s current membership state is not ``join``, reject. - -#. If type is ``m.room.third_party_invite``: - - a. Allow if and only if ``sender``'s current power level is greater than - or equal to the *invite level*. - -#. If the event type's *required power level* is greater than the ``sender``'s power - level, reject. - -#. If the event has a ``state_key`` that starts with an ``@`` and does not match - the ``sender``, reject. - -#. If type is ``m.room.power_levels``: - - a. If ``users`` key in ``content`` is not a dictionary with keys that are - valid user IDs with values that are integers (or a string that is an - integer), reject. - - #. If there is no previous ``m.room.power_levels`` event in the room, allow. - - #. For each of the keys ``users_default``, ``events_default``, - ``state_default``, ``ban``, ``redact``, ``kick``, ``invite``, as well as - each entry being changed under the ``events`` or ``users`` keys: - - i. If the current value is higher than the ``sender``'s current power level, - reject. - - #. If the new value is higher than the ``sender``'s current power level, - reject. - - #. For each entry being changed under the ``users`` key, other than the - ``sender``'s own entry: - - i. If the current value is equal to the ``sender``'s current power level, - reject. - - #. Otherwise, allow. - -#. If type is ``m.room.redaction``: - - a. If the ``sender``'s power level is greater than or equal to the *redact - level*, allow. - - #. If the domain of the ``event_id`` of the event being redacted is the same - as the domain of the ``event_id`` of the ``m.room.redaction``, allow. - - #. Otherwise, reject. - -#. Otherwise, allow. - -.. NOTE:: - - Some consequences of these rules: - - * Unless you are a member of the room, the only permitted operations (apart - from the intial create/join) are: joining a public room; accepting or - rejecting an invitation to a room. - - * To unban somebody, you must have power level greater than or equal to both - the kick *and* ban levels, *and* greater than the target user's power - level. - +specified above. Each room version can have a different algorithm for how the +rules work, and which rules are applied. For more detailed information, please +see the `room version specification`_. Rejection +++++++++ diff --git a/specification/targets.yaml b/specification/targets.yaml index f78d124b9..830449ae5 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -34,6 +34,10 @@ targets: files: - rooms/v2.rst version_label: v2 + rooms@v3: # this is translated to be rooms/v3.html + files: + - rooms/v3.rst + version_label: v3 appendices: files: - appendices.rst From ccce6c196d9a941b8ceeb6b47c2781bb65cfc1ac Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 30 Jan 2019 19:43:55 -0700 Subject: [PATCH 073/170] Specify how capabilities work in the c2s API Original proposals: * https://github.com/matrix-org/matrix-doc/pull/1753 * https://github.com/matrix-org/matrix-doc/pull/1804 Implementation proof: * https://github.com/matrix-org/synapse/pull/4472 * https://github.com/matrix-org/matrix-js-sdk/pull/830 There is one change to MSC1753 which is included in this commit. MSC1804 remains unchanged. In the original proposal, the change password capability being present was an indication that password changes were possible. It was found that this doesn't really communicate the state very well to clients in that lack of a capability (or a 404, etc) would mean that users would erroneously not be able to change their passwords. A simple boolean flag was added to assist clients in detecting this capability. --- api/client-server/capabilities.yaml | 108 ++++++++++++++++++ .../client_server/newsfragments/1829.feature | 1 + proposals/1753-capabilities.md | 6 +- specification/client_server_api.rst | 101 ++++++++++++++++ 4 files changed, 214 insertions(+), 2 deletions(-) create mode 100644 api/client-server/capabilities.yaml create mode 100644 changelogs/client_server/newsfragments/1829.feature diff --git a/api/client-server/capabilities.yaml b/api/client-server/capabilities.yaml new file mode 100644 index 000000000..acec600da --- /dev/null +++ b/api/client-server/capabilities.yaml @@ -0,0 +1,108 @@ +# Copyright 2019 New Vector Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +swagger: '2.0' +info: + title: "Matrix Client-Server Capabiltiies API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/%CLIENT_MAJOR_VERSION% +produces: + - application/json +securityDefinitions: + $ref: definitions/security.yaml +paths: + "/capabilities": + get: + summary: Gets information about the server's capabilities. + description: |- + Gets information about the server's supported feature set + and other relevant capabilities. + operationId: getCapabilities + security: + - accessToken: [] + parameters: [] + responses: + 200: + description: + The capabilities of the server + examples: + application/json: { + "capabilities": { + "m.change_password": { + "enabled": false + }, + "m.room_versions": { + "default": "1", + "available": { + "1": "stable", + "2": "stable", + "3": "unstable", + "test-version": "unstable" + } + }, + "com.example.custom.ratelimit": { + "max_requests_per_hour": 600 + } + } + } + schema: + type: object + required: ["capabilities"] + additionalProperties: + type: object + description: |- + The custom capabilities the server supports, using the + Java package naming convention. + properties: + "m.change_password": + type: object + description: |- + Capability to indicate if the user can change their password. + title: ChangePasswordCapability + properties: + enabled: + type: boolean + description: |- + True if the user can change their password, false otherwise. + example: false + required: ['enabled'] + "m.room_versions": + type: object + description: The room versions the server supports. + title: RoomVersionsCapability + properties: + default: + type: string + description: |- + The default room version the server is using for new rooms. + example: "1" + available: + type: object + description: |- + A detailed description of the room versions the server supports. + additionalProperties: + type: string + title: RoomVersionStability + enum: [stable, unstable] + description: The stability of the room version. + required: ['default', 'available'] + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/errors/rate_limited.yaml" + tags: + - Capabilities diff --git a/changelogs/client_server/newsfragments/1829.feature b/changelogs/client_server/newsfragments/1829.feature new file mode 100644 index 000000000..107291f34 --- /dev/null +++ b/changelogs/client_server/newsfragments/1829.feature @@ -0,0 +1 @@ +Support optional features by having clients query for capabilities. diff --git a/proposals/1753-capabilities.md b/proposals/1753-capabilities.md index ec5169aba..5a56eacaa 100644 --- a/proposals/1753-capabilities.md +++ b/proposals/1753-capabilities.md @@ -40,8 +40,10 @@ As a starting point, a single capability identifier is proposed: change the user's password via the `POST /_matrix/client/r0/account/password` API. -The value of the `capabilities` object in the response should be the empty -object. +The value of the `capabilities` object in the response should contain a single +boolean flag, `enabled`, to indicate whether a password change is possible. If +the capability is not listed, the client should assume that password changes +are possible. ### Fallback behaviour diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 4df838142..40ac55884 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1070,6 +1070,107 @@ Current account information {{whoami_cs_http_api}} +Capabilities negotiation +------------------------ + +A homeserver may not support certain operations and clients must be able to +query for what the homeserver can and can't offer. For example, a homeserver +may not support users changing their password as it is configured to perform +authentication against an external system. + +The capabilities advertised through this system are intended to advertise +functionality which is optional in the API, or which depend in some way on +the state of the user or server. This system should not be used to advertise +unstable or experimental features - this is better done by the ``/versions`` +endpoint. + +Some examples of what a reasonable capability could be are: + +* Whether the server supports user presence. + +* Whether the server supports optional features, such as the user or room + directories. + +* The rate limits or file type restrictions imposed on clients by the server. + +Some examples of what should **not** be a capability are: + +* Whether the server supports a feature in the ``unstable`` specification. + +* Media size limits - these are handled by the ``/media/%CLIENT_MAJOR_VERSION%/config`` + API. + +* Optional encodings or alternative transports for communicating with the + server. + +Capabilities prefixed with ``m.`` are reserved for definition in the Matrix +specification while other values may be used by servers using the Java package +naming convention. The capabilities supported by the Matrix specification are +defined later in this section. + +{{capabilities_cs_http_api}} + + +``m.change_password`` capability +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This capability has a single flag, ``enabled``, which indicates whether or not +the user can use the ``/account/password`` API to change their password. If not +present, the client should assume that password changes are possible via the +API. When present, clients SHOULD respect the capability's ``enabled`` flag +and indicate to the user if they are unable to change their password. + +An example of the capability API's response for this capability is:: + + { + "capabilities": { + "m.change_password": { + "enabled": false + } + } + } + + +``m.room_versions`` capability +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This capability describes the default and available room versions a server +supports, and at what level of stability. Clients should make use of this +capability to determine if users need to be encouraged to upgrade their rooms. + +An example of the capability API's response for this capability is:: + + { + "capabilities": { + "m.room_versions": { + "default": "1", + "available": { + "1": "stable", + "2": "stable", + "3": "unstable", + "custom-version": "unstable" + } + } + } + } + +This capability mirrors the same restrictions of `room versions`_ to describe +which versions are stable and unstable. Clients should assume that the ``default`` +version is ``stable``. Any version not explicitly labelled as ``stable`` in the +``available`` versions is to be treated as ``unstable``. For example, a version +listed as ``future-stable`` should be treated as ``unstable``. + +The ``default`` version is the version the server is using to create new rooms. +Clients should encourage users with sufficient permissions in a room to upgrade +their room to the ``default`` version when the room is using an ``unstable`` +version. + +When this capability is not listed, clients should use ``"1"`` as the default +and only stable ``available`` room version. + +.. _`room versions`: ../index.html#room-versions + + Pagination ---------- From 9193d57dfdbc176d65f7582e22faa667bac81477 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 30 Jan 2019 19:47:16 -0700 Subject: [PATCH 074/170] full stop --- api/client-server/capabilities.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/client-server/capabilities.yaml b/api/client-server/capabilities.yaml index acec600da..63a3ea91a 100644 --- a/api/client-server/capabilities.yaml +++ b/api/client-server/capabilities.yaml @@ -38,7 +38,7 @@ paths: responses: 200: description: - The capabilities of the server + The capabilities of the server. examples: application/json: { "capabilities": { From 0347e873efca4f80460492bdff9271601c7bc341 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 30 Jan 2019 22:11:31 -0700 Subject: [PATCH 075/170] Specify .well-known s2s discovery and X.509 validation Original proposals: * https://github.com/matrix-org/matrix-doc/pull/1708 (note: the JSON requirements were softened by https://github.com/matrix-org/matrix-doc/pull/1824) * https://github.com/matrix-org/matrix-doc/pull/1711 Implementation proofs: * https://github.com/matrix-org/synapse/pull/4489 * No explicit PRs for MSC1711 could be found, however Synapse is known to implement it. There are no intentional changes which differ from the proposals in this commit, however the author has relied upon various historical conversations outside of the proposals to gain the required context. Inaccuracies introduced by the author are purely accidental. --- api/server-server/definitions/keys.yaml | 23 +--- api/server-server/examples/server_key.json | 5 +- api/server-server/wellknown.yaml | 53 +++++++++ specification/server_server_api.rst | 124 ++++++++++++++++----- 4 files changed, 155 insertions(+), 50 deletions(-) create mode 100644 api/server-server/wellknown.yaml diff --git a/api/server-server/definitions/keys.yaml b/api/server-server/definitions/keys.yaml index 738e9e469..06619641a 100644 --- a/api/server-server/definitions/keys.yaml +++ b/api/server-server/definitions/keys.yaml @@ -25,9 +25,9 @@ properties: verify_keys: type: object description: |- - Public keys of the homeserver for verifying digital signatures. - - The object's key is the algorithm and version combined (``ed25519`` being the + Public keys of the homeserver for verifying digital signatures. + + The object's key is the algorithm and version combined (``ed25519`` being the algorithm and ``abc123`` being the version in the example below). Together, this forms the Key ID. The version must have characters matching the regular expression ``[a-zA-Z0-9_]``. @@ -49,9 +49,9 @@ properties: old_verify_keys: type: object description: |- - The public keys that the server used to use and when it stopped using them. - - The object's key is the algorithm and version combined (``ed25519`` being the + The public keys that the server used to use and when it stopped using them. + + The object's key is the algorithm and version combined (``ed25519`` being the algorithm and ``0ldK3y`` being the version in the example below). Together, this forms the Key ID. The version must have characters matching the regular expression ``[a-zA-Z0-9_]``. @@ -90,17 +90,6 @@ properties: additionalProperties: type: string name: Encoded Signature Verification Key - tls_fingerprints: - type: array - description: Hashes of X.509 TLS certificates used by this server. - items: - type: object - title: TLS Fingerprint - properties: - sha256: - type: string - description: The `Unpadded Base64`_ encoded fingerprint. - example: "VGhpcyBpcyBoYXNoIHdoaWNoIHNob3VsZCBiZSBieXRlcw" valid_until_ts: type: integer format: int64 diff --git a/api/server-server/examples/server_key.json b/api/server-server/examples/server_key.json index bebd8445e..ffdfcb5a5 100644 --- a/api/server-server/examples/server_key.json +++ b/api/server-server/examples/server_key.json @@ -16,8 +16,5 @@ "ed25519:auto2": "VGhpcyBzaG91bGQgYWN0dWFsbHkgYmUgYSBzaWduYXR1cmU" } }, - "tls_fingerprints": [{ - "sha256": "VGhpcyBpcyBoYXNoIHdoaWNoIHNob3VsZCBiZSBieXRlcw" - }], "valid_until_ts": 1652262000000 -} \ No newline at end of file +} diff --git a/api/server-server/wellknown.yaml b/api/server-server/wellknown.yaml new file mode 100644 index 000000000..273da7eb3 --- /dev/null +++ b/api/server-server/wellknown.yaml @@ -0,0 +1,53 @@ +# Copyright 2019 New Vector Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +swagger: '2.0' +info: + title: "Matrix Federation Server Discovery API" + version: "1.0.0" +host: localhost:443 +schemes: + - https +basePath: /.well-known +produces: + - application/json +paths: + "/matrix/server": + get: + summary: Gets information about the delegated server for server-server communication. + description: |- + Gets information about the delegated server for server-server communication + between Matrix homeservers. Servers should follow 30x redirects, carefully + avoiding redirect loops, and use normal X.509 certificate validation. + responses: + 200: + description: + The delegated server information. The ``Content-Type`` for this response SHOULD + be ``application/json``, however servers parsing the response should assume that + the body is JSON regardless of type. Failures parsing the JSON or invalid data + provided in the resulting parsed JSON must result in server discovery failure (no + attempts should be made to continue finding an IP address/port number to connect + to). + examples: + application/json: { + "m.server": "delegated.example.com:1234" + } + schema: + type: object + properties: + "m.server": + type: string + description: |- + The server name to delegate server-server communciations to, with optional + port. The delegated server name uses the same grammar as + `server names in the appendices <../appendices.html#server-name>`_. diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index a63bf0a6b..c88bd7ae0 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -90,35 +90,94 @@ Server discovery Resolving server names ~~~~~~~~~~~~~~~~~~~~~~ -Each matrix homeserver is identified by a server name consisting of a hostname +Each Matrix homeserver is identified by a server name consisting of a hostname and an optional port, as described by the `grammar -<../appendices.html#server-name>`_. Server names should be resolved to an IP -address and port using the following process: - -* If the hostname is an IP literal, then that IP address should be used, - together with the given port number, or 8448 if no port is given. - -* Otherwise, if the port is present, then an IP address is discovered by - looking up an AAAA or A record for the hostname, and the specified port is - used. - -* If the hostname is not an IP literal and no port is given, the server is - discovered by first looking up a ``_matrix._tcp`` SRV record for the - hostname, which may give a hostname (to be looked up using AAAA or A queries) - and port. If the SRV record does not exist, then the server is discovered by - looking up an AAAA or A record on the hostname and taking the default - fallback port number of 8448. - - Homeservers may use SRV records to load balance requests between multiple TLS - endpoints or to failover to another endpoint if an endpoint fails. - -When making requests to servers, use the hostname of the target server in the -``Host`` header, regardless of any hostname given in the SRV record. For -example, if the server name is ``example.org``, and the SRV record resolves to -``matrix.example.org``, the ``Host`` header in the request should be -``example.org``. If an explicit port was given in the server name, it should be -included in the ``Host`` header; otherwise, no port number should be given in -the ``Host`` header. +<../appendices.html#server-name>`_. Where applicable, a delegated server name +uses the same grammar. + +Server names are resolved to an IP address and port to connect to, and have +various conditions affecting which certificates and ``Host`` headers to send. +The process overall is as follows: + +.. Note from the author: The repetitive "use this Host header and this cert" + comments are intentional. The process is overall quite complicated, and + explaining explicitly what requests look like at each step helps ease the + understanding and ensure everyone is on the same page. Implementations + are of course welcome to realize where the process can be optimized, and + do so - just ensure that the result is the same! + +1. If the hostname is an IP literal, then that IP address should be used, + together with the given port number, or 8448 if no port is given. A + valid TLS certificate must be provided by the target server for the + IP address on all requests. Requests must be made with a ``Host`` + header containing the IP address, without port. + +2. If the hostname is not an IP literal, a server is found by resolving + an SRV record for ``_matrix._tcp.``. This may result in + a hostname (to be resolved using AAAA or A records) and port. Requests + are made to the resolved IP address and port, using 8448 as a default + port, with a ``Host`` header of ````. A valid TLS certificate + for ```` must be provided by the target server on all requests. + +3. If the SRV record yielded no results, a ``/.well-known`` request is + made to the hostname (using port 443 exclusively, ignoring the port + provided in the server name). The target must present a valid TLS + certificate for the hostname, and a ``Host`` header containing the + hostname is used to make the request. The schema of the ``/.well-known`` + request is later in this section. Assuming the response is valid, + the ``m.server`` property is parsed as ``[:]`` + and processed as follows: + + * If ```` is an IP literal, then that IP address + should be used together with the ```` or 8448 if no + port is provided. A valid TLS certificate must be provided by the + target server for that IP address. Requests must be made with a + ``Host`` header containing the IP address, without port. + + * If ```` is not an IP literal, and ```` + is present, an IP address is disovered by looking up an AAAA or A + record for ````. The resulting IP address is + used, alongside the ````, to make requests with a + ``Host`` header of ````. A valid TLS certificate + must be provided by the target server for ````. + + * If ```` is not an IP literal and no + ```` is present, an SRV record is looked up for + ``_matrix._tcp.``. This may result in another + hostname (to be resolved using AAAA or A records) and port. Requests + should be made to the resolved IP address and port with a ``Host`` + header containing the ````. Additionally, a + valid TLS certificate must be provided by the target server for the + ````. + + * If no SRV record is found, an IP address is resolved using AAAA + or A records. Requests are then made to the resolve IP address + and a port of 8448, using a ``Host`` header of ````. + A valid TLS certificate for ```` must be + provided by the target server. + +4. If the `/.well-known` request was invalid or returned an error response, + and the SRV record was not found, an IP address is resolved using AAAA + and A records. Requests are made to the resolved IP address using port + 8448 and a ``Host`` header containing the ````. A valid TLS + certificate for ```` must be provided by the target server + on all requests. + + +The TLS certificate provided by the target server must be present on all +requests made to the server. The TLS certificate must be signed by a known +Certificate Authority. Servers are ultimately responsible for determining +the trusted Certificate Authorities, however are strongly encouraged to +rely on the operating system's judgement. Servers can offer administrators +a means to override the trusted authorities list. Servers can additionally +skip the certificate validation for a given whitelist of domains or netmasks +for the purposes of testing or in networks where verification is done +elsewhere, such as with ``.onion`` addresses. + +Servers are encouraged to make use of the +`Certificate Transparency `_ project. + +{{wellknown_ss_http_api}} Server implementation ~~~~~~~~~~~~~~~~~~~~~~ @@ -130,9 +189,16 @@ Retrieving server keys .. NOTE:: There was once a "version 1" of the key exchange. It has been removed from the - specification due to lack of significance. It may be reviewed `here + specification due to lack of significance. It may be reviewed `from the historical draft `_. +.. NOTE:: + Older drafts of this specification made use of a different style of key verification, + however for reasons discussed in `MSC1711 `_, + the approach was removed from the initial version of the specification. The older + draft may be reviewed `thanks to web.archive.org + `_. + Each homeserver publishes its public keys under ``/_matrix/key/v2/server/{keyId}``. Homeservers query for keys by either getting ``/_matrix/key/v2/server/{keyId}`` directly or by querying an intermediate notary server using a From c888f3f080a12393abae7577eea1038fe1d44c79 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 00:09:30 -0700 Subject: [PATCH 076/170] Make example strings more legible --- api/server-server/definitions/pdu.yaml | 6 +++--- api/server-server/definitions/pdu_v3.yaml | 10 +++++----- api/server-server/definitions/unsigned_pdu_base.yaml | 8 ++++---- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/api/server-server/definitions/pdu.yaml b/api/server-server/definitions/pdu.yaml index d86b85384..93de20e66 100644 --- a/api/server-server/definitions/pdu.yaml +++ b/api/server-server/definitions/pdu.yaml @@ -26,13 +26,13 @@ allOf: description: |- Content hashes of the PDU, following the algorithm specified in `Signing Events`_. example: { - "sha256": "thishashcoversallfieldsincasethisisredacted" + "sha256": "ThisHashCoversAllFieldsInCaseThisIsRedacted" } properties: sha256: type: string description: The hash. - example: thishashcoversallfieldsincasthisisredacted + example: ThisHashCoversAllFieldsInCaseThisIsRedacted required: ['sha256'] signatures: type: object @@ -40,7 +40,7 @@ allOf: Signatures for the PDU, following the algorithm specified in `Signing Events`_. example: { "example.com": { - "ed25519:key_version:": "these86bytesofbase64signaturecoveressentialfieldsincludinghashessocancheckredactedpdus" + "ed25519:key_version:": "86BytesOfUnpaddedBase64ToCoverAllFieldsIncludingHashes" } } additionalProperties: diff --git a/api/server-server/definitions/pdu_v3.yaml b/api/server-server/definitions/pdu_v3.yaml index 97747ca8c..40093962c 100644 --- a/api/server-server/definitions/pdu_v3.yaml +++ b/api/server-server/definitions/pdu_v3.yaml @@ -28,7 +28,7 @@ allOf: description: |- Event IDs and reference hashes for the authorization events that would allow this event to be in the room. - example: ["$abase64encodedhash", "$anotherevent"] + example: ["$base64EncodedHash", "$AnotherEvent"] prev_events: type: array items: @@ -37,20 +37,20 @@ allOf: description: |- Event IDs and reference hashes for the most recent events in the room that the homeserver was aware of when it made this event. - example: ["$abase64encodedhash", "$anotherevent"] + example: ["$base64EncodedHash", "$AnotherEvent"] hashes: type: object title: Event Hash description: |- Content hashes of the PDU, following the algorithm specified in `Signing Events`_. example: { - "sha256": "thishashcoversallfieldsincasethisisredacted" + "sha256": "ThisHashCoversAllFieldsInCaseThisIsRedacted" } properties: sha256: type: string description: The hash. - example: thishashcoversallfieldsincasthisisredacted + example: ThisHashCoversAllFieldsInCaseThisIsRedacted required: ['sha256'] signatures: type: object @@ -58,7 +58,7 @@ allOf: Signatures for the PDU, following the algorithm specified in `Signing Events`_. example: { "example.com": { - "ed25519:key_version:": "these86bytesofbase64signaturecoveressentialfieldsincludinghashessocancheckredactedpdus" + "ed25519:key_version:": "86BytesOfUnpaddedBase64ToCoverAllFieldsIncludingHashes" } } additionalProperties: diff --git a/api/server-server/definitions/unsigned_pdu_base.yaml b/api/server-server/definitions/unsigned_pdu_base.yaml index d447b3a3d..283e4fed6 100644 --- a/api/server-server/definitions/unsigned_pdu_base.yaml +++ b/api/server-server/definitions/unsigned_pdu_base.yaml @@ -64,13 +64,13 @@ properties: - type: object title: Event Hash example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" + "sha256": "Base64EncodedSha256HashesShouldBe43BytesLong" } properties: sha256: type: string description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong + example: Base64EncodedSha256HashesShouldBe43BytesLong required: ['sha256'] depth: type: integer @@ -95,13 +95,13 @@ properties: - type: object title: Event Hash example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" + "sha256": "Base64EncodedSha256HashesShouldBe43BytesLong" } properties: sha256: type: string description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong + example: Base64EncodedSha256HashesShouldBe43BytesLong required: ['sha256'] redacts: type: string From 0e90cf6a0c8529cb17dd5a4ed57bc2e221667a02 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 09:38:58 -0700 Subject: [PATCH 077/170] Proposal to change the order of .well-known and SRV discovery techniques --- proposals/1831-srv-after-wellknown.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 proposals/1831-srv-after-wellknown.md diff --git a/proposals/1831-srv-after-wellknown.md b/proposals/1831-srv-after-wellknown.md new file mode 100644 index 000000000..53f5ddacc --- /dev/null +++ b/proposals/1831-srv-after-wellknown.md @@ -0,0 +1,19 @@ +# Proposal to do SRV lookups after .well-known to discover homeservers + +Currently there is a logistical error proposed by [MSC1708](https://github.com/matrix-org/matrix-doc/pull/1708) +which results in some homeservers unable to migrate to the new functionality +proposed by [MSC1711](https://github.com/matrix-org/matrix-doc/pull/1711). This +can happen if the delegated homeserver cannot obtain a valid TLS certificate for +the top level domain, and an SRV record is used for backwards compatibility reasons. + +## Proposal + +We change the order of operations to perform a .well-known lookup before falling +back to resolving the SRV record. This allows for domains to delegate to other +hostnames and maintains backwards compatibility with older homeservers. + +## Tradeoffs + +More HTTP hits will be made due to the .well-known lookup being first. This is +somewhat mitigated by servers caching the responses appropriately, and using +connection pools where possible. From ab7876cbef86ba669710c1a07aa4560764361cfc Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 09:55:58 -0700 Subject: [PATCH 078/170] Update 1831-srv-after-wellknown.md --- proposals/1831-srv-after-wellknown.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1831-srv-after-wellknown.md b/proposals/1831-srv-after-wellknown.md index 53f5ddacc..f1b125f27 100644 --- a/proposals/1831-srv-after-wellknown.md +++ b/proposals/1831-srv-after-wellknown.md @@ -4,7 +4,7 @@ Currently there is a logistical error proposed by [MSC1708](https://github.com/m which results in some homeservers unable to migrate to the new functionality proposed by [MSC1711](https://github.com/matrix-org/matrix-doc/pull/1711). This can happen if the delegated homeserver cannot obtain a valid TLS certificate for -the top level domain, and an SRV record is used for backwards compatibility reasons. +the domain, and an SRV record is used for backwards compatibility reasons. ## Proposal From 2fe6b2cb5c1fc93088378ee86c639582acac1a95 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 10:53:06 -0700 Subject: [PATCH 079/170] Plagiarize from richvdh for a better explanation --- proposals/1831-srv-after-wellknown.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/proposals/1831-srv-after-wellknown.md b/proposals/1831-srv-after-wellknown.md index 53f5ddacc..c0f8ef7a2 100644 --- a/proposals/1831-srv-after-wellknown.md +++ b/proposals/1831-srv-after-wellknown.md @@ -6,6 +6,12 @@ proposed by [MSC1711](https://github.com/matrix-org/matrix-doc/pull/1711). This can happen if the delegated homeserver cannot obtain a valid TLS certificate for the top level domain, and an SRV record is used for backwards compatibility reasons. +Specifically, in order to be compatible with requests from both Synapse 0.34 and 1.0, +servers can have both a SRV and a .well-known file, with Synapse presenting a certificate +corresponding to the target of the .well-known. Synapse 0.34 is then happy because it +will follow the SRV (and won't care about the incorrect certificate); Synapse 1.0 is +happy because it will follow the .well-known (and will see the correct cert). + ## Proposal We change the order of operations to perform a .well-known lookup before falling From d2f012f4ea8f525ace16f75f7abaa5ce3351e3cd Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 11:17:56 -0700 Subject: [PATCH 080/170] Incorporate MSC1831 Original proposal: https://github.com/matrix-org/matrix-doc/pull/1831 Implementation proof: pending --- specification/server_server_api.rst | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index c88bd7ae0..e680c7924 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -112,14 +112,7 @@ The process overall is as follows: IP address on all requests. Requests must be made with a ``Host`` header containing the IP address, without port. -2. If the hostname is not an IP literal, a server is found by resolving - an SRV record for ``_matrix._tcp.``. This may result in - a hostname (to be resolved using AAAA or A records) and port. Requests - are made to the resolved IP address and port, using 8448 as a default - port, with a ``Host`` header of ````. A valid TLS certificate - for ```` must be provided by the target server on all requests. - -3. If the SRV record yielded no results, a ``/.well-known`` request is +2. If the hostname is not an IP literal, a ``/.well-known`` request is made to the hostname (using port 443 exclusively, ignoring the port provided in the server name). The target must present a valid TLS certificate for the hostname, and a ``Host`` header containing the @@ -156,12 +149,19 @@ The process overall is as follows: A valid TLS certificate for ```` must be provided by the target server. -4. If the `/.well-known` request was invalid or returned an error response, - and the SRV record was not found, an IP address is resolved using AAAA - and A records. Requests are made to the resolved IP address using port - 8448 and a ``Host`` header containing the ````. A valid TLS - certificate for ```` must be provided by the target server - on all requests. +3. If the `/.well-known` request returned an error response, a server is + found by resolving an SRV record for ``_matrix._tcp.``. This + may result in a hostname (to be resolved using AAAA or A records) and + port. Requests are made to the resolved IP address and port, using 8448 + as a default port, with a ``Host`` header of ````. A valid TLS + certificate for ```` must be provided by the target server on + all requests. + +4. If the `/.well-known` request returned an error response, and the SRV + record was not found, an IP address is resolved using AAAA and A records. + Requests are made to the resolved IP address using port 8448 and a ``Host`` + header containing the ````. A valid TLS certificate for + ```` must be provided by the target server on all requests. The TLS certificate provided by the target server must be present on all From 48e4d6e412077a48aeb39d374dae17e60f8e64df Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 12:27:09 -0700 Subject: [PATCH 081/170] Document domain reuse concerns Fixes https://github.com/matrix-org/matrix-doc/issues/1783 --- specification/server_server_api.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index a63bf0a6b..b65060d9d 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -1303,6 +1303,16 @@ Example code known hash functions like SHA-256 when none of the keys have been redacted]] +Security considerations +----------------------- + +When a domain's ownership changes, the new controller of the domain can masquerade +as the previous owner, receiving messages (similarly to email) and request past +messages from other servers. In the future, proposals like +`MSC1228 `_ will address this +issue. + + .. |/query/directory| replace:: ``/query/directory`` .. _/query/directory: #get-matrix-federation-v1-query-directory From b971bcee7d42e0a497c7476d3272ac80c16e4057 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 16:24:07 -0700 Subject: [PATCH 082/170] Accurately represent the 3 proposals and provide more detail --- specification/server_server_api.rst | 57 +++++++++++++++-------------- 1 file changed, 30 insertions(+), 27 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index e680c7924..380fe6174 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -112,14 +112,22 @@ The process overall is as follows: IP address on all requests. Requests must be made with a ``Host`` header containing the IP address, without port. -2. If the hostname is not an IP literal, a ``/.well-known`` request is +2. If the hostname is not an IP literal, and has an explicit port given, + resolve the IP address using AAAA or A records. Requests are made to + the resolved IP address and given port with a ``Host`` header of the + original hostname (without port). A valid TLS certificate must be + provided by the target server for the hostname. + +3. If the hostname is not an IP literal, a ``/.well-known`` request is made to the hostname (using port 443 exclusively, ignoring the port - provided in the server name). The target must present a valid TLS - certificate for the hostname, and a ``Host`` header containing the - hostname is used to make the request. The schema of the ``/.well-known`` - request is later in this section. Assuming the response is valid, - the ``m.server`` property is parsed as ``[:]`` - and processed as follows: + provided in the server name). This is done as a plain HTTPS request + which follows 30x redirects, being careful to avoid redirect loops. + The schema of the ``/.well-known`` request is later in this section. + If the response is invalid (bad JSON, missing properties, etc), + attempts to connect to the target server are aborted - no connections + should be attempted. If the response is valid, the ``m.server`` property + is parsed as ``[:]`` and processed + as follows: * If ```` is an IP literal, then that IP address should be used together with the ```` or 8448 if no @@ -131,8 +139,8 @@ The process overall is as follows: is present, an IP address is disovered by looking up an AAAA or A record for ````. The resulting IP address is used, alongside the ````, to make requests with a - ``Host`` header of ````. A valid TLS certificate - must be provided by the target server for ````. + ``Host`` header of ``:``. A valid + TLS certificate must be provided by the target server for ````. * If ```` is not an IP literal and no ```` is present, an SRV record is looked up for @@ -143,36 +151,38 @@ The process overall is as follows: valid TLS certificate must be provided by the target server for the ````. - * If no SRV record is found, an IP address is resolved using AAAA - or A records. Requests are then made to the resolve IP address - and a port of 8448, using a ``Host`` header of ````. - A valid TLS certificate for ```` must be - provided by the target server. + * If no SRV record is found, an IP address is resolved using AAAA + or A records. Requests are then made to the resolve IP address + and a port of 8448, using a ``Host`` header of ````. + A valid TLS certificate for ```` must be + provided by the target server. -3. If the `/.well-known` request returned an error response, a server is - found by resolving an SRV record for ``_matrix._tcp.``. This +4. If the `/.well-known` request did not result in a 200 response, a server + is found by resolving an SRV record for ``_matrix._tcp.``. This may result in a hostname (to be resolved using AAAA or A records) and port. Requests are made to the resolved IP address and port, using 8448 as a default port, with a ``Host`` header of ````. A valid TLS certificate for ```` must be provided by the target server on all requests. -4. If the `/.well-known` request returned an error response, and the SRV +5. If the `/.well-known` request returned an error response, and the SRV record was not found, an IP address is resolved using AAAA and A records. Requests are made to the resolved IP address using port 8448 and a ``Host`` header containing the ````. A valid TLS certificate for ```` must be provided by the target server on all requests. -The TLS certificate provided by the target server must be present on all -requests made to the server. The TLS certificate must be signed by a known +The TLS certificate provided by the target server must be signed by a known Certificate Authority. Servers are ultimately responsible for determining the trusted Certificate Authorities, however are strongly encouraged to rely on the operating system's judgement. Servers can offer administrators a means to override the trusted authorities list. Servers can additionally skip the certificate validation for a given whitelist of domains or netmasks for the purposes of testing or in networks where verification is done -elsewhere, such as with ``.onion`` addresses. +elsewhere, such as with ``.onion`` addresses. Servers should respect SNI +when making requests where possible: a SNI should be sent for the certificate +which is expected, unless that certificate is expected to be an IP address in +which case SNI is not supported and should not be sent. Servers are encouraged to make use of the `Certificate Transparency `_ project. @@ -192,13 +202,6 @@ Retrieving server keys specification due to lack of significance. It may be reviewed `from the historical draft `_. -.. NOTE:: - Older drafts of this specification made use of a different style of key verification, - however for reasons discussed in `MSC1711 `_, - the approach was removed from the initial version of the specification. The older - draft may be reviewed `thanks to web.archive.org - `_. - Each homeserver publishes its public keys under ``/_matrix/key/v2/server/{keyId}``. Homeservers query for keys by either getting ``/_matrix/key/v2/server/{keyId}`` directly or by querying an intermediate notary server using a From 3c17a0e53bad28cc38f2df0828e83a71b2f8e1ac Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 16:29:07 -0700 Subject: [PATCH 083/170] Mention caching --- specification/server_server_api.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 380fe6174..a2672e927 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -122,12 +122,19 @@ The process overall is as follows: made to the hostname (using port 443 exclusively, ignoring the port provided in the server name). This is done as a plain HTTPS request which follows 30x redirects, being careful to avoid redirect loops. - The schema of the ``/.well-known`` request is later in this section. - If the response is invalid (bad JSON, missing properties, etc), - attempts to connect to the target server are aborted - no connections - should be attempted. If the response is valid, the ``m.server`` property - is parsed as ``[:]`` and processed - as follows: + Responses (successful or otherwise) to the ``/.well-known`` endpoint + should be cached by the requesting server. Servers should respect + the cache control headers present on the response, or use a sensible + default when headers are not present. The recommended sensible default + is 24 hours. Servers should additionally impose a maximum cache time + for responses: 48 hours is recommended. Errors are recommended to be + cached for up to an hour, and servers are encouraged to exponentially + back off for repeated failures. The schema of the ``/.well-known`` + request is later in this section. If the response is invalid (bad JSON, + missing properties, etc), attempts to connect to the target server are + aborted - no connections should be attempted. If the response is valid, + the ``m.server`` property is parsed as ``[:]`` + and processed as follows: * If ```` is an IP literal, then that IP address should be used together with the ```` or 8448 if no From 01556e5b179ab5fd9beaf1c07d6cc327d5fadde3 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 16:44:24 -0700 Subject: [PATCH 084/170] Clarifications and bug fixes in how v3 rooms work --- api/server-server/definitions/pdu.yaml | 4 +- api/server-server/definitions/pdu_v3.yaml | 8 ++-- specification/index.rst | 2 +- specification/rooms/v3.rst | 47 +++++++++++++---------- 4 files changed, 33 insertions(+), 28 deletions(-) diff --git a/api/server-server/definitions/pdu.yaml b/api/server-server/definitions/pdu.yaml index 93de20e66..87064c22c 100644 --- a/api/server-server/definitions/pdu.yaml +++ b/api/server-server/definitions/pdu.yaml @@ -13,7 +13,7 @@ # limitations under the License. type: object title: Persistent Data Unit -description: A persistent data unit (event) +description: A persistent data unit (event) for room versions 1 and 2. example: $ref: "../examples/pdu.json" allOf: @@ -40,7 +40,7 @@ allOf: Signatures for the PDU, following the algorithm specified in `Signing Events`_. example: { "example.com": { - "ed25519:key_version:": "86BytesOfUnpaddedBase64ToCoverAllFieldsIncludingHashes" + "ed25519:key_version:": "86BytesOfSignatureOfTheRedactedEvent" } } additionalProperties: diff --git a/api/server-server/definitions/pdu_v3.yaml b/api/server-server/definitions/pdu_v3.yaml index 40093962c..8d41fbdaf 100644 --- a/api/server-server/definitions/pdu_v3.yaml +++ b/api/server-server/definitions/pdu_v3.yaml @@ -13,7 +13,7 @@ # limitations under the License. type: object title: Persistent Data Unit -description: A persistent data unit (event) for room version 3. +description: A persistent data unit (event) for room version 3 and beyond. example: $ref: "../examples/pdu_v3.json" allOf: @@ -26,7 +26,7 @@ allOf: type: string description: Event ID. description: |- - Event IDs and reference hashes for the authorization events that would + Event IDs for the authorization events that would allow this event to be in the room. example: ["$base64EncodedHash", "$AnotherEvent"] prev_events: @@ -35,7 +35,7 @@ allOf: type: string description: Event ID. description: |- - Event IDs and reference hashes for the most recent events in the room + Event IDs for the most recent events in the room that the homeserver was aware of when it made this event. example: ["$base64EncodedHash", "$AnotherEvent"] hashes: @@ -58,7 +58,7 @@ allOf: Signatures for the PDU, following the algorithm specified in `Signing Events`_. example: { "example.com": { - "ed25519:key_version:": "86BytesOfUnpaddedBase64ToCoverAllFieldsIncludingHashes" + "ed25519:key_version:": "86BytesOfSignatureOfTheRedactedEvent" } } additionalProperties: diff --git a/specification/index.rst b/specification/index.rst index fa9ca3d65..05b85ab7c 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -484,7 +484,7 @@ The available room versions are: * `Version 1 `_ - **Stable**. The current version of most rooms. * `Version 2 `_ - **Stable**. Implements State Resolution Version 2. -* `Version 3 `_ - **Stable**. Introduces a new event format. +* `Version 3 `_ - **Stable**. Introduces events whose IDs are the event's hash. Specification Versions ---------------------- diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index 22276ab32..3f383e7b7 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -15,7 +15,7 @@ Room Version 3 ============== -This room version builds off of `version 2 `_ with an improved event format. +This room version builds on `version 2 `_ with an improved event format. .. note: All requirements listed in this room version specification are scoped to rooms @@ -49,29 +49,32 @@ Server implementation components Room version 3 uses the state resolution algorithm defined in `room version 2 `_, -and the event format defined here. Other event formats and applicable algorithms -may be used in other room versions. +and the event format defined here. Event IDs ~~~~~~~~~ -In other room versions (namely version 1 and 2) the event ID is a distinct field -from the remainder of the event, which must be tracked as such. This leads to -complications where servers receive multiple events with the same ID in either the -same or different rooms where the server cannot easily keep track of which event it -should be using. By removing the use of a dedicated event ID, servers are required -to track the hashes on an event to determine its ID. +.. admonition:: Rationale -The event ID must be a sha256 hash of the event, encoded using `Unpadded Base64`_ -and prefixed with `$`. For example, an event ID might be -``$CD66HAED5npg6074c6pDtLKalHjVfYb2q4Q3LZgrW6o``. + In other room versions (namely version 1 and 2) the event ID is a distinct field + from the remainder of the event, which must be tracked as such. This leads to + complications where servers receive multiple events with the same ID in either the + same or different rooms where the server cannot easily keep track of which event it + should be using. By removing the use of a dedicated event ID, servers are required + to track the hashes on an event to determine its ID. -The hash itself is calculated the same as previous reference hashes are: +The event ID is calculated using the following algorithm. Note that the hashing +algorithm used is the same as in previous room versions. 1. Redact the event. 2. Remove the `signatures` field from the event. 3. Serialize the event into `Canonical JSON`_. 4. Compute the hash of the JSON bytes. +5. Encode the sha256 hash using `Unpadded Base64`_. +6. Use the resulting string as the event ID prefixed with ``$``. + +A resulting event ID using this approach should look similar to +``$CD66HAED5npg6074c6pDtLKalHjVfYb2q4Q3LZgrW6o``. Event IDs should not be sent over federation to servers when the room uses this room version. On the receiving end of an event, the server should compute @@ -91,11 +94,10 @@ APIs which currently accept an event ID must do so with the new format. Servers must append the calculated event ID to all events sent to clients where an event ID would normally be expected. -Because servers are not communicating the event ID over the wire to each other, -servers must be aware of the room version where the event resides so that the -server may parse and handle the event. The federation API has taken this concern -into consideration by ensuring that servers are aware of (or can find) the room -version during a request. +Because the format of events has changed, servers must be aware of the room version +where the event resides so that the server may parse and handle the event. The +federation API has taken this concern into consideration by ensuring that servers +are aware of (or can find) the room version during a request. Authorization rules for events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -108,9 +110,12 @@ to the change in event format: domain. * Previously, redactions were allowed if the sender's domain matched the domain - in the event ID it was redacting, allowing self redaction. This check is removed - and redaction events are always accepted. Redaction events only take effect - when the original event is received, and the domain of the each event matches. + in the event ID it was redacting, allowing self redaction. Due to changes in + the event format, this check is now impossible to do. Instead, servers should + allow redactions from servers of the same origin to redact other events as a + self-redaction mechanism. The rules for allowing other servers to redact events + (as done by moderators) is unchanged. Redaction events only take effect when + the original event is received, and the domain of the each event matches. Servers should not send redactions down to clients until the redaction has taken effect. From ad64af3f016275d6f5e2f2acd954b5c7cbb7d6d8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 17:22:55 -0700 Subject: [PATCH 085/170] Clarify how reference hashes are done --- specification/rooms/v3.rst | 13 ++----------- specification/server_server_api.rst | 16 ++++++++++++++++ 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index 3f383e7b7..9efcc463a 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -63,17 +63,8 @@ Event IDs should be using. By removing the use of a dedicated event ID, servers are required to track the hashes on an event to determine its ID. -The event ID is calculated using the following algorithm. Note that the hashing -algorithm used is the same as in previous room versions. - -1. Redact the event. -2. Remove the `signatures` field from the event. -3. Serialize the event into `Canonical JSON`_. -4. Compute the hash of the JSON bytes. -5. Encode the sha256 hash using `Unpadded Base64`_. -6. Use the resulting string as the event ID prefixed with ``$``. - -A resulting event ID using this approach should look similar to +The event ID is the reference hash of the event encoded using `Unpadded Base64`_, +prefixed with ``$``. A resulting event ID using this approach should look similar to ``$CD66HAED5npg6074c6pDtLKalHjVfYb2q4Q3LZgrW6o``. Event IDs should not be sent over federation to servers when the room uses diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 8f021e620..6eaf74e0d 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -1050,6 +1050,22 @@ been given a redacted version of the event. To enforce this, the receiving server should use the redacted copy it calculated rather than the full copy it received. +Calculating the reference hash for an event +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The *reference hash* of an event covers the essential fields of an event, +includuing content hashes. It is calculated as follows. + +1. The event is put through the redaction algorithm. + +2. The ``signatures``, ``age_ts``, and ``unsigned`` properties are removed + from the event, if present. + +3. The event is converted into `Canonical JSON`_. + +4. A sha256 hash is calculed on the resulting JSON object. + + Calculating the content hash for an event ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 48912a73202121f86f1a8f971539df55eabea0d8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 17:35:25 -0700 Subject: [PATCH 086/170] Fix auth rules of redactions in v3 --- specification/rooms/v3.rst | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index 9efcc463a..040a7ff40 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -100,15 +100,17 @@ to the change in event format: is no domain in the event ID), but still needs to be signed by the sender's domain. -* Previously, redactions were allowed if the sender's domain matched the domain - in the event ID it was redacting, allowing self redaction. Due to changes in - the event format, this check is now impossible to do. Instead, servers should - allow redactions from servers of the same origin to redact other events as a - self-redaction mechanism. The rules for allowing other servers to redact events - (as done by moderators) is unchanged. Redaction events only take effect when - the original event is received, and the domain of the each event matches. - Servers should not send redactions down to clients until the redaction has - taken effect. +* In past room versions, redactions were only permitted to enter the DAG if the + sender's domain matched the domain in the event ID being redacted, or the sender + had appropriate permissions per the power levels. Due to servers now not being + able to determine where an event came from during event authorization, redaction + events are always accepted (provided the event is allowed by ``events`` and + ``events_default`` in the power levels). However, servers should not apply or send + redactions to clients until both the redaction event and original event have been + seen, and are valid. Servers should only apply redactions to events where the + origin sender's domains match, or the sender has the appropriate permissions per + the power levels. + The remaining rules are the same as room version 1. From a6243da03fef85e14bd8024c91914f138d218bb4 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 17:58:42 -0700 Subject: [PATCH 087/170] Wording changes and links --- specification/rooms/v3.rst | 9 +++++---- specification/server_server_api.rst | 6 +++++- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index 040a7ff40..e5b0c44c7 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -63,7 +63,7 @@ Event IDs should be using. By removing the use of a dedicated event ID, servers are required to track the hashes on an event to determine its ID. -The event ID is the reference hash of the event encoded using `Unpadded Base64`_, +The event ID is the `reference hash`_ of the event encoded using `Unpadded Base64`_, prefixed with ``$``. A resulting event ID using this approach should look similar to ``$CD66HAED5npg6074c6pDtLKalHjVfYb2q4Q3LZgrW6o``. @@ -108,13 +108,14 @@ to the change in event format: ``events_default`` in the power levels). However, servers should not apply or send redactions to clients until both the redaction event and original event have been seen, and are valid. Servers should only apply redactions to events where the - origin sender's domains match, or the sender has the appropriate permissions per - the power levels. + sender's domains match, or the sender of the redaction has the appropriate + permissions per the power levels. -The remaining rules are the same as room version 1. +The remaining rules are the same as `room version 1 `_. .. _`Unpadded Base64`: ../../appendices.html#unpadded-base64 .. _`Canonical JSON`: ../../appendices.html#canonical-json .. _`Signing Events`: ../../server_server/unstable.html#signing-events +.. _`reference hash`: ../../server_server/unstable.html#reference-hashes diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 6eaf74e0d..f5831639c 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -281,6 +281,8 @@ Transactions are limited in size; they can have at most 50 PDUs and 100 EDUs. {{transactions_ss_http_api}} +.. _`Persistent Data Unit schema`: + PDUs ---- @@ -1050,11 +1052,13 @@ been given a redacted version of the event. To enforce this, the receiving server should use the redacted copy it calculated rather than the full copy it received. +.. _`reference hashes`: + Calculating the reference hash for an event ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The *reference hash* of an event covers the essential fields of an event, -includuing content hashes. It is calculated as follows. +including content hashes. It is calculated as follows. 1. The event is put through the redaction algorithm. From 1c30f5eba986335635f1740b1689e9b011beefd1 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 21:15:47 -0700 Subject: [PATCH 088/170] Clarify certificate usage and ports --- specification/server_server_api.rst | 75 ++++++++++++++--------------- 1 file changed, 36 insertions(+), 39 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index a2672e927..46a485af3 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -107,76 +107,73 @@ The process overall is as follows: do so - just ensure that the result is the same! 1. If the hostname is an IP literal, then that IP address should be used, - together with the given port number, or 8448 if no port is given. A - valid TLS certificate must be provided by the target server for the - IP address on all requests. Requests must be made with a ``Host`` - header containing the IP address, without port. + together with the given port number, or 8448 if no port is given. The + target server must present a valid certificate for the IP address. + Requests must be made with a ``Host`` header containing the IP address, + without port. 2. If the hostname is not an IP literal, and has an explicit port given, resolve the IP address using AAAA or A records. Requests are made to the resolved IP address and given port with a ``Host`` header of the - original hostname (without port). A valid TLS certificate must be - provided by the target server for the hostname. - -3. If the hostname is not an IP literal, a ``/.well-known`` request is - made to the hostname (using port 443 exclusively, ignoring the port - provided in the server name). This is done as a plain HTTPS request - which follows 30x redirects, being careful to avoid redirect loops. - Responses (successful or otherwise) to the ``/.well-known`` endpoint - should be cached by the requesting server. Servers should respect - the cache control headers present on the response, or use a sensible - default when headers are not present. The recommended sensible default - is 24 hours. Servers should additionally impose a maximum cache time - for responses: 48 hours is recommended. Errors are recommended to be - cached for up to an hour, and servers are encouraged to exponentially - back off for repeated failures. The schema of the ``/.well-known`` - request is later in this section. If the response is invalid (bad JSON, - missing properties, etc), attempts to connect to the target server are - aborted - no connections should be attempted. If the response is valid, - the ``m.server`` property is parsed as ``[:]`` - and processed as follows: + original hostname (with port). The target server must present a valid + certificate for the hostname. + +3. If the hostname is not an IP literal, a regular HTTPS request is made + to ``https:///.well-known/matrix/server``, expecting the + schema defined later in this section. 30x redirects should be followed, + however redirection loops should be avoided. Responses (successful or + otherwise) to the ``/.well-known`` endpoint should be cached by the + requesting server. Servers should respect the cache control headers + present on the response, or use a sensible default when headers are not + present. The recommended sensible default is 24 hours. Servers should + additionally impose a maximum cache time for responses: 48 hours is + recommended. Errors are recommended to be cached for up to an hour, + and servers are encouraged to exponentially back off for repeated + failures. The schema of the ``/.well-known`` request is later in this + section. If the response is invalid (bad JSON, missing properties, etc), + attempts to connect to the target server are aborted - no connections + should be attempted. If the response is valid, the ``m.server`` property + is parsed as ``[:]`` and processed + as follows: * If ```` is an IP literal, then that IP address should be used together with the ```` or 8448 if no - port is provided. A valid TLS certificate must be provided by the - target server for that IP address. Requests must be made with a - ``Host`` header containing the IP address, without port. + port is provided. The target server must present a valid TLS certificate + for the IP address. Requests must be made with a ``Host`` header containing + the IP address, with port. * If ```` is not an IP literal, and ```` is present, an IP address is disovered by looking up an AAAA or A record for ````. The resulting IP address is used, alongside the ````, to make requests with a - ``Host`` header of ``:``. A valid - TLS certificate must be provided by the target server for ````. + ``Host`` header of ``:``. The + target server must present a valid certificate for ````. * If ```` is not an IP literal and no ```` is present, an SRV record is looked up for ``_matrix._tcp.``. This may result in another hostname (to be resolved using AAAA or A records) and port. Requests should be made to the resolved IP address and port with a ``Host`` - header containing the ````. Additionally, a - valid TLS certificate must be provided by the target server for the - ````. + header containing the ````. The target server + must present a valid certificate for ````. * If no SRV record is found, an IP address is resolved using AAAA or A records. Requests are then made to the resolve IP address and a port of 8448, using a ``Host`` header of ````. - A valid TLS certificate for ```` must be - provided by the target server. + The target server must present a valid certificate for ````. 4. If the `/.well-known` request did not result in a 200 response, a server is found by resolving an SRV record for ``_matrix._tcp.``. This may result in a hostname (to be resolved using AAAA or A records) and port. Requests are made to the resolved IP address and port, using 8448 - as a default port, with a ``Host`` header of ````. A valid TLS - certificate for ```` must be provided by the target server on - all requests. + as a default port, with a ``Host`` header of ````. The target + server must present a valid certificate for ````. 5. If the `/.well-known` request returned an error response, and the SRV record was not found, an IP address is resolved using AAAA and A records. Requests are made to the resolved IP address using port 8448 and a ``Host`` - header containing the ````. A valid TLS certificate for - ```` must be provided by the target server on all requests. + header containing the ````. The target server must present a + valid certificate for ````. The TLS certificate provided by the target server must be signed by a known From 39b71413be1f20d7071c2d8babe784b22b015537 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 1 Feb 2019 07:40:39 -0700 Subject: [PATCH 089/170] Update specification/server_server_api.rst Co-Authored-By: turt2live --- specification/server_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 46a485af3..17f3e646b 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -112,7 +112,7 @@ The process overall is as follows: Requests must be made with a ``Host`` header containing the IP address, without port. -2. If the hostname is not an IP literal, and has an explicit port given, +2. If the hostname is not an IP literal, and the server name includes an explicit port, resolve the IP address using AAAA or A records. Requests are made to the resolved IP address and given port with a ``Host`` header of the original hostname (with port). The target server must present a valid From 6421582bf2f29f766a151f93264126c3f69f4139 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 08:18:12 -0700 Subject: [PATCH 090/170] Clarifications and alter the error handling of .well-known Note that MSC1831 changes the order, so the changes to MSC1708 might not make sense when combining all the proposals together. However, independently the change should make sense. --- proposals/1708-well-known-for-federation.md | 4 +- specification/server_server_api.rst | 51 ++++++++++----------- 2 files changed, 27 insertions(+), 28 deletions(-) diff --git a/proposals/1708-well-known-for-federation.md b/proposals/1708-well-known-for-federation.md index d23c0e2ff..c9f90631e 100644 --- a/proposals/1708-well-known-for-federation.md +++ b/proposals/1708-well-known-for-federation.md @@ -44,8 +44,8 @@ redirect loops). If the request does not return a 200, continue to step 4, otherwise: The response must be valid JSON which follows the structure documented -below. Otherwise, the request is aborted. It is NOT necessary for the response -to have a `Content-Type` of `application/json`. +below. Otherwise, continue to the next step in the discovery process. It is +NOT necessary for the response to have a `Content-Type` of `application/json`. If the response is valid, the `m.server` property is parsed as `[:]`, and processed as follows: diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 17f3e646b..8de5ecc54 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -109,14 +109,14 @@ The process overall is as follows: 1. If the hostname is an IP literal, then that IP address should be used, together with the given port number, or 8448 if no port is given. The target server must present a valid certificate for the IP address. - Requests must be made with a ``Host`` header containing the IP address, - without port. + The ``Host`` header in the request should be set to the server name, + including the port if the server name included one. -2. If the hostname is not an IP literal, and the server name includes an explicit port, - resolve the IP address using AAAA or A records. Requests are made to - the resolved IP address and given port with a ``Host`` header of the - original hostname (with port). The target server must present a valid - certificate for the hostname. +2. If the hostname is not an IP literal, and the server name includes an + explicit port, resolve the IP address using AAAA or A records. Requests + are made to the resolved IP address and given port with a ``Host`` header + of the original server name (with port). The target server must present a + valid certificate for the hostname. 3. If the hostname is not an IP literal, a regular HTTPS request is made to ``https:///.well-known/matrix/server``, expecting the @@ -130,39 +130,38 @@ The process overall is as follows: recommended. Errors are recommended to be cached for up to an hour, and servers are encouraged to exponentially back off for repeated failures. The schema of the ``/.well-known`` request is later in this - section. If the response is invalid (bad JSON, missing properties, etc), - attempts to connect to the target server are aborted - no connections - should be attempted. If the response is valid, the ``m.server`` property - is parsed as ``[:]`` and processed - as follows: + section. If the response is invalid (bad JSON, missing properties, non-200 + response, etc), skip to step 4. If the response is valid, the ``m.server`` + property is parsed as ``[:]`` and + processed as follows: - * If ```` is an IP literal, then that IP address + * If ```` is an IP literal, then that IP address should be used together with the ```` or 8448 if no port is provided. The target server must present a valid TLS certificate for the IP address. Requests must be made with a ``Host`` header containing - the IP address, with port. + the IP address, including the port if one was provided. - * If ```` is not an IP literal, and ```` + * If ```` is not an IP literal, and ```` is present, an IP address is disovered by looking up an AAAA or A - record for ````. The resulting IP address is - used, alongside the ````, to make requests with a - ``Host`` header of ``:``. The - target server must present a valid certificate for ````. + record for ````. The resulting IP address is + used, alongside the ````. Requests must be made with a + ``Host`` header of ``:``. The + target server must present a valid certificate for ````. - * If ```` is not an IP literal and no + * If ```` is not an IP literal and no ```` is present, an SRV record is looked up for - ``_matrix._tcp.``. This may result in another + ``_matrix._tcp.``. This may result in another hostname (to be resolved using AAAA or A records) and port. Requests should be made to the resolved IP address and port with a ``Host`` - header containing the ````. The target server - must present a valid certificate for ````. + header containing the ````. The target server + must present a valid certificate for ````. * If no SRV record is found, an IP address is resolved using AAAA or A records. Requests are then made to the resolve IP address - and a port of 8448, using a ``Host`` header of ````. - The target server must present a valid certificate for ````. + and a port of 8448, using a ``Host`` header of ````. + The target server must present a valid certificate for ````. -4. If the `/.well-known` request did not result in a 200 response, a server +4. If the `/.well-known` request resulted in an error response, a server is found by resolving an SRV record for ``_matrix._tcp.``. This may result in a hostname (to be resolved using AAAA or A records) and port. Requests are made to the resolved IP address and port, using 8448 From 9b214ec16d1fe8db4f59100a35fcc9b5872da32c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 23:46:22 -0700 Subject: [PATCH 091/170] Make the backfill response aware of event format changes --- api/server-server/backfill.yaml | 29 +++++-------------- .../definitions/transaction.yaml | 15 ++++++++-- api/server-server/examples/transaction.json | 6 ++-- 3 files changed, 24 insertions(+), 26 deletions(-) diff --git a/api/server-server/backfill.yaml b/api/server-server/backfill.yaml index b196d17c8..738e6efcb 100644 --- a/api/server-server/backfill.yaml +++ b/api/server-server/backfill.yaml @@ -65,22 +65,6 @@ paths: event(s), up to the given limit. schema: $ref: "definitions/transaction.yaml" - # Override the example to show the response of the request a bit better - examples: - application/json: { - "$ref": "examples/transaction.json", - "pdus": [ - { - "$ref": "pdu.json", - "room_id": "!SomeRoom:matrix.org", - "event_id": "$abc123:matrix.org" - }, - { - "$ref": "pdu.json", - "room_id": "!SomeRoom:matrix.org" - }, - ] - } "/get_missing_events/{roomId}": post: summary: Retrieves events that the sender is missing @@ -114,14 +98,14 @@ paths: earliest_events: type: array description: |- - The latest events that the sender already has. These are skipped when retrieving + The latest event IDs that the sender already has. These are skipped when retrieving the previous events of ``latest_events``. items: type: string example: ["$missing_event:example.org"] latest_events: type: array - description: The events to retrieve the previous events for. + description: The event IDs to retrieve the previous events for. items: type: string example: ["$event_that_has_the_missing_event_as_a_previous_event:example.org"] @@ -136,13 +120,16 @@ paths: properties: events: type: array - description: The missing events. + description: |- + The missing events. The event format varies depending on the room version - check + the `room version specification`_ for precise event formats. items: - $ref: definitions/pdu.yaml + type: object + title: PDU required: ['events'] examples: application/json: { "events": [ - {"$ref": "examples/pdu.json"} + {"see_room_version_spec": "The event format changes depending on the room version."} ] } diff --git a/api/server-server/definitions/transaction.yaml b/api/server-server/definitions/transaction.yaml index 9833f7852..7ec9386e5 100644 --- a/api/server-server/definitions/transaction.yaml +++ b/api/server-server/definitions/transaction.yaml @@ -26,12 +26,21 @@ properties: type: integer format: int64 description: |- - POSIX timestamp in milliseconds on originating homeserver when this + POSIX timestamp in milliseconds on originating homeserver when this transaction started. example: 1532991320875 pdus: type: array - description: List of persistent updates to rooms. Must not include more than 50 PDUs. + description: |- + List of persistent updates to rooms. Must not include more than 50 PDUs. Note that + events have a different version depending on the room version - check the + `room version specification`_ for precise event formats. items: - $ref: "pdu.yaml" + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ contained in the transaction. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. + properties: [] + example: {"see_room_version_spec": "The event format changes depending on the room version."} required: ['origin', 'origin_server_ts', 'pdus'] diff --git a/api/server-server/examples/transaction.json b/api/server-server/examples/transaction.json index bd8ac3dc7..6d77f79ec 100644 --- a/api/server-server/examples/transaction.json +++ b/api/server-server/examples/transaction.json @@ -1,5 +1,7 @@ { "origin": "matrix.org", "origin_server_ts": 1234567890, - "pdus": [{"$ref": "pdu.json"}] -} \ No newline at end of file + "pdus": [{ + "see_room_version_spec": "The event format changes depending on the room version." + }] +} From d94a70f49de4770ec84a700058737384938d7ce4 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 31 Jan 2019 23:54:14 -0700 Subject: [PATCH 092/170] Warn clients about changes in event format --- specification/client_server_api.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 40ac55884..8f4a9a1d3 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1267,7 +1267,21 @@ point in time:: [E0]->[E1]->[E2]->[E3]->[E4]->[E5] +.. WARNING:: + + The format of events can change depending on room version. Check the + `room version specification`_ for specific details on what to expect for + event formats. Examples contained within the client-server specification + are expected to be compatible with all specified room versions, however + some differences may still apply. + + For this version of the specification, clients only need to worry about + the event ID format being different depending on room version. Clients + should not be parsing the event ID, and instead be treating it as an + opaque string. No changes should be required to support the currently + available room versions. +.. _`room version specification`: ../index.html#room-versions Types of room events ~~~~~~~~~~~~~~~~~~~~ From 33406e4662a101eaa23f9b74612f18856b902b25 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 10:08:30 -0700 Subject: [PATCH 093/170] Apply event format warnings to the remainder of the s2s spec --- api/server-server/backfill.yaml | 2 +- .../definitions/invite_event.yaml | 44 +-- .../definitions/transaction.yaml | 3 +- api/server-server/event_auth.yaml | 50 +++- api/server-server/events.yaml | 30 +- api/server-server/examples/minimal_pdu.json | 7 + api/server-server/examples/transaction.json | 2 +- api/server-server/invites-v1.yaml | 12 +- api/server-server/invites-v2.yaml | 12 +- api/server-server/joins.yaml | 266 ++++++++---------- api/server-server/leaving.yaml | 248 +++++++--------- api/server-server/transactions.yaml | 3 + 12 files changed, 316 insertions(+), 363 deletions(-) create mode 100644 api/server-server/examples/minimal_pdu.json diff --git a/api/server-server/backfill.yaml b/api/server-server/backfill.yaml index 738e6efcb..0da0e2341 100644 --- a/api/server-server/backfill.yaml +++ b/api/server-server/backfill.yaml @@ -130,6 +130,6 @@ paths: examples: application/json: { "events": [ - {"see_room_version_spec": "The event format changes depending on the room version."} + {"$ref": "examples/minimal_pdu.json"} ] } diff --git a/api/server-server/definitions/invite_event.yaml b/api/server-server/definitions/invite_event.yaml index d196339a6..1f8b29d05 100644 --- a/api/server-server/definitions/invite_event.yaml +++ b/api/server-server/definitions/invite_event.yaml @@ -1,4 +1,4 @@ -# Copyright 2018 New Vector Ltd +# Copyright 2018-2019 New Vector Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,15 +13,14 @@ # limitations under the License. type: object title: Invite Event -description: An invite event +description: |- + An invite event. Note that events have a different version depending on the + room version - check the `room version specification`_ for precise event formats. allOf: - - $ref: "pdu.yaml" - type: object properties: - # Note: we override a bunch of parameters to change their descriptions sender: type: string - # TODO: Verify/clarify this - it doesn't seem right, given this is a 'regular' invite description: |- The matrix ID of the user who sent the original ``m.room.third_party_invite``. example: "@someone:example.org" @@ -46,7 +45,7 @@ allOf: type: object title: Membership Event Content description: |- - The content of the event, matching what is available in the + The content of the event, matching what is available in the `Client-Server API`_. Must include a ``membership`` of ``invite``. example: {"membership": "invite"} properties: @@ -55,33 +54,10 @@ allOf: description: The value ``invite``. example: "invite" required: ['membership'] - auth_events: - type: array - description: |- - An event reference list containing the authorization events that would - allow the member to be invited to the room. - items: - type: array - maxItems: 2 - minItems: 2 - items: - - type: string - title: Event ID - example: "$abc123:matrix.org" - - type: object - title: Event Hash - example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" - } - properties: - sha256: - type: string - description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong - required: ['sha256'] - redacts: - type: string - description: Not used. required: - # Every other field is already flagged as required by the $ref - state_key + - sender + - origin + - origin_server_ts + - type + - content diff --git a/api/server-server/definitions/transaction.yaml b/api/server-server/definitions/transaction.yaml index 7ec9386e5..a77f6f8a3 100644 --- a/api/server-server/definitions/transaction.yaml +++ b/api/server-server/definitions/transaction.yaml @@ -42,5 +42,6 @@ properties: The `PDUs <#pdus>`_ contained in the transaction. The event format varies depending on the room version - check the `room version specification`_ for precise event formats. properties: [] - example: {"see_room_version_spec": "The event format changes depending on the room version."} + example: + $ref: "../examples/minimal_pdu.json" required: ['origin', 'origin_server_ts', 'pdus'] diff --git a/api/server-server/event_auth.yaml b/api/server-server/event_auth.yaml index 905b112f2..262ea2827 100644 --- a/api/server-server/event_auth.yaml +++ b/api/server-server/event_auth.yaml @@ -20,7 +20,7 @@ host: localhost:8448 schemes: - https basePath: /_matrix/federation/v1 -consumes: +consumes: - application/json produces: - application/json @@ -58,10 +58,19 @@ paths: type: array description: |- The full set of authorization events that make up the state of - the room, and their authorization events, recursively. + the room, and their authorization events, recursively. Note that + events have a different version depending on the room version - + check the `room version specification`_ for precise event formats. items: - $ref: "definitions/pdu.yaml" - example: [{"$ref": "examples/pdu.json"}] + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ contained in the auth chain. The event format + varies depending on the room version - check the `room version specification`_ + for precise event formats. + properties: [] + example: + $ref: "examples/minimal_pdu.json" required: ['auth_chain'] "/query_auth/{roomId}/{eventId}": post: @@ -98,10 +107,20 @@ paths: properties: auth_chain: type: array - description: The auth chain (the "remote auth"). + description: |- + The auth chain (the "remote auth"). Note that events have a different + version depending on the room version - check the `room version specification`_ + for precise event formats. items: - $ref: "definitions/pdu.yaml" - example: [{"$ref": "examples/pdu.json"}] + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ contained in the auth chain. The event format + varies depending on the room version - check the `room version specification`_ + for precise event formats. + properties: [] + example: + $ref: "examples/minimal_pdu.json" missing: type: array description: |- @@ -142,14 +161,23 @@ paths: type: array description: |- The auth chain the receiver has, and used to determine the auth - chain differences (the "local auth"). + chain differences (the "local auth"). Note that events have a different + version depending on the room version - check the `room version specification`_ + for precise event formats. items: - $ref: "definitions/pdu.yaml" - example: [{"$ref": "examples/pdu.json"}] + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ contained in the auth chain. The event format + varies depending on the room version - check the `room version specification`_ + for precise event formats. + properties: [] + example: + $ref: "examples/minimal_pdu.json" missing: type: array description: |- - The list of event IDs that the receiver believes it is missing, + The list of event IDs that the receiver believes it is missing, after comparing the "remote auth" and "local auth" chains. items: type: string diff --git a/api/server-server/events.yaml b/api/server-server/events.yaml index d23456d1c..e7cb52cde 100644 --- a/api/server-server/events.yaml +++ b/api/server-server/events.yaml @@ -59,17 +59,35 @@ paths: type: array description: |- The full set of authorization events that make up the state - of the room, and their authorization events, recursively. + of the room, and their authorization events, recursively. Note that + events have a different version depending on the room version - + check the `room version specification`_ for precise event formats. items: - $ref: "definitions/pdu.yaml" - example: [{"$ref": "examples/pdu.json"}] + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ contained in the auth chain. The event format + varies depending on the room version - check the `room version specification`_ + for precise event formats. + properties: [] + example: + $ref: "examples/minimal_pdu.json" pdus: type: array description: |- - The fully resolved state of the room at the given event. + The fully resolved state of the room at the given event. Note that + events have a different version depending on the room version - + check the `room version specification`_ for precise event formats. items: - $ref: "definitions/pdu.yaml" - example: [{"$ref": "examples/pdu.json"}] + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ for the fully resolved state of the room. The event format + varies depending on the room version - check the `room version specification`_ + for precise event formats. + properties: [] + example: + $ref: "examples/minimal_pdu.json" required: ['auth_chain', 'pdus'] "/state_ids/{roomId}": get: diff --git a/api/server-server/examples/minimal_pdu.json b/api/server-server/examples/minimal_pdu.json new file mode 100644 index 000000000..f8b8efc35 --- /dev/null +++ b/api/server-server/examples/minimal_pdu.json @@ -0,0 +1,7 @@ +{ + "type": "m.room.minimal_pdu", + "room_id": "!somewhere:example.org", + "content": { + "see_room_version_spec": "The event format changes depending on the room version." + } +} diff --git a/api/server-server/examples/transaction.json b/api/server-server/examples/transaction.json index 6d77f79ec..bbc661d99 100644 --- a/api/server-server/examples/transaction.json +++ b/api/server-server/examples/transaction.json @@ -2,6 +2,6 @@ "origin": "matrix.org", "origin_server_ts": 1234567890, "pdus": [{ - "see_room_version_spec": "The event format changes depending on the room version." + "$ref": "minimal_pdu.json" }] } diff --git a/api/server-server/invites-v1.yaml b/api/server-server/invites-v1.yaml index 44db1f8da..8daaabab7 100644 --- a/api/server-server/invites-v1.yaml +++ b/api/server-server/invites-v1.yaml @@ -38,6 +38,11 @@ paths: Servers should prefer to use the v2 API for invites instead of the v1 API. Servers which receive a v1 invite request must assume that the room version is either ``"1"`` or ``"2"``. + + Note that events have a different version depending on the room version - check the + `room version specification`_ for precise event formats. **The request and response + bodies here describe the common event fields in more detail and may be missing other + required fields for a PDU.** operationId: sendInviteV1 security: - signedRequest: [] @@ -107,7 +112,7 @@ paths: } ] example: { - "$ref": "examples/pdu.json", + "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@joe:elsewhere.com", "unsigned": { @@ -143,7 +148,8 @@ paths: 200: description: |- The event with the invited server's signature added. All other fields of the events - should remain untouched. + should remain untouched. Note that events have a different version depending on the + room version - check the `room version specification`_ for precise event formats. schema: type: array minItems: 2 @@ -164,7 +170,7 @@ paths: 200, { "event": { - "$ref": "examples/pdu.json", + "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", "unsigned": { diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml index f86951958..91e95d7f4 100644 --- a/api/server-server/invites-v2.yaml +++ b/api/server-server/invites-v2.yaml @@ -42,6 +42,11 @@ paths: This endpoint is preferred over the v1 API as it is more useful for servers. Senders which receive a 400 or 404 response to this endpoint should retry using the v1 API as the server may be older, if the room version is "1" or "2". + + Note that events have a different version depending on the room version - check the + `room version specification`_ for precise event formats. **The request and response + bodies here describe the common event fields in more detail and may be missing other + required fields for a PDU.** operationId: sendInviteV2 security: - signedRequest: [] @@ -111,7 +116,7 @@ paths: example: { "room_version": "2", "event": { - "$ref": "examples/pdu.json", + "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@joe:elsewhere.com", "content": { @@ -146,7 +151,8 @@ paths: 200: description: |- The event with the invited server's signature added. All other fields of the events - should remain untouched. + should remain untouched. Note that events have a different version depending on the + room version - check the `room version specification`_ for precise event formats. schema: type: object description: An object containing the signed invite event. @@ -158,7 +164,7 @@ paths: examples: application/json: { "event": { - "$ref": "examples/pdu.json", + "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", "unsigned": { diff --git a/api/server-server/joins.yaml b/api/server-server/joins.yaml index 3c0ec48f0..83f8d1c5a 100644 --- a/api/server-server/joins.yaml +++ b/api/server-server/joins.yaml @@ -61,7 +61,11 @@ paths: responses: 200: description: |- - A template to be used for the rest of the `Joining Rooms`_ handshake. + A template to be used for the rest of the `Joining Rooms`_ handshake. Note that + events have a different version depending on the room version - check the + `room version specification`_ for precise event formats. **The response body + here describes the common event fields in more detail and may be missing other + required fields for a PDU.** schema: type: object properties: @@ -72,96 +76,61 @@ paths: the room version is assumed to be either "1" or "2". example: "2" event: - allOf: - - $ref: "definitions/unsigned_pdu.yaml" - - description: |- - An unsigned template event. + description: |- + An unsigned template event. Note that events have a different version + depending on the room version - check the `room version specification`_ + for precise event formats. + type: object + properties: + sender: + type: string + description: The user ID of the joining member. + example: "@someone:example.org" + origin: + type: string + description: The name of the resident homeserver. + example: "matrix.org" + origin_server_ts: + type: integer + format: int64 + description: A timestamp added by the resident homeserver. + example: 1234567890 + type: + type: string + description: The value ``m.room.member``. + example: "m.room.member" + state_key: + type: string + description: The user ID of the joining member. + example: "@someone:example.org" + content: type: object + title: Membership Event Content + description: The content of the event. + example: {"membership": "join"} properties: - # Note: we override a bunch of parameters to change their descriptions - sender: - type: string - description: The user ID of the joining member. - example: "@someone:example.org" - origin: - type: string - description: The name of the resident homeserver. - example: "matrix.org" - origin_server_ts: - type: integer - format: int64 - description: A timestamp added by the resident homeserver. - example: 1234567890 - type: - type: string - description: The value ``m.room.member``. - example: "m.room.member" - state_key: - type: string - description: The user ID of the joining member. - example: "@someone:example.org" - content: - type: object - title: Membership Event Content - description: The content of the event. - example: {"membership": "join"} - properties: - membership: - type: string - description: The value ``join``. - example: "join" - required: ['membership'] - depth: - type: integer - description: This field must be present but is ignored; it may be 0. - example: 12 - auth_events: - type: array - description: |- - An event reference list containing the authorization events that would - allow the member to join the room. This should normally be the - ``m.room.create``, ``m.room.power_levels``, and ``m.room.join_rules`` - events. - items: - type: array - maxItems: 2 - minItems: 2 - items: - - type: string - title: Event ID - example: "$abc123:matrix.org" - - type: object - title: Event Hash - example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" - } - properties: - sha256: - type: string - description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong - required: ['sha256'] - redacts: + membership: type: string - description: Not used. - required: - # Every other field is already flagged as required by the $ref - - state_key + description: The value ``join``. + example: "join" + required: ['membership'] + required: + - state_key + - origin + - origin_server_ts + - type + - state_key + - content examples: application/json: { "room_version": "2", "event": { - "$ref": "examples/unsigned_pdu.json", + "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", "content": { "membership": "join" - }, - "auth_events": [ - ["$room_cre4te_3vent:matrix.org", {"sha256": "abase64encodedsha256hashshouldbe43byteslong"}], - ["$room_j0in_rul3s_3vent:matrix.org", {"sha256": "abase64encodedsha256hashshouldbe43byteslong"}], - ["$room_p0wer_l3vels_3vent:matrix.org", {"sha256": "abase64encodedsha256hashshouldbe43byteslong"}] - ] + } } } 400: @@ -193,7 +162,12 @@ paths: summary: Submit a signed join event to a resident server description: |- Submits a signed join event to the resident server for it - to accept it into the room's graph. + to accept it into the room's graph. Note that events have + a different version depending on the room version - check + the `room version specification`_ for precise event formats. + **The request and response body here describes the common + event fields in more detail and may be missing other required + fields for a PDU.** operationId: sendJoin security: - signedRequest: [] @@ -215,79 +189,50 @@ paths: type: object required: true schema: - allOf: - - $ref: "definitions/pdu.yaml" - - type: object + type: object + properties: + sender: + type: string + description: The user ID of the joining member. + example: "@someone:example.org" + origin: + type: string + description: The name of the joining homeserver. + example: "matrix.org" + origin_server_ts: + type: integer + format: int64 + description: A timestamp added by the joining homeserver. + example: 1234567890 + type: + type: string + description: The value ``m.room.member``. + example: "m.room.member" + state_key: + type: string + description: The user ID of the joining member. + example: "@someone:example.org" + content: + type: object + title: Membership Event Content + description: The content of the event. + example: {"membership": "join"} properties: - # Note: we override a bunch of parameters to change their descriptions - sender: + membership: type: string - description: The user ID of the joining member. - example: "@someone:example.org" - origin: - type: string - description: The name of the joining homeserver. - example: "matrix.org" - origin_server_ts: - type: integer - format: int64 - description: A timestamp added by the joining homeserver. - example: 1234567890 - type: - type: string - description: The value ``m.room.member``. - example: "m.room.member" - state_key: - type: string - description: The user ID of the joining member. - example: "@someone:example.org" - content: - type: object - title: Membership Event Content - description: The content of the event. - example: {"membership": "join"} - properties: - membership: - type: string - description: The value ``join``. - example: "join" - required: ['membership'] - depth: - type: integer - description: This field must be present but is ignored; it may be 0. - example: 12 - auth_events: - type: array - description: |- - An event reference list containing the authorization events that would - allow the member to join the room. - items: - type: array - maxItems: 2 - minItems: 2 - items: - - type: string - title: Event ID - example: "$abc123:matrix.org" - - type: object - title: Event Hash - example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" - } - properties: - sha256: - type: string - description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong - required: ['sha256'] - redacts: - type: string - description: Not used. - required: - # Every other field is already flagged as required by the $ref - - state_key + description: The value ``join``. + example: "join" + required: ['membership'] + required: + - state_key + - sender + - origin + - origin_server_ts + - type + - state_key + - content example: { - "$ref": "examples/pdu.json", + "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", "content": { @@ -315,11 +260,20 @@ paths: description: The resident server's DNS name. auth_chain: type: array - description: The auth chain. + description: |- + The auth chain. Note that events have a different version depending on + the room version - check the `room version specification`_ for precise + event formats. items: type: object - schema: - $ref: "definitions/pdu.yaml" + title: PDU + description: |- + The `PDUs <#pdus>`_ contained in the auth chain. The event format + varies depending on the room version - check the `room version specification`_ + for precise event formats. + properties: [] + example: + $ref: "examples/minimal_pdu.json" state: type: array description: The room state. @@ -333,7 +287,7 @@ paths: 200, { "origin": "matrix.org", - "auth_chain": [{"$ref": "examples/pdu.json"}], - "state": [{"$ref": "examples/pdu.json"}] + "auth_chain": [{"$ref": "examples/minimal_pdu.json"}], + "state": [{"$ref": "examples/minimal_pdu.json"}] } ] diff --git a/api/server-server/leaving.yaml b/api/server-server/leaving.yaml index 68ada9d70..1a112ad38 100644 --- a/api/server-server/leaving.yaml +++ b/api/server-server/leaving.yaml @@ -52,7 +52,11 @@ paths: responses: 200: description: |- - A template to be used to call ``/send_leave``. + A template to be used to call ``/send_leave``. Note that + events have a different version depending on the room version - check the + `room version specification`_ for precise event formats. **The response body + here describes the common event fields in more detail and may be missing other + required fields for a PDU.** schema: schema: type: object @@ -64,92 +68,62 @@ paths: the room version is assumed to be either "1" or "2". example: "2" event: - allOf: - - $ref: "definitions/unsigned_pdu.yaml" - - description: |- - An unsigned template event. + description: |- + An unsigned template event. Note that events have a different version + depending on the room version - check the `room version specification`_ + for precise event formats. + type: object + properties: + sender: + type: string + description: The user ID of the leaving member. + example: "@someone:example.org" + origin: + type: string + description: The name of the resident homeserver. + example: "matrix.org" + origin_server_ts: + type: integer + format: int64 + description: A timestamp added by the resident homeserver. + example: 1234567890 + type: + type: string + description: The value ``m.room.member``. + example: "m.room.member" + state_key: + type: string + description: The user ID of the leaving member. + example: "@someone:example.org" + content: type: object + title: Membership Event Content + description: The content of the event. + example: {"membership": "leave"} properties: - # Note: we override a bunch of parameters to change their descriptions - sender: - type: string - description: The user ID of the leaving member. - example: "@someone:example.org" - origin: - type: string - description: The name of the resident homeserver. - example: "matrix.org" - origin_server_ts: - type: integer - format: int64 - description: A timestamp added by the resident homeserver. - example: 1234567890 - type: - type: string - description: The value ``m.room.member``. - example: "m.room.member" - state_key: - type: string - description: The user ID of the leaving member. - example: "@someone:example.org" - content: - type: object - title: Membership Event Content - description: The content of the event. - example: {"membership": "leave"} - properties: - membership: - type: string - description: The value ``leave``. - example: "leave" - required: ['membership'] - auth_events: - type: array - description: |- - An event reference list containing the authorization events that would - allow the member to leave the room. This should normally be the - ``m.room.create``, ``m.room.power_levels``, and ``m.room.join_rules`` - events. - items: - type: array - maxItems: 2 - minItems: 2 - items: - - type: string - title: Event ID - example: "$abc123:matrix.org" - - type: object - title: Event Hash - example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" - } - properties: - sha256: - type: string - description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong - required: ['sha256'] - redacts: + membership: type: string - description: Not used. - required: - # Every other field is already flagged as required by the $ref - - state_key + description: The value ``leave``. + example: "leave" + required: ['membership'] + required: + - state_key + - sender + - origin + - origin_server_ts + - type + - state_key + - content examples: application/json: { "room_version": "2", "event": { - "$ref": "examples/unsigned_pdu.json", + "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", "content": { "membership": "leave" - }, - "auth_events": [ - ["$room_cre4te_3vent:matrix.org", {"sha256": "abase64encodedsha256hashshouldbe43byteslong"}], - ["$room_j0in_rul3s_3vent:matrix.org", {"sha256": "abase64encodedsha256hashshouldbe43byteslong"}], - ["$room_p0wer_l3vels_3vent:matrix.org", {"sha256": "abase64encodedsha256hashshouldbe43byteslong"}] - ] + } } } 403: @@ -167,7 +141,12 @@ paths: summary: Submit a signed leave event to a resident server description: |- Submits a signed leave event to the resident server for it - to accept it into the room's graph. + to accept it into the room's graph. Note that events have + a different version depending on the room version - check + the `room version specification`_ for precise event formats. + **The request and response body here describes the common + event fields in more detail and may be missing other required + fields for a PDU.** operationId: sendLeave security: - signedRequest: [] @@ -189,79 +168,54 @@ paths: type: object required: true schema: - allOf: - - $ref: "definitions/pdu.yaml" - - type: object + type: object + properties: + sender: + type: string + description: The user ID of the leaving member. + example: "@someone:example.org" + origin: + type: string + description: The name of the leaving homeserver. + example: "matrix.org" + origin_server_ts: + type: integer + format: int64 + description: A timestamp added by the leaving homeserver. + example: 1234567890 + type: + type: string + description: The value ``m.room.member``. + example: "m.room.member" + state_key: + type: string + description: The user ID of the leaving member. + example: "@someone:example.org" + content: + type: object + title: Membership Event Content + description: The content of the event. + example: {"membership": "leave"} properties: - # Note: we override a bunch of parameters to change their descriptions - sender: - type: string - description: The user ID of the leaving member. - example: "@someone:example.org" - origin: - type: string - description: The name of the leaving homeserver. - example: "matrix.org" - origin_server_ts: - type: integer - format: int64 - description: A timestamp added by the leaving homeserver. - example: 1234567890 - type: - type: string - description: The value ``m.room.member``. - example: "m.room.member" - state_key: + membership: type: string - description: The user ID of the leaving member. - example: "@someone:example.org" - content: - type: object - title: Membership Event Content - description: The content of the event. - example: {"membership": "leave"} - properties: - membership: - type: string - description: The value ``leave``. - example: "leave" - required: ['membership'] - depth: - type: integer - description: This field must be present but is ignored; it may be 0. - example: 12 - auth_events: - type: array - description: |- - An event reference list containing the authorization events that would - allow the member to leave the room. - items: - type: array - maxItems: 2 - minItems: 2 - items: - - type: string - title: Event ID - example: "$abc123:matrix.org" - - type: object - title: Event Hash - example: { - "sha256": "abase64encodedsha256hashshouldbe43byteslong" - } - properties: - sha256: - type: string - description: The event hash. - example: abase64encodedsha256hashshouldbe43byteslong - required: ['sha256'] - redacts: - type: string - description: Not used. - required: - # Every other field is already flagged as required by the $ref - - state_key + description: The value ``leave``. + example: "leave" + required: ['membership'] + depth: + type: integer + description: This field must be present but is ignored; it may be 0. + example: 12 + required: + - state_key + - sender + - origin + - origin_server_ts + - type + - state_key + - content example: { - "$ref": "examples/pdu.json", + "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", "content": { diff --git a/api/server-server/transactions.yaml b/api/server-server/transactions.yaml index 355be2c67..38073e408 100644 --- a/api/server-server/transactions.yaml +++ b/api/server-server/transactions.yaml @@ -37,6 +37,9 @@ paths: The sending server must wait and retry for a 200 OK response before sending a transaction with a different ``txnId`` to the receiving server. + + Note that events have a different version depending on the room version - check + the `room version specification`_ for precise event formats. operationId: sendTransaction security: - signedRequest: [] From 5d8fa65e6e5c06573f744e8656a9a7db9ed600aa Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 10:11:39 -0700 Subject: [PATCH 094/170] De-duplicate state keys --- api/server-server/joins.yaml | 1 - api/server-server/leaving.yaml | 2 -- 2 files changed, 3 deletions(-) diff --git a/api/server-server/joins.yaml b/api/server-server/joins.yaml index 83f8d1c5a..c35392974 100644 --- a/api/server-server/joins.yaml +++ b/api/server-server/joins.yaml @@ -119,7 +119,6 @@ paths: - origin - origin_server_ts - type - - state_key - content examples: application/json: { diff --git a/api/server-server/leaving.yaml b/api/server-server/leaving.yaml index 1a112ad38..556e0800c 100644 --- a/api/server-server/leaving.yaml +++ b/api/server-server/leaving.yaml @@ -112,7 +112,6 @@ paths: - origin - origin_server_ts - type - - state_key - content examples: application/json: { @@ -212,7 +211,6 @@ paths: - origin - origin_server_ts - type - - state_key - content example: { "$ref": "examples/minimal_pdu.json", From 890fb1a01928dfa77a6ac86714c823e33460ae0a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 10:15:32 -0700 Subject: [PATCH 095/170] Fix examples for new schema Fix missed example in make_join Fix state array in response of send_join Try removing examples from send_join? Try printing more information about the error Copy/paste known working examples Try schema definitions in the response? --- api/check_examples.py | 6 ++++++ api/server-server/invites-v1.yaml | 6 ++++++ api/server-server/invites-v2.yaml | 6 ++++++ api/server-server/joins.yaml | 29 ++++++++++++++++++++++------- api/server-server/leaving.yaml | 6 ++++++ 5 files changed, 46 insertions(+), 7 deletions(-) diff --git a/api/check_examples.py b/api/check_examples.py index 0fb275b17..94f3495e3 100755 --- a/api/check_examples.py +++ b/api/check_examples.py @@ -75,6 +75,12 @@ def check_response(filepath, request, code, response): filepath, request, code )) check_schema(filepath, example, schema) + except jsonschema.SchemaError as error: + for suberror in sorted(error.context, key=lambda e: e.schema_path): + print(list(suberror.schema_path), suberror.message, sep=", ") + raise ValueError("Error validating JSON schema for %r %r" % ( + request, code + ), e) except Exception as e: raise ValueError("Error validating JSON schema for %r %r" % ( request, code diff --git a/api/server-server/invites-v1.yaml b/api/server-server/invites-v1.yaml index 8daaabab7..f397fc91b 100644 --- a/api/server-server/invites-v1.yaml +++ b/api/server-server/invites-v1.yaml @@ -115,6 +115,9 @@ paths: "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@joe:elsewhere.com", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", "unsigned": { "invite_room_state": [ { @@ -173,6 +176,9 @@ paths: "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", "unsigned": { "invite_room_state": [ { diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml index 91e95d7f4..d57a7ba29 100644 --- a/api/server-server/invites-v2.yaml +++ b/api/server-server/invites-v2.yaml @@ -119,6 +119,9 @@ paths: "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@joe:elsewhere.com", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", "content": { "membership": "invite" }, @@ -167,6 +170,9 @@ paths: "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", "unsigned": { "invite_room_state": [ { diff --git a/api/server-server/joins.yaml b/api/server-server/joins.yaml index c35392974..fca273b57 100644 --- a/api/server-server/joins.yaml +++ b/api/server-server/joins.yaml @@ -127,6 +127,9 @@ paths: "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", "content": { "membership": "join" } @@ -228,12 +231,14 @@ paths: - origin - origin_server_ts - type - - state_key - content example: { "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", "content": { "membership": "join" } @@ -267,19 +272,29 @@ paths: type: object title: PDU description: |- - The `PDUs <#pdus>`_ contained in the auth chain. The event format - varies depending on the room version - check the `room version specification`_ - for precise event formats. - properties: [] + The `PDUs <#pdus>`_ that make up the auth chain. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. + schema: + type: object + properties: [] example: $ref: "examples/minimal_pdu.json" state: type: array - description: The room state. + description: |- + The room state. The event format varies depending on the room version - + check the `room version specification`_ for precise event formats. items: type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ for the fully resolved state of the room. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. schema: - $ref: "definitions/pdu.yaml" + type: object + properties: [] + example: + $ref: "examples/minimal_pdu.json" required: ["auth_chain", "state", "origin"] examples: application/json: [ diff --git a/api/server-server/leaving.yaml b/api/server-server/leaving.yaml index 556e0800c..6c0d49598 100644 --- a/api/server-server/leaving.yaml +++ b/api/server-server/leaving.yaml @@ -120,6 +120,9 @@ paths: "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", "content": { "membership": "leave" } @@ -216,6 +219,9 @@ paths: "$ref": "examples/minimal_pdu.json", "type": "m.room.member", "state_key": "@someone:example.org", + "origin": "example.org", + "origin_server_ts": 1549041175876, + "sender": "@someone:example.org", "content": { "membership": "leave" } From 82bed06d3f433dc9655246931e2553efb546c5e9 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 13:19:24 -0700 Subject: [PATCH 096/170] The event *format* changes, not the version --- api/server-server/definitions/invite_event.yaml | 2 +- api/server-server/definitions/transaction.yaml | 2 +- api/server-server/event_auth.yaml | 6 +++--- api/server-server/events.yaml | 4 ++-- api/server-server/invites-v1.yaml | 4 ++-- api/server-server/invites-v2.yaml | 4 ++-- api/server-server/joins.yaml | 8 ++++---- api/server-server/leaving.yaml | 6 +++--- api/server-server/transactions.yaml | 2 +- 9 files changed, 19 insertions(+), 19 deletions(-) diff --git a/api/server-server/definitions/invite_event.yaml b/api/server-server/definitions/invite_event.yaml index 1f8b29d05..674ef2c94 100644 --- a/api/server-server/definitions/invite_event.yaml +++ b/api/server-server/definitions/invite_event.yaml @@ -14,7 +14,7 @@ type: object title: Invite Event description: |- - An invite event. Note that events have a different version depending on the + An invite event. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. allOf: - type: object diff --git a/api/server-server/definitions/transaction.yaml b/api/server-server/definitions/transaction.yaml index a77f6f8a3..08f3738ab 100644 --- a/api/server-server/definitions/transaction.yaml +++ b/api/server-server/definitions/transaction.yaml @@ -33,7 +33,7 @@ properties: type: array description: |- List of persistent updates to rooms. Must not include more than 50 PDUs. Note that - events have a different version depending on the room version - check the + events have a different format depending on the room version - check the `room version specification`_ for precise event formats. items: type: object diff --git a/api/server-server/event_auth.yaml b/api/server-server/event_auth.yaml index 262ea2827..0db0d4015 100644 --- a/api/server-server/event_auth.yaml +++ b/api/server-server/event_auth.yaml @@ -59,7 +59,7 @@ paths: description: |- The full set of authorization events that make up the state of the room, and their authorization events, recursively. Note that - events have a different version depending on the room version - + events have a different format depending on the room version - check the `room version specification`_ for precise event formats. items: type: object @@ -109,7 +109,7 @@ paths: type: array description: |- The auth chain (the "remote auth"). Note that events have a different - version depending on the room version - check the `room version specification`_ + format depending on the room version - check the `room version specification`_ for precise event formats. items: type: object @@ -162,7 +162,7 @@ paths: description: |- The auth chain the receiver has, and used to determine the auth chain differences (the "local auth"). Note that events have a different - version depending on the room version - check the `room version specification`_ + format depending on the room version - check the `room version specification`_ for precise event formats. items: type: object diff --git a/api/server-server/events.yaml b/api/server-server/events.yaml index e7cb52cde..1f1a802dd 100644 --- a/api/server-server/events.yaml +++ b/api/server-server/events.yaml @@ -60,7 +60,7 @@ paths: description: |- The full set of authorization events that make up the state of the room, and their authorization events, recursively. Note that - events have a different version depending on the room version - + events have a different format depending on the room version - check the `room version specification`_ for precise event formats. items: type: object @@ -76,7 +76,7 @@ paths: type: array description: |- The fully resolved state of the room at the given event. Note that - events have a different version depending on the room version - + events have a different format depending on the room version - check the `room version specification`_ for precise event formats. items: type: object diff --git a/api/server-server/invites-v1.yaml b/api/server-server/invites-v1.yaml index f397fc91b..2ad0f2201 100644 --- a/api/server-server/invites-v1.yaml +++ b/api/server-server/invites-v1.yaml @@ -39,7 +39,7 @@ paths: which receive a v1 invite request must assume that the room version is either ``"1"`` or ``"2"``. - Note that events have a different version depending on the room version - check the + Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. **The request and response bodies here describe the common event fields in more detail and may be missing other required fields for a PDU.** @@ -151,7 +151,7 @@ paths: 200: description: |- The event with the invited server's signature added. All other fields of the events - should remain untouched. Note that events have a different version depending on the + should remain untouched. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. schema: type: array diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml index d57a7ba29..c459a8485 100644 --- a/api/server-server/invites-v2.yaml +++ b/api/server-server/invites-v2.yaml @@ -43,7 +43,7 @@ paths: which receive a 400 or 404 response to this endpoint should retry using the v1 API as the server may be older, if the room version is "1" or "2". - Note that events have a different version depending on the room version - check the + Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. **The request and response bodies here describe the common event fields in more detail and may be missing other required fields for a PDU.** @@ -154,7 +154,7 @@ paths: 200: description: |- The event with the invited server's signature added. All other fields of the events - should remain untouched. Note that events have a different version depending on the + should remain untouched. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. schema: type: object diff --git a/api/server-server/joins.yaml b/api/server-server/joins.yaml index fca273b57..ad033001d 100644 --- a/api/server-server/joins.yaml +++ b/api/server-server/joins.yaml @@ -62,7 +62,7 @@ paths: 200: description: |- A template to be used for the rest of the `Joining Rooms`_ handshake. Note that - events have a different version depending on the room version - check the + events have a different format depending on the room version - check the `room version specification`_ for precise event formats. **The response body here describes the common event fields in more detail and may be missing other required fields for a PDU.** @@ -77,7 +77,7 @@ paths: example: "2" event: description: |- - An unsigned template event. Note that events have a different version + An unsigned template event. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. type: object @@ -165,7 +165,7 @@ paths: description: |- Submits a signed join event to the resident server for it to accept it into the room's graph. Note that events have - a different version depending on the room version - check + a different format depending on the room version - check the `room version specification`_ for precise event formats. **The request and response body here describes the common event fields in more detail and may be missing other required @@ -265,7 +265,7 @@ paths: auth_chain: type: array description: |- - The auth chain. Note that events have a different version depending on + The auth chain. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. items: diff --git a/api/server-server/leaving.yaml b/api/server-server/leaving.yaml index 6c0d49598..95ee68805 100644 --- a/api/server-server/leaving.yaml +++ b/api/server-server/leaving.yaml @@ -53,7 +53,7 @@ paths: 200: description: |- A template to be used to call ``/send_leave``. Note that - events have a different version depending on the room version - check the + events have a different format depending on the room version - check the `room version specification`_ for precise event formats. **The response body here describes the common event fields in more detail and may be missing other required fields for a PDU.** @@ -69,7 +69,7 @@ paths: example: "2" event: description: |- - An unsigned template event. Note that events have a different version + An unsigned template event. Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. type: object @@ -144,7 +144,7 @@ paths: description: |- Submits a signed leave event to the resident server for it to accept it into the room's graph. Note that events have - a different version depending on the room version - check + a different format depending on the room version - check the `room version specification`_ for precise event formats. **The request and response body here describes the common event fields in more detail and may be missing other required diff --git a/api/server-server/transactions.yaml b/api/server-server/transactions.yaml index 38073e408..9cc8be75d 100644 --- a/api/server-server/transactions.yaml +++ b/api/server-server/transactions.yaml @@ -38,7 +38,7 @@ paths: The sending server must wait and retry for a 200 OK response before sending a transaction with a different ``txnId`` to the receiving server. - Note that events have a different version depending on the room version - check + Note that events have a different format depending on the room version - check the `room version specification`_ for precise event formats. operationId: sendTransaction security: From 985d02d95e9c265a446e41c2c5765349817538ba Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 13:35:38 -0700 Subject: [PATCH 097/170] Fix titles in schemas --- api/server-server/joins.yaml | 1 + api/server-server/leaving.yaml | 1 + api/server-server/third_party_invite.yaml | 1 + 3 files changed, 3 insertions(+) diff --git a/api/server-server/joins.yaml b/api/server-server/joins.yaml index ad033001d..38586adb1 100644 --- a/api/server-server/joins.yaml +++ b/api/server-server/joins.yaml @@ -81,6 +81,7 @@ paths: depending on the room version - check the `room version specification`_ for precise event formats. type: object + title: Event Template properties: sender: type: string diff --git a/api/server-server/leaving.yaml b/api/server-server/leaving.yaml index 95ee68805..0713da6f1 100644 --- a/api/server-server/leaving.yaml +++ b/api/server-server/leaving.yaml @@ -73,6 +73,7 @@ paths: depending on the room version - check the `room version specification`_ for precise event formats. type: object + title: Event Template properties: sender: type: string diff --git a/api/server-server/third_party_invite.yaml b/api/server-server/third_party_invite.yaml index 37c3a1890..a401ae1b6 100644 --- a/api/server-server/third_party_invite.yaml +++ b/api/server-server/third_party_invite.yaml @@ -85,6 +85,7 @@ paths: third_party_invite: type: object description: The third party invite + title: Third Party Invite properties: display_name: type: string From 1d0156ad787776966436c12785684c9211486e0a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 13:37:19 -0700 Subject: [PATCH 098/170] Fix more titles --- api/server-server/third_party_invite.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/api/server-server/third_party_invite.yaml b/api/server-server/third_party_invite.yaml index a401ae1b6..0b7aac5bb 100644 --- a/api/server-server/third_party_invite.yaml +++ b/api/server-server/third_party_invite.yaml @@ -98,6 +98,7 @@ paths: description: |- A block of content which has been signed, which servers can use to verify the event. + title: Invite Signatures properties: signatures: type: object From 222957157f5cadcaf095bc38f19dadd3e21d62ef Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 13:56:30 -0700 Subject: [PATCH 099/170] Add missing required flags --- api/server-server/joins.yaml | 1 + api/server-server/leaving.yaml | 1 + 2 files changed, 2 insertions(+) diff --git a/api/server-server/joins.yaml b/api/server-server/joins.yaml index 38586adb1..1b5f4632d 100644 --- a/api/server-server/joins.yaml +++ b/api/server-server/joins.yaml @@ -121,6 +121,7 @@ paths: - origin_server_ts - type - content + - sender examples: application/json: { "room_version": "2", diff --git a/api/server-server/leaving.yaml b/api/server-server/leaving.yaml index 0713da6f1..c088cb5df 100644 --- a/api/server-server/leaving.yaml +++ b/api/server-server/leaving.yaml @@ -215,6 +215,7 @@ paths: - origin - origin_server_ts - type + - depth - content example: { "$ref": "examples/minimal_pdu.json", From fe7f58223324d28559a460f60c1be8241456f318 Mon Sep 17 00:00:00 2001 From: Aaron Raimist Date: Fri, 1 Feb 2019 16:42:49 -0600 Subject: [PATCH 100/170] Fix several spelling mistakes Signed-off-by: Aaron Raimist --- specification/appendices/base64.rst | 2 +- specification/appendices/identifier_grammar.rst | 2 +- specification/client_server_api.rst | 6 +++--- specification/index.rst | 2 +- specification/modules/account_data.rst | 2 +- specification/modules/end_to_end_encryption.rst | 4 ++-- specification/modules/mentions.rst | 2 +- specification/modules/tags.rst | 2 +- specification/modules/third_party_invites.rst | 2 +- specification/push_gateway.rst | 2 +- specification/rooms/v1.rst | 2 +- specification/rooms/v2.rst | 2 +- specification/server_server_api.rst | 8 ++++---- 13 files changed, 19 insertions(+), 19 deletions(-) diff --git a/specification/appendices/base64.rst b/specification/appendices/base64.rst index d046e0fc1..a918c2f9c 100644 --- a/specification/appendices/base64.rst +++ b/specification/appendices/base64.rst @@ -52,6 +52,6 @@ Examples of strings encoded using unpadded Base64:: UNPADDED_BASE64("foobar") = "Zm9vYmFy" When decoding Base64, implementations SHOULD accept input with or without -padding characters whereever possible, to ensure maximum interoperability. +padding characters wherever possible, to ensure maximum interoperability. .. _`RFC 4648`: https://tools.ietf.org/html/rfc4648 diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index b46789886..a0cdf298f 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -203,7 +203,7 @@ a homeserver creating a user ID for a new user based on the username passed to Implementations are free to do this mapping however they choose. Since the user ID is opaque except to the implementation which created it, the only -requirement is that the implemention can perform the mapping +requirement is that the implementation can perform the mapping consistently. However, we suggest the following algorithm: 1. Encode character strings as UTF-8. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 8f4a9a1d3..c506af4e4 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -158,7 +158,7 @@ Other error codes the client might encounter are: Sent when the room alias given to the ``createRoom`` API is already in use. :``M_INVALID_ROOM_STATE``: - Sent when the intial state given to the ``createRoom`` API is invalid. + Sent when the initial state given to the ``createRoom`` API is invalid. :``M_THREEPID_IN_USE``: Sent when a threepid given to an API cannot be used because the same threepid is already in use. @@ -636,7 +636,7 @@ To use this authentication type, clients should submit an auth dict as follows: where the ``identifier`` property is a user identifier object, as described in `Identifier types`_. -For example, to authenticate using the user's Matrix ID, clients whould submit: +For example, to authenticate using the user's Matrix ID, clients would submit: .. code:: json @@ -928,7 +928,7 @@ Third-party ID :Type: ``m.id.thirdparty`` :Description: - The user is identified by a third-party identifer in canonicalised form. + The user is identified by a third-party identifier in canonicalised form. A client can identify a user using a 3pid associated with the user's account on the homeserver, where the 3pid was previously associated using the diff --git a/specification/index.rst b/specification/index.rst index 05b85ab7c..aa4471c4b 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -474,7 +474,7 @@ Room versions are divided into two distinct groups: stable and unstable. Stable room versions may be used by rooms safely. Unstable room versions are everything else which is either not listed in the specification or flagged as unstable for some other reason. Versions can switch between stable and unstable periodically -for a variety of reasons, including discovered security vulnerabilites and age. +for a variety of reasons, including discovered security vulnerabilities and age. Clients should not ask room administrators to upgrade their rooms if the room is running a stable version. Servers SHOULD use room version 1 as the default room diff --git a/specification/modules/account_data.rst b/specification/modules/account_data.rst index f0bf285f0..a67e503a6 100644 --- a/specification/modules/account_data.rst +++ b/specification/modules/account_data.rst @@ -27,7 +27,7 @@ The account_data may be either global or scoped to a particular rooms. Events ------ -The client recieves the account data as events in the ``account_data`` sections +The client receives the account data as events in the ``account_data`` sections of a ``/sync``. These events can also be received in a ``/events`` response or in the diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 72bfae357..a605875b4 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -450,7 +450,7 @@ previously-received ``request`` message with the same ``request_id`` and .. NOTE:: Key sharing can be a big attack vector, thus it must be done very carefully. - A reasonable stategy is for a user's client to only send keys requested by the + A reasonable strategy is for a user's client to only send keys requested by the verified devices of the same user. Key exports @@ -696,7 +696,7 @@ An event encrypted using Megolm has the following format: "sender_key": "", "device_id": "", "session_id": "", - "ciphertext": "" + "ciphertext": "" } } diff --git a/specification/modules/mentions.rst b/specification/modules/mentions.rst index 4501b7766..dc078f3b6 100644 --- a/specification/modules/mentions.rst +++ b/specification/modules/mentions.rst @@ -44,7 +44,7 @@ In addition to using the appropriate ``matrix.to URI`` for the mention, clients should use the following guidelines when making mentions in events to be sent: -* When mentioning users, use the user's potentially ambigious display name for +* When mentioning users, use the user's potentially ambiguous display name for the anchor's text. If the user does not have a display name, use the user's ID. diff --git a/specification/modules/tags.rst b/specification/modules/tags.rst index 739ead2c7..a4b0becf6 100644 --- a/specification/modules/tags.rst +++ b/specification/modules/tags.rst @@ -48,7 +48,7 @@ The tag namespace is defined as follows: * The namespace ``u.*`` is reserved for user-defined tags. The portion of the string after the ``u.`` is defined to be the display name of this tag. No other semantics should be inferred from tags in this namespace. -* A client or app willing to use special tags for advanced functionnality should namespace them similarly to state keys: ``tld.name.*`` +* A client or app willing to use special tags for advanced functionality should namespace them similarly to state keys: ``tld.name.*`` * Any tag in the ``tld.name.*`` form but not matching the namespace of the current client should be ignored * Any tag not matching the above rules should be interpreted as a user tag from the ``u.*`` namespace, as if the name had already had ``u.`` stripped from the start (ie. the name of the tag is used as the diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst index df71d2154..3e11d9295 100644 --- a/specification/modules/third_party_invites.rst +++ b/specification/modules/third_party_invites.rst @@ -229,7 +229,7 @@ verification must still be performed, so the attack surface here is minimized. Security considerations ----------------------- -There are a number of privary and trust implications to this module. +There are a number of privacy and trust implications to this module. It is important for user privacy that leaking the mapping between a matrix user ID and a third party identifier is hard. In particular, being able to look up diff --git a/specification/push_gateway.rst b/specification/push_gateway.rst index a77d43db0..cba38b6c0 100644 --- a/specification/push_gateway.rst +++ b/specification/push_gateway.rst @@ -84,7 +84,7 @@ This describes the format used by "HTTP" pushers to send notifications of events to Push Gateways. If the endpoint returns an HTTP error code, the homeserver SHOULD retry for a reasonable amount of time using exponential backoff. -When pushing notifications for events, the hoemserver is expected to include all of +When pushing notifications for events, the homeserver is expected to include all of the event-related fields in the ``/notify`` request. When the homeserver is performing a push where the ``format`` is ``"event_id_only"``, only the ``event_id``, ``room_id``, ``counts``, and ``devices`` are required to be populated. diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index e09420e47..d7939c61e 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -275,7 +275,7 @@ The rules are as follows: Some consequences of these rules: * Unless you are a member of the room, the only permitted operations (apart - from the intial create/join) are: joining a public room; accepting or + from the initial create/join) are: joining a public room; accepting or rejecting an invitation to a room. * To unban somebody, you must have power level greater than or equal to both diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index d39a7caa3..c95fc4c9b 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -116,7 +116,7 @@ Mainline ordering of condition 1 below. The *mainline ordering based on* :math:`P` of a set of events is the - ordering, from smallest to largest, using the following comparision relation + ordering, from smallest to largest, using the following comparison relation on events: for events :math:`x` and :math:`y`, :math:`x`` is not an IP literal, and ```` - is present, an IP address is disovered by looking up an AAAA or A + is present, an IP address is discovered by looking up an AAAA or A record for ````. The resulting IP address is used, alongside the ````. Requests must be made with a ``Host`` header of ``:``. The @@ -1125,7 +1125,7 @@ including content hashes. It is calculated as follows. 3. The event is converted into `Canonical JSON`_. -4. A sha256 hash is calculed on the resulting JSON object. +4. A sha256 hash is calculated on the resulting JSON object. Calculating the content hash for an event @@ -1170,7 +1170,7 @@ Example code # Keys under "unsigned" can be modified by other servers. # They are useful for conveying information like the age of an # event that will change in transit. - # Since they can be modifed we need to exclude them from the hash. + # Since they can be modified we need to exclude them from the hash. event_object.pop("unsigned", None) # Signatures will depend on the current value of the "hashes" key. @@ -1192,7 +1192,7 @@ Example code [[TODO(markjh): Since the ``hash`` object cannot be redacted a server shouldn't allow too many hashes to be listed, otherwise a server might embed - illict data within the ``hash`` object. + illicit data within the ``hash`` object. We might want to specify a maximum number of keys for the ``hash`` and we might want to specify the maximum output size of a hash]] From 9364787b9cedb641f9d268eb00bf2dac3098d0c8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 12:12:34 -0700 Subject: [PATCH 101/170] Remove stability warning --- specification/server_server_api.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 30479b750..0c794ea53 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -16,10 +16,6 @@ Federation API ============== -.. WARNING:: - This API is unstable and will change without warning or discussion while - we work towards a r0 release (scheduled for August 2018). - Matrix homeservers use the Federation APIs (also known as server-server APIs) to communicate with each other. Homeservers use these APIs to push messages to each other in real-time, to retrieve historic messages from each other, and to From 8ace64bab719deb190cfb43b1fe1f896ab02e690 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 12:36:35 -0700 Subject: [PATCH 102/170] r0.1.0 of the s2s specification --- changelogs/server_server.rst | 6 ++++++ specification/identity_service_api.rst | 2 +- specification/rooms/v1.rst | 4 ++-- specification/rooms/v2.rst | 2 +- specification/rooms/v3.rst | 4 ++-- specification/server_server_api.rst | 2 +- 6 files changed, 13 insertions(+), 7 deletions(-) diff --git a/changelogs/server_server.rst b/changelogs/server_server.rst index e69de29bb..5dabc4ac2 100644 --- a/changelogs/server_server.rst +++ b/changelogs/server_server.rst @@ -0,0 +1,6 @@ +r0.1.0 +====== + +This is the first release of the Server Server (Federation) specification. +It includes support for homeservers being able to interact with other +homeservers in a decentralized and standard way. diff --git a/specification/identity_service_api.rst b/specification/identity_service_api.rst index 1bcdeff0a..ecd2c99fd 100644 --- a/specification/identity_service_api.rst +++ b/specification/identity_service_api.rst @@ -279,4 +279,4 @@ this isn't possible. .. _`Unpadded Base64`: ../appendices.html#unpadded-base64 .. _`3PID Types`: ../appendices.html#pid-types .. _`Signing JSON`: ../appendices.html#signing-json -.. _`/3pid/onbind`: ../server_server/unstable.html#put-matrix-federation-v1-3pid-onbind +.. _`/3pid/onbind`: ../server_server/r0.1.0.html#put-matrix-federation-v1-3pid-onbind diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index d7939c61e..63bb1d7ac 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -290,5 +290,5 @@ Events in version 1 rooms have the following structure: {{definition_ss_pdu}} -.. _`auth events selection`: ../../server_server/unstable.html#auth-events-selection -.. _`Signing Events`: ../../server_server/unstable.html#signing-events +.. _`auth events selection`: ../../server_server/r0.1.0.html#auth-events-selection +.. _`Signing Events`: ../../server_server/r0.1.0.html#signing-events diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index c95fc4c9b..b73662ead 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -159,7 +159,7 @@ The *resolution* of a set of states is obtained as follows: resolved state. -.. _`authorization rules`: ../server_server/unstable.html#authorization-rules +.. _`authorization rules`: ../server_server/r0.1.0.html#authorization-rules Rejected events +++++++++++++++ diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index e5b0c44c7..368485197 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -117,5 +117,5 @@ The remaining rules are the same as `room version 1 `_: Includes all changes since the latest versioned release. +- `r0.1.0 `_ Server discovery ---------------- From 9631e4bcb1927bb69995346a02fb8183347c7527 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 18:00:26 -0700 Subject: [PATCH 103/170] Add a bit of text to ensure that the intro page isn't missed --- specification/index.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specification/index.rst b/specification/index.rst index aa4471c4b..db6ff99fd 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -40,10 +40,14 @@ The specification consists of the following parts: {{apis}} +Additionally, this introduction page is a valuable resource for all Matrix developers, +including the sections on `room versions <#room-versions>`_ and `overall architecture <#architecture>`_. + The `Appendices `_ contain supplemental information not specific to one of the above APIs. -The `Matrix Client-Server API Swagger Viewer `_ is useful for browsing the Client-Server API. +The `Matrix Client-Server API Swagger Viewer `_ +is useful for browsing the Client-Server API. Introduction to the Matrix APIs ------------------------------- From 1f5783b329f1d5b5e58206ff0a0f54d895868f10 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 1 Feb 2019 18:07:55 -0700 Subject: [PATCH 104/170] fix links and wording --- specification/index.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index db6ff99fd..d6176c93a 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -40,8 +40,9 @@ The specification consists of the following parts: {{apis}} -Additionally, this introduction page is a valuable resource for all Matrix developers, -including the sections on `room versions <#room-versions>`_ and `overall architecture <#architecture>`_. +Additionally, this introduction page contains the key baseline information required to +understand the specific APIs, including the sections on `room versions`_ +and `overall architecture <#architecture>`_. The `Appendices `_ contain supplemental information not specific to one of the above APIs. @@ -132,6 +133,8 @@ To propose a change to the Matrix Spec, see the explanations at `Proposals for Spec Changes to Matrix `_. +.. _`architecture`: + Architecture ------------ @@ -422,6 +425,7 @@ dedicated API. The API is symmetrical to managing Profile data. Would it really be overengineered to use the same API for both profile & private user data, but with different ACLs? +.. _`room versions`: Room Versions ------------- From e9bac1b011957c87e3791b916b70755fbafa401e Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 4 Feb 2019 13:49:29 -0700 Subject: [PATCH 105/170] Remove wrong references to TLS fingerprints Also fix some styling in the server discovery section - this didn't feel like it needed its own commit. --- specification/server_server_api.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 778a5819a..d00ca135c 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -157,14 +157,14 @@ The process overall is as follows: and a port of 8448, using a ``Host`` header of ````. The target server must present a valid certificate for ````. -4. If the `/.well-known` request resulted in an error response, a server +4. If the ``/.well-known`` request resulted in an error response, a server is found by resolving an SRV record for ``_matrix._tcp.``. This may result in a hostname (to be resolved using AAAA or A records) and port. Requests are made to the resolved IP address and port, using 8448 as a default port, with a ``Host`` header of ````. The target server must present a valid certificate for ````. -5. If the `/.well-known` request returned an error response, and the SRV +5. If the ``/.well-known`` request returned an error response, and the SRV record was not found, an IP address is resolved using AAAA and A records. Requests are made to the resolved IP address using port 8448 and a ``Host`` header containing the ````. The target server must present a @@ -220,12 +220,11 @@ server by querying other servers. Publishing Keys +++++++++++++++ -Homeservers publish the allowed TLS fingerprints and signing keys in a JSON +Homeservers publish their signing keys in a JSON object at ``/_matrix/key/v2/server/{key_id}``. The response contains a list of ``verify_keys`` that are valid for signing federation requests made by the homeserver and for signing events. It contains a list of ``old_verify_keys`` which -are only valid for signing events. Finally the response contains a list of TLS -certificate fingerprints to validate any connection made to the homeserver. +are only valid for signing events. {{keys_server_ss_http_api}} From 3dd0601a96be6f285753d72aa9fd212bb9ae3bc7 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 4 Feb 2019 13:52:20 -0700 Subject: [PATCH 106/170] Remove more TLS fingerprint talk --- api/server-server/keys_server.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/server-server/keys_server.yaml b/api/server-server/keys_server.yaml index 8734f2edd..69985ab7b 100644 --- a/api/server-server/keys_server.yaml +++ b/api/server-server/keys_server.yaml @@ -27,7 +27,7 @@ paths: get: summary: Get the homeserver's public key(s) description: |- - Gets the homeserver's published TLS fingerprints and signing keys. + Gets the homeserver's published signing keys. The homeserver may have any number of active keys and may have a number of old keys. @@ -49,7 +49,7 @@ paths: type: string description: |- **Deprecated**. Servers should not use this parameter and instead - opt to return all keys, not just the requested one. The key ID to + opt to return all keys, not just the requested one. The key ID to look up. required: false x-example: "ed25519:abc123" From 6067a4ad3cfcfde21f812484162704728d144a96 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 4 Feb 2019 13:56:59 -0700 Subject: [PATCH 107/170] Use the real identity server version in the APIs table --- scripts/templating/matrix_templates/units.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 721501ff5..f697dbdf0 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -774,7 +774,7 @@ class MatrixUnits(Units): "Privileged server plugins", ), TypeTableRow( "`Identity Service API `_", - "unstable", + is_ver, "Mapping of third party IDs to Matrix IDs", ), TypeTableRow( "`Push Gateway API `_", From f37a6d2ef554a4534ac6834e9c37fb5c1e008275 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 4 Feb 2019 14:00:34 -0700 Subject: [PATCH 108/170] Changelog --- changelogs/server_server/newsfragments/1844.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/server_server/newsfragments/1844.clarification diff --git a/changelogs/server_server/newsfragments/1844.clarification b/changelogs/server_server/newsfragments/1844.clarification new file mode 100644 index 000000000..f80eef518 --- /dev/null +++ b/changelogs/server_server/newsfragments/1844.clarification @@ -0,0 +1 @@ +Remove legacy references to TLS fingerprints. From 54ee861b5faf85bea35361cdbb89f28bae15711b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 4 Feb 2019 14:47:20 -0700 Subject: [PATCH 109/170] Fix changelog generation for non-default versions Currently if you generate a changelog for r0.1.1 of an API, you'd get "No significant changes" which is wrong. You should get a real changelog for the version. This is now handled by generating a "preferred" changelog which acts as the default for version variables in the RST. Using a specific version's changelog is still supported for the rare cases where that is desired. --- scripts/templating/matrix_templates/sections.py | 2 +- scripts/templating/matrix_templates/units.py | 14 +++++++++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/scripts/templating/matrix_templates/sections.py b/scripts/templating/matrix_templates/sections.py index af4976744..4451d21d9 100644 --- a/scripts/templating/matrix_templates/sections.py +++ b/scripts/templating/matrix_templates/sections.py @@ -41,7 +41,7 @@ class MatrixSections(Sections): version_var = "%s_%s" % (spec_var, version) logger.info("Rendering changelog for %s" % version_var) rendered[version_var] = changelog - if version == "unstable": + if version == "preferred": rendered[spec_var] = changelog return rendered diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 721501ff5..fd2618385 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -903,9 +903,17 @@ class MatrixUnits(Units): return schema - def load_changelogs(self): + def load_changelogs(self, substitutions): changelogs = {} + preferred_versions = { + "server_server": substitutions.get("%SERVER_RELEASE_LABEL%", "unstable"), + "client_server": substitutions.get("%CLIENT_RELEASE_LABEL%", "unstable"), + "identity_service": substitutions.get("%IDENTITY_RELEASE_LABEL%", "unstable"), + "push_gateway": substitutions.get("%PUSH_GATEWAY_RELEASE_LABEL%", "unstable"), + "application_service": substitutions.get("%APPSERVICE_RELEASE_LABEL%", "unstable"), + } + # Changelog generation is a bit complicated. We rely on towncrier to # generate the unstable/current changelog, but otherwise use the RST # edition to record historical changelogs. This is done by prepending @@ -1007,6 +1015,10 @@ class MatrixUnits(Units): title_part = keyword_versions[title_part] changelog = "".join(changelog_lines) changelogs[name][title_part.replace("^[a-zA-Z0-9]", "_").lower()] = changelog + preferred_changelog = changelogs[name]["unstable"] + if name in preferred_versions: + preferred_changelog = changelogs[name][preferred_versions[name]] + changelogs[name]["preferred"] = preferred_changelog return changelogs From 3581368f1f9066bf0d2679f8909a921c83d5486c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 4 Feb 2019 14:48:31 -0700 Subject: [PATCH 110/170] Add the "please use latest.html" warning to the s2s spec Now that we have a release, we should be warning people who try and use the unstable spec as fact. --- specification/server_server_api.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 778a5819a..0993cc144 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -16,6 +16,8 @@ Federation API ============== +{{unstable_warning_block_SERVER_RELEASE_LABEL}} + Matrix homeservers use the Federation APIs (also known as server-server APIs) to communicate with each other. Homeservers use these APIs to push messages to each other in real-time, to retrieve historic messages from each other, and to From 4ec3a43a8534a1f17d861e14271d5b51853f27dd Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 5 Feb 2019 12:58:11 +0000 Subject: [PATCH 111/170] Replace "3pid" with "3PID" --- specification/client_server_api.rst | 12 ++++++------ specification/identity_service_api.rst | 22 +++++++++++----------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index c506af4e4..1dce010ca 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -650,7 +650,7 @@ For example, to authenticate using the user's Matrix ID, clients would submit: "session": "" } -Alternatively reply using a 3pid bound to the user's account on the homeserver +Alternatively reply using a 3PID bound to the user's account on the homeserver using the |/account/3pid|_ API rather then giving the ``user`` explicitly as follows: @@ -667,7 +667,7 @@ follows: "session": "" } -In the case that the homeserver does not know about the supplied 3pid, the +In the case that the homeserver does not know about the supplied 3PID, the homeserver must respond with 403 Forbidden. Google ReCaptcha @@ -930,8 +930,8 @@ Third-party ID :Description: The user is identified by a third-party identifier in canonicalised form. -A client can identify a user using a 3pid associated with the user's account on -the homeserver, where the 3pid was previously associated using the +A client can identify a user using a 3PID associated with the user's account on +the homeserver, where the 3PID was previously associated using the |/account/3pid|_ API. See the `3PID Types`_ Appendix for a list of Third-party ID media. @@ -987,7 +987,7 @@ request as follows: "password": "" } -Alternatively, a client can use a 3pid bound to the user's account on the +Alternatively, a client can use a 3PID bound to the user's account on the homeserver using the |/account/3pid|_ API rather then giving the ``user`` explicitly, as follows: @@ -1002,7 +1002,7 @@ explicitly, as follows: "password": "" } -In the case that the homeserver does not know about the supplied 3pid, the +In the case that the homeserver does not know about the supplied 3PID, the homeserver must respond with ``403 Forbidden``. To log in using a login token, clients should submit a ``/login`` request as diff --git a/specification/identity_service_api.rst b/specification/identity_service_api.rst index ecd2c99fd..c6c0e5ec7 100644 --- a/specification/identity_service_api.rst +++ b/specification/identity_service_api.rst @@ -22,10 +22,10 @@ Identity Service API The Matrix client-server and server-server APIs are largely expressed in Matrix user identifiers. From time to time, it is useful to refer to users by other -("third-party") identifiers, or "3pid"s, e.g. their email address or phone +("third-party") identifiers, or "3PID"s, e.g. their email address or phone number. This Identity Service Specification describes how mappings between third-party identifiers and Matrix user identifiers can be established, -validated, and used. This description technically may apply to any 3pid, but in +validated, and used. This description technically may apply to any 3PID, but in practice has only been applied specifically to email addresses and phone numbers. .. contents:: Table of Contents @@ -150,9 +150,9 @@ Identity is a privacy-sensitive issue. While the identity server exists to provide identity information, access should be restricted to avoid leaking potentially sensitive data. In particular, being able to construct large-scale connections between identities should be avoided. To this end, in general APIs -should allow a 3pid to be mapped to a Matrix user identity, but not in the other -direction (i.e. one should not be able to get all 3pids associated with a Matrix -user ID, or get all 3pids associated with a 3pid). +should allow a 3PID to be mapped to a Matrix user identity, but not in the other +direction (i.e. one should not be able to get all 3PIDs associated with a Matrix +user ID, or get all 3PIDs associated with a 3PID). Web browser clients ------------------- @@ -204,10 +204,10 @@ Establishing associations The flow for creating an association is session-based. -Within a session, one may prove that one has ownership of a 3pid. +Within a session, one may prove that one has ownership of a 3PID. Once this has been established, the user can form an association between that -3pid and a Matrix user ID. Note that this association is only proved one way; -a user can associate *any* Matrix user ID with a validated 3pid, +3PID and a Matrix user ID. Note that this association is only proved one way; +a user can associate *any* Matrix user ID with a validated 3PID, i.e. I can claim that any email address I own is associated with @billg:microsoft.com. @@ -255,11 +255,11 @@ General Invitation storage ------------------ -An identity server can store pending invitations to a user's 3pid, which will -be retrieved and can be either notified on or look up when the 3pid is +An identity server can store pending invitations to a user's 3PID, which will +be retrieved and can be either notified on or look up when the 3PID is associated with a Matrix user ID. -At a later point, if the owner of that particular 3pid binds it with a Matrix user +At a later point, if the owner of that particular 3PID binds it with a Matrix user ID, the identity server will attempt to make an HTTP POST to the Matrix user's homeserver via the `/3pid/onbind`_ endpoint. The request MUST be signed with a long-term private key for the identity server. From c8428b1f8bf38ec88a40f9ee38fbaead3bad052d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 5 Feb 2019 22:39:36 -0700 Subject: [PATCH 112/170] Fix contradiction in wellknown discovery for servers Fixes https://github.com/matrix-org/matrix-doc/issues/1854 --- api/server-server/wellknown.yaml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/api/server-server/wellknown.yaml b/api/server-server/wellknown.yaml index 273da7eb3..756766465 100644 --- a/api/server-server/wellknown.yaml +++ b/api/server-server/wellknown.yaml @@ -35,9 +35,8 @@ paths: The delegated server information. The ``Content-Type`` for this response SHOULD be ``application/json``, however servers parsing the response should assume that the body is JSON regardless of type. Failures parsing the JSON or invalid data - provided in the resulting parsed JSON must result in server discovery failure (no - attempts should be made to continue finding an IP address/port number to connect - to). + provided in the resulting parsed JSON should not result in discovery failure - + consult the server discovery process for information on how to continue. examples: application/json: { "m.server": "delegated.example.com:1234" From 4b68b5c9392b44f72c7022686c766c408240d4dc Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 6 Feb 2019 12:20:56 -0700 Subject: [PATCH 113/170] Changelog --- changelogs/server_server/newsfragments/1855.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/server_server/newsfragments/1855.clarification diff --git a/changelogs/server_server/newsfragments/1855.clarification b/changelogs/server_server/newsfragments/1855.clarification new file mode 100644 index 000000000..e61e14c6d --- /dev/null +++ b/changelogs/server_server/newsfragments/1855.clarification @@ -0,0 +1 @@ +Clarify that servers should not fail to contact servers if ``/.well-known`` fails. From 8bd9ca4edd7a6f20da3a1d0e4a13854e70f03fa2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 6 Feb 2019 12:35:04 -0700 Subject: [PATCH 114/170] Prep for r0.1.1 of s2s --- changelogs/server_server.rst | 10 ++++++++++ .../server_server/newsfragments/1844.clarification | 1 - .../server_server/newsfragments/1855.clarification | 1 - specification/identity_service_api.rst | 2 +- specification/rooms/v1.rst | 4 ++-- specification/rooms/v2.rst | 2 +- specification/rooms/v3.rst | 4 ++-- specification/server_server_api.rst | 1 + 8 files changed, 17 insertions(+), 8 deletions(-) delete mode 100644 changelogs/server_server/newsfragments/1844.clarification delete mode 100644 changelogs/server_server/newsfragments/1855.clarification diff --git a/changelogs/server_server.rst b/changelogs/server_server.rst index 5dabc4ac2..a21da177a 100644 --- a/changelogs/server_server.rst +++ b/changelogs/server_server.rst @@ -1,3 +1,13 @@ +r0.1.1 +====== + +Spec Clarifications +------------------- + +- Remove legacy references to TLS fingerprints. (`#1844 `_) +- Clarify that servers should not fail to contact servers if ``/.well-known`` fails. (`#1855 `_) + + r0.1.0 ====== diff --git a/changelogs/server_server/newsfragments/1844.clarification b/changelogs/server_server/newsfragments/1844.clarification deleted file mode 100644 index f80eef518..000000000 --- a/changelogs/server_server/newsfragments/1844.clarification +++ /dev/null @@ -1 +0,0 @@ -Remove legacy references to TLS fingerprints. diff --git a/changelogs/server_server/newsfragments/1855.clarification b/changelogs/server_server/newsfragments/1855.clarification deleted file mode 100644 index e61e14c6d..000000000 --- a/changelogs/server_server/newsfragments/1855.clarification +++ /dev/null @@ -1 +0,0 @@ -Clarify that servers should not fail to contact servers if ``/.well-known`` fails. diff --git a/specification/identity_service_api.rst b/specification/identity_service_api.rst index ecd2c99fd..e00cee813 100644 --- a/specification/identity_service_api.rst +++ b/specification/identity_service_api.rst @@ -279,4 +279,4 @@ this isn't possible. .. _`Unpadded Base64`: ../appendices.html#unpadded-base64 .. _`3PID Types`: ../appendices.html#pid-types .. _`Signing JSON`: ../appendices.html#signing-json -.. _`/3pid/onbind`: ../server_server/r0.1.0.html#put-matrix-federation-v1-3pid-onbind +.. _`/3pid/onbind`: ../server_server/r0.1.1.html#put-matrix-federation-v1-3pid-onbind diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 63bb1d7ac..f50131943 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -290,5 +290,5 @@ Events in version 1 rooms have the following structure: {{definition_ss_pdu}} -.. _`auth events selection`: ../../server_server/r0.1.0.html#auth-events-selection -.. _`Signing Events`: ../../server_server/r0.1.0.html#signing-events +.. _`auth events selection`: ../../server_server/r0.1.1.html#auth-events-selection +.. _`Signing Events`: ../../server_server/r0.1.1.html#signing-events diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index b73662ead..7ad7668c9 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -159,7 +159,7 @@ The *resolution* of a set of states is obtained as follows: resolved state. -.. _`authorization rules`: ../server_server/r0.1.0.html#authorization-rules +.. _`authorization rules`: ../server_server/r0.1.1.html#authorization-rules Rejected events +++++++++++++++ diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index 368485197..0915d1962 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -117,5 +117,5 @@ The remaining rules are the same as `room version 1 `_: Includes all changes since the latest versioned release. +- `r0.1.1 `_ - `r0.1.0 `_ Server discovery From 76946a8a7c6ba2bffb26ef91993e7b993bba03a9 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 6 Feb 2019 22:02:21 -0700 Subject: [PATCH 115/170] Simplify changelog generation We don'e need `{{server_server_changelog_r0.1.0}}` (for example), so don't go through the hassle of generating it. Instead, we'll generate the changelog for the requested versions of each API and put that in place. In the future, we may wish to consider bringing back more complicated variables when/if we start generating released versions of the spec on the fly rather than manually. --- .../templating/matrix_templates/sections.py | 9 +- scripts/templating/matrix_templates/units.py | 181 +++++++++--------- 2 files changed, 92 insertions(+), 98 deletions(-) diff --git a/scripts/templating/matrix_templates/sections.py b/scripts/templating/matrix_templates/sections.py index 4451d21d9..5961aa246 100644 --- a/scripts/templating/matrix_templates/sections.py +++ b/scripts/templating/matrix_templates/sections.py @@ -34,15 +34,10 @@ class MatrixSections(Sections): def render_changelogs(self): rendered = {} changelogs = self.units.get("changelogs") - for spec, versioned in changelogs.items(): + for spec, changelog_text in changelogs.items(): spec_var = "%s_changelog" % spec logger.info("Rendering changelog for spec: %s" % spec) - for version, changelog in versioned.items(): - version_var = "%s_%s" % (spec_var, version) - logger.info("Rendering changelog for %s" % version_var) - rendered[version_var] = changelog - if version == "preferred": - rendered[spec_var] = changelog + rendered[spec_var] = changelog_text return rendered def _render_events(self, filterFn, sortFn): diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index fd2618385..0e3546cba 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -904,9 +904,20 @@ class MatrixUnits(Units): return schema def load_changelogs(self, substitutions): + """Loads the changelog unit for later rendering in a section. + + Args: + substitutions: dict of variable name to value. Provided by the gendoc script. + + Returns: + A dict of API name ("client_server", for example) to changelog. + """ changelogs = {} - preferred_versions = { + # The APIs and versions we'll prepare changelogs for. We use the substitutions + # to ensure that we pick up the right version for generated documentation. This + # defaults to "unstable" as a version for incremental generated documentation (CI). + prepare_versions = { "server_server": substitutions.get("%SERVER_RELEASE_LABEL%", "unstable"), "client_server": substitutions.get("%CLIENT_RELEASE_LABEL%", "unstable"), "identity_service": substitutions.get("%IDENTITY_RELEASE_LABEL%", "unstable"), @@ -914,112 +925,100 @@ class MatrixUnits(Units): "application_service": substitutions.get("%APPSERVICE_RELEASE_LABEL%", "unstable"), } - # Changelog generation is a bit complicated. We rely on towncrier to - # generate the unstable/current changelog, but otherwise use the RST - # edition to record historical changelogs. This is done by prepending - # the towncrier output to the RST in memory, then parsing the RST by - # hand. We parse the entire changelog to create a changelog for each - # version which may be of use in some APIs. - - # Map specific headers to specific keys that'll be used eventually - # in variables. Things not listed here will get lowercased and formatted - # such that characters not [a-z0-9] will be replaced with an underscore. - keyword_versions = { - "Unreleased Changes": "unstable" - } - - # Only generate changelogs for things that have an RST document - for f in os.listdir(CHANGELOG_DIR): - if not f.endswith(".rst"): - continue - path = os.path.join(CHANGELOG_DIR, f) - name = f[:-4] # take off ".rst" - - # If there's a directory with the same name, we'll try to generate - # a towncrier changelog and prepend it to the general changelog. - tc_path = os.path.join(CHANGELOG_DIR, name) - tc_lines = [] - if os.path.isdir(tc_path): - logger.info("Generating towncrier changelog for: %s" % name) - p = subprocess.Popen( - ['towncrier', '--version', 'Unreleased Changes', '--name', name, '--draft'], - cwd=tc_path, - stderr=subprocess.PIPE, - stdout=subprocess.PIPE, - ) - stdout, stderr = p.communicate() - if p.returncode != 0: - # Something broke - dump as much information as we can - logger.error("Towncrier exited with code %s" % p.returncode) - logger.error(stdout.decode('UTF-8')) - logger.error(stderr.decode('UTF-8')) - raw_log = "" - else: - raw_log = stdout.decode('UTF-8') - - # This is a bit of a hack, but it does mean that the log at least gets *something* - # to tell us it broke - if not raw_log.startswith("Unreleased Changes"): - logger.error("Towncrier appears to have failed to generate a changelog") - logger.error(raw_log) - raw_log = "" - tc_lines = raw_log.splitlines() + # Changelogs are split into two places: towncrier for the unstable changelog and + # the RST file for historical versions. If the prepare_versions dict above has + # a version other than "unstable" specified for an API, we'll use the historical + # changelog and otherwise generate the towncrier log in-memory. - title_part = None + for api_name, target_version in prepare_versions.items(): + logger.info("Generating changelog for %s at %s" % (api_name, target_version,)) changelog_lines = [] - with open(path, "r", encoding="utf-8") as f: - lines = f.readlines() + if target_version == 'unstable': + # generate towncrier log + tc_path = os.path.join(CHANGELOG_DIR, api_name) + if os.path.isdir(tc_path): + logger.info("Generating towncrier changelog for: %s" % api_name) + p = subprocess.Popen( + ['towncrier', '--version', 'unstable', '--name', api_name, '--draft'], + cwd=tc_path, + stderr=subprocess.PIPE, + stdout=subprocess.PIPE, + ) + stdout, stderr = p.communicate() + if p.returncode != 0: + # Something broke - dump as much information as we can + logger.error("Towncrier exited with code %s" % p.returncode) + logger.error(stdout.decode('UTF-8')) + logger.error(stderr.decode('UTF-8')) + raw_log = "" + else: + raw_log = stdout.decode('UTF-8') + + # This is a bit of a hack, but it does mean that the log at least gets *something* + # to tell us it broke + if not raw_log.startswith("unstable"): + logger.error("Towncrier appears to have failed to generate a changelog") + logger.error(raw_log) + raw_log = "" + changelog_lines = raw_log.splitlines() + else: + # read in the existing RST changelog + logger.info("Reading changelog RST for %s" % api_name) + rst_path = os.path.join(CHANGELOG_DIR, "%s.rst" % api_name) + with open(rst_path, 'r', encoding="utf-8") as f: + changelog_lines = f.readlines() + + # Parse the changelog lines to find the header we're looking for and therefore + # the changelog body. prev_line = None - for line in (tc_lines + lines): + title_part = None + changelog_body_lines = [] + have_changelog = False + for line in changelog_lines: if prev_line is None: prev_line = line continue if not title_part: - # find the title underline (at least 3 =) + # Titles we care about are underlined with at least 3 equal signs if re.match("^[=]{3,}$", line.strip()): - title_part = prev_line + logger.info("Found header %s" % prev_line) + title_part = prev_line.strip() continue prev_line = line - else: # have title, get body (stop on next title or EOF) + else: + # we have a title, start parsing the body if re.match("^[=]{3,}$", line.strip()): - # we hit another title, so pop the last line of - # the changelog and record the changelog - new_title = changelog_lines.pop() - if name not in changelogs: - changelogs[name] = {} - if title_part in keyword_versions: - title_part = keyword_versions[title_part] - title_part = title_part.strip().replace("^[a-zA-Z0-9]", "_").lower() - changelog = "".join(changelog_lines) - changelogs[name][title_part] = changelog - - # reset for the next version - changelog_lines = [] - title_part = new_title.strip() + # we hit another title. prev_line will be the new section's header. + # do a check to see if the section we just read is the one we want - if + # it is, use that changelog and move on. If it isn't, keep reading. + if title_part == target_version: + changelogs[api_name] = "".join(changelog_body_lines) + have_changelog = True + break + # not the section we want - start the next section + title_part = changelog_body_lines.pop().strip() + changelog_body_lines = [] continue - # Don't generate subheadings (we'll keep the title though) if re.match("^[-]{3,}$", line.strip()): - continue - if line.strip().startswith(".. version: "): - # The changelog is directing us to use a different title - # for the changelog. - title_part = line.strip()[len(".. version: "):] + # the last line is a subheading - drop this line because it's the underline + # and that causes problems with rendering. We'll keep the header text though. continue if line.strip().startswith(".. "): - continue # skip comments - changelog_lines.append(" " + line + '\n') - if len(changelog_lines) > 0 and title_part is not None: - if name not in changelogs: - changelogs[name] = {} - if title_part in keyword_versions: - title_part = keyword_versions[title_part] - changelog = "".join(changelog_lines) - changelogs[name][title_part.replace("^[a-zA-Z0-9]", "_").lower()] = changelog - preferred_changelog = changelogs[name]["unstable"] - if name in preferred_versions: - preferred_changelog = changelogs[name][preferred_versions[name]] - changelogs[name]["preferred"] = preferred_changelog + # skip comments + continue + # if we made it this far, append the line to the changelog body. We indent it so + # that it renders correctly in the section. We also add newlines so that there's + # intentionally blank lines that make rst2html happy. + changelog_body_lines.append(" " + line + '\n') + # do some quick checks to see if the last read section is our changelog + if not have_changelog: + logger.info("No changelog - testing %s == %s" % (target_version, title_part,)) + if title_part == target_version and len(changelog_body_lines) > 0: + changelogs[api_name] = "".join(changelog_body_lines) + else: + raise ValueError("No changelog for %s at %s" % (api_name, target_version,)) + # return our `dict[api_name] => changelog` as the last step. return changelogs def load_unstable_warnings(self, substitutions): From 375104127cc25fc5cb3d46456e4b7d08623f490c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 6 Feb 2019 22:03:16 -0700 Subject: [PATCH 116/170] Fix spec release process to match new changelog stuff Also while we're here, make it accurate. Fixes https://github.com/matrix-org/matrix-doc/issues/1858 --- meta/releasing_a_spec.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/meta/releasing_a_spec.md b/meta/releasing_a_spec.md index 2d9cd34ec..1d7aaea7b 100644 --- a/meta/releasing_a_spec.md +++ b/meta/releasing_a_spec.md @@ -18,16 +18,18 @@ The remainder of the process is as follows: 1. Activate your Python 3 virtual environment. 1. Having checked out the new release branch, navigate your way over to `./changelogs`. 1. Follow the release instructions provided in the README.md located there. -1. Update the changelog section of the specification you're releasing to make a reference - to the new version. 1. Update any version/link references across all specifications. -1. Ensure the `targets.yml` file lists the version correctly. -1. Commit the changes and PR them to master. -1. Tag the release with the format `client_server/r0.4.0`. -1. Add the changes to the matrix-org/matrix.org repository (for historic tracking). +1. Generate the specification using `./scripts/gendoc.py -c r0.4.0`, specifying all the + API versions at the time of generation. +1. PR the changes to the matrix-org/matrix.org repository (for historic tracking). * This is done by making a PR to the `unstyled_docs/spec` folder for the version and specification you're releasing. * Don't forget to symlink the new release as `latest`. + * For the client-server API, don't forget to generate the swagger JSON by using + `./scripts/dump-swagger.py -c r0.4.0`. This will also need symlinking to `latest`. +1. Commit the changes and PR them to master. **Wait for review from the spec core team.** + * Link to your matrix-org/matrix.org so both can be reviewed at the same time. +1. Tag the release with the format `client_server/r0.4.0`. 1. Perform a release on GitHub to tag the release. 1. Yell from the mountaintop to the world about the new release. From cf11965a8ee6c9bbe2de6e107b9291ecf666b8aa Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 6 Feb 2019 22:15:13 -0700 Subject: [PATCH 117/170] Change notice about room v2's scope to represent reality Fixes https://github.com/matrix-org/matrix-doc/issues/1851 --- specification/rooms/v2.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index 7ad7668c9..6188966a1 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -27,9 +27,8 @@ Server implementation components details contained here, and can safely ignore their presence. -The algorithms defined here should only apply to version 2 rooms. Other algorithms -may be used by other room versions, and as such servers should be aware of which -version room they are dealing with prior to executing a given algorithm. +Room version 2 uses the base compoennts of `room version 1 `_, changing +only the state resolution algorithm. State resolution From 85578f984240b3eda83b074f0ee286175806e5a6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 6 Feb 2019 22:18:56 -0700 Subject: [PATCH 118/170] Fix spelling mistake: endponts -> endpoints Fixes https://github.com/matrix-org/matrix-doc/issues/1677 --- changelogs/client_server/newsfragments/1838.clarification | 1 + changelogs/client_server/newsfragments/1860.clarification | 1 + specification/client_server_api.rst | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/1838.clarification create mode 100644 changelogs/client_server/newsfragments/1860.clarification diff --git a/changelogs/client_server/newsfragments/1838.clarification b/changelogs/client_server/newsfragments/1838.clarification new file mode 100644 index 000000000..b0f052035 --- /dev/null +++ b/changelogs/client_server/newsfragments/1838.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. diff --git a/changelogs/client_server/newsfragments/1860.clarification b/changelogs/client_server/newsfragments/1860.clarification new file mode 100644 index 000000000..b0f052035 --- /dev/null +++ b/changelogs/client_server/newsfragments/1860.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index c506af4e4..b41c34607 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -73,7 +73,7 @@ MUST be encoded as UTF-8. Clients are authenticated using opaque ``access_token`` strings (see `Client Authentication`_ for details), passed as a query string parameter on all requests. -The names of the API endponts for the HTTP transport follow a convention of +The names of the API endpoints for the HTTP transport follow a convention of using underscores to separate words (for example ``/delete_devices``). The key names in JSON objects passed over the API also follow this convention. From 772ba8dc2af8c9236aa2947603ae270c78f99c9b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 6 Feb 2019 22:26:41 -0700 Subject: [PATCH 119/170] Add a table of contents to each room version spec Fixes https://github.com/matrix-org/matrix-doc/issues/1852 We get clickable headers for free by doing this. --- specification/rooms/v1.rst | 3 +++ specification/rooms/v2.rst | 3 +++ specification/rooms/v3.rst | 3 +++ 3 files changed, 9 insertions(+) diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index f50131943..1c7a56c4c 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -18,6 +18,9 @@ Room Version 1 This room version is the first ever version for rooms, and contains the building blocks for other room versions. +.. contents:: Table of Contents +.. sectnum:: + Server implementation components -------------------------------- diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index 7ad7668c9..b262f47f0 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -18,6 +18,9 @@ Room Version 2 This room version builds off of `version 1 `_ with an improved state resolution algorithm. +.. contents:: Table of Contents +.. sectnum:: + Server implementation components -------------------------------- diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index 0915d1962..863f1c3a3 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -24,6 +24,9 @@ This room version builds on `version 2 `_ with an improved event format where the contextual room of the request is using this room version. Rooms using other room versions should not be affected by these sweeping requirements. +.. contents:: Table of Contents +.. sectnum:: + Client considerations --------------------- From 0ed0fee26147af548d2684f25ed8aa06028fdc59 Mon Sep 17 00:00:00 2001 From: "J. Ryan Stinnett" Date: Thu, 7 Feb 2019 08:09:35 -0700 Subject: [PATCH 120/170] Update specification/rooms/v2.rst Co-Authored-By: turt2live --- specification/rooms/v2.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index 6188966a1..1ff2d3fcc 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -27,7 +27,7 @@ Server implementation components details contained here, and can safely ignore their presence. -Room version 2 uses the base compoennts of `room version 1 `_, changing +Room version 2 uses the base components of `room version 1 `_, changing only the state resolution algorithm. From 7eb8b5d7f3992c4e5bcf295e1e672eff0f1ce0c7 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 8 Feb 2019 14:36:12 +0000 Subject: [PATCH 121/170] Add proposal for invite error code --- .../1866-invite-unsupported-version-error-code.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 proposals/1866-invite-unsupported-version-error-code.md diff --git a/proposals/1866-invite-unsupported-version-error-code.md b/proposals/1866-invite-unsupported-version-error-code.md new file mode 100644 index 000000000..98098637f --- /dev/null +++ b/proposals/1866-invite-unsupported-version-error-code.md @@ -0,0 +1,14 @@ +# MSC 1866 - Unsupported Room Version Error Code for Invites + +It is currently unspecified what error code should be relayed to clients when +they attempt to invite a user on a remote server that does not support the room +version. + +The proposal is to reuse the `M_UNSUPPORTED_ROOM_VERSION` error code that is +currently returned by the create room API. + +Strictly, the error returned by the create room API would mean the local server +didn't support the room version, while for the invite API it would mean the +remote server didn't. However, there is sufficient overlap that it makes sense +to reuse the same error code and rely on the context to differentiate the two +cases. From 6bbf22cd04733c1a0fc04ef1f05aab708068aed7 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 10 Feb 2019 17:07:42 -0700 Subject: [PATCH 122/170] Add routes for retrieving account data Original proposal: https://github.com/matrix-org/matrix-doc/issues/1339 This contains no known differences to what was ultimately decided upon and implemented. --- api/client-server/account-data.yaml | 87 +++++++++++++++++-- .../client_server/newsfragments/1873.new | 1 + 2 files changed, 83 insertions(+), 5 deletions(-) create mode 100644 changelogs/client_server/newsfragments/1873.new diff --git a/api/client-server/account-data.yaml b/api/client-server/account-data.yaml index 76b2b1565..ae845b251 100644 --- a/api/client-server/account-data.yaml +++ b/api/client-server/account-data.yaml @@ -43,8 +43,8 @@ paths: name: userId required: true description: |- - The id of the user to set account_data for. The access token must be - authorized to make requests for this user id. + The ID of the user to set account_data for. The access token must be + authorized to make requests for this user ID. x-example: "@alice:example.com" - in: path type: string @@ -69,6 +69,41 @@ paths: The account_data was successfully added. tags: - User data + get: + summary: Get some account_data for the user. + description: |- + Get some account_data for the client. This config is only visible to the user + that set the account_data. + operationId: getAccountData + security: + - accessToken: [] + parameters: + - in: path + type: string + name: userId + required: true + description: |- + The ID of the user to get account_data for. The access token must be + authorized to make requests for this user ID. + x-example: "@alice:example.com" + - in: path + type: string + name: type + required: true + description: |- + The event type of the account_data to get. Custom types should be + namespaced to avoid clashes. + x-example: "org.example.custom.config" + responses: + 200: + description: + The account data content for the given type. + schema: + type: object + example: { + "custom_account_data_key": "custom_config_value"} + tags: + - User data "/user/{userId}/rooms/{roomId}/account_data/{type}": put: summary: Set some account_data for the user. @@ -85,15 +120,15 @@ paths: name: userId required: true description: |- - The id of the user to set account_data for. The access token must be - authorized to make requests for this user id. + The ID of the user to set account_data for. The access token must be + authorized to make requests for this user ID. x-example: "@alice:example.com" - in: path type: string name: roomId required: true description: |- - The id of the room to set account_data on. + The ID of the room to set account_data on. x-example: "!726s6s6q:example.com" - in: path type: string @@ -118,3 +153,45 @@ paths: The account_data was successfully added. tags: - User data + get: + summary: Get some account_data for the user. + description: |- + Get some account_data for the client on a given room. This config is only + visible to the user that set the account_data. + operationId: getAccountDataPerRoom + security: + - accessToken: [] + parameters: + - in: path + type: string + name: userId + required: true + description: |- + The ID of the user to set account_data for. The access token must be + authorized to make requests for this user ID. + x-example: "@alice:example.com" + - in: path + type: string + name: roomId + required: true + description: |- + The ID of the room to get account_data for. + x-example: "!726s6s6q:example.com" + - in: path + type: string + name: type + required: true + description: |- + The event type of the account_data to get. Custom types should be + namespaced to avoid clashes. + x-example: "org.example.custom.room.config" + responses: + 200: + description: + The account data content for the given type. + schema: + type: object + example: { + "custom_account_data_key": "custom_config_value"} + tags: + - User data diff --git a/changelogs/client_server/newsfragments/1873.new b/changelogs/client_server/newsfragments/1873.new new file mode 100644 index 000000000..724a43083 --- /dev/null +++ b/changelogs/client_server/newsfragments/1873.new @@ -0,0 +1 @@ +``GET /account_data`` routes. From 5721712eae1ea6e039d8452c5539bb08f358b76e Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 10 Feb 2019 17:47:17 -0700 Subject: [PATCH 123/170] Add M_RESOURCE_LIMIT_EXCEEDED Original proposal: https://github.com/matrix-org/matrix-doc/issues/1504 No changes from the original proposal or implementations have been made intentionally here. --- changelogs/client_server/newsfragments/1874.feature | 1 + specification/client_server_api.rst | 10 +++++++++- 2 files changed, 10 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/1874.feature diff --git a/changelogs/client_server/newsfragments/1874.feature b/changelogs/client_server/newsfragments/1874.feature new file mode 100644 index 000000000..03b891e25 --- /dev/null +++ b/changelogs/client_server/newsfragments/1874.feature @@ -0,0 +1 @@ +Add `M_RESOURCE_LIMIT_EXCEEDED` as an error code for when homeservers exceed limits imposed on them. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index b41c34607..618fdc10a 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -210,10 +210,18 @@ Other error codes the client might encounter are: The resource being requested is reserved by an application service, or the application service making the request has not created the resource. +:``M_RESOURCE_LIMIT_EXCEEDED``: + The request cannot be completed because the homeserver has reached a resource + limit imposed on it. For example, a homeserver held in a shared hosting environment + may reach a resource limit if it starts using too much memory or disk space. The + error MUST have an ``admin_contact`` field to provide the user receiving the error + a place to reach out to. Typically, this error will appear on routes which attempt + to modify state (eg: sending messages, account data, etc) and not routes which only + read state (eg: ``/sync``, get account data, etc). + .. TODO: More error codes (covered by other issues) .. * M_CONSENT_NOT_GIVEN - GDPR: https://github.com/matrix-org/matrix-doc/issues/1512 .. * M_CANNOT_LEAVE_SERVER_NOTICE_ROOM - GDPR: https://github.com/matrix-org/matrix-doc/issues/1254 -.. * M_RESOURCE_LIMIT_EXCEEDED - Limits: https://github.com/matrix-org/matrix-doc/issues/1504 .. _sect:txn_ids: From b1689a3036878cbf152e2e4de45f1ac28a08d5ee Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 10 Feb 2019 18:03:17 -0700 Subject: [PATCH 124/170] Misc improvements --- meta/releasing_a_spec.md | 4 +- scripts/templating/matrix_templates/units.py | 120 +++++++++---------- 2 files changed, 56 insertions(+), 68 deletions(-) diff --git a/meta/releasing_a_spec.md b/meta/releasing_a_spec.md index 1d7aaea7b..f186c4be1 100644 --- a/meta/releasing_a_spec.md +++ b/meta/releasing_a_spec.md @@ -19,8 +19,8 @@ The remainder of the process is as follows: 1. Having checked out the new release branch, navigate your way over to `./changelogs`. 1. Follow the release instructions provided in the README.md located there. 1. Update any version/link references across all specifications. -1. Generate the specification using `./scripts/gendoc.py -c r0.4.0`, specifying all the - API versions at the time of generation. +1. Generate the specification using `./scripts/gendoc.py`, specifying all the + API versions at the time of generation. For example: `./scripts/gendoc.py -c r0.4.0 -s r0.1.0 -i r0.1.0 #etc` 1. PR the changes to the matrix-org/matrix.org repository (for historic tracking). * This is done by making a PR to the `unstyled_docs/spec` folder for the version and specification you're releasing. diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index a061c6936..c1755119e 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -935,92 +935,80 @@ class MatrixUnits(Units): changelog_lines = [] if target_version == 'unstable': # generate towncrier log - tc_path = os.path.join(CHANGELOG_DIR, api_name) - if os.path.isdir(tc_path): - logger.info("Generating towncrier changelog for: %s" % api_name) - p = subprocess.Popen( - ['towncrier', '--version', 'unstable', '--name', api_name, '--draft'], - cwd=tc_path, - stderr=subprocess.PIPE, - stdout=subprocess.PIPE, - ) - stdout, stderr = p.communicate() - if p.returncode != 0: - # Something broke - dump as much information as we can - logger.error("Towncrier exited with code %s" % p.returncode) - logger.error(stdout.decode('UTF-8')) - logger.error(stderr.decode('UTF-8')) - raw_log = "" - else: - raw_log = stdout.decode('UTF-8') - - # This is a bit of a hack, but it does mean that the log at least gets *something* - # to tell us it broke - if not raw_log.startswith("unstable"): - logger.error("Towncrier appears to have failed to generate a changelog") - logger.error(raw_log) - raw_log = "" - changelog_lines = raw_log.splitlines() + changelog_lines = self._read_towncrier_changelog(api_name) else: # read in the existing RST changelog - logger.info("Reading changelog RST for %s" % api_name) - rst_path = os.path.join(CHANGELOG_DIR, "%s.rst" % api_name) - with open(rst_path, 'r', encoding="utf-8") as f: - changelog_lines = f.readlines() + changelog_lines = self._read_rst_changelog(api_name) # Parse the changelog lines to find the header we're looking for and therefore # the changelog body. prev_line = None title_part = None changelog_body_lines = [] - have_changelog = False for line in changelog_lines: if prev_line is None: prev_line = line continue - if not title_part: - # Titles we care about are underlined with at least 3 equal signs - if re.match("^[=]{3,}$", line.strip()): - logger.info("Found header %s" % prev_line) - title_part = prev_line.strip() - continue - prev_line = line - else: - # we have a title, start parsing the body - if re.match("^[=]{3,}$", line.strip()): - # we hit another title. prev_line will be the new section's header. - # do a check to see if the section we just read is the one we want - if - # it is, use that changelog and move on. If it isn't, keep reading. - if title_part == target_version: - changelogs[api_name] = "".join(changelog_body_lines) - have_changelog = True - break - # not the section we want - start the next section - title_part = changelog_body_lines.pop().strip() - changelog_body_lines = [] - continue - if re.match("^[-]{3,}$", line.strip()): - # the last line is a subheading - drop this line because it's the underline - # and that causes problems with rendering. We'll keep the header text though. - continue - if line.strip().startswith(".. "): - # skip comments - continue + if re.match("^[=]{3,}$", line.strip()): + # the last line was a header - use that as our new title_part + title_part = prev_line.strip() + continue + if re.match("^[-]{3,}$", line.strip()): + # the last line is a subheading - drop this line because it's the underline + # and that causes problems with rendering. We'll keep the header text though. + continue + if line.strip().startswith(".. "): + # skip comments + continue + if title_part == target_version: # if we made it this far, append the line to the changelog body. We indent it so # that it renders correctly in the section. We also add newlines so that there's # intentionally blank lines that make rst2html happy. changelog_body_lines.append(" " + line + '\n') - # do some quick checks to see if the last read section is our changelog - if not have_changelog: - logger.info("No changelog - testing %s == %s" % (target_version, title_part,)) - if title_part == target_version and len(changelog_body_lines) > 0: - changelogs[api_name] = "".join(changelog_body_lines) - else: - raise ValueError("No changelog for %s at %s" % (api_name, target_version,)) + + if len(changelog_body_lines) > 0: + changelogs[api_name] = "".join(changelog_body_lines) + else: + raise ValueError("No changelog for %s at %s" % (api_name, target_version,)) # return our `dict[api_name] => changelog` as the last step. return changelogs + def _read_towncrier_changelog(self, api_name): + tc_path = os.path.join(CHANGELOG_DIR, api_name) + if os.path.isdir(tc_path): + logger.info("Generating towncrier changelog for: %s" % api_name) + p = subprocess.Popen( + ['towncrier', '--version', 'unstable', '--name', api_name, '--draft'], + cwd=tc_path, + stderr=subprocess.PIPE, + stdout=subprocess.PIPE, + ) + stdout, stderr = p.communicate() + if p.returncode != 0: + # Something broke - dump as much information as we can + logger.error("Towncrier exited with code %s" % p.returncode) + logger.error(stdout.decode('UTF-8')) + logger.error(stderr.decode('UTF-8')) + raw_log = "" + else: + raw_log = stdout.decode('UTF-8') + + # This is a bit of a hack, but it does mean that the log at least gets *something* + # to tell us it broke + if not raw_log.startswith("unstable"): + logger.error("Towncrier appears to have failed to generate a changelog") + logger.error(raw_log) + raw_log = "" + return raw_log.splitlines() + return [] + + def _read_rst_changelog(self, api_name): + logger.info("Reading changelog RST for %s" % api_name) + rst_path = os.path.join(CHANGELOG_DIR, "%s.rst" % api_name) + with open(rst_path, 'r', encoding="utf-8") as f: + return f.readlines() + def load_unstable_warnings(self, substitutions): warning = """ .. WARNING:: From ef13aef8c3751166032907ab2041fb1386fd7077 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 10 Feb 2019 19:31:25 -0700 Subject: [PATCH 125/170] Clarify the recommendations for "transferable state" Fixes https://github.com/matrix-org/matrix-doc/issues/1843 --- api/client-server/room_upgrades.yaml | 4 +--- specification/modules/room_upgrades.rst | 22 +++++++++++++++++----- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/api/client-server/room_upgrades.yaml b/api/client-server/room_upgrades.yaml index 6511d9fcf..55ee14dcd 100644 --- a/api/client-server/room_upgrades.yaml +++ b/api/client-server/room_upgrades.yaml @@ -31,9 +31,7 @@ paths: post: summary: Upgrades a room to a new room version. description: |- - Upgrades the given room to a particular room version, migrating as much - data as possible over to the new room. See the `room_upgrades <#room-upgrades>`_ - module for more information on what this entails. + Upgrades the given room to a particular room version. operationId: upgradeRoom security: - accessToken: [] diff --git a/specification/modules/room_upgrades.rst b/specification/modules/room_upgrades.rst index 49ff4414d..20efbabdb 100644 --- a/specification/modules/room_upgrades.rst +++ b/specification/modules/room_upgrades.rst @@ -47,9 +47,21 @@ When the client requests to upgrade a known room to a known version, the server: 1. Checks that the user has permission to send ``m.room.tombstone`` events in the room. 2. Creates a replacement room with a ``m.room.create`` event containing a ``predecessor`` field and the applicable ``room_version``. -3. Replicates the power levels, privacy, topic, and other transferable state events to - the new room. This generally excludes membership events but may include client-specified - events and other presentation details. +3. Replicates transferable state events to the new room. The exact details for what is + transferred is left as an implementation detail, however the recommended state events + to transfer are: + + * ``m.room.server_acl`` + * ``m.room.encryption`` + * ``m.room.name`` + * ``m.room.avatar`` + * ``m.room.topic`` + * ``m.room.guest_access`` + * ``m.room.history_visibility`` + * ``m.room.join_rules`` + + Membership events should not be transferred to the new room. + 4. Moves any local aliases to the new room. 5. Sends a ``m.room.tombstone`` event to the old room to indicate that it is not intended to be used any further. @@ -57,5 +69,5 @@ When the client requests to upgrade a known room to a known version, the server: of events and inviting new users. For example, setting ``events_default`` and ``invite`` to the greater of ``50`` and ``users_default + 1``. -When a user joins the new room, the server may wish to automatically transfer/replicate -some of the user's personalized settings such as notifications, tags, etc. +When a user joins the new room, the server should automatically transfer/replicate some of +the user's personalized settings such as notifications, tags, etc. From f67782230a6e26792b4a05f0cb5d63d7a54bd277 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 10 Feb 2019 19:35:23 -0700 Subject: [PATCH 126/170] changelog --- changelogs/client_server/newsfragments/1791.feature | 2 +- changelogs/client_server/newsfragments/1817.deprecation | 2 +- changelogs/client_server/newsfragments/1875.feature | 1 + 3 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/1875.feature diff --git a/changelogs/client_server/newsfragments/1791.feature b/changelogs/client_server/newsfragments/1791.feature index ae961ad3f..0a854c8f9 100644 --- a/changelogs/client_server/newsfragments/1791.feature +++ b/changelogs/client_server/newsfragments/1791.feature @@ -1 +1 @@ -Add room version upgrades +Add room version upgrades. diff --git a/changelogs/client_server/newsfragments/1817.deprecation b/changelogs/client_server/newsfragments/1817.deprecation index 9a888e85d..2c52d198e 100644 --- a/changelogs/client_server/newsfragments/1817.deprecation +++ b/changelogs/client_server/newsfragments/1817.deprecation @@ -1 +1 @@ -Remove references to presence lists +Remove references to presence lists. diff --git a/changelogs/client_server/newsfragments/1875.feature b/changelogs/client_server/newsfragments/1875.feature new file mode 100644 index 000000000..0a854c8f9 --- /dev/null +++ b/changelogs/client_server/newsfragments/1875.feature @@ -0,0 +1 @@ +Add room version upgrades. From f058a0f40a404d8a2711b168e183724285787bf6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 10 Feb 2019 19:36:33 -0700 Subject: [PATCH 127/170] also power levels --- specification/modules/room_upgrades.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/specification/modules/room_upgrades.rst b/specification/modules/room_upgrades.rst index 20efbabdb..889c1068f 100644 --- a/specification/modules/room_upgrades.rst +++ b/specification/modules/room_upgrades.rst @@ -59,6 +59,7 @@ When the client requests to upgrade a known room to a known version, the server: * ``m.room.guest_access`` * ``m.room.history_visibility`` * ``m.room.join_rules`` + * ``m.room.power_levels`` Membership events should not be transferred to the new room. From 946acbf380a4fc510737886f3a94ec024d39f227 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 10 Feb 2019 19:49:55 -0700 Subject: [PATCH 128/170] Clarify v3 event representation in identifier grammar Fixes https://github.com/matrix-org/matrix-doc/issues/1870 Fixes https://github.com/matrix-org/matrix-doc/issues/1869 Fixes https://github.com/matrix-org/matrix-doc/issues/1867 --- specification/appendices/identifier_grammar.rst | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index a0cdf298f..e10348034 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -19,7 +19,7 @@ Identifier Grammar Some identifiers are specific to given room versions, please refer to the `room versions specification`_ for more information. -.. _`room versions specification`: ../index.html#room-versions +.. _`room versions specification`: index.html#room-versions Server Name @@ -234,18 +234,17 @@ A room has exactly one room ID. A room ID has the format:: !opaque_id:domain -An event has exactly one event ID. An event ID has the format:: +An event has exactly one event ID. The format of an event ID depends upon the +`room version specification `_. - $opaque_id:domain - -The ``domain`` of a room/event ID is the `server name`_ of the homeserver which +The ``domain`` of a room ID is the `server name`_ of the homeserver which created the room/event. The domain is used only for namespacing to avoid the risk of clashes of identifiers between different homeservers. There is no implication that the room or event in question is still available at the corresponding homeserver. Event IDs and Room IDs are case-sensitive. They are not meant to be human -readable. +readable. They are intended to be treated as fully opaque strings by clients. .. TODO-spec What is the grammar for the opaque part? https://matrix.org/jira/browse/SPEC-389 @@ -333,6 +332,10 @@ Examples of matrix.to URIs are: * User: ``https://matrix.to/#/@alice:example.org`` * Group: ``https://matrix.to/#/+example:example.org`` +.. Note:: + Event IDs have the potential to contain slashes in some `room versions `_. + No component of the URI should be URL encoded. + .. Note:: Room ID permalinks are unroutable as there is no reliable domain to send requests to upon receipt of the permalink. Clients should do their best route Room IDs to From 22188ebfebb291ffb30a879045499aef12a9fef1 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 10 Feb 2019 19:54:20 -0700 Subject: [PATCH 129/170] Further clarify why membership events are not to be transferred --- specification/modules/room_upgrades.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specification/modules/room_upgrades.rst b/specification/modules/room_upgrades.rst index 889c1068f..f1861f720 100644 --- a/specification/modules/room_upgrades.rst +++ b/specification/modules/room_upgrades.rst @@ -61,7 +61,11 @@ When the client requests to upgrade a known room to a known version, the server: * ``m.room.join_rules`` * ``m.room.power_levels`` - Membership events should not be transferred to the new room. + Membership events should not be transferred to the new room due to technical limitations + of servers not being able to impersonate people from other homeservers. Additionally, + servers should not transfer state events which are sensitive to who sent them, such as + events outside of the Matrix namespace where clients may rely on the sender to match + certain criteria. 4. Moves any local aliases to the new room. 5. Sends a ``m.room.tombstone`` event to the old room to indicate that it is not intended From d31d2f5e57c58903d0fdbdba8d215d1e1e10d2df Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 11 Feb 2019 20:31:48 -0700 Subject: [PATCH 130/170] Correctly nest the capabilities response object Everything is contained in a "capabilities" property, which is not represented by the schema. The example was correct. --- api/client-server/capabilities.yaml | 68 ++++++++++--------- .../client_server/newsfragments/1879.feature | 1 + 2 files changed, 37 insertions(+), 32 deletions(-) create mode 100644 changelogs/client_server/newsfragments/1879.feature diff --git a/api/client-server/capabilities.yaml b/api/client-server/capabilities.yaml index 63a3ea91a..a50908f75 100644 --- a/api/client-server/capabilities.yaml +++ b/api/client-server/capabilities.yaml @@ -62,44 +62,48 @@ paths: schema: type: object required: ["capabilities"] - additionalProperties: - type: object - description: |- - The custom capabilities the server supports, using the - Java package naming convention. properties: - "m.change_password": + capabilities: type: object + title: Capabilities description: |- - Capability to indicate if the user can change their password. - title: ChangePasswordCapability + The custom capabilities the server supports, using the + Java package naming convention. + additionalProperties: + type: object properties: - enabled: - type: boolean - description: |- - True if the user can change their password, false otherwise. - example: false - required: ['enabled'] - "m.room_versions": - type: object - description: The room versions the server supports. - title: RoomVersionsCapability - properties: - default: - type: string - description: |- - The default room version the server is using for new rooms. - example: "1" - available: + "m.change_password": type: object description: |- - A detailed description of the room versions the server supports. - additionalProperties: - type: string - title: RoomVersionStability - enum: [stable, unstable] - description: The stability of the room version. - required: ['default', 'available'] + Capability to indicate if the user can change their password. + title: ChangePasswordCapability + properties: + enabled: + type: boolean + description: |- + True if the user can change their password, false otherwise. + example: false + required: ['enabled'] + "m.room_versions": + type: object + description: The room versions the server supports. + title: RoomVersionsCapability + properties: + default: + type: string + description: |- + The default room version the server is using for new rooms. + example: "1" + available: + type: object + description: |- + A detailed description of the room versions the server supports. + additionalProperties: + type: string + title: RoomVersionStability + enum: [stable, unstable] + description: The stability of the room version. + required: ['default', 'available'] 429: description: This request was rate-limited. schema: diff --git a/changelogs/client_server/newsfragments/1879.feature b/changelogs/client_server/newsfragments/1879.feature new file mode 100644 index 000000000..107291f34 --- /dev/null +++ b/changelogs/client_server/newsfragments/1879.feature @@ -0,0 +1 @@ +Support optional features by having clients query for capabilities. From 82258fc0fc90c8f2cc5d96967229a029335b1b7e Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 13 Feb 2019 23:11:14 +0000 Subject: [PATCH 131/170] Proposal for changing event ids. Again. --- .../1884-replace-slashes-in-event_ids.md | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 proposals/1884-replace-slashes-in-event_ids.md diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md new file mode 100644 index 000000000..5d9d0bc23 --- /dev/null +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -0,0 +1,91 @@ +# MSC1884: Proposal to replace slashes in event IDs + +[MSC1659](https://github.com/matrix-org/matrix-doc/pull/1659) mandated that, +starting in version 3 rooms, event IDs must be calculated as a base64-encoding +of a hash. This implies that event IDs may contain any character in the +standard Base64 alphabet, which notably includes the slash character, `/`. + +Event IDs are often embedded in URI paths, and since the slash character is +used as a separator in URI paths, this presents a problem. The immediate +solution is to ensure that event IDs are URL-encoded, so that `/` is instead +represented as `%2F`. However, this is not entirely satisfactory for a number +of reasons: + + * There exist a number of client (and possibly server) implementations which + do not currently URL-encode such parameters; these are therefore broken by + such event IDs and must be updated. Furthermore, all future client + implementers must remember to do the encoding correctly. + + * Even if client implementations do rembember to URL-encode their parameters, + they may not do it correctly: many URL-encoding implementations may be + intended to encode parameters in the query-string (which can of course + contain literal slashes) rather tha the path component. + + * Some proxy software may treat `%2F` specially: for instance, Apache, when + configured as a reverse-proxy, will reject requests for a path containing + `%2F` unless it is also configured with `nocanon`. Again this means that + existing setups will be broken by this change, and it is a trap for new + users of the software. + +## Proposal + +This MSC proposes that we should introduce a new room version, in which event +IDs are encoded using the [URL-safe Base64 +encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and +`_` as the 62nd and 63rd characters instead of `+` and `/`). + +## Counterarguments + +1. Inconsistency. Base64 encoding is used heavily elsewhere in the matrix + protocol and in all cases the standard encoding is used (though with some + variation as to the inclusion of padding characters). Further, SHA256 hashes + are used in a number of places and are universally included with standard, + unpadded Base64. + + Changing event IDs alone would therefore leave us with a confusing mix of + encodings. + + A potential extension would be to change *all* Base64 encodings to be + URL-safe. This would address the inconsistency. However, it feels like a + large job which would span the entire matrix ecosystem (far larger than + updating clients to URL-encode their URL prarameters), and again the + situation would be confusing while the transition was in progress. + +2. Incompleteness. Event IDs are certainly not the only identifier which can + contain slashes - Room aliases, Room IDs, Group IDs, User IDs [1], and state + keys can all contain slashes, as well as a number of identifiers whose + grammars are currently underspecified (eg transaction ids, event types, + device IDs). (Indeed, there was nothing preventing Event IDs from containing + slashes before room v3 - it just happened that Synapse used an algorithm + which didn't generate them). + + All of these other identifiers can appear in URLs in either or both the + client-server or server-server APIs, and all have the potential to cause + misbehaviour if software does not correctly URL-encode them. + + It can be argued that it is better for software to fail 50% of the time [2] + so that it can be fixed than it is to fail only on edge-cases or, worse, + when deliberately provoked by a malicious or "curious" actor. + + Of course, an alternative is to modify the grammars of all of these + identifiers to forbid slashes. + +[1] Discussion remains open as to whether allowing slashes in User IDs was a +good idea. + +[2] 48% of random 32-byte sequences will contain a slash when Base64-encoded. + +## Alternatives + +An alternative would be to modify all REST endpoints to use query or body +parameters instead of path parameters. This would of course be a significant +and incompatible change, but it would also bring the benefit of solving a +common problem where forgetting to use `nocanon` in a reverse-proxy +configuration [breaks +federation](https://github.com/matrix-org/synapse/issues/3294) (though other +solutions to that are also possible). + +## Conclusion + +It's unclear to me that changing the format of event IDs alone solves any +problems. From 2de7ef9a3d1305edffd7774fe22ea140b9903966 Mon Sep 17 00:00:00 2001 From: Anatoly Sablin Date: Fri, 15 Feb 2019 23:23:38 +0300 Subject: [PATCH 132/170] #1865 Add the m.push_rules schema. --- event-schemas/examples/m.push_rules | 192 ++++++++++++++++++ .../schema/core-event-schema/push_rule.yaml | 85 ++++++++ event-schemas/schema/m.push_rules | 59 ++++++ specification/modules/push.rst | 2 + 4 files changed, 338 insertions(+) create mode 100644 event-schemas/examples/m.push_rules create mode 100644 event-schemas/schema/core-event-schema/push_rule.yaml create mode 100644 event-schemas/schema/m.push_rules diff --git a/event-schemas/examples/m.push_rules b/event-schemas/examples/m.push_rules new file mode 100644 index 000000000..9785da253 --- /dev/null +++ b/event-schemas/examples/m.push_rules @@ -0,0 +1,192 @@ +{ + "$ref": "core/state_event.json", + "type": "m.push_rules", + "content": { + "global": { + "content": [ + { + "actions": [ + "notify", + { + "set_tweak": "sound", + "value": "default" + }, + { + "set_tweak": "highlight" + } + ], + "default": true, + "enabled": true, + "pattern": "alice", + "rule_id": ".m.rule.contains_user_name" + } + ], + "override": [ + { + "actions": [ + "dont_notify" + ], + "conditions": [], + "default": true, + "enabled": false, + "rule_id": ".m.rule.master" + }, + { + "actions": [ + "dont_notify" + ], + "conditions": [ + { + "key": "content.msgtype", + "kind": "event_match", + "pattern": "m.notice" + } + ], + "default": true, + "enabled": true, + "rule_id": ".m.rule.suppress_notices" + } + ], + "room": [], + "sender": [], + "underride": [ + { + "actions": [ + "notify", + { + "set_tweak": "sound", + "value": "ring" + }, + { + "set_tweak": "highlight", + "value": false + } + ], + "conditions": [ + { + "key": "type", + "kind": "event_match", + "pattern": "m.call.invite" + } + ], + "default": true, + "enabled": true, + "rule_id": ".m.rule.call" + }, + { + "actions": [ + "notify", + { + "set_tweak": "sound", + "value": "default" + }, + { + "set_tweak": "highlight" + } + ], + "conditions": [ + { + "kind": "contains_display_name" + } + ], + "default": true, + "enabled": true, + "rule_id": ".m.rule.contains_display_name" + }, + { + "actions": [ + "notify", + { + "set_tweak": "sound", + "value": "default" + }, + { + "set_tweak": "highlight", + "value": false + } + ], + "conditions": [ + { + "is": "2", + "kind": "room_member_count" + } + ], + "default": true, + "enabled": true, + "rule_id": ".m.rule.room_one_to_one" + }, + { + "actions": [ + "notify", + { + "set_tweak": "sound", + "value": "default" + }, + { + "set_tweak": "highlight", + "value": false + } + ], + "conditions": [ + { + "key": "type", + "kind": "event_match", + "pattern": "m.room.member" + }, + { + "key": "content.membership", + "kind": "event_match", + "pattern": "invite" + }, + { + "key": "state_key", + "kind": "event_match", + "pattern": "@alice:example.com" + } + ], + "default": true, + "enabled": true, + "rule_id": ".m.rule.invite_for_me" + }, + { + "actions": [ + "notify", + { + "set_tweak": "highlight", + "value": false + } + ], + "conditions": [ + { + "key": "type", + "kind": "event_match", + "pattern": "m.room.member" + } + ], + "default": true, + "enabled": true, + "rule_id": ".m.rule.member_event" + }, + { + "actions": [ + "notify", + { + "set_tweak": "highlight", + "value": false + } + ], + "conditions": [ + { + "key": "type", + "kind": "event_match", + "pattern": "m.room.message" + } + ], + "default": true, + "enabled": true, + "rule_id": ".m.rule.message" + } + ] + } + } +} diff --git a/event-schemas/schema/core-event-schema/push_rule.yaml b/event-schemas/schema/core-event-schema/push_rule.yaml new file mode 100644 index 000000000..a7626bbeb --- /dev/null +++ b/event-schemas/schema/core-event-schema/push_rule.yaml @@ -0,0 +1,85 @@ +# Copyright 2016 OpenMarket Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +title: PushRule +type: object +properties: + actions: + items: + type: + - object + - string + type: array + description: |- + The actions to perform when this rule is matched. + default: + type: boolean + description: |- + Whether this is a default rule, or has been set explicitly. + enabled: + type: boolean + description: |- + Whether the push rule is enabled or not. + rule_id: + type: string + description: |- + The ID of this rule. + conditions: + type: array + items: + title: PushCondition + type: object + properties: + kind: + enum: + - event_match + - contains_display_name + - room_member_count + type: string + key: + type: string + description: |- + Required for ``event_match`` conditions. The dot-separated field of the + event to match. + x-example: content.body + pattern: + type: string + description: |- + Required for ``event_match`` conditions. The glob-style pattern to + match against. Patterns with no special glob characters should be + treated as having asterisks prepended and appended when testing the + condition. + is: + type: string + description: |- + Required for ``room_member_count`` conditions. A decimal integer + optionally prefixed by one of, ==, <, >, >= or <=. A prefix of < matches + rooms where the member count is strictly less than the given number and + so forth. If no prefix is present, this parameter defaults to ==. + required: + - kind + description: |- + The conditions that must hold true for an event in order for a rule to be + applied to an event. A rule with no conditions always matches. Only + applicable to ``underride`` and ``override`` rules. + pattern: + type: string + description: |- + The glob-style pattern to match against. Only applicable to ``content`` + rules. +required: + - actions + - default + - enabled + - rule_id diff --git a/event-schemas/schema/m.push_rules b/event-schemas/schema/m.push_rules new file mode 100644 index 000000000..b487a6f54 --- /dev/null +++ b/event-schemas/schema/m.push_rules @@ -0,0 +1,59 @@ +--- +allOf: + - $ref: core-event-schema/state_event.yaml +description: Describes all push rules for this user. +properties: + content: + properties: + global: + type: object + title: Ruleset + description: The global ruleset + properties: + content: + items: + allOf: + - $ref: core-event-schema/push_rule.yaml + title: PushRule + type: object + type: array + override: + items: + allOf: + - $ref: core-event-schema/push_rule.yaml + title: PushRule + type: object + type: array + room: + items: + allOf: + - $ref: core-event-schema/push_rule.yaml + title: PushRule + type: object + type: array + sender: + items: + allOf: + - $ref: core-event-schema/push_rule.yaml + title: PushRule + type: object + type: array + underride: + items: + allOf: + - $ref: core-event-schema/push_rule.yaml + title: PushRule + type: object + type: array + type: object + state_key: + description: A zero-length string. + pattern: '^$' + type: string + type: + enum: + - m.push_rules + type: string +title: Push rules +type: object + diff --git a/specification/modules/push.rst b/specification/modules/push.rst index 008eaa8b6..9830729a1 100644 --- a/specification/modules/push.rst +++ b/specification/modules/push.rst @@ -660,6 +660,8 @@ When a user changes their push rules a ``m.push_rules`` event is sent to all clients in the ``account_data`` section of their next ``/sync`` request. The content of the event is the current push rules for the user. +{{m_push_rules_event}} + Examples ++++++++ From d4b4d92b827a41ec1fcda1e799dd2a3f67edf8bb Mon Sep 17 00:00:00 2001 From: Anatoly Sablin Date: Sat, 16 Feb 2019 00:24:26 +0300 Subject: [PATCH 133/170] Fix the m.push_rules type (switch from the state event to the common event). --- event-schemas/examples/m.push_rules | 2 +- event-schemas/schema/m.push_rules | 6 +----- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/event-schemas/examples/m.push_rules b/event-schemas/examples/m.push_rules index 9785da253..e4f0a959f 100644 --- a/event-schemas/examples/m.push_rules +++ b/event-schemas/examples/m.push_rules @@ -1,5 +1,5 @@ { - "$ref": "core/state_event.json", + "$ref": "core/event.json", "type": "m.push_rules", "content": { "global": { diff --git a/event-schemas/schema/m.push_rules b/event-schemas/schema/m.push_rules index b487a6f54..ededb33ed 100644 --- a/event-schemas/schema/m.push_rules +++ b/event-schemas/schema/m.push_rules @@ -1,6 +1,6 @@ --- allOf: - - $ref: core-event-schema/state_event.yaml + - $ref: core-event-schema/event.yaml description: Describes all push rules for this user. properties: content: @@ -46,10 +46,6 @@ properties: type: object type: array type: object - state_key: - description: A zero-length string. - pattern: '^$' - type: string type: enum: - m.push_rules From 17420c537a8a12199e7447f9610ba9bb0058232f Mon Sep 17 00:00:00 2001 From: Anatoly Sablin Date: Sat, 16 Feb 2019 08:56:02 +0300 Subject: [PATCH 134/170] Fix links. --- specification/rooms/v3.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index 863f1c3a3..c17c0fe03 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -118,7 +118,7 @@ to the change in event format: The remaining rules are the same as `room version 1 `_. -.. _`Unpadded Base64`: ../../appendices.html#unpadded-base64 -.. _`Canonical JSON`: ../../appendices.html#canonical-json -.. _`Signing Events`: ../../server_server/r0.1.1.html#signing-events -.. _`reference hash`: ../../server_server/r0.1.1.html#reference-hashes +.. _`Unpadded Base64`: ../../spec/appendices.html#unpadded-base64 +.. _`Canonical JSON`: ../../spec/appendices.html#canonical-json +.. _`Signing Events`: ../../spec/server_server/r0.1.1.html#signing-events +.. _`reference hash`: ../../spec/server_server/r0.1.1.html#reference-hashes From befd76f8533d3bdd1af82490893af6d6f66d46e5 Mon Sep 17 00:00:00 2001 From: Anatoly Sablin Date: Sat, 16 Feb 2019 10:38:51 +0300 Subject: [PATCH 135/170] Fix links. --- specification/rooms/v3.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/rooms/v3.rst b/specification/rooms/v3.rst index c17c0fe03..733c69846 100644 --- a/specification/rooms/v3.rst +++ b/specification/rooms/v3.rst @@ -118,7 +118,7 @@ to the change in event format: The remaining rules are the same as `room version 1 `_. -.. _`Unpadded Base64`: ../../spec/appendices.html#unpadded-base64 -.. _`Canonical JSON`: ../../spec/appendices.html#canonical-json -.. _`Signing Events`: ../../spec/server_server/r0.1.1.html#signing-events -.. _`reference hash`: ../../spec/server_server/r0.1.1.html#reference-hashes +.. _`Unpadded Base64`: ../appendices.html#unpadded-base64 +.. _`Canonical JSON`: ../appendices.html#canonical-json +.. _`Signing Events`: ../server_server/r0.1.1.html#signing-events +.. _`reference hash`: ../server_server/r0.1.1.html#reference-hashes From 989d7ff192cc2414606fea061255f55df31ac1be Mon Sep 17 00:00:00 2001 From: Kitsune Ral Date: Sat, 16 Feb 2019 18:49:13 +0900 Subject: [PATCH 136/170] Remove extended text about life without transaction IDs Signed-off-by: Kitsune Ral --- specification/modules/instant_messaging.rst | 27 ++++++--------------- 1 file changed, 8 insertions(+), 19 deletions(-) diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 2adade5e7..86daa7006 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -185,25 +185,14 @@ reduced through clients making use of the transaction ID they used to send a particular event. The transaction ID used will be included in the event's ``unsigned`` data as ``transaction_id`` when it arrives through the event stream. -Clients unable to make use of the transaction ID are more likely to experience -flickering due to the following two scenarios, however the effect can be mitigated -to a degree: - -- The client sends a message and the remote echo arrives on the event stream - *after* the request to send the message completes. -- The client sends a message and the remote echo arrives on the event stream - *before* the request to send the message completes. - -In the first scenario, the client will receive an event ID when the request to -send the message completes. This ID can be used to identify the duplicate event -when it arrives on the event stream. However, in the second scenario, the event -arrives before the client has obtained an event ID. This makes it impossible to -identify it as a duplicate event. This results in the client displaying the -message twice for a fraction of a second before the the original request to send -the message completes. Once it completes, the client can take remedial actions -to remove the duplicate event by looking for duplicate event IDs. A future version -of the client-server API will resolve this by attaching the transaction ID of the -sending request to the event itself. +Clients unable to make use of the transaction ID are likely to experience +flickering when the remote echo arrives on the event stream *before* +the request to send the message completes. In that case the event +arrives before the client has obtained an event ID, making it impossible to +identify it as a remote echo. This results in the client displaying the message +twice for some time (depending on the server responsiveness) before the original +request to send the message completes. Once it completes, the client can take +remedial actions to remove the duplicate event by looking for duplicate event IDs. Calculating the display name for a user From e825224b5b9649cdf3e3a71f3eb4b45e846abb3a Mon Sep 17 00:00:00 2001 From: Hristo Venev Date: Thu, 21 Feb 2019 21:27:21 +0000 Subject: [PATCH 137/170] Restrict identifier port numbers to 5 digits. --- specification/appendices/identifier_grammar.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index a0cdf298f..b0d4f6f7a 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -34,7 +34,7 @@ following grammar:: server_name = hostname [ ":" port ] - port = *DIGIT + port = 1*5DIGIT hostname = IPv4address / "[" IPv6address "]" / dns-name From 4bec3d6dd866e9ba4843a7f55dd87f841d50f1e9 Mon Sep 17 00:00:00 2001 From: Anatoly Sablin Date: Sun, 24 Feb 2019 19:43:03 +0300 Subject: [PATCH 138/170] Avoid duplicates. --- .../schema/core-event-schema/push_rule.yaml | 85 ------------------- event-schemas/schema/m.push_rules | 38 +-------- 2 files changed, 2 insertions(+), 121 deletions(-) delete mode 100644 event-schemas/schema/core-event-schema/push_rule.yaml diff --git a/event-schemas/schema/core-event-schema/push_rule.yaml b/event-schemas/schema/core-event-schema/push_rule.yaml deleted file mode 100644 index a7626bbeb..000000000 --- a/event-schemas/schema/core-event-schema/push_rule.yaml +++ /dev/null @@ -1,85 +0,0 @@ -# Copyright 2016 OpenMarket Ltd -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -title: PushRule -type: object -properties: - actions: - items: - type: - - object - - string - type: array - description: |- - The actions to perform when this rule is matched. - default: - type: boolean - description: |- - Whether this is a default rule, or has been set explicitly. - enabled: - type: boolean - description: |- - Whether the push rule is enabled or not. - rule_id: - type: string - description: |- - The ID of this rule. - conditions: - type: array - items: - title: PushCondition - type: object - properties: - kind: - enum: - - event_match - - contains_display_name - - room_member_count - type: string - key: - type: string - description: |- - Required for ``event_match`` conditions. The dot-separated field of the - event to match. - x-example: content.body - pattern: - type: string - description: |- - Required for ``event_match`` conditions. The glob-style pattern to - match against. Patterns with no special glob characters should be - treated as having asterisks prepended and appended when testing the - condition. - is: - type: string - description: |- - Required for ``room_member_count`` conditions. A decimal integer - optionally prefixed by one of, ==, <, >, >= or <=. A prefix of < matches - rooms where the member count is strictly less than the given number and - so forth. If no prefix is present, this parameter defaults to ==. - required: - - kind - description: |- - The conditions that must hold true for an event in order for a rule to be - applied to an event. A rule with no conditions always matches. Only - applicable to ``underride`` and ``override`` rules. - pattern: - type: string - description: |- - The glob-style pattern to match against. Only applicable to ``content`` - rules. -required: - - actions - - default - - enabled - - rule_id diff --git a/event-schemas/schema/m.push_rules b/event-schemas/schema/m.push_rules index ededb33ed..6fde9e148 100644 --- a/event-schemas/schema/m.push_rules +++ b/event-schemas/schema/m.push_rules @@ -9,42 +9,8 @@ properties: type: object title: Ruleset description: The global ruleset - properties: - content: - items: - allOf: - - $ref: core-event-schema/push_rule.yaml - title: PushRule - type: object - type: array - override: - items: - allOf: - - $ref: core-event-schema/push_rule.yaml - title: PushRule - type: object - type: array - room: - items: - allOf: - - $ref: core-event-schema/push_rule.yaml - title: PushRule - type: object - type: array - sender: - items: - allOf: - - $ref: core-event-schema/push_rule.yaml - title: PushRule - type: object - type: array - underride: - items: - allOf: - - $ref: core-event-schema/push_rule.yaml - title: PushRule - type: object - type: array + allOf: + - $ref: ../../api/client-server/definitions/push_ruleset.yaml type: object type: enum: From d52918a8923acc65acb91ddfc7d064bf7b88e7a1 Mon Sep 17 00:00:00 2001 From: Anatoly Sablin Date: Sun, 24 Feb 2019 22:26:32 +0300 Subject: [PATCH 139/170] Fix #1904. --- api/server-server/openid.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/server-server/openid.yaml b/api/server-server/openid.yaml index 0eac48c84..0761618d9 100644 --- a/api/server-server/openid.yaml +++ b/api/server-server/openid.yaml @@ -33,7 +33,7 @@ paths: User ID of the owner. operationId: exchangeOpenIdToken parameters: - - in: path + - in: query name: access_token type: string description: |- From 060c5c7b21243ef737cb9ec6d537a0e3ab7674df Mon Sep 17 00:00:00 2001 From: Anatoly Sablin Date: Sun, 24 Feb 2019 23:01:01 +0300 Subject: [PATCH 140/170] Add clarification changelog entry. --- changelogs/server_server/newsfragments/1904.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/server_server/newsfragments/1904.clarification diff --git a/changelogs/server_server/newsfragments/1904.clarification b/changelogs/server_server/newsfragments/1904.clarification new file mode 100644 index 000000000..94174ebd1 --- /dev/null +++ b/changelogs/server_server/newsfragments/1904.clarification @@ -0,0 +1 @@ +Fix the `access_token` parameter in the open_id endpoint. From baf8948eb5b3be04f06bba0fce87153834bda16f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 24 Feb 2019 21:53:50 -0700 Subject: [PATCH 141/170] Describe M_UNSUPPORTED_ROOM_VERSION on invite and createRoom endpoints Spec for MSC1866: https://github.com/matrix-org/matrix-doc/pull/1866 --- api/client-server/create_room.yaml | 6 ++++++ api/client-server/inviting.yaml | 14 ++++++++++++++ .../client_server/newsfragments/1903.feature | 1 + 3 files changed, 21 insertions(+) create mode 100644 changelogs/client_server/newsfragments/1903.feature diff --git a/api/client-server/create_room.yaml b/api/client-server/create_room.yaml index d361d9738..bce61aad3 100644 --- a/api/client-server/create_room.yaml +++ b/api/client-server/create_room.yaml @@ -244,6 +244,12 @@ paths: invalid: for example, the user's ``power_level`` is set below that necessary to set the room name (``errcode`` set to ``M_INVALID_ROOM_STATE``). + + - The homeserver doesn't support the requested room version, or + one or more users being invited to the new room are residents + of a homeserver which does not support the requested room version. + The ``errcode`` will be ``M_UNSUPPORTED_ROOM_VERSION`` in these + cases. schema: "$ref": "definitions/errors/error.yaml" tags: diff --git a/api/client-server/inviting.yaml b/api/client-server/inviting.yaml index f312d5ceb..dfa66b9b0 100644 --- a/api/client-server/inviting.yaml +++ b/api/client-server/inviting.yaml @@ -82,6 +82,20 @@ paths: } schema: type: object + 400: + description: |- + + The request is invalid. A meaningful ``errcode`` and description + error text will be returned. Example reasons for rejection include: + + - The request body is malformed (``errcode`` set to ``M_BAD_JSON`` + or ``M_NOT_JSON``). + + - One or more users being invited to the room are residents of a + homeserver which does not support the requested room version. The + ``errcode`` will be ``M_UNSUPPORTED_ROOM_VERSION`` in these cases. + schema: + "$ref": "definitions/errors/error.yaml" 403: description: |- You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are: diff --git a/changelogs/client_server/newsfragments/1903.feature b/changelogs/client_server/newsfragments/1903.feature new file mode 100644 index 000000000..1c64d826e --- /dev/null +++ b/changelogs/client_server/newsfragments/1903.feature @@ -0,0 +1 @@ +Emit ``M_UNSUPPORTED_ROOM_VERSION`` error codes where applicable on ``/createRoom`` and ``/invite`` APIs. From adb721bc0b101bcf458a17275792eec20fe96e63 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 24 Feb 2019 21:55:57 -0700 Subject: [PATCH 142/170] Fix unrelated changelog entry's RST representation --- changelogs/client_server/newsfragments/1874.feature | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/client_server/newsfragments/1874.feature b/changelogs/client_server/newsfragments/1874.feature index 03b891e25..bdab54644 100644 --- a/changelogs/client_server/newsfragments/1874.feature +++ b/changelogs/client_server/newsfragments/1874.feature @@ -1 +1 @@ -Add `M_RESOURCE_LIMIT_EXCEEDED` as an error code for when homeservers exceed limits imposed on them. +Add ``M_RESOURCE_LIMIT_EXCEEDED`` as an error code for when homeservers exceed limits imposed on them. From 4049ca92085b98b29d19eb24323ebaab41765529 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Mon, 25 Feb 2019 09:47:19 +0000 Subject: [PATCH 143/170] Specify news fragments must be done in RST. --- CONTRIBUTING.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 0666c5797..7a6c6be20 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -98,8 +98,10 @@ the ``newsfragments`` directory. The ``type`` can be one of the following: * ``deprecation`` - Used when deprecating something -All news fragments must have a brief summary explaining the change in the contents -of the file. The summary must end in a full stop to be in line with the style guide. +All news fragments must have a brief summary explaining the change in the +contents of the file. The summary must end in a full stop to be in line with +the style guide and and formatting must be done using `Restructured Text +`_. Changes that do not change the spec, such as changes to the build script, formatting, CSS, etc should not get a news fragment. From 99405418c9408004ce099cf21b01a7838d97b51e Mon Sep 17 00:00:00 2001 From: Ben Parsons Date: Tue, 26 Feb 2019 18:11:02 +0000 Subject: [PATCH 144/170] in the appendices, thanks earfolds --- specification/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/index.rst b/specification/index.rst index d6176c93a..19c5ad223 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -196,7 +196,7 @@ allocated the account and has the form:: @localpart:domain -See `'Identifier Grammar' the appendices `_ for full details of +See `'Identifier Grammar' in the appendices `_ for full details of the structure of user IDs. Devices From 5303ba54b72a926e029528a9e96c7fb75925d51a Mon Sep 17 00:00:00 2001 From: Anatoly Sablin Date: Thu, 28 Feb 2019 21:04:23 +0300 Subject: [PATCH 145/170] Add a clarification changelog. --- changelogs/client_server/newsfragments/1889.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1889.clarification diff --git a/changelogs/client_server/newsfragments/1889.clarification b/changelogs/client_server/newsfragments/1889.clarification new file mode 100644 index 000000000..5026dab38 --- /dev/null +++ b/changelogs/client_server/newsfragments/1889.clarification @@ -0,0 +1 @@ +Add the missing `m.push_rules` event schema. From f92925ed456d51c71dd064ee30b93bad40f1fe44 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 6 Mar 2019 11:32:28 +0000 Subject: [PATCH 146/170] Add a 3PID unbind API --- .../1915-unbind-identity-server-param.md | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 proposals/1915-unbind-identity-server-param.md diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md new file mode 100644 index 000000000..f018d261d --- /dev/null +++ b/proposals/1915-unbind-identity-server-param.md @@ -0,0 +1,97 @@ +# MSC 1915 - Add unbind 3PID APIs + +Note that this is a simplified version of MSC1194. + + +## Motivation + +Currently we do not have a reasonable route for a user to unbind/remove a 3PID +from their account, particularly when deactivating their account. Users have an +expectation to be able to do this, and thus we should have an API to provide it. + +This is meant as a simple extension to the current APIs, and so this explicitly +does not try and solve any existing usability concerns. + + +## New APIs + +### Client-Server API + +Add `POST /_matrix/client/r0/account/3pid/delete` API, which expects a JSON body +with `medium`, `address` and `id_server` fields (as per existing APIs). + +The `id_server` parameter is optional and if missing the server will attempt to +unbind from a suitable identity server (e.g. its default identity server or the +server used when originally binding the 3pid). + +The 200 response is a JSON object with an `id_server_unbind_result` field whose +value is either `success` or `no-support`, where the latter indicates that the +identity server (IS) does not support unbinding 3PIDs directly. + +Example: + +``` +POST /_matrix/client/r0/account/3pid/delete HTTP/1.1 + +{ + "medium": "email", + "address": "foobar@example.com", + "id_server": "https://matrix.org +} + +HTTP/1.1 200 OK +{ + "id_server_unbind_result": "success" +} +``` + + +### Identity Server API + +Add `POST /_matrix/identity/api/v1/unbind` with `mxid` and `threepid` fields. +The `mxid` is the user's `user_id` and `threepid` is a dict with the usual +`medium` and `address` fields. + +If the server returns a 400, 404 or 501 HTTP error code then the homeserver +should assume that the identity server doesn't support the `/unbind` API, unless +it returns a specific matrix error response (i.e. the body is a JSON object with +`error` and `errcode` fields). + +The identity server should accept any request to unbind a 3PID for a `user_id` from +the homeserver controlling that user ID. + +Example: + +``` +POST /_matrix/identity/api/v1/unbind HTTP/1.1 + +{ + "mxid": "@foobar:example.com", + "threepid": { + "medium": "email", + "address": "foobar@example.com" + } +} + +HTTP/1.1 200 OK + +{} +``` + +# Trade-offs + +A homeserver can unbind any 3PID associated with one of its users, and +specifically does not require a re-validation of control of the 3PID. This means +that users have to trust that their homeserver will not arbitrarily remove valid +3PIDs, however users must already trust their homeserver to a large extent. The +flip side is that this provides a mechanism for homeservers and users to remove +3PIDs directed at their user IDs that they no longer (or never did) have control +over. + +Removing a 3PID does not require user interactive auth (UIA), which opens a +potential attack whereby a logged in device can remove all associated 3PIDs and +then log out all devices. If the user has forgotten their password they would no +longer be able to reset their password via a 3PID (e.g. email), resulting in +losing access to their account. However, given that clients and servers have +implemented these APIs in the wild this is considered a sufficient edge case +that adding UIA is unlikely to be worthwhile. From ee6513d608cf8004f67a9fbae1b69d02477719dd Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 7 Mar 2019 16:52:58 +0000 Subject: [PATCH 147/170] Add alternative sid/client_secret authentication --- proposals/1915-unbind-identity-server-param.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md index f018d261d..1dc2d8333 100644 --- a/proposals/1915-unbind-identity-server-param.md +++ b/proposals/1915-unbind-identity-server-param.md @@ -57,8 +57,11 @@ should assume that the identity server doesn't support the `/unbind` API, unless it returns a specific matrix error response (i.e. the body is a JSON object with `error` and `errcode` fields). -The identity server should accept any request to unbind a 3PID for a `user_id` from -the homeserver controlling that user ID. +The identity server should authenticate the request in one of two ways: + +1. The request is signed by the homeserver which controls the `user_id`. +2. The request includes the `sid` and `client_server` params (as per `/bind`), + which proves ownership of the given 3PID. Example: From 77110b46df7000d0ea78f2cdd4778c2122b7de73 Mon Sep 17 00:00:00 2001 From: David Baker Date: Fri, 8 Mar 2019 12:14:08 +0000 Subject: [PATCH 148/170] Update proposals/1915-unbind-identity-server-param.md Co-Authored-By: erikjohnston --- proposals/1915-unbind-identity-server-param.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md index 1dc2d8333..9ffb99b30 100644 --- a/proposals/1915-unbind-identity-server-param.md +++ b/proposals/1915-unbind-identity-server-param.md @@ -60,7 +60,7 @@ it returns a specific matrix error response (i.e. the body is a JSON object with The identity server should authenticate the request in one of two ways: 1. The request is signed by the homeserver which controls the `user_id`. -2. The request includes the `sid` and `client_server` params (as per `/bind`), +2. The request includes the `sid` and `client_secret` params (as per `/bind`), which proves ownership of the given 3PID. Example: From 60f1cf79de80221029234949483b0643f246df43 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 12 Mar 2019 14:20:37 +0000 Subject: [PATCH 149/170] Update wording and add deactivate account API --- .../1915-unbind-identity-server-param.md | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md index 9ffb99b30..8dbfe2e6c 100644 --- a/proposals/1915-unbind-identity-server-param.md +++ b/proposals/1915-unbind-identity-server-param.md @@ -13,16 +13,16 @@ This is meant as a simple extension to the current APIs, and so this explicitly does not try and solve any existing usability concerns. -## New APIs +## API Changes -### Client-Server API +### Client-Server 3PID Delete API -Add `POST /_matrix/client/r0/account/3pid/delete` API, which expects a JSON body -with `medium`, `address` and `id_server` fields (as per existing APIs). +Add an `id_server` param to `POST /_matrix/client/r0/account/3pid/delete` API, +which matches the 3PID creation APIs. -The `id_server` parameter is optional and if missing the server will attempt to -unbind from a suitable identity server (e.g. its default identity server or the -server used when originally binding the 3pid). +The new `id_server` parameter is optional and if missing the server will attempt +to unbind from a suitable identity server (e.g. its default identity server or +the server used when originally binding the 3pid). The 200 response is a JSON object with an `id_server_unbind_result` field whose value is either `success` or `no-support`, where the latter indicates that the @@ -45,8 +45,14 @@ HTTP/1.1 200 OK } ``` +### Client-Server Deactivate account API -### Identity Server API +Add an `id_server` param to `POST /_matrix/client/r0/account/deactivate` API, +with the same semantics as above. This is used to unbind any bound threepids +from the given identity server. + + +### Identity Server 3PID Unbind API Add `POST /_matrix/identity/api/v1/unbind` with `mxid` and `threepid` fields. The `mxid` is the user's `user_id` and `threepid` is a dict with the usual From 0e07a6d243f2fc11c1ce95504e842d2f96f24c00 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 Mar 2019 13:32:38 -0600 Subject: [PATCH 150/170] Proposal to have a push rule for m.room.tombstone events --- proposals/1930-tombstone-notifications.md | 40 +++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 proposals/1930-tombstone-notifications.md diff --git a/proposals/1930-tombstone-notifications.md b/proposals/1930-tombstone-notifications.md new file mode 100644 index 000000000..715921aa4 --- /dev/null +++ b/proposals/1930-tombstone-notifications.md @@ -0,0 +1,40 @@ +# Proposal to add a default push rule for m.room.tombstone events + +Currently users are unaware of when a room becomes upgraded, leaving them potentially in the old room +without knowing until they visit the room again. By having a notification for when the room is upgraded, +users are able to ensure they are able to stay relevant in rooms by joining the upgraded room. + + +## Proposal + +A new default override rule is to be added which is similar to `@room` notifications: + +```json +{ + "rule_id": ".m.rule.tombstone", + "default": true, + "enabled": true, + "conditions": [ + { + "kind": "event_match", + "key": "type", + "pattern": "m.room.tombstone" + } + ], + "actions": [ + "notify", + { + "set_tweak": "highlight", + "value": true + } + ] +} +``` + + +## Tradeoffs + +Clients could calculate this on their own and show some sort of "room upgraded" notification instead, +however by doing it this way it means that all clients would need to be aware of room upgrades. Having +a default push rule means that clients get this notification for free. Clients which want a more diverse +UX can still do so by ignoring this push rule locally. From bd4fab7a15e9086473714fd9d54656b396212467 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 18 Mar 2019 09:28:46 +0000 Subject: [PATCH 151/170] Be more explicit identity server selection and errors --- proposals/1915-unbind-identity-server-param.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md index 8dbfe2e6c..438dfa44f 100644 --- a/proposals/1915-unbind-identity-server-param.md +++ b/proposals/1915-unbind-identity-server-param.md @@ -21,12 +21,13 @@ Add an `id_server` param to `POST /_matrix/client/r0/account/3pid/delete` API, which matches the 3PID creation APIs. The new `id_server` parameter is optional and if missing the server will attempt -to unbind from a suitable identity server (e.g. its default identity server or -the server used when originally binding the 3pid). +to unbind from the identity server used when originally binding the 3pid (if +known by the homeserver). The 200 response is a JSON object with an `id_server_unbind_result` field whose value is either `success` or `no-support`, where the latter indicates that the -identity server (IS) does not support unbinding 3PIDs directly. +identity server (IS) does not support unbinding 3PIDs directly. If the identity +server returns an error than that should be returned to the client. Example: From 410a5dbbff6dc37132ee22ce2e0bf4df3aaa51a6 Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 18 Mar 2019 09:36:01 +0000 Subject: [PATCH 152/170] Update proposals/1915-unbind-identity-server-param.md Co-Authored-By: erikjohnston --- proposals/1915-unbind-identity-server-param.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md index 438dfa44f..053552f6f 100644 --- a/proposals/1915-unbind-identity-server-param.md +++ b/proposals/1915-unbind-identity-server-param.md @@ -27,7 +27,7 @@ known by the homeserver). The 200 response is a JSON object with an `id_server_unbind_result` field whose value is either `success` or `no-support`, where the latter indicates that the identity server (IS) does not support unbinding 3PIDs directly. If the identity -server returns an error than that should be returned to the client. +server returns an error then that should be returned to the client. Example: From 1a739ec97b54faa3e8719c1fdb271c0d3388f15a Mon Sep 17 00:00:00 2001 From: Vikingat-RAGE Date: Mon, 18 Mar 2019 21:16:52 +0000 Subject: [PATCH 153/170] E2E typo fix in spec --- specification/modules/end_to_end_encryption.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index a605875b4..76b36c4f9 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -291,8 +291,8 @@ v string **Required.** Version of the encrypted attachments ========= ========= ============================================================ Parameter Type Description ========= ========= ============================================================ -key string **Required.** Key type. Must be ``oct``. -key_opts [string] **Required.** Key operations. Must at least contain +kty string **Required.** Key type. Must be ``oct``. +key_ops [string] **Required.** Key operations. Must at least contain ``encrypt`` and ``decrypt``. alg string **Required.** Algorithm. Must be ``A256CTR``. k string **Required.** The key, encoded as urlsafe unpadded base64. From 76ee13b9bb4c65a8af1eaaba22e443c4648c8500 Mon Sep 17 00:00:00 2001 From: Vikingat-RAGE Date: Mon, 18 Mar 2019 21:54:56 +0000 Subject: [PATCH 154/170] Fixed bad spacing. --- changelogs/client_server/newsfragments/1933.clarification | 1 + specification/modules/end_to_end_encryption.rst | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/1933.clarification diff --git a/changelogs/client_server/newsfragments/1933.clarification b/changelogs/client_server/newsfragments/1933.clarification new file mode 100644 index 000000000..b0f052035 --- /dev/null +++ b/changelogs/client_server/newsfragments/1933.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 76b36c4f9..3b5bd4d9e 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -292,7 +292,7 @@ v string **Required.** Version of the encrypted attachments Parameter Type Description ========= ========= ============================================================ kty string **Required.** Key type. Must be ``oct``. -key_ops [string] **Required.** Key operations. Must at least contain +key_ops [string] **Required.** Key operations. Must at least contain ``encrypt`` and ``decrypt``. alg string **Required.** Algorithm. Must be ``A256CTR``. k string **Required.** The key, encoded as urlsafe unpadded base64. From 42cf72c60f9f5fe6036fc32f65c0cd5d2ed73b6e Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 Mar 2019 14:43:16 -0600 Subject: [PATCH 155/170] URL encode matrix.to URIs --- .../appendices/identifier_grammar.rst | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index e10348034..17c557d97 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -323,18 +323,27 @@ and instead should perform some sort of action within the client. For example, i the user were to click on a matrix.to URI for a room alias, the client may open a view for the user to participate in the room. +The components of the matrix.to URI (```` and ````) +are to be percent-encoded as per RFC 3986. + Examples of matrix.to URIs are: -* Room alias: ``https://matrix.to/#/#somewhere:example.org`` -* Room: ``https://matrix.to/#/!somewhere:example.org`` -* Permalink by room: ``https://matrix.to/#/!somewhere:example.org/$event:example.org`` +* Room alias: ``https://matrix.to/#/%23somewhere%3Aexample.org`` +* Room: ``https://matrix.to/#/!somewhere%3Aexample.org`` +* Permalink by room: ``https://matrix.to/#/!somewhere%3Aexample.org/%24event%3Aexample.org`` * Permalink by room alias: ``https://matrix.to/#/#somewhere:example.org/$event:example.org`` * User: ``https://matrix.to/#/@alice:example.org`` * Group: ``https://matrix.to/#/+example:example.org`` .. Note:: - Event IDs have the potential to contain slashes in some `room versions `_. - No component of the URI should be URL encoded. + Historically, clients have not produced URIs which are fully encoded. Clients should + try to interpret these cases to the best of their ability. For example, an unencoded + room alias should still work within the client if possible. + +.. Note:: + Clients should be aware that decoding a matrix.to URI may result in extra slashes + appearing due to some `room versions `_. These slashes + should normally be encoded when producing matrix.to URIs, however. .. Note:: Room ID permalinks are unroutable as there is no reliable domain to send requests From 0ec34039e85e777c2b50b532dac9c594513174fe Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 Mar 2019 14:44:54 -0600 Subject: [PATCH 156/170] URL encode all the things --- specification/appendices/identifier_grammar.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index 17c557d97..ad0794bde 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -331,9 +331,9 @@ Examples of matrix.to URIs are: * Room alias: ``https://matrix.to/#/%23somewhere%3Aexample.org`` * Room: ``https://matrix.to/#/!somewhere%3Aexample.org`` * Permalink by room: ``https://matrix.to/#/!somewhere%3Aexample.org/%24event%3Aexample.org`` -* Permalink by room alias: ``https://matrix.to/#/#somewhere:example.org/$event:example.org`` -* User: ``https://matrix.to/#/@alice:example.org`` -* Group: ``https://matrix.to/#/+example:example.org`` +* Permalink by room alias: ``https://matrix.to/#/#somewhere:example.org/%24event%3Aexample.org`` +* User: ``https://matrix.to/#/%40alice%3Aexample.org`` +* Group: ``https://matrix.to/#/%2Bexample%3Aexample.org`` .. Note:: Historically, clients have not produced URIs which are fully encoded. Clients should From d224c4ff715456cb1d65fbac0c82e1e2eaf39801 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 25 Mar 2019 18:13:14 -0400 Subject: [PATCH 157/170] fix some errors in key export format - empirically, we don't acatually wrap the array in an object - fix an incorrect type --- specification/modules/end_to_end_encryption.rst | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 3b5bd4d9e..59d241b67 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -496,18 +496,9 @@ passphrase, and is created as follows: Key export format <<<<<<<<<<<<<<<<< -The exported sessions are formatted as a JSON object of type ``ExportData`` +The exported sessions are formatted as a JSON array of ``SessionData`` objects described as follows: -``ExportData`` - -=============== ================= ============================================== -Parameter Type Description -=============== ================= ============================================== -sessions ``[SessionData]`` Required. The sessions that are being - exported. -=============== ================= ============================================== - ``SessionData`` .. table:: @@ -529,7 +520,7 @@ sessions ``[SessionData]`` Required. The sessions that are being device which initiated the session originally. sender_claimed_keys {string: Required. The Ed25519 key of the - integer} device which initiated the session + string} device which initiated the session originally. session_id string Required. The ID of the session. session_key string Required. The key for the session. From 207d6cf8512231eb121d9696369b1d1d01449673 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sat, 30 Mar 2019 13:12:18 +0000 Subject: [PATCH 158/170] update MSC1884 to reflect new conclusions following discussion on the PR --- .../1884-replace-slashes-in-event_ids.md | 80 ++++++++++++++++++- 1 file changed, 77 insertions(+), 3 deletions(-) diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md index 5d9d0bc23..9c3b7ea75 100644 --- a/proposals/1884-replace-slashes-in-event_ids.md +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -11,12 +11,20 @@ solution is to ensure that event IDs are URL-encoded, so that `/` is instead represented as `%2F`. However, this is not entirely satisfactory for a number of reasons: + * The act of escaping and unescaping slash characters when doing casual + development and ops work becomes an constant and annoying chore which + is entirely avoidable. Whenever using tools like `curl` and `grep` or + manipulating SQL, developers will have to constantly keep in mind whether + they are dealing with escaped or unescaped IDs, and manually convert between + the two as needed. This will only get worse with further keys-as-IDs + landing with MSC1228. + * There exist a number of client (and possibly server) implementations which do not currently URL-encode such parameters; these are therefore broken by such event IDs and must be updated. Furthermore, all future client implementers must remember to do the encoding correctly. - * Even if client implementations do rembember to URL-encode their parameters, + * Even if client implementations do remember to URL-encode their parameters, they may not do it correctly: many URL-encoding implementations may be intended to encode parameters in the query-string (which can of course contain literal slashes) rather tha the path component. @@ -27,6 +35,14 @@ of reasons: existing setups will be broken by this change, and it is a trap for new users of the software. + * Cosmetically, URL-escaping base64 in otherwise-constant-length IDs results + in variable length IDs, making it harder to visually scan lists of IDs and + manipulate them in columnar form when doing devops work. + + * Those developing against the CS API might reasonably expect us to use + URL-safe identifiers in URLs where available, rather than deliberately + choosing non-URL-safe IDs, which could be seen as developer-unfriendly. + ## Proposal This MSC proposes that we should introduce a new room version, in which event @@ -34,6 +50,22 @@ IDs are encoded using the [URL-safe Base64 encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and `_` as the 62nd and 63rd characters instead of `+` and `/`). +URL-safe Base64 encoding is then used consistently for encoding binary +identifiers in the CS API - particularly in upcoming MSC1228 IDs for rooms and +users, such that typical CS API developers should be able to safely assume +that for all common cases they should use URL-safe Base64 when decoding base64 +strings. + +The exception would be for E2EE data (device keys and signatures etc) which +currently use normal Base64 with no easy mechanism to migrate to a new encoding. +Given E2EE development is rare and requires expert skills, it seems acceptable +to expect E2EE developers to be able to use the right encoding without tripping +up significantly. + +Similarly, the S2S API could continue to use standard base64-encoded hashes and +signatures, given they are only exposed to S2S API developers who are necessarily +expert and should be able to correctly pick the right encoding. + ## Counterarguments 1. Inconsistency. Base64 encoding is used heavily elsewhere in the matrix @@ -45,6 +77,14 @@ encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and Changing event IDs alone would therefore leave us with a confusing mix of encodings. + However, the current uses of standard Base64 encodings are not exposed to + common CS API developers, and so whilst this might be slightly confusing + for the minority of expert homeserver developers, the confusion does not + exist today for client developers. Therefore it seems safe to standardise + on URL-safe Base64 for identifiers exposed to the client developers, who + form by far the majority of the Matrix ecosystem today, and expect as + simple an API as possible. + A potential extension would be to change *all* Base64 encodings to be URL-safe. This would address the inconsistency. However, it feels like a large job which would span the entire matrix ecosystem (far larger than @@ -70,6 +110,16 @@ encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and Of course, an alternative is to modify the grammars of all of these identifiers to forbid slashes. + The counter-counterargument to this is that it is of course best practice + for implementations is to URL-escape any IDs used in URLs, and human-selected + IDs such as Room aliases, Group IDs, Matrix user IDs etc apply an adequate + forcing function already to remind developers to do this. However, + it doesn't follow that we should then also deliberately pick URL-unsafe + encodings for machine-selected IDs - the argument that it is better for software + to fail 50% of the time to force a fix is irrelevant when the possibility + exists for the software to fail 0% of the time in the first place by picking + an identifier format which cannot fail. + [1] Discussion remains open as to whether allowing slashes in User IDs was a good idea. @@ -87,5 +137,29 @@ solutions to that are also possible). ## Conclusion -It's unclear to me that changing the format of event IDs alone solves any -problems. +There are two main questions here: + + 1. Whether it's worth forcing casual CS API developers to juggle escaping of + machine-selected IDs in order to remind them to escape all variables in + their URIs correctly. + + 2. Whether it's a significant problem for E2EE & SS API developers to have to + handle strings which are a mix of standard Base64 and URL-safe Base64 + encodings. + +Both of these are a subjective judgement call. + +Given we wish the CS API particularly to be as easy for casual developers to +use as possible, it feels that we should find another way to encourage +developers to escape variables in their URLs in general - e.g. by recommending +that developers test their clients against a 'torture room' full of exotic IDs +and data, or by improving warnings in the spec... rather than (ab)using +machine-selected IDs as a reminder. + +Meanwhile, given we have many more CS API developers than SS or E2EE developers, +and we wish to make the CS API particularly easy for casual developers to use, +it feels we should not prioritise consistency of encodings for SS/E2EE developers +over the usability of the CS API. + +Therefore, on balance, it seems plausible that changing the format of event IDs +does solve sufficient problems to make it desirable. \ No newline at end of file From 9dcf2d6a2805e4c81b5c04af71034a10cb68282a Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 1 Apr 2019 00:43:31 +0100 Subject: [PATCH 159/170] Update proposals/1884-replace-slashes-in-event_ids.md Co-Authored-By: ara4n --- proposals/1884-replace-slashes-in-event_ids.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md index 9c3b7ea75..ae2c0e948 100644 --- a/proposals/1884-replace-slashes-in-event_ids.md +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -27,7 +27,7 @@ of reasons: * Even if client implementations do remember to URL-encode their parameters, they may not do it correctly: many URL-encoding implementations may be intended to encode parameters in the query-string (which can of course - contain literal slashes) rather tha the path component. + contain literal slashes) rather than the path component. * Some proxy software may treat `%2F` specially: for instance, Apache, when configured as a reverse-proxy, will reject requests for a path containing @@ -162,4 +162,4 @@ it feels we should not prioritise consistency of encodings for SS/E2EE developer over the usability of the CS API. Therefore, on balance, it seems plausible that changing the format of event IDs -does solve sufficient problems to make it desirable. \ No newline at end of file +does solve sufficient problems to make it desirable. From 417f3a3e8b1b0510513306139ba1fd05f39e295e Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 1 Apr 2019 00:58:10 +0100 Subject: [PATCH 160/170] incorporate further feedback from vdh --- .../1884-replace-slashes-in-event_ids.md | 44 +++++++++---------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md index ae2c0e948..9f1a2ad08 100644 --- a/proposals/1884-replace-slashes-in-event_ids.md +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -11,8 +11,8 @@ solution is to ensure that event IDs are URL-encoded, so that `/` is instead represented as `%2F`. However, this is not entirely satisfactory for a number of reasons: - * The act of escaping and unescaping slash characters when doing casual - development and ops work becomes an constant and annoying chore which + * The act of escaping and unescaping slash characters when manually calling + the API during devops work becomes an constant and annoying chore which is entirely avoidable. Whenever using tools like `curl` and `grep` or manipulating SQL, developers will have to constantly keep in mind whether they are dealing with escaped or unescaped IDs, and manually convert between @@ -50,11 +50,10 @@ IDs are encoded using the [URL-safe Base64 encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and `_` as the 62nd and 63rd characters instead of `+` and `/`). -URL-safe Base64 encoding is then used consistently for encoding binary -identifiers in the CS API - particularly in upcoming MSC1228 IDs for rooms and -users, such that typical CS API developers should be able to safely assume -that for all common cases they should use URL-safe Base64 when decoding base64 -strings. +We will then aim to use URL-safe Base64 encoding across Matrix in future, +such that typical CS API developers should be able to safely assume +that for all common cases (including upcoming MSC1228 identifiers) they should +use URL-safe Base64 when decoding base64 strings. The exception would be for E2EE data (device keys and signatures etc) which currently use normal Base64 with no easy mechanism to migrate to a new encoding. @@ -63,8 +62,9 @@ to expect E2EE developers to be able to use the right encoding without tripping up significantly. Similarly, the S2S API could continue to use standard base64-encoded hashes and -signatures, given they are only exposed to S2S API developers who are necessarily -expert and should be able to correctly pick the right encoding. +signatures in the places it does today, given they are only exposed to S2S API +developers who are necessarily expert and should be able to correctly pick the +right encoding. ## Counterarguments @@ -81,9 +81,9 @@ expert and should be able to correctly pick the right encoding. common CS API developers, and so whilst this might be slightly confusing for the minority of expert homeserver developers, the confusion does not exist today for client developers. Therefore it seems safe to standardise - on URL-safe Base64 for identifiers exposed to the client developers, who - form by far the majority of the Matrix ecosystem today, and expect as - simple an API as possible. + (except those implementing E2EE) on URL-safe Base64 for identifiers exposed + to the client developers, who form by far the majority of the Matrix + ecosystem today, and expect as simple an API as possible. A potential extension would be to change *all* Base64 encodings to be URL-safe. This would address the inconsistency. However, it feels like a @@ -139,7 +139,7 @@ solutions to that are also possible). There are two main questions here: - 1. Whether it's worth forcing casual CS API developers to juggle escaping of + 1. Whether it's worth forcing manual CS API users to juggle escaping of machine-selected IDs in order to remind them to escape all variables in their URIs correctly. @@ -149,17 +149,17 @@ There are two main questions here: Both of these are a subjective judgement call. -Given we wish the CS API particularly to be as easy for casual developers to -use as possible, it feels that we should find another way to encourage -developers to escape variables in their URLs in general - e.g. by recommending -that developers test their clients against a 'torture room' full of exotic IDs -and data, or by improving warnings in the spec... rather than (ab)using +Given we wish the CS API particularly to be as easy as possible for manual +use, it feels that we should find another way to encourage developers to +escape variables in their URLs in general - e.g. by recommending that +developers test their clients against a 'torture room' full of exotic IDs and +data, or by improving warnings in the spec... rather than (ab)using machine-selected IDs as a reminder. -Meanwhile, given we have many more CS API developers than SS or E2EE developers, -and we wish to make the CS API particularly easy for casual developers to use, -it feels we should not prioritise consistency of encodings for SS/E2EE developers -over the usability of the CS API. +Meanwhile, given we have many more CS API developers than SS or E2EE +developers, and we wish to make the CS API particularly easy for developers to +manually invoke, it feels we should not prioritise consistency of encodings +for SS/E2EE developers over the usability of the CS API. Therefore, on balance, it seems plausible that changing the format of event IDs does solve sufficient problems to make it desirable. From 88f533f0db33518e48c489872d8cbc68a204ab2c Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 1 Apr 2019 10:27:30 +0100 Subject: [PATCH 161/170] incorporate further feedback --- .../1884-replace-slashes-in-event_ids.md | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md index 9f1a2ad08..bec8d7adc 100644 --- a/proposals/1884-replace-slashes-in-event_ids.md +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -80,10 +80,10 @@ right encoding. However, the current uses of standard Base64 encodings are not exposed to common CS API developers, and so whilst this might be slightly confusing for the minority of expert homeserver developers, the confusion does not - exist today for client developers. Therefore it seems safe to standardise - (except those implementing E2EE) on URL-safe Base64 for identifiers exposed - to the client developers, who form by far the majority of the Matrix - ecosystem today, and expect as simple an API as possible. + exist today for client developers (except those implementing E2EE). + Therefore it seems safe to standardise on URL-safe Base64 for identifiers + exposed to the client developers, who form by far the majority of the + Matrix ecosystem today, and expect as simple an API as possible. A potential extension would be to change *all* Base64 encodings to be URL-safe. This would address the inconsistency. However, it feels like a @@ -139,9 +139,9 @@ solutions to that are also possible). There are two main questions here: - 1. Whether it's worth forcing manual CS API users to juggle escaping of - machine-selected IDs in order to remind them to escape all variables in - their URIs correctly. + 1. Whether it's worth forcing CS API developers to juggle escaping of + machine-selected IDs during manual use of the API in order to remind them + to escape all variables in their URIs correctly when writing code. 2. Whether it's a significant problem for E2EE & SS API developers to have to handle strings which are a mix of standard Base64 and URL-safe Base64 @@ -156,10 +156,11 @@ developers test their clients against a 'torture room' full of exotic IDs and data, or by improving warnings in the spec... rather than (ab)using machine-selected IDs as a reminder. -Meanwhile, given we have many more CS API developers than SS or E2EE -developers, and we wish to make the CS API particularly easy for developers to -manually invoke, it feels we should not prioritise consistency of encodings -for SS/E2EE developers over the usability of the CS API. +Meanwhile, given we have many more people manually invoking the CS API than +developing on the SS or E2EE APIs, and we wish to make the CS API particularly +easy for developers to manually invoke, it feels we should not prioritise +consistency of encodings for SS/E2EE developers over the usability of the CS +API. Therefore, on balance, it seems plausible that changing the format of event IDs does solve sufficient problems to make it desirable. From 743eeca27a821b0939780d4fd3713c3f514b3660 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Fri, 5 Apr 2019 15:19:22 +0100 Subject: [PATCH 162/170] MSC to remove prev_content from the essential keys list --- ...ove-prev_event-from-essential-keys-list.md | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 proposals/1954-remove-prev_event-from-essential-keys-list.md diff --git a/proposals/1954-remove-prev_event-from-essential-keys-list.md b/proposals/1954-remove-prev_event-from-essential-keys-list.md new file mode 100644 index 000000000..992b3e802 --- /dev/null +++ b/proposals/1954-remove-prev_event-from-essential-keys-list.md @@ -0,0 +1,64 @@ +# Remove prev_content from the essential keys list + +Matrix supports the concept of event redaction. The ability to redact rather +than delete is necessary because some events e.g. membership events are +essential to the protocol and _cannot_ be deleted. Therefore we do not delete +events outright and instead redact them. This involves removing all keys from +an event that are not required by the protocol. The stripped down event is +thereafter returned anytime a client or remote server requests it. + + +## Proposal + +[The redaction algorithm](https://matrix.org/docs/spec/client_server/r0.4.0.html#redactions) +defines which keys must be retained through a redaction. Currently it lists +```prev_content``` as a key to retain, though in practice there is no need to +do so at the protocol level. + +The proposal is simply to remove ```prev_content``` from the essential keys +list. + +Note: the inclusion of ```prev_content``` in the essential keys list was +unintentional and should be considered a spec bug. Synapse (and other server +implementations) have not implemented the bug and already omit +```prev_content``` from redacted events. + + +## Tradeoffs + +When sending events over federation the events are [hashed and +signed](https://matrix.org/docs/spec/server_server/unstable.html#adding-hashes-and-signatures-to-outgoing-events), +this involves operating not only on the original event but also the redacted +form of the event. The redacted hash and redacted signed event are necessary if +the event is ever redacted in future. As a result, any change of the essential +keys list must be managed carefully. If disparate servers implement different +versions of the redaction algorithm (for a given event) attempts to send the +event over federation will fail. + +We _could_ manage this change via room versioning and create a new room +version that implements this MSC. However, because the federation already +omits the ```prev_content``` key by convention, implementing this MSC only in +the new room version would mean that the entire existing federation would not +be spec compliant. + +As a result it seems pragmatic to have the spec reflect reality, acknowledge +that the spec and federation have deviated and instead update the spec +retrospectively to describe the de-facto redaction algorithm. + +## Potential issues + +It is theoretically possible that a closed federation could exist whose servers +do follow the spec as is. This MSC would render those servers uncompliant with +the spec. On balance this seems unlikely and in the worst case those +implementors could add the change to a subsequent room version, eventually +reaching spec consistency as older room versions are deprecated. + +## Security considerations + +I am unaware of any security issues related to this proposal, but can certainly +see issues with a precedent that the federation deviates from the spec. + +## Conclusions +Removing ```prev_content``` is pragmatic response to the current situation. It +alligns the federation and the spec, and does so in a way that removes +unecessary overhead. From 8ad82ce3c0783cacb29f0730546f738acaacd92a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 5 Apr 2019 16:24:51 -0600 Subject: [PATCH 163/170] Add permalink routing through ?via parameters on matrix.to URIs Spec for [MSC1704](https://github.com/matrix-org/matrix-doc/pull/1704) Reference implementations: * Original: https://github.com/matrix-org/matrix-react-sdk/pull/2250 * Modern recommendations: https://github.com/matrix-org/matrix-react-sdk/blob/2ca281f6b7b735bc96945370584c5cb1b5f7e1f1/src/matrix-to.js#L29-L70 The only deviation from the original MSC is the recommendation for which servers to pick. The original MSC failed to consider server ACLs and IP addresses correctly, and during implementation it was realized that both of these cases should be handled. The core principles of the original MSC are left unaltered. --- .../appendices/identifier_grammar.rst | 61 +++++++++++++++++-- 1 file changed, 55 insertions(+), 6 deletions(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index 729246d07..ea805955d 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -311,13 +311,16 @@ in the room's history (a permalink). A matrix.to URI has the following format, based upon the specification defined in RFC 3986: - https://matrix.to/#// + https://matrix.to/#//? The identifier may be a room ID, room alias, user ID, or group ID. The extra parameter is only used in the case of permalinks where an event ID is referenced. The matrix.to URI, when referenced, must always start with ``https://matrix.to/#/`` followed by the identifier. +The ```` and the preceeding question mark are optional and +only apply in certain circumstances, documented below. + Clients should not rely on matrix.to URIs falling back to a web server if accessed and instead should perform some sort of action within the client. For example, if the user were to click on a matrix.to URI for a room alias, the client may open @@ -331,7 +334,7 @@ Examples of matrix.to URIs are: * Room alias: ``https://matrix.to/#/%23somewhere%3Aexample.org`` * Room: ``https://matrix.to/#/!somewhere%3Aexample.org`` * Permalink by room: ``https://matrix.to/#/!somewhere%3Aexample.org/%24event%3Aexample.org`` -* Permalink by room alias: ``https://matrix.to/#/#somewhere:example.org/%24event%3Aexample.org`` +* Permalink by room alias: ``https://matrix.to/#/%23somewhere:example.org/%24event%3Aexample.org`` * User: ``https://matrix.to/#/%40alice%3Aexample.org`` * Group: ``https://matrix.to/#/%2Bexample%3Aexample.org`` @@ -345,7 +348,53 @@ Examples of matrix.to URIs are: appearing due to some `room versions `_. These slashes should normally be encoded when producing matrix.to URIs, however. -.. Note:: - Room ID permalinks are unroutable as there is no reliable domain to send requests - to upon receipt of the permalink. Clients should do their best route Room IDs to - where they need to go, however they should also be aware of `issue #1579 `_. +Routing +<<<<<<< + +Room IDs are not routable on their own as there is no reliable domain to send requests +to. This is partially mitigated with the addition of a ``via`` argument on a matrix.to +URI, however the problem of routability is still present. Clients should do their best +to route Room IDs to where they need to go, however they should also be aware of +`issue #1579 `_. + +A room (or room permalink) which isn't using a room alias should supply at least one +server using ``via`` in the ````, like so: +``https://matrix.to/!somewhere%3Aexample.org?via=example.org&via=alt.example.org``. The +parameter can be supplied multiple times to specify multiple servers to try. + +The values of ``via`` are intended to be passed along as the ``server_name`` parameters +on the Client Server ``/join`` API. + +When generating room links and permalinks, the application should pick servers which +have a high probability of being in the room in the distant future. How these servers +are picked is left as an implementation detail, however the current recommendation is +to pick 3 unique servers based on the following criteria: + +* The first server should be the server of the highest power level user in the room, + provided they are at least power level 50. If no user meets this criteria, pick the + most popular server in the room (most joined users). The rationale for not picking + users with power levels under 50 is that they are unlikely to be around into the + distant future while higher ranking users (and therefore servers) are less likely + to give up their power and move somewhere else. Most rooms in the public federation + have a power level 100 user and have not deviated from the default structure where + power level 50 users have moderator-style privileges. + +* The second server should be the next highest server by population, or the first + highest by population if the first server was based on a user's power level. The + rationale for picking popular servers is that the server is unlikely to be removed + as the room naturally grows in membership due to that server joining users. The + server could be refused participation in the future due to server ACLs or similar, + however the chance of that happening to a server which is organically joining the + room is unlikely. + +* The third server should be the next highest server by population. + +* Servers which are blocked due to server ACLs should never be chosen. + +* Servers which are IP addresses should never be chosen. Servers which use a domain + name are less likely to be unroutable in the future whereas IP addresses cannot be + pointed to a different location and therefore higher risk options. + +* All 3 servers should be unique from each other. If the room does not have enough users + to supply 3 servers, the application should only specify the servers it can. For example, + a room with only 2 users in it would result in maximum 2 ``via`` parameters. From 99cd064f2623ce762319f571c10b79912db3bc68 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 5 Apr 2019 16:37:27 -0600 Subject: [PATCH 164/170] Update original MSC as per proposal guidelines --- proposals/1704-matrix.to-permalinks.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/1704-matrix.to-permalinks.md b/proposals/1704-matrix.to-permalinks.md index 1430565cf..7e1a2b28d 100644 --- a/proposals/1704-matrix.to-permalinks.md +++ b/proposals/1704-matrix.to-permalinks.md @@ -33,4 +33,7 @@ is to pick up to 3 unique servers where the first one is that of the user with t highest power level in the room, provided that power level is 50 or higher. The other 2 servers should be the most popular servers in the room based on the number of joined users. This same heuristic should apply to the first server if no user meets the power -level requirements. +level requirements. Servers blocked by server ACLs should not be picked because they +are unlikely to continue being residents of the room. Similarly, IP addresses should +not be picked because they cannot be redirected to another location like domain names +can, making them a higher risk option. From b41fbc86b6d07a93c10f074c97befe9fe7e4733e Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Tue, 9 Apr 2019 13:56:45 +0100 Subject: [PATCH 165/170] add further potential issues and security concerns --- ...emove-prev_event-from-essential-keys-list.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/proposals/1954-remove-prev_event-from-essential-keys-list.md b/proposals/1954-remove-prev_event-from-essential-keys-list.md index 992b3e802..1cd0fd361 100644 --- a/proposals/1954-remove-prev_event-from-essential-keys-list.md +++ b/proposals/1954-remove-prev_event-from-essential-keys-list.md @@ -53,10 +53,23 @@ the spec. On balance this seems unlikely and in the worst case those implementors could add the change to a subsequent room version, eventually reaching spec consistency as older room versions are deprecated. +Another scenario is that a client may redact events according to the spec as is +and persist prev_content through the redaction, thereby diverting from that on +the server(s). Client authors will have to update their code to drop +```prev_content``` - however, given that prev_content should not be used in +important calculations and/or visualisations, this ought to be relatively +uninvaisive change. + + ## Security considerations -I am unaware of any security issues related to this proposal, but can certainly -see issues with a precedent that the federation deviates from the spec. +A further reason to support removal of ```prev_content``` is the case where a +malicious user adds illegal or abusive content into a state event and then +overwrites that state event. The content would then be preserved through the +redaction. + +Additionally, there are plenty of reasons to have security concerns over a +precedent that the federation can deviate from the spec. ## Conclusions Removing ```prev_content``` is pragmatic response to the current situation. It From 24e0ec4bcecee55e34d54bd8a747f619b57384b5 Mon Sep 17 00:00:00 2001 From: aqtusia <47819902+aqtusia@users.noreply.github.com> Date: Sun, 14 Apr 2019 18:17:44 +0200 Subject: [PATCH 166/170] Replace /bind with /3pid/bind --- api/identity/associations.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/identity/associations.yaml b/api/identity/associations.yaml index edd43f5d9..152a0a9b9 100644 --- a/api/identity/associations.yaml +++ b/api/identity/associations.yaml @@ -90,7 +90,7 @@ paths: } schema: $ref: "../client-server/definitions/errors/error.yaml" - "/bind": + "/3pid/bind": post: summary: Publish an association between a session and a Matrix user ID. description: |- From 2eb9708f7fcdead5e21ad22bd8a054e1ae4bfd31 Mon Sep 17 00:00:00 2001 From: aqtusia <47819902+aqtusia@users.noreply.github.com> Date: Sun, 14 Apr 2019 18:21:59 +0200 Subject: [PATCH 167/170] Replace /unbind with /3pid/unbind --- proposals/1915-unbind-identity-server-param.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md index 053552f6f..6817ece3d 100644 --- a/proposals/1915-unbind-identity-server-param.md +++ b/proposals/1915-unbind-identity-server-param.md @@ -55,12 +55,12 @@ from the given identity server. ### Identity Server 3PID Unbind API -Add `POST /_matrix/identity/api/v1/unbind` with `mxid` and `threepid` fields. +Add `POST /_matrix/identity/api/v1/3pid/unbind` with `mxid` and `threepid` fields. The `mxid` is the user's `user_id` and `threepid` is a dict with the usual `medium` and `address` fields. If the server returns a 400, 404 or 501 HTTP error code then the homeserver -should assume that the identity server doesn't support the `/unbind` API, unless +should assume that the identity server doesn't support the `/3pid/unbind` API, unless it returns a specific matrix error response (i.e. the body is a JSON object with `error` and `errcode` fields). @@ -73,7 +73,7 @@ The identity server should authenticate the request in one of two ways: Example: ``` -POST /_matrix/identity/api/v1/unbind HTTP/1.1 +POST /_matrix/identity/api/v1/3pid/unbind HTTP/1.1 { "mxid": "@foobar:example.com", From 911fb94ea0d7213ea363293c1db39ca4391669f7 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Mon, 15 Apr 2019 17:08:09 +0100 Subject: [PATCH 168/170] typos --- .../1954-remove-prev_event-from-essential-keys-list.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/1954-remove-prev_event-from-essential-keys-list.md b/proposals/1954-remove-prev_event-from-essential-keys-list.md index 1cd0fd361..735fac012 100644 --- a/proposals/1954-remove-prev_event-from-essential-keys-list.md +++ b/proposals/1954-remove-prev_event-from-essential-keys-list.md @@ -48,7 +48,7 @@ retrospectively to describe the de-facto redaction algorithm. ## Potential issues It is theoretically possible that a closed federation could exist whose servers -do follow the spec as is. This MSC would render those servers uncompliant with +do follow the spec as is. This MSC would render those servers non-compliant with the spec. On balance this seems unlikely and in the worst case those implementors could add the change to a subsequent room version, eventually reaching spec consistency as older room versions are deprecated. @@ -57,8 +57,8 @@ Another scenario is that a client may redact events according to the spec as is and persist prev_content through the redaction, thereby diverting from that on the server(s). Client authors will have to update their code to drop ```prev_content``` - however, given that prev_content should not be used in -important calculations and/or visualisations, this ought to be relatively -uninvaisive change. +important calculations and/or visualisations, this ought to be a relatively +non-invasive change. ## Security considerations From 043dddc49061f4a8837a0b9d5dc0fb0c0d79f85d Mon Sep 17 00:00:00 2001 From: Jonas Platte Date: Sat, 20 Apr 2019 21:56:43 +0200 Subject: [PATCH 169/170] Fix a typo in m.call.invite --- event-schemas/schema/m.call.invite | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/event-schemas/schema/m.call.invite b/event-schemas/schema/m.call.invite index ebf09267b..65796e1e9 100644 --- a/event-schemas/schema/m.call.invite +++ b/event-schemas/schema/m.call.invite @@ -10,7 +10,7 @@ "properties": { "call_id": { "type": "string", - "description": "A unique identifer for the call." + "description": "A unique identifier for the call." }, "offer": { "type": "object", From 6cdc8982fa84fc25b6ea9d20d241df8e2795b643 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Fri, 26 Apr 2019 15:58:31 +0100 Subject: [PATCH 170/170] jenkins is dead, long live buildkite --- .buildkite/pipeline.yaml | 11 +++++++++++ jenkins.sh | 3 --- scripts/requirements.txt | 8 ++++++-- 3 files changed, 17 insertions(+), 5 deletions(-) create mode 100644 .buildkite/pipeline.yaml delete mode 100755 jenkins.sh diff --git a/.buildkite/pipeline.yaml b/.buildkite/pipeline.yaml new file mode 100644 index 000000000..e98d70265 --- /dev/null +++ b/.buildkite/pipeline.yaml @@ -0,0 +1,11 @@ +steps: + - label: ":books: Build spec" + command: + - python3 -m venv env + - env/bin/pip install -r scripts/requirements.txt + - ". env/bin/activate; scripts/generate-matrix-org-assets" + artifact_paths: + - assets.tar.gz + plugins: + - docker#v3.0.1: + image: "python:3.6" diff --git a/jenkins.sh b/jenkins.sh deleted file mode 100755 index 79b77acb5..000000000 --- a/jenkins.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/sh - -exec ./scripts/test-and-build.sh diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 2a7d7ff85..66027f91e 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -4,8 +4,12 @@ docutils >= 0.14 pygments >= 2.2.0 Jinja2 >= 2.9.6 -jsonschema >= 2.6.0 + +# jsonschema 3.0.0 objects to the $refs in our schema file. TODO: figure out +# why. +jsonschema >= 2.6.0, < 3.0.0 + PyYAML >= 3.12 requests >= 2.18.4 towncrier == 18.6.0 -six >= 1.11.0 \ No newline at end of file +six >= 1.11.0