Specify .well-known s2s discovery and X.509 validation

Original proposals:
* https://github.com/matrix-org/matrix-doc/pull/1708 (note: the JSON requirements were softened by https://github.com/matrix-org/matrix-doc/pull/1824)
* https://github.com/matrix-org/matrix-doc/pull/1711

Implementation proofs:
* https://github.com/matrix-org/synapse/pull/4489
* No explicit PRs for MSC1711 could be found, however Synapse is known to implement it.

There are no intentional changes which differ from the proposals in this commit, however the author has relied upon various historical conversations outside of the proposals to gain the required context. Inaccuracies introduced by the author are purely accidental.

api/server-server/definitions/keys.yaml

@@ -90,17 +90,6 @@ properties:
       additionalProperties:
         type: string
         name: Encoded Signature Verification Key
-  tls_fingerprints:
-    type: array
-    description: Hashes of X.509 TLS certificates used by this server.
-    items:
-      type: object
-      title: TLS Fingerprint
-      properties:
-        sha256:
-          type: string
-          description: The `Unpadded Base64`_ encoded fingerprint.
-          example: "VGhpcyBpcyBoYXNoIHdoaWNoIHNob3VsZCBiZSBieXRlcw"
   valid_until_ts:
     type: integer
     format: int64

api/server-server/examples/server_key.json

@@ -16,8 +16,5 @@
             "ed25519:auto2": "VGhpcyBzaG91bGQgYWN0dWFsbHkgYmUgYSBzaWduYXR1cmU"
         }
     },
-    "tls_fingerprints": [{
-        "sha256": "VGhpcyBpcyBoYXNoIHdoaWNoIHNob3VsZCBiZSBieXRlcw"
-    }],
     "valid_until_ts": 1652262000000
 }

api/server-server/wellknown.yaml

@@ -0,0 +1,53 @@
+# Copyright 2019 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+swagger: '2.0'
+info:
+  title: "Matrix Federation Server Discovery API"
+  version: "1.0.0"
+host: localhost:443
+schemes:
+  - https
+basePath: /.well-known
+produces:
+  - application/json
+paths:
+  "/matrix/server":
+    get:
+      summary: Gets information about the delegated server for server-server communication.
+      description: |-
+        Gets information about the delegated server for server-server communication
+        between Matrix homeservers. Servers should follow 30x redirects, carefully
+        avoiding redirect loops, and use normal X.509 certificate validation.
+      responses:
+        200:
+          description:
+            The delegated server information. The ``Content-Type`` for this response SHOULD
+            be ``application/json``, however servers parsing the response should assume that
+            the body is JSON regardless of type. Failures parsing the JSON or invalid data
+            provided in the resulting parsed JSON must result in server discovery failure (no
+            attempts should be made to continue finding an IP address/port number to connect
+            to).
+          examples:
+            application/json: {
+              "m.server": "delegated.example.com:1234"
+            }
+          schema:
+            type: object
+            properties:
+              "m.server":
+                type: string
+                description: |-
+                  The server name to delegate server-server communciations to, with optional
+                  port. The delegated server name uses the same grammar as
+                  `server names in the appendices <../appendices.html#server-name>`_.

specification/server_server_api.rst

@@ -90,35 +90,94 @@ Server discovery
 Resolving server names
 ~~~~~~~~~~~~~~~~~~~~~~
 
-Each matrix homeserver is identified by a server name consisting of a hostname
+Each Matrix homeserver is identified by a server name consisting of a hostname
 and an optional port, as described by the `grammar
-<../appendices.html#server-name>`_.  Server names should be resolved to an IP
+<../appendices.html#server-name>`_. Where applicable, a delegated server name
-address and port using the following process:
+uses the same grammar.
-
+
-* If the hostname is an IP literal, then that IP address should be used,
+Server names are resolved to an IP address and port to connect to, and have
-  together with the given port number, or 8448 if no port is given.
+various conditions affecting which certificates and ``Host`` headers to send.
-
+The process overall is as follows:
-* Otherwise, if the port is present, then an IP address is discovered by
+
-  looking up an AAAA or A record for the hostname, and the specified port is
+.. Note from the author: The repetitive "use this Host header and this cert"
-  used.
+   comments are intentional. The process is overall quite complicated, and
-
+   explaining explicitly what requests look like at each step helps ease the
-* If the hostname is not an IP literal and no port is given, the server is
+   understanding and ensure everyone is on the same page. Implementations
-  discovered by first looking up a ``_matrix._tcp`` SRV record for the
+   are of course welcome to realize where the process can be optimized, and
-  hostname, which may give a hostname (to be looked up using AAAA or A queries)
+   do so - just ensure that the result is the same!
-  and port.  If the SRV record does not exist, then the server is discovered by
+
-  looking up an AAAA or A record on the hostname and taking the default
+1. If the hostname is an IP literal, then that IP address should be used,
-  fallback port number of 8448.
+   together with the given port number, or 8448 if no port is given. A
-
+   valid TLS certificate must be provided by the target server for the
-  Homeservers may use SRV records to load balance requests between multiple TLS
+   IP address on all requests. Requests must be made with a ``Host``
-  endpoints or to failover to another endpoint if an endpoint fails.
+   header containing the IP address, without port.
-
+
-When making requests to servers, use the hostname of the target server in the
+2. If the hostname is not an IP literal, a server is found by resolving
-``Host`` header, regardless of any hostname given in the SRV record. For
+   an SRV record for ``_matrix._tcp.<hostname>``. This may result in
-example, if the server name is ``example.org``, and the SRV record resolves to
+   a hostname (to be resolved using AAAA or A records) and port. Requests
-``matrix.example.org``, the ``Host`` header in the request should be
+   are made to the resolved IP address and port, using 8448 as a default
-``example.org``.  If an explicit port was given in the server name, it should be
+   port, with a ``Host`` header of ``<hostname>``. A valid TLS certificate
-included in the ``Host`` header; otherwise, no port number should be given in
+   for ``<hostname>`` must be provided by the target server on all requests.
-the ``Host`` header.
+
+3. If the SRV record yielded no results, a ``/.well-known`` request is
+   made to the hostname (using port 443 exclusively, ignoring the port
+   provided in the server name). The target must present a valid TLS
+   certificate for the hostname, and a ``Host`` header containing the
+   hostname is used to make the request. The schema of the ``/.well-known``
+   request is later in this section. Assuming the response is valid,
+   the ``m.server`` property is parsed as ``<delegated_server_name>[:<delegated_port>]``
+   and processed as follows:
+
+   * If ``<delegated_server_name>`` is an IP literal, then that IP address
+     should be used together with the ``<delegated_port>`` or 8448 if no
+     port is provided. A valid TLS certificate must be provided by the
+     target server for that IP address. Requests must be made with a
+     ``Host`` header containing the IP address, without port.
+
+   * If ``<delegated_server_name>`` is not an IP literal, and ``<delegated_port>``
+     is present, an IP address is disovered by looking up an AAAA or A
+     record for ``<delegated_server_name>``. The resulting IP address is
+     used, alongside the ``<delegated_port>``, to make requests with a
+     ``Host`` header of ``<delegated_server_name>``. A valid TLS certificate
+     must be provided by the target server for ``<delegated_server_name>``.
+
+   * If ``<delegated_server_name>`` is not an IP literal and no
+     ``<delegated_port>`` is present, an SRV record is looked up for
+     ``_matrix._tcp.<delegated_server_name>``. This may result in another
+     hostname (to be resolved using AAAA or A records) and port. Requests
+     should be made to the resolved IP address and port with a ``Host``
+     header containing the ``<delegated_server_name>``. Additionally, a
+     valid TLS certificate must be provided by the target server for the
+     ``<delegated_server_name>``.
+
+      * If no SRV record is found, an IP address is resolved using AAAA
+        or A records. Requests are then made to the resolve IP address
+        and a port of 8448, using a ``Host`` header of ``<delegated_server_name>``.
+        A valid TLS certificate for ``<delegated_server_name>`` must be
+        provided by the target server.
+
+4. If the `/.well-known` request was invalid or returned an error response,
+   and the SRV record was not found, an IP address is resolved using AAAA
+   and A records. Requests are made to the resolved IP address using port
+   8448 and a ``Host`` header containing the ``<hostname>``. A valid TLS
+   certificate for ``<hostname>`` must be provided by the target server
+   on all requests.
+
+
+The TLS certificate provided by the target server must be present on all
+requests made to the server. The TLS certificate must be signed by a known
+Certificate Authority. Servers are ultimately responsible for determining
+the trusted Certificate Authorities, however are strongly encouraged to
+rely on the operating system's judgement. Servers can offer administrators
+a means to override the trusted authorities list. Servers can additionally
+skip the certificate validation for a given whitelist of domains or netmasks
+for the purposes of testing or in networks where verification is done
+elsewhere, such as with ``.onion`` addresses.
+
+Servers are encouraged to make use of the
+`Certificate Transparency <https://www.certificate-transparency.org/>`_ project.
+
+{{wellknown_ss_http_api}}
 
 Server implementation
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -130,9 +189,16 @@ Retrieving server keys
 
 .. NOTE::
   There was once a "version 1" of the key exchange. It has been removed from the
-  specification due to lack of significance. It may be reviewed `here
+  specification due to lack of significance. It may be reviewed `from the historical draft
   <https://github.com/matrix-org/matrix-doc/blob/51faf8ed2e4a63d4cfd6d23183698ed169956cc0/specification/server_server_api.rst#232version-1>`_.
 
+.. NOTE::
+   Older drafts of this specification made use of a different style of key verification,
+   however for reasons discussed in `MSC1711 <https://github.com/matrix-org/matrix-doc/pull/1711>`_,
+   the approach was removed from the initial version of the specification. The older
+   draft may be reviewed `thanks to web.archive.org
+   <https://web.archive.org/web/20181019044236/https://matrix.org/docs/spec/server_server/unstable.html>`_.
+
 Each homeserver publishes its public keys under ``/_matrix/key/v2/server/{keyId}``.
 Homeservers query for keys by either getting ``/_matrix/key/v2/server/{keyId}``
 directly or by querying an intermediate notary server using a