Federation API =============== Federation is the term used to describe how to communicate between Matrix home servers. Federation is a mechanism by which two home servers can exchange Matrix event messages, both as a real-time push of current events, and as a historic fetching mechanism to synchronise past history for clients to view. It uses HTTPS connections between each pair of servers involved as the underlying transport. Messages are exchanged between servers in real-time by active pushing from each server's HTTP client into the server of the other. Queries to fetch historic data for the purpose of back-filling scrollback buffers and the like can also be performed. Currently routing of messages between homeservers is full mesh (like email) - however, fan-out refinements to this design are currently under consideration. There are three main kinds of communication that occur between home servers: :Queries: These are single request/response interactions between a given pair of servers, initiated by one side sending an HTTPS GET request to obtain some information, and responded by the other. They are not persisted and contain no long-term significant history. They simply request a snapshot state at the instant the query is made. :Ephemeral Data Units (EDUs): These are notifications of events that are pushed from one home server to another. They are not persisted and contain no long-term significant history, nor does the receiving home server have to reply to them. :Persisted Data Units (PDUs): These are notifications of events that are broadcast from one home server to any others that are interested in the same "context" (namely, a Room ID). They are persisted to long-term storage and form the record of history for that context. EDUs and PDUs are further wrapped in an envelope called a Transaction, which is transferred from the origin to the destination home server using an HTTP PUT request. Transactions ------------ .. WARNING:: This section may be misleading or inaccurate. The transfer of EDUs and PDUs between home servers is performed by an exchange of Transaction messages, which are encoded as JSON objects, passed over an HTTP PUT request. A Transaction is meaningful only to the pair of home servers that exchanged it; they are not globally-meaningful. Each transaction has: - An opaque transaction ID. - A timestamp (UNIX epoch time in milliseconds) generated by its origin server. - An origin and destination server name. - A list of "previous IDs". - A list of PDUs and EDUs - the actual message payload that the Transaction carries. ``origin`` Type: String Description: DNS name of homeserver making this transaction. ``ts`` Type: Integer Description: Timestamp in milliseconds on originating homeserver when this transaction started. ``previous_ids`` Type: List of strings Description: List of transactions that were sent immediately prior to this transaction. ``pdus`` Type: List of Objects. Description: List of updates contained in this transaction. :: { "transaction_id":"916d630ea616342b42e98a3be0b74113", "ts":1404835423000, "origin":"red", "destination":"blue", "prev_ids":["e1da392e61898be4d2009b9fecce5325"], "pdus":[...], "edus":[...] } The ``prev_ids`` field contains a list of previous transaction IDs that the ``origin`` server has sent to this ``destination``. Its purpose is to act as a sequence checking mechanism - the destination server can check whether it has successfully received that Transaction, or ask for a retransmission if not. The ``pdus`` field of a transaction is a list, containing zero or more PDUs.[*] Each PDU is itself a JSON object containing a number of keys, the exact details of which will vary depending on the type of PDU. Similarly, the ``edus`` field is another list containing the EDUs. This key may be entirely absent if there are no EDUs to transfer. (* Normally the PDU list will be non-empty, but the server should cope with receiving an "empty" transaction.) PDUs and EDUs ------------- .. WARNING:: This section may be misleading or inaccurate. All PDUs have: - An ID - A context - A declaration of their type - A list of other PDU IDs that have been seen recently on that context (regardless of which origin sent them) ``context`` Type: String Description: Event context identifier ``origin`` Type: String Description: DNS name of homeserver that created this PDU. ``pdu_id`` Type: String Description: Unique identifier for PDU within the context for the originating homeserver ``ts`` Type: Integer Description: Timestamp in milliseconds on originating homeserver when this PDU was created. ``pdu_type`` Type: String Description: PDU event type. ``prev_pdus`` Type: List of pairs of strings Description: The originating homeserver and PDU ids of the most recent PDUs the homeserver was aware of for this context when it made this PDU. ``depth`` Type: Integer Description: The maximum depth of the previous PDUs plus one. .. TODO-spec paul - Update this structure so that 'pdu_id' is a two-element [origin,ref] pair like the prev_pdus are For state updates: ``is_state`` Type: Boolean Description: True if this PDU is updating state. ``state_key`` Type: String Description: Optional key identifying the updated state within the context. ``power_level`` Type: Integer Description: The asserted power level of the user performing the update. ``required_power_level`` Type: Integer Description: The required power level needed to replace this update. ``prev_state_id`` Type: String Description: PDU event type. ``prev_state_origin`` Type: String Description: The PDU id of the update this replaces. ``user_id`` Type: String Description: The user updating the state. :: { "pdu_id":"a4ecee13e2accdadf56c1025af232176", "context":"#example.green", "origin":"green", "ts":1404838188000, "pdu_type":"m.text", "prev_pdus":[["blue","99d16afbc857975916f1d73e49e52b65"]], "content":... "is_state":false } In contrast to Transactions, it is important to note that the ``prev_pdus`` field of a PDU refers to PDUs that any origin server has sent, rather than previous IDs that this ``origin`` has sent. This list may refer to other PDUs sent by the same origin as the current one, or other origins. Because of the distributed nature of participants in a Matrix conversation, it is impossible to establish a globally-consistent total ordering on the events. However, by annotating each outbound PDU at its origin with IDs of other PDUs it has received, a partial ordering can be constructed allowing causality relationships to be preserved. A client can then display these messages to the end-user in some order consistent with their content and ensure that no message that is semantically in reply of an earlier one is ever displayed before it. PDUs fall into two main categories: those that deliver Events, and those that synchronise State. For PDUs that relate to State synchronisation, additional keys exist to support this: :: {..., "is_state":true, "state_key":TODO-doc "power_level":TODO-doc "prev_state_id":TODO-doc "prev_state_origin":TODO-doc} EDUs, by comparison to PDUs, do not have an ID, a context, or a list of "previous" IDs. The only mandatory fields for these are the type, origin and destination home server names, and the actual nested content. :: {"edu_type":"m.presence", "origin":"blue", "destination":"orange", "content":...} Protocol URLs ------------- .. WARNING:: This section may be misleading or inaccurate. All these URLs are namespaced within a prefix of:: /_matrix/federation/v1/... For active pushing of messages representing live activity "as it happens":: PUT .../send/:transaction_id/ Body: JSON encoding of a single Transaction Response: TODO-doc The transaction_id path argument will override any ID given in the JSON body. The destination name will be set to that of the receiving server itself. Each embedded PDU in the transaction body will be processed. To fetch a particular PDU:: GET .../pdu/:origin/:pdu_id/ Response: JSON encoding of a single Transaction containing one PDU Retrieves a given PDU from the server. The response will contain a single new Transaction, inside which will be the requested PDU. To fetch all the state of a given context:: GET .../state/:context/ Response: JSON encoding of a single Transaction containing multiple PDUs Retrieves a snapshot of the entire current state of the given context. The response will contain a single Transaction, inside which will be a list of PDUs that encode the state. To backfill events on a given context:: GET .../backfill/:context/ Query args: v, limit Response: JSON encoding of a single Transaction containing multiple PDUs Retrieves a sliding-window history of previous PDUs that occurred on the given context. Starting from the PDU ID(s) given in the "v" argument, the PDUs that preceeded it are retrieved, up to a total number given by the "limit" argument. These are then returned in a new Transaction containing all of the PDUs. To stream events all the events:: GET .../pull/ Query args: origin, v Response: JSON encoding of a single Transaction consisting of multiple PDUs Retrieves all of the transactions later than any version given by the "v" arguments. To make a query:: GET .../query/:query_type Query args: as specified by the individual query types Response: JSON encoding of a response object Performs a single query request on the receiving home server. The Query Type part of the path specifies the kind of query being made, and its query arguments have a meaning specific to that kind of query. The response is a JSON-encoded object whose meaning also depends on the kind of query. Backfilling ----------- .. NOTE:: This section is a work in progress. .. TODO-doc - What it is, when is it used, how is it done SRV Records ----------- .. NOTE:: This section is a work in progress. .. TODO-doc - Why it is needed State Conflict Resolution ------------------------- .. NOTE:: This section is a work in progress. .. TODO-doc - How do conflicts arise (diagrams?) - How are they resolved (incl tie breaks) - How does this work with deleting current state Presence -------- The server API for presence is based entirely on exchange of the following EDUs. There are no PDUs or Federation Queries involved. Performing a presence update and poll subscription request:: EDU type: m.presence Content keys: push: (optional): list of push operations. Each should be an object with the following keys: user_id: string containing a User ID presence: "offline"|"unavailable"|"online"|"free_for_chat" status_msg: (optional) string of freeform text last_active_ago: miliseconds since the last activity by the user poll: (optional): list of strings giving User IDs unpoll: (optional): list of strings giving User IDs The presence of this combined message is two-fold: it informs the recipient server of the current status of one or more users on the sending server (by the ``push`` key), and it maintains the list of users on the recipient server that the sending server is interested in receiving updates for, by adding (by the ``poll`` key) or removing them (by the ``unpoll`` key). The ``poll`` and ``unpoll`` lists apply *changes* to the implied list of users; any existing IDs that the server sent as ``poll`` operations in a previous message are not removed until explicitly requested by a later ``unpoll``. On receipt of a message containing a non-empty ``poll`` list, the receiving server should immediately send the sending server a presence update EDU of its own, containing in a ``push`` list the current state of every user that was in the orginal EDU's ``poll`` list. Sending a presence invite:: EDU type: m.presence_invite Content keys: observed_user: string giving the User ID of the user whose presence is requested (i.e. the recipient of the invite) observer_user: string giving the User ID of the user who is requesting to observe the presence (i.e. the sender of the invite) Accepting a presence invite:: EDU type: m.presence_accept Content keys - as for m.presence_invite Rejecting a presence invite:: EDU type: m.presence_deny Content keys - as for m.presence_invite .. TODO-doc - Explain the timing-based roundtrip reduction mechanism for presence messages - Explain the zero-byte presence inference logic See also: docs/client-server/model/presence Profiles -------- The server API for profiles is based entirely on the following Federation Queries. There are no additional EDU or PDU types involved, other than the implicit ``m.presence`` and ``m.room.member`` events (see section below). Querying profile information:: Query type: profile Arguments: user_id: the ID of the user whose profile to return field: (optional) string giving a field name Returns: JSON object containing the following keys: displayname: string of freeform text avatar_url: string containing an http-scheme URL If the query contains the optional ``field`` key, it should give the name of a result field. If such is present, then the result should contain only a field of that name, with no others present. If not, the result should contain as much of the user's profile as the home server has available and can make public. Server-Server Authentication ---------------------------- .. TODO-doc - Why is this needed. - High level overview of process. - Transaction/PDU signing - How does this work with redactions? (eg hashing required keys only) Threat Model ------------ Denial of Service ~~~~~~~~~~~~~~~~~ The attacker could attempt to prevent delivery of messages to or from the victim in order to: * Disrupt service or marketing campaign of a commercial competitor. * Censor a discussion or censor a participant in a discussion. * Perform general vandalism. Threat: Resource Exhaustion +++++++++++++++++++++++++++ An attacker could cause the victims server to exhaust a particular resource (e.g. open TCP connections, CPU, memory, disk storage) Threat: Unrecoverable Consistency Violations ++++++++++++++++++++++++++++++++++++++++++++ An attacker could send messages which created an unrecoverable "split-brain" state in the cluster such that the victim's servers could no longer dervive a consistent view of the chatroom state. Threat: Bad History +++++++++++++++++++ An attacker could convince the victim to accept invalid messages which the victim would then include in their view of the chatroom history. Other servers in the chatroom would reject the invalid messages and potentially reject the victims messages as well since they depended on the invalid messages. .. TODO-spec Track trustworthiness of HS or users based on if they try to pretend they haven't seen recent events, and fake a splitbrain... --M Threat: Block Network Traffic +++++++++++++++++++++++++++++ An attacker could try to firewall traffic between the victim's server and some or all of the other servers in the chatroom. Threat: High Volume of Messages +++++++++++++++++++++++++++++++ An attacker could send large volumes of messages to a chatroom with the victim making the chatroom unusable. Threat: Banning users without necessary authorisation +++++++++++++++++++++++++++++++++++++++++++++++++++++ An attacker could attempt to ban a user from a chatroom with the necessary authorisation. Spoofing ~~~~~~~~ An attacker could try to send a message claiming to be from the victim without the victim having sent the message in order to: * Impersonate the victim while performing illict activity. * Obtain privileges of the victim. Threat: Altering Message Contents +++++++++++++++++++++++++++++++++ An attacker could try to alter the contents of an existing message from the victim. Threat: Fake Message "origin" Field +++++++++++++++++++++++++++++++++++ An attacker could try to send a new message purporting to be from the victim with a phony "origin" field. Spamming ~~~~~~~~ The attacker could try to send a high volume of solicicted or unsolicted messages to the victim in order to: * Find victims for scams. * Market unwanted products. Threat: Unsoliticted Messages +++++++++++++++++++++++++++++ An attacker could try to send messages to victims who do not wish to receive them. Threat: Abusive Messages ++++++++++++++++++++++++ An attacker could send abusive or threatening messages to the victim Spying ~~~~~~ The attacker could try to access message contents or metadata for messages sent by the victim or to the victim that were not intended to reach the attacker in order to: * Gain sensitive personal or commercial information. * Impersonate the victim using credentials contained in the messages. (e.g. password reset messages) * Discover who the victim was talking to and when. Threat: Disclosure during Transmission ++++++++++++++++++++++++++++++++++++++ An attacker could try to expose the message contents or metadata during transmission between the servers. Threat: Disclosure to Servers Outside Chatroom ++++++++++++++++++++++++++++++++++++++++++++++ An attacker could try to convince servers within a chatroom to send messages to a server it controls that was not authorised to be within the chatroom. Threat: Disclosure to Servers Within Chatroom ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ An attacker could take control of a server within a chatroom to expose message contents or metadata for messages in that room. Identity Servers ================ .. NOTE:: This section is a work in progress. .. TODO-doc Dave - 3PIDs and identity server, functions Lawful Interception ------------------- Key Escrow Servers ~~~~~~~~~~~~~~~~~~ Policy Servers ============== .. NOTE:: This section is a work in progress. .. TODO-spec We should mention them in the Architecture section at least: how they fit into the picture. Enforcing policies ------------------ .. Links through the external API docs are below .. ============================================= .. |createRoom| replace:: ``/createRoom`` .. _createRoom: /docs/api/client-server/#!/-rooms/create_room .. |initialSync| replace:: ``/initialSync`` .. _initialSync: /docs/api/client-server/#!/-events/initial_sync .. |/rooms//initialSync| replace:: ``/rooms//initialSync`` .. _/rooms//initialSync: /docs/api/client-server/#!/-rooms/get_room_sync_data .. |login| replace:: ``/login`` .. _login: /docs/api/client-server/#!/-login .. |register| replace:: ``/register`` .. _register: /docs/api/client-server/#!/-registration .. |/rooms//messages| replace:: ``/rooms//messages`` .. _/rooms//messages: /docs/api/client-server/#!/-rooms/get_messages .. |/rooms//members| replace:: ``/rooms//members`` .. _/rooms//members: /docs/api/client-server/#!/-rooms/get_members .. |/rooms//state| replace:: ``/rooms//state`` .. _/rooms//state: /docs/api/client-server/#!/-rooms/get_state_events .. |/rooms//send/| replace:: ``/rooms//send/`` .. _/rooms//send/: /docs/api/client-server/#!/-rooms/send_non_state_event .. |/rooms//state//| replace:: ``/rooms//state//`` .. _/rooms//state//: /docs/api/client-server/#!/-rooms/send_state_event .. |/rooms//invite| replace:: ``/rooms//invite`` .. _/rooms//invite: /docs/api/client-server/#!/-rooms/invite .. |/rooms//join| replace:: ``/rooms//join`` .. _/rooms//join: /docs/api/client-server/#!/-rooms/join_room .. |/rooms//leave| replace:: ``/rooms//leave`` .. _/rooms//leave: /docs/api/client-server/#!/-rooms/leave .. |/rooms//ban| replace:: ``/rooms//ban`` .. _/rooms//ban: /docs/api/client-server/#!/-rooms/ban .. |/join/| replace:: ``/join/`` .. _/join/: /docs/api/client-server/#!/-rooms/join .. _`Event Stream`: /docs/api/client-server/#!/-events/get_event_stream