From 458383585f6e58b588171d37584daccba49ff5bb Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 2 Oct 2015 15:03:55 +0100 Subject: [PATCH 01/41] Stub sections --- specification/modules/instant_messaging.rst | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 43a06aa1..619226df 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -3,6 +3,9 @@ Instant Messaging .. _module:im: +This module adds support for sending human-readable messages to a room. It also +adds human-readable information to the room itself such as a room name and topic. + Events ------ @@ -15,7 +18,7 @@ Events {{m_room_topic_event}} m.room.message msgtypes ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ .. TODO-spec How a client should handle unknown message types. @@ -27,3 +30,13 @@ outlined below. {{msgtype_events}} + +Client behaviour +---------------- + +Server behaviour +---------------- + +Security considerations +----------------------- + From 8e5c832ff9fa8d8e674346e3776c08e659c306c9 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 5 Oct 2015 13:45:23 +0100 Subject: [PATCH 02/41] Flesh out more of the IM module --- .../schema/v1/m.room.message.feedback | 2 +- specification/modules/instant_messaging.rst | 59 +++++++++++++++++-- 2 files changed, 54 insertions(+), 7 deletions(-) diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback index 1bbfc1ba..2eaed999 100644 --- a/event-schemas/schema/v1/m.room.message.feedback +++ b/event-schemas/schema/v1/m.room.message.feedback @@ -1,7 +1,7 @@ { "type": "object", "title": "MessageFeedback", - "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.", + "description": "**NB: Usage of this event is discouraged in favour of the** `receipts module`_. **Most clients will not recognise this event.** Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged.", "allOf": [{ "$ref": "core-event-schema/room_event.json" }], diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 619226df..7934b80a 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -4,15 +4,37 @@ Instant Messaging .. _module:im: This module adds support for sending human-readable messages to a room. It also -adds human-readable information to the room itself such as a room name and topic. +adds support for associating human-readable information with the room itself +such as a room name and topic. Events ------ {{m_room_message_event}} + +.. admonition:: Rationale + + Not all clients can display all message types. The most commonly supported + message type is raw text. As a result, we chose to have a textual fallback + display method represented by the ``body`` key. This means that even if the + client cannot display a particular ``msgtype``, they can still display + *something*, even if it is just plain text. + {{m_room_message_feedback_event}} +Usage of this event is discouraged for several reasons: + - The number of feedback events will grow very quickly with the number of users + in the room. This event provides no way to "batch" feedback, unlike the + `receipts module`_. + - Pairing feedback to messages gets complicated when paginating as feedback + arrives before the message it is acknowledging. + - There are no guarantees that the client has seen the event ID being + acknowledged. + + +.. _`receipts module`: `module:receipts`_ + {{m_room_name_event}} {{m_room_topic_event}} @@ -20,13 +42,10 @@ Events m.room.message msgtypes ~~~~~~~~~~~~~~~~~~~~~~~ -.. TODO-spec - How a client should handle unknown message types. - - Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type of message being sent. Each type has their own required and optional keys, as -outlined below. +outlined below. If a client cannot display the given ``msgtype`` then it MUST +display the fallback plain text ``body`` key instead. {{msgtype_events}} @@ -34,9 +53,37 @@ outlined below. Client behaviour ---------------- +Events which have attachments (e.g. ``m.image``, ``m.file``) are advised to be +uploaded using the `content repository module`_ where available. The +resulting ``mxc://`` URI can then be used in the ``url`` key. The +attachment SHOULD be uploaded *prior* to sending the event in order to stop a +race condition where the recipient receives a link to a non-existent attachment. + +.. _`content repository module`: `module:content`_ + +Recommendations when sending messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- Advise using idempotent PUTs to send messages (and why) +- Retries (exp backoff, giveup eventually allowing manual retry) +- Queueing (bucket per room) + +Implementing local echo +~~~~~~~~~~~~~~~~~~~~~~~ +- Local echo (document bug with races) - sending state. Pairing returned event ID. + +Displaying membership information with messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Member name linking (incl. pagination aka historical display names) + Server behaviour ---------------- +- SHOULD enforce the body/msgtype keys are present (can 400 them) + Security considerations ----------------------- +- Not encrypted, link to E2E module. +- XSS: Should sanitise ALL KEYS before injecting as unsafe HTML (name/topic/body/etc) + From 91ca36509b3112a83680a97a0d672a428b4d7fef Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 7 Oct 2015 11:51:49 +0100 Subject: [PATCH 03/41] Flesh out IM module --- specification/0-intro.rst | 2 + specification/modules/instant_messaging.rst | 89 ++++++++++++++++++--- 2 files changed, 80 insertions(+), 11 deletions(-) diff --git a/specification/0-intro.rst b/specification/0-intro.rst index 445b32ef..7c27e0d7 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -434,6 +434,8 @@ Some requests have unique error codes: :``M_LOGIN_EMAIL_URL_NOT_YET``: Encountered when polling for an email link which has not been clicked yet. +.. _sect:txn_ids: + The C-S API typically uses ``HTTP POST`` to submit requests. This means these requests are not idempotent. The C-S API also allows ``HTTP PUT`` to make requests idempotent. In order to use a ``PUT``, paths should be suffixed with diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 7934b80a..426189b5 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -53,7 +53,14 @@ display the fallback plain text ``body`` key instead. Client behaviour ---------------- -Events which have attachments (e.g. ``m.image``, ``m.file``) are advised to be +Clients SHOULD verify the structure of incoming events to ensure that the +expected keys exist and that they are of the right type. Clients can discard +malformed events or display a placeholder message to the user. Redacted +``m.room.message`` events MUST be removed from the client. This can either be +replaced with placeholder text (e.g. "[REDACTED]") or the redacted message can +be removed entirely from the messages view. + +Events which have attachments (e.g. ``m.image``, ``m.file``) SHOULD be uploaded using the `content repository module`_ where available. The resulting ``mxc://`` URI can then be used in the ``url`` key. The attachment SHOULD be uploaded *prior* to sending the event in order to stop a @@ -63,27 +70,87 @@ race condition where the recipient receives a link to a non-existent attachment. Recommendations when sending messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Advise using idempotent PUTs to send messages (and why) -- Retries (exp backoff, giveup eventually allowing manual retry) -- Queueing (bucket per room) -Implementing local echo -~~~~~~~~~~~~~~~~~~~~~~~ -- Local echo (document bug with races) - sending state. Pairing returned event ID. +Clients can send messages using ``POST`` or ``PUT`` requests. Clients SHOULD use +``PUT`` requests with `transaction IDs`_ to make requests idempotent. This +ensures that messages are sent exactly once even under poor network conditions. +Clients SHOULD retry requests using an exponential-backoff algorithm for a +certain amount of time T. It is recommended that T is no longer than 5 minutes. +After this time, the client should stop retrying and mark the message as "unsent". +Users should be able to manually resend unsent messages. + +Users may type several messages at once and send them all in quick succession. +Clients SHOULD preserve the order in which they were sent by the user. This +means that clients should wait for the response to the previous request before +sending the next request. This can lead to head-of-line blocking. In order to +reduce the impact of head-of-line blocking, clients should use a queue per room +rather than a global queue, as ordering is only relevant within a single room +rather than between rooms. + +.. _`transaction IDs`: `sect:txn_ids`_ + +Local echo +~~~~~~~~~~ + +Messages SHOULD appear immediately in the message view when a user presses the +"send" button. This should occur even if the message is still sending. This is +referred to as "local echo". Clients SHOULD implement "local echo" of messages. + +Clients need to be able to pair up the "remote echo" from the server with the +"local echo" to prevent duplicate messages being displayed. Ideally this pairing +would occur transparently to the user: the UI would not flicker as it transitions +from local to remote. Flickering cannot be fully avoided in version 1 of the +client-server API. Two scenarios need to be considered: + +- The client sends a message and the remote echo arrives on the event stream + *after* the request to send the message completes. +- The client sends a message and the remote echo arrives on the event stream + *before* the request to send the message completes. + +In the first scenario, the client will receive an event ID when the request to +send the message completes. This ID can be used to identify the duplicate event +when it arrives on the event stream. However, in the second scenario, the event +arrives before the client has obtained an event ID. This makes it impossible to +identify it as a duplicate event. This results in the client displaying the +message twice for a fraction of a second before the the original request to send +the message completes. Once it completes, the client can take remedial actions +to remove the duplicate event by looking for duplicate event IDs. Version 2 of +the client-server API resolves this by attaching the transaction ID of the +sending request to the event itself. + Displaying membership information with messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Member name linking (incl. pagination aka historical display names) +Clients may wish to show the display name and avatar URL of the room member who +sent a message. This can be achieved by inspecting the ``m.room.member`` event +for that user ID. + +When a user paginates the message history, clients may wish to show the +**historical** display name and avatar URL for a room member. This is possible +because older ``m.room.member`` events are returned when paginating. This can +be implemented efficiently by keeping two sets of room state: old and current. +As new events arrive and/or the user paginates back in time, these two sets of +state diverge from each other. New events update the current state and paginated +events update the old state. When paginated events are processed sequentially, +the old state represents the state of the room *at the time the event was sent*. +This can then be used to set the historical display name and avatar URL. Server behaviour ---------------- -- SHOULD enforce the body/msgtype keys are present (can 400 them) +Homeservers SHOULD enforce that ``m.room.message`` events have textual ``body`` +and ``msgtype`` keys by 400ing the request to send a message. Security considerations ----------------------- -- Not encrypted, link to E2E module. -- XSS: Should sanitise ALL KEYS before injecting as unsafe HTML (name/topic/body/etc) +Messages sent using this module are not encrypted. Encryption can be layered +over the top of this module: where the plaintext format is an ``m.room.message`` +conforming to this module. This can be achieved using the `E2E module`_. + +Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site +Scripting (XSS) attacks. This includes room names and topics. + +.. _`E2E module`: `module:e2e`_ From e378de8379abcb85db8048b7dbd80b3851b4ac51 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 9 Oct 2015 11:04:11 +0100 Subject: [PATCH 04/41] Shuffle history visibility module to be in the module format --- specification/0-events.rst | 2 - specification/modules/history_visibility.rst | 44 +++++++++++++++----- 2 files changed, 33 insertions(+), 13 deletions(-) diff --git a/specification/0-events.rst b/specification/0-events.rst index 16948462..e2373f96 100644 --- a/specification/0-events.rst +++ b/specification/0-events.rst @@ -26,8 +26,6 @@ prefixed with ``m.`` {{m_room_create_event}} -{{m_room_history_visibility_event}} - {{m_room_join_rules_event}} {{m_room_member_event}} diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 371282bd..4f0c8976 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -1,21 +1,36 @@ Room History Visibility ------------------------ +======================= .. _module:history-visibility: -Whether a member of a room can see the events that happened in a room from -before they joined the room is controlled by the ``history_visibility`` key -of the ``m.room.history_visibility`` state event. The valid values for -``history_visibility`` are: +This module adds support for controlling the visibility of previous events in a +room. Whether a member of a room can see the events that happened in a room from +before they joined the room is controlled by the ``m.room.history_visibility`` +event outlined below. Visibility can take the form of one of three options: + +- ``shared`` - Previous events are always shown to newly joined members. +- ``invited`` - Previous events are shown to newly joined members from the point + they were invited onwards. +- ``joined`` - Previous events are shown to newly joined members from the point + they joined the room onwards. + +Events +------ + +{{m_room_history_visibility_event}} -- ``shared`` -- ``invited`` -- ``joined`` +Client behaviour +---------------- -By default if no ``history_visibility`` is set it is assumed to be ``shared``. +Clients that implement this module MUST present to the user the possible options +for setting history visibility when creating a room. -The rules governing whether a user is allowed to see an event depend solely on -the state of the room at that event: +Server behaviour +---------------- + +By default if no ``history_visibility`` is set the visibility is assumed to be +``shared``. The rules governing whether a user is allowed to see an event depend +solely on the state of the room *at that event*: 1. If the user was joined, allow. 2. If the user was invited and the ``history_visibility`` was set to @@ -24,3 +39,10 @@ the state of the room at that event: was set to ``shared``, allow. 4. Otherwise, deny. +Security considerations +----------------------- + +The default value for ``history_visibility`` is ``shared`` for +backwards-compatibility reasons. Clients need to be aware that by not setting +this event they are exposing all of their room history to anyone in the room. + From 173d00cea04e3c607992396b7395741971578e35 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 13 Oct 2015 11:08:19 +0100 Subject: [PATCH 05/41] Review comments --- specification/modules/history_visibility.rst | 24 ++++++++++++++++---- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 4f0c8976..12ea7b22 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -6,13 +6,27 @@ Room History Visibility This module adds support for controlling the visibility of previous events in a room. Whether a member of a room can see the events that happened in a room from before they joined the room is controlled by the ``m.room.history_visibility`` -event outlined below. Visibility can take the form of one of three options: - -- ``shared`` - Previous events are always shown to newly joined members. +event outlined below. In all cases, the member still needs to be joined to the +room to receive events for that room. The visibility option simply determines +which subset of events in the room are presented to the client. Visibility can +take the form of one of three options: + +- ``shared`` - Previous events are always shown to newly joined members. All + events in the room are shown, even those sent when the member was not a part + of the room. - ``invited`` - Previous events are shown to newly joined members from the point - they were invited onwards. + they were invited onwards. Events stop being shown when the member's state + changes to something other than ``invite`` or ``join``. - ``joined`` - Previous events are shown to newly joined members from the point - they joined the room onwards. + they joined the room onwards. Events stop being shown when the members state + changes to something other than ``join``. + +.. WARNING:: + These options are applied at the point an event is *sent*. Checks are + performed with the state of the ``m.room.history_visibility`` event when the + event in question is added to the DAG. This means clients cannot + retrospectively choose to show or hide history to new users if the setting at + that time was more restrictive. Events ------ From 4170dbd5cf065f7e1c7fde32f4c5a6e622c98055 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 13 Oct 2015 11:29:54 +0100 Subject: [PATCH 06/41] Review comments --- event-schemas/schema/v1/m.room.message | 2 +- specification/modules/instant_messaging.rst | 44 +++++++++------------ 2 files changed, 19 insertions(+), 27 deletions(-) diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message index 27b7e925..91c04b7f 100644 --- a/event-schemas/schema/v1/m.room.message +++ b/event-schemas/schema/v1/m.room.message @@ -1,7 +1,7 @@ { "type": "object", "title": "Message", - "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.", + "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message. This allows clients to display *something* even if it is just plain text.", "allOf": [{ "$ref": "core-event-schema/room_event.json" }], diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 426189b5..487240de 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -12,15 +12,6 @@ Events {{m_room_message_event}} - -.. admonition:: Rationale - - Not all clients can display all message types. The most commonly supported - message type is raw text. As a result, we chose to have a textual fallback - display method represented by the ``body`` key. This means that even if the - client cannot display a particular ``msgtype``, they can still display - *something*, even if it is just plain text. - {{m_room_message_feedback_event}} Usage of this event is discouraged for several reasons: @@ -44,7 +35,7 @@ m.room.message msgtypes Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type of message being sent. Each type has their own required and optional keys, as -outlined below. If a client cannot display the given ``msgtype`` then it MUST +outlined below. If a client cannot display the given ``msgtype`` then it SHOULD display the fallback plain text ``body`` key instead. {{msgtype_events}} @@ -62,9 +53,7 @@ be removed entirely from the messages view. Events which have attachments (e.g. ``m.image``, ``m.file``) SHOULD be uploaded using the `content repository module`_ where available. The -resulting ``mxc://`` URI can then be used in the ``url`` key. The -attachment SHOULD be uploaded *prior* to sending the event in order to stop a -race condition where the recipient receives a link to a non-existent attachment. +resulting ``mxc://`` URI can then be used in the ``url`` key. .. _`content repository module`: `module:content`_ @@ -95,12 +84,15 @@ Local echo Messages SHOULD appear immediately in the message view when a user presses the "send" button. This should occur even if the message is still sending. This is referred to as "local echo". Clients SHOULD implement "local echo" of messages. - -Clients need to be able to pair up the "remote echo" from the server with the -"local echo" to prevent duplicate messages being displayed. Ideally this pairing -would occur transparently to the user: the UI would not flicker as it transitions -from local to remote. Flickering cannot be fully avoided in version 1 of the -client-server API. Two scenarios need to be considered: + +Clients need to be able to match the message they are sending with the same +message which they receive from the event stream. The echo of the same message +from the event stream is referred to as "remote echo". Both echoes need to be +identified as the same message in order to prevent duplicate messages being +displayed. Ideally this pairing would occur transparently to the user: the UI +would not flicker as it transitions from local to remote. Flickering cannot be +fully avoided in version 1 of the client-server API. Two scenarios need to be +considered: - The client sends a message and the remote echo arrives on the event stream *after* the request to send the message completes. @@ -123,8 +115,8 @@ Displaying membership information with messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Clients may wish to show the display name and avatar URL of the room member who -sent a message. This can be achieved by inspecting the ``m.room.member`` event -for that user ID. +sent a message. This can be achieved by inspecting the ``m.room.member`` state +event for that user ID. When a user paginates the message history, clients may wish to show the **historical** display name and avatar URL for a room member. This is possible @@ -139,15 +131,15 @@ This can then be used to set the historical display name and avatar URL. Server behaviour ---------------- -Homeservers SHOULD enforce that ``m.room.message`` events have textual ``body`` -and ``msgtype`` keys by 400ing the request to send a message. +Homeservers SHOULD reject ``m.room.message`` events which don't have a +``msgtype`` key, or who don't have a textual ``body`` key, with an HTTP status +code of 400. Security considerations ----------------------- -Messages sent using this module are not encrypted. Encryption can be layered -over the top of this module: where the plaintext format is an ``m.room.message`` -conforming to this module. This can be achieved using the `E2E module`_. +Messages sent using this module are not encrypted. Messages can be encrypted +using the `E2E module`_. Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site Scripting (XSS) attacks. This includes room names and topics. From e561a663d30810c0e58ca30b89bc3bc759e92aec Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 13 Oct 2015 16:53:27 +0100 Subject: [PATCH 07/41] Tweaks to intro, start using swagger APIs in the CS API section. --- specification/0-intro.rst | 46 ++++++++++++--------------- specification/1-client_server_api.rst | 44 ++----------------------- 2 files changed, 22 insertions(+), 68 deletions(-) diff --git a/specification/0-intro.rst b/specification/0-intro.rst index 445b32ef..1c18a4f0 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -99,23 +99,23 @@ Architecture ------------ Matrix defines APIs for synchronising extensible JSON objects known as -``events`` between compatible clients, servers and services. Clients are +"events" between compatible clients, servers and services. Clients are typically messaging/VoIP applications or IoT devices/hubs and communicate by -synchronising communication history with their ``homeserver`` using the -``Client-Server API``. Each homeserver stores the communication history and +synchronising communication history with their "homeserver" using the +"Client-Server API". Each homeserver stores the communication history and account information for all of its clients, and shares data with the wider Matrix ecosystem by synchronising communication history with other homeservers and their clients. Clients typically communicate with each other by emitting events in the -context of a virtual ``room``. Room data is replicated across *all of the +context of a virtual "room". Room data is replicated across *all of the homeservers* whose users are participating in a given room. As such, *no single homeserver has control or ownership over a given room*. Homeservers model communication history as a partially ordered graph of events known as -the room's ``event graph``, which is synchronised with eventual consistency -between the participating servers using the ``Server-Server API``. This process +the room's "event graph", which is synchronised with eventual consistency +between the participating servers using the "Server-Server API". This process of synchronising shared conversation history between homeservers run by -different parties is called ``Federation``. Matrix optimises for the the +different parties is called "Federation". Matrix optimises for the the Availability and Partitioned properties of CAP theorem at the expense of Consistency. @@ -151,13 +151,13 @@ Users ~~~~~ Each client is associated with a user account, which is identified in Matrix -using a unique "User ID". This ID is namespaced to the home server which +using a unique "User ID". This ID is namespaced to the homeserver which allocated the account and has the form:: @localpart:domain The ``localpart`` of a user ID may be a user name, or an opaque ID identifying -this user. They are case-insensitive. +this user. The ``domain`` of a user ID is the domain of the homeserver. .. TODO-spec - Need to specify precise grammar for Matrix IDs @@ -183,9 +183,9 @@ Event Graphs .. _sect:event-graph: Events exchanged in the context of a room are stored in a directed acyclic graph -(DAG) called an ``event graph``. The partial ordering of this graph gives the +(DAG) called an "event graph". The partial ordering of this graph gives the chronological ordering of events within the room. Each event in the graph has a -list of zero or more ``parent`` events, which refer to any preceding events +list of zero or more "parent" events, which refer to any preceding events which have no chronological successor from the perspective of the homeserver which created the event. @@ -292,11 +292,10 @@ Each room can also have multiple "Room Aliases", which look like:: A room alias "points" to a room ID and is the human-readable label by which rooms are publicised and discovered. The room ID the alias is pointing to can -be obtained by visiting the domain specified. They are case-insensitive. Note -that the mapping from a room alias to a room ID is not fixed, and may change -over time to point to a different room ID. For this reason, Clients SHOULD -resolve the room alias to a room ID once and then use that ID on subsequent -requests. +be obtained by visiting the domain specified. Note that the mapping from a room +alias to a room ID is not fixed, and may change over time to point to a +different room ID. For this reason, Clients SHOULD resolve the room alias to a +room ID once and then use that ID on subsequent requests. When resolving a room alias the server will also respond with a list of servers that are in the room that can be used to join via. @@ -339,12 +338,9 @@ Profiles ~~~~~~~~ Users may publish arbitrary key/value data associated with their account - such -as a human readable ``display name``, a profile photo URL, contact information +as a human readable display name, a profile photo URL, contact information (email address, phone numbers, website URLs etc). -In Client-Server API v2, profile data is typed using namespaced keys for -interoperability, much like events - e.g. ``m.profile.display_name``. - .. TODO Actually specify the different types of data - e.g. what format are display names allowed to be? @@ -431,13 +427,11 @@ Some requests have unique error codes: :``M_BAD_PAGINATION``: Encountered when specifying bad pagination query parameters. -:``M_LOGIN_EMAIL_URL_NOT_YET``: - Encountered when polling for an email link which has not been clicked yet. -The C-S API typically uses ``HTTP POST`` to submit requests. This means these -requests are not idempotent. The C-S API also allows ``HTTP PUT`` to make -requests idempotent. In order to use a ``PUT``, paths should be suffixed with -``/{txnId}``. ``{txnId}`` is a unique client-generated transaction ID which +The Client-Server API typically uses ``HTTP POST`` to submit requests. This +means these requests are not idempotent. The C-S API also allows ``HTTP PUT`` to +make requests idempotent. In order to use a ``PUT``, paths should be suffixed +with ``/{txnId}``. ``{txnId}`` is a unique client-generated transaction ID which identifies the request, and is scoped to a given Client (identified by that client's ``access_token``). Crucially, it **only** serves to identify new requests from retransmits. After the request has finished, the ``{txnId}`` diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 59e6b68e..94d1a184 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -528,7 +528,7 @@ Room events are split into two categories: :Message events: These are events which describe transient "once-off" activity in a room: typically communication such as sending an instant message or setting up a - VoIP call. These used to be called 'non-state' events. + VoIP call. This specification outlines several events, all with the event type prefix ``m.``. However, applications may wish to add their own type of event, and this @@ -600,9 +600,6 @@ See `Room Events`_ for the ``m.`` event specification. Syncing rooms ~~~~~~~~~~~~~ -.. NOTE:: - This section is a work in progress. - When a client logs in, they may have a list of rooms which they have already joined. These rooms may also have a list of events associated with them. The purpose of 'syncing' is to present the current room and event information in a @@ -620,45 +617,8 @@ presence events will also be returned. A single syncing API is provided: onwards. The event stream cannot do this for a single room currently. As a result, commenting room-scoped initial sync at this time. -The |initialSync|_ API contains the following keys: - -``presence`` - Description: - Contains a list of presence information for users the client is interested - in. - Format: - A JSON array of ``m.presence`` events. - -``end`` - Description: - Contains an event stream token which can be used with the `Event Stream`_. - Format: - A string containing the event stream token. -``rooms`` - Description: - Contains a list of room information for all rooms the client has joined, - and limited room information on rooms the client has been invited to. - Format: - A JSON array containing Room Information JSON objects. - -Room Information: - Description: - Contains all state events for the room, along with a limited amount of - the most recent events, configured via the ``limit`` query - parameter. Also contains additional keys with room metadata, such as the - ``room_id`` and the client's ``membership`` to the room. - Format: - A JSON object with the following keys: - ``room_id`` - A string containing the ID of the room being described. - ``membership`` - A string representing the client's membership status in this room. - ``messages`` - An event stream JSON object containing a ``chunk`` of recent - events (both state events and non-state events), along with an ``end`` token. - ``state`` - A JSON array containing all the current state events for this room. +{{sync_http_api}} Getting events for a room ~~~~~~~~~~~~~~~~~~~~~~~~~ From e716e819635babb0216c971940f9e7a8cdbeacc4 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 09:58:03 +0100 Subject: [PATCH 08/41] Add `dir` parameter to pagination. Remove path references This section needs a lot of work to not lie. --- specification/1-client_server_api.rst | 30 +++++++++++++++------------ 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 94d1a184..a4f62b4f 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -348,11 +348,13 @@ Pagination Querying large datasets in Matrix always uses the same pagination API pattern to to give clients a consistent way of selecting subsets of a potentially changing -dataset. Requests pass in ``from``, ``to`` and ``limit`` parameters which describe -where to read from the stream. ``from`` and ``to`` are opaque textual 'stream -tokens' which describe positions in the dataset. The response returns new -``start`` and ``end`` stream token values which can then be passed to subsequent -requests to continue pagination. +dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters +which describe where to read from the stream. ``from`` and ``to`` are opaque +textual 'stream tokens' which describe the current position in the dataset. +The ``dir`` parameter is an enum representing the direction of events to return: +either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and +``end`` stream token values which can then be passed to subsequent requests to +continue pagination. Pagination Request Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -367,24 +369,26 @@ Query parameters: limit: integer - An integer representing the maximum number of items to return. + dir: + f|b - The direction to return events in. Typically this is ``b`` to paginate + backwards in time. 'START' and 'END' are placeholder values used in these examples to describe the start and end of the dataset respectively. -Unless specified, the default pagination parameters are from=START, to=END, -without a limit set. This allows you to hit an API like -/events without any query parameters to get everything. +Unless specified, the default pagination parameters are ``from=START``, +``to=END``, without a limit set. -For example, the event stream has events E1 -> E15. The client wants the last 5 +For example, if an endpoint had events E1 -> E15. The client wants the last 5 events and doesn't know any previous events:: S E |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| | | | | _____| | - |__________________ | ___________________| - | | | - GET /events?to=START&limit=5&from=END + |__________________ | _______________| + | | | + GET /somepath?to=START&limit=5&from=END Returns: E15,E14,E13,E12,E11 @@ -401,7 +405,7 @@ now show page 3 (rooms R11 -> 15):: Currently | viewing | | - GET /rooms/list?from=9&to=END&limit=5 + GET /roomslist?from=9&to=END&limit=5 Returns: R11,R12,R13,R14,R15 Note that tokens are treated in an *exclusive*, not inclusive, manner. The end From 7bdb71b1c9697eb9ded20e028723a500b1d71a02 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 10:25:39 +0100 Subject: [PATCH 09/41] Tweak the syncing section Bring together disparate sections to make it more cohesive. --- api/client-server/v1/sync.yaml | 6 +- specification/1-client_server_api.rst | 99 +++++++++++---------------- 2 files changed, 43 insertions(+), 62 deletions(-) diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml index 7c1d43f3..398b0773 100644 --- a/api/client-server/v1/sync.yaml +++ b/api/client-server/v1/sync.yaml @@ -30,7 +30,9 @@ paths: - in: query type: string name: from - description: The token to stream from. + description: |- + The token to stream from. This token is either from the a previous + request to this API or from the initial sync API. required: false x-example: "s3456_9_0" - in: query @@ -40,7 +42,7 @@ paths: required: false x-example: "35000" - in: query - type: string + type: boolean name: archived description: |- Whether to include rooms that the user has left. If absent then diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index a4f62b4f..fefdf511 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -1,9 +1,6 @@ Client-Server API ================= -Overview --------- - The client-server API provides a simple lightweight API to let clients send messages, control rooms and synchronise conversation history. It is designed to support both lightweight clients which store no state and lazy-load data from @@ -31,7 +28,7 @@ return with a status of 401 and the error code, ``M_MISSING_TOKEN`` or ``M_UNKNOWN_TOKEN`` respectively. User-Interactive Authentication API ------------------------------------ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This section refers to API Version 2. Some API endpoints such as ``login`` or ``register`` require authentication that @@ -159,7 +156,7 @@ absence of that login stage type in the 'completed' array indicating whether that stage is complete. Example -~~~~~~~ ++++++++ At a high level, the requests made for an API call completing an auth flow with three stages will resemble the following diagram:: @@ -201,7 +198,7 @@ This specification defines the following login types: - ``m.login.dummy`` Password-based -~~~~~~~~~~~~~~ +++++++++++++++ :Type: ``m.login.password`` :Description: @@ -216,7 +213,7 @@ To respond to this type, reply with an auth dict as follows:: } Google ReCaptcha -~~~~~~~~~~~~~~~~ +++++++++++++++++ :Type: ``m.login.recaptcha`` :Description: @@ -230,7 +227,7 @@ To respond to this type, reply with an auth dict as follows:: } Token-based -~~~~~~~~~~~ ++++++++++++ :Type: ``m.login.token`` :Description: @@ -261,7 +258,7 @@ newly provisioned access_token). The ``token`` must be a macaroon. OAuth2-based -~~~~~~~~~~~~ +++++++++++++ :Type: ``m.login.oauth2`` :Description: @@ -285,7 +282,7 @@ the OAuth flow has completed, the client retries the request with the session only, as above. Email-based (identity server) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++++++++ :Type: ``m.login.email.identity`` :Description: @@ -310,7 +307,7 @@ To respond to this type, reply with an auth dict as follows:: } Dummy Auth -~~~~~~~~~~ +++++++++++ :Type: ``m.login.dummy`` :Description: @@ -327,7 +324,7 @@ if provided:: Fallback -~~~~~~~~ +++++++++ Clients cannot be expected to be able to know how to process every single login type. If a client does not know how to handle a given login type, it can direct the user to a web browser with the URL of a fallback page which will allow the @@ -346,7 +343,15 @@ the authentication has been completed. Pagination ---------- -Querying large datasets in Matrix always uses the same pagination API pattern to +.. NOTE:: + The paths referred to in this section are not actual endpoints. They only + serve as examples to explain how pagination functions. + +Pagination is the process of dividing a dataset into multiple discrete pages. +Matrix makes use of pagination to allow clients to view extremely large datasets. +These datasets are not limited to events in a room (for example clients may want +to paginate rooms in addition to events within those rooms). Regardless of *what* +is being paginated, there is a common underlying API which is used to to give clients a consistent way of selecting subsets of a potentially changing dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters which describe where to read from the stream. ``from`` and ``to`` are opaque @@ -354,7 +359,8 @@ textual 'stream tokens' which describe the current position in the dataset. The ``dir`` parameter is an enum representing the direction of events to return: either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and ``end`` stream token values which can then be passed to subsequent requests to -continue pagination. +continue pagination. Not all endpoints will make use of all the parameters +outlined here: see the specific endpoint in question for more information. Pagination Request Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -385,10 +391,10 @@ events and doesn't know any previous events:: S E |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| | | | - | _____| | - |__________________ | _______________| - | | | - GET /somepath?to=START&limit=5&from=END + | _____| <--backwards-- | + |__________________ | | ________| + | | | | + GET /somepath?to=START&limit=5&dir=b&from=END Returns: E15,E14,E13,E12,E11 @@ -433,9 +439,6 @@ Events .. _sect:events: -Overview -~~~~~~~~ - The model of conversation history exposed by the client-server API can be considered as a list of events. The server 'linearises' the eventually-consistent event graph of events into an 'event stream' at any given @@ -463,7 +466,7 @@ You can visualise the range of events being returned as:: | | start: '1-2-3' end: 'a-b-c' -Now, to receive future events in real-time on the eventstream, you simply GET +Now, to receive future events in real-time on the event stream, you simply GET $PREFIX/events with a ``from`` parameter of 'a-b-c': in other words passing in the ``end`` token returned by initial sync. The request blocks until new events are available or until your specified timeout elapses, and then returns a @@ -493,31 +496,30 @@ To continue paginating backwards, one calls the /messages API again, supplying the new ``start`` value as the ``from`` parameter. -Receiving live updates on a client -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Syncing +~~~~~~~ -Clients receive new events by long-polling the home server via the -$PREFIX/events API, specifying a timeout in milliseconds in the timeout -parameter. This will hold open the HTTP connection for a short period of time -waiting for new events, returning early if an event occurs. This is called the -`Event Stream`_. All events which are visible to the client will appear in the -event stream. When the request returns, an ``end`` token is included in the +Clients receive new events by "long-polling" the home server via the events API. +This involves specifying a timeout in the request which will hold +open the HTTP connection for a short period of time waiting for new events, +returning early if an event occurs. Only the events API supports long-polling. +All events which are visible to the client will appear in the +events API. When the request returns, an ``end`` token is included in the response. This token can be used in the next request to continue where the -last request left off. - -All events must be de-duplicated based on their event ID. +last request left off. Multiple events can be returned per long-poll. All events +must be de-duplicated based on their event ID. .. TODO is deduplication actually a hard requirement in CS v2? .. TODO-spec - Do we ever return multiple events in a single request? - Don't we get lots of request setup RTT latency if we only do one event per request? Do we ever support streaming requests? Why not websockets? When the client first logs in, they will need to initially synchronise with -their home server. This is achieved via the |initialSync|_ API. This API also -returns an ``end`` token which can be used with the event stream. See the 'Room Sync' section below. +their home server. This is achieved via the initial sync API. This API also +returns an ``end`` token which can be used with the event stream. + +{{sync_http_api}} Events in a room ~~~~~~~~~~~~~~~~ @@ -601,29 +603,6 @@ example:: See `Room Events`_ for the ``m.`` event specification. -Syncing rooms -~~~~~~~~~~~~~ - -When a client logs in, they may have a list of rooms which they have already -joined. These rooms may also have a list of events associated with them. The -purpose of 'syncing' is to present the current room and event information in a -convenient, compact manner. The events returned are not limited to room events; -presence events will also be returned. A single syncing API is provided: - - - |initialSync|_ : A global sync which will present room and event information - for all rooms the user has joined. - -.. TODO-spec room-scoped initial sync - - |/rooms//initialSync|_ : A sync scoped to a single room. Presents - room and event information for this room only. - - Room-scoped initial sync is Very Tricky because typically people would - want to sync the room then listen for any new content from that point - onwards. The event stream cannot do this for a single room currently. - As a result, commenting room-scoped initial sync at this time. - - -{{sync_http_api}} - Getting events for a room ~~~~~~~~~~~~~~~~~~~~~~~~~ From 07d7a3fa3a4d0c9d7e4db70d62a24bc266824dbb Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 11:03:53 +0100 Subject: [PATCH 10/41] Shuffle pagination section around --- specification/1-client_server_api.rst | 201 +++++++++++++------------- 1 file changed, 103 insertions(+), 98 deletions(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index fefdf511..d8b9299a 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -340,100 +340,6 @@ This MUST return an HTML page which can perform this authentication stage. This page must attempt to call the JavaScript function ``window.onAuthDone`` when the authentication has been completed. -Pagination ----------- - -.. NOTE:: - The paths referred to in this section are not actual endpoints. They only - serve as examples to explain how pagination functions. - -Pagination is the process of dividing a dataset into multiple discrete pages. -Matrix makes use of pagination to allow clients to view extremely large datasets. -These datasets are not limited to events in a room (for example clients may want -to paginate rooms in addition to events within those rooms). Regardless of *what* -is being paginated, there is a common underlying API which is used to -to give clients a consistent way of selecting subsets of a potentially changing -dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters -which describe where to read from the stream. ``from`` and ``to`` are opaque -textual 'stream tokens' which describe the current position in the dataset. -The ``dir`` parameter is an enum representing the direction of events to return: -either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and -``end`` stream token values which can then be passed to subsequent requests to -continue pagination. Not all endpoints will make use of all the parameters -outlined here: see the specific endpoint in question for more information. - -Pagination Request Query Parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Query parameters: - from: - $streamtoken - The opaque token to start streaming from. - to: - $streamtoken - The opaque token to end streaming at. Typically, - clients will not know the item of data to end at, so this will usually be - omitted. - limit: - integer - An integer representing the maximum number of items to - return. - dir: - f|b - The direction to return events in. Typically this is ``b`` to paginate - backwards in time. - -'START' and 'END' are placeholder values used in these examples to describe the -start and end of the dataset respectively. - -Unless specified, the default pagination parameters are ``from=START``, -``to=END``, without a limit set. - -For example, if an endpoint had events E1 -> E15. The client wants the last 5 -events and doesn't know any previous events:: - - S E - |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| - | | | - | _____| <--backwards-- | - |__________________ | | ________| - | | | | - GET /somepath?to=START&limit=5&dir=b&from=END - Returns: - E15,E14,E13,E12,E11 - - -Another example: a public room list has rooms R1 -> R17. The client is showing 5 -rooms at a time on screen, and is on page 2. They want to -now show page 3 (rooms R11 -> 15):: - - S E - | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | stream token - |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room - |____________| |________________| - | | - Currently | - viewing | - | - GET /roomslist?from=9&to=END&limit=5 - Returns: R11,R12,R13,R14,R15 - -Note that tokens are treated in an *exclusive*, not inclusive, manner. The end -token from the initial request was '9' which corresponded to R10. When the 2nd -request was made, R10 did not appear again, even though from=9 was specified. If -you know the token, you already have the data. - -Pagination Response -~~~~~~~~~~~~~~~~~~~ - -Responses to pagination requests MUST follow the format:: - - { - "chunk": [ ... , Responses , ... ], - "start" : $streamtoken, - "end" : $streamtoken - } - -Where $streamtoken is an opaque token which can be used in another query to -get the next set of results. The "start" and "end" keys can only be omitted if -the complete dataset is provided in "chunk". - Events ------ @@ -521,8 +427,8 @@ returns an ``end`` token which can be used with the event stream. {{sync_http_api}} -Events in a room -~~~~~~~~~~~~~~~~ +Types of room events +~~~~~~~~~~~~~~~~~~~~ Room events are split into two categories: @@ -544,7 +450,7 @@ convention, e.g. ``com.example.myapp.event``. This ensures event types are suitably namespaced for each application and reduces the risk of clashes. State events -~~~~~~~~~~~~ +++++++++++++ State events can be sent by ``PUT`` ing to |/rooms//state//|_. These events will be @@ -587,7 +493,7 @@ In some cases, there may be no need for a ``state_key``, so it can be omitted:: See `Room Events`_ for the ``m.`` event specification. Message events -~~~~~~~~~~~~~~ +++++++++++++++ Message events can be sent by sending a request to |/rooms//send/|_. These requests *can* use transaction @@ -655,6 +561,105 @@ The redaction event should be added under the key ``redacted_because``. When a client receives a redaction event it should change the redacted event in the same way a server does. +Pagination +---------- + +.. NOTE:: + The paths referred to in this section are not actual endpoints. They only + serve as examples to explain how pagination functions. + +Pagination is the process of dividing a dataset into multiple discrete pages. +Matrix makes use of pagination to allow clients to view extremely large datasets. +These datasets are not limited to events in a room (for example clients may want +to paginate rooms in addition to events within those rooms). Regardless of *what* +is being paginated, there is a common underlying API which is used to +to give clients a consistent way of selecting subsets of a potentially changing +dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters +which describe where to read from the stream. ``from`` and ``to`` are opaque +textual 'stream tokens' which describe the current position in the dataset. +The ``dir`` parameter is an enum representing the direction of events to return: +either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and +``end`` stream token values which can then be passed to subsequent requests to +continue pagination. Not all endpoints will make use of all the parameters +outlined here: see the specific endpoint in question for more information. + +Pagination Request Query Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Query parameters: + from: + $streamtoken - The opaque token to start streaming from. + to: + $streamtoken - The opaque token to end streaming at. Typically, + clients will not know the item of data to end at, so this will usually be + omitted. + limit: + integer - An integer representing the maximum number of items to + return. + dir: + f|b - The direction to return events in. Typically this is ``b`` to paginate + backwards in time. + +'START' and 'END' are placeholder values used in these examples to describe the +start and end of the dataset respectively. + +Unless specified, the default pagination parameters are ``from=START``, +``to=END``, without a limit set. + +For example, if an endpoint had events E1 -> E15. The client wants the last 5 +events and doesn't know any previous events:: + + S E + |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| + | | | + | _____| <--backwards-- | + |__________________ | | ________| + | | | | + GET /somepath?to=START&limit=5&dir=b&from=END + Returns: + E15,E14,E13,E12,E11 + + +Another example: a public room list has rooms R1 -> R17. The client is showing 5 +rooms at a time on screen, and is on page 2. They want to +now show page 3 (rooms R11 -> 15):: + + S E + | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | stream token + |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room + |____________| |________________| + | | + Currently | + viewing | + | + GET /roomslist?from=9&to=END&limit=5 + Returns: R11,R12,R13,R14,R15 + +Note that tokens are treated in an *exclusive*, not inclusive, manner. The end +token from the initial request was '9' which corresponded to R10. When the 2nd +request was made, R10 did not appear again, even though from=9 was specified. If +you know the token, you already have the data. + +Pagination Response +~~~~~~~~~~~~~~~~~~~ + +Responses to pagination requests MUST follow the format:: + + { + "chunk": [ ... , Responses , ... ], + "start" : $streamtoken, + "end" : $streamtoken + } + +Where $streamtoken is an opaque token which can be used in another query to +get the next set of results. The "start" and "end" keys can only be omitted if +the complete dataset is provided in "chunk". + +Pagination APIs +~~~~~~~~~~~~~~~ + +{{message_pagination_http_api}} + Rooms ----- From 41fb0645a975d5bfb4e6394db1f7c4fef59463eb Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 11:16:58 +0100 Subject: [PATCH 11/41] Add message pagination API --- api/client-server/v1/message_pagination.yaml | 131 +++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 api/client-server/v1/message_pagination.yaml diff --git a/api/client-server/v1/message_pagination.yaml b/api/client-server/v1/message_pagination.yaml new file mode 100644 index 00000000..b3317c13 --- /dev/null +++ b/api/client-server/v1/message_pagination.yaml @@ -0,0 +1,131 @@ +swagger: '2.0' +info: + title: "Matrix Client-Server v1 Rooms API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/api/v1 +consumes: + - application/json +produces: + - application/json +securityDefinitions: + accessToken: + type: apiKey + description: The user_id or application service access_token + name: access_token + in: query +paths: + "/rooms/{roomId}/messages": + get: + summary: Get a list of events for this room + description: |- + This API returns a list of message and state events for a room. It uses + pagination query parameters to paginate history in the room. + security: + - accessToken: [] + parameters: + - in: path + type: string + name: roomId + description: The room to get events from. + required: true + x-example: "!636q39766251:example.com" + - in: query + type: string + name: from + description: |- + The token to start returning events from. This token can be obtained + from the initial sync API. + required: true + x-example: "s345_678_333" + - in: query + type: string + enum: ["b", "f"] + name: dir + description: |- + The direction to return events from. + required: true + x-example: "b" + - in: query + type: integer + name: limit + description: |- + The maximum number of events to return. Default: 10. + x-example: "3" + responses: + 200: + description: A list of messages with a new token to request more. + schema: + type: object + description: A list of messages with a new token to request more. + properties: + start: + type: string + description: |- + The token to start paginating from. If ``dir=b`` this will be + the token supplied in ``from``. + end: + type: string + description: |- + The token the pagination ends at. If ``dir=b`` this token should + be used again to request even earlier events. + chunk: + type: array + description: |- + A list of room events. + items: + type: object + title: RoomEvent + examples: + application/json: |- + { + "start": "t47429-4392820_219380_26003_2265", + "end": "t47409-4357353_219380_26003_2265", + "chunk": [ + { + "origin_server_ts": 1444812213737, + "user_id": "@alice:example.com", + "event_id": "$1444812213350496Caaaa:example.com", + "content": { + "body": "hello world", + "msgtype":"m.text" + }, + "room_id":"!Xq3620DUiqCaoxq:example.com", + "type":"m.room.message", + "age": 1042 + }, + { + "origin_server_ts": 1444812194656 , + "user_id": "@bob:example.com", + "event_id": "$1444812213350496Cbbbb:example.com", + "content": { + "body": "the world is big", + "msgtype":"m.text" + }, + "room_id":"!Xq3620DUiqCaoxq:example.com", + "type":"m.room.message", + "age": 20123 + }, + { + "origin_server_ts": 1444812163990, + "user_id": "@bob:example.com", + "event_id": "$1444812213350496Ccccc:example.com", + "content": { + "name": "New room name" + }, + "prev_content": { + "name": "Old room name" + }, + "state_key": "", + "room_id":"!Xq3620DUiqCaoxq:example.com", + "type":"m.room.name", + "age": 50789 + } + ] + } + 403: + description: > + You aren't a member of the room. \ No newline at end of file From 6f6861a11d45b86c5cbcad7b864fd527a77f5724 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 13:17:51 +0100 Subject: [PATCH 12/41] Swaggerify the createRoom API --- api/client-server/v1/create_room.yaml | 148 ++++++++++++++++++++++++++ specification/1-client_server_api.rst | 135 ++--------------------- 2 files changed, 155 insertions(+), 128 deletions(-) create mode 100644 api/client-server/v1/create_room.yaml diff --git a/api/client-server/v1/create_room.yaml b/api/client-server/v1/create_room.yaml new file mode 100644 index 00000000..5bb4e17c --- /dev/null +++ b/api/client-server/v1/create_room.yaml @@ -0,0 +1,148 @@ +swagger: '2.0' +info: + title: "Matrix Client-Server v1 Room Creation API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/api/v1 +consumes: + - application/json +produces: + - application/json +securityDefinitions: + accessToken: + type: apiKey + description: The user_id or application service access_token + name: access_token + in: query +paths: + "/createRoom": + post: + summary: Create a new room + description: |- + Create a new room with various configuration options. + security: + - accessToken: [] + parameters: + - in: body + name: body + description: The desired room configuration. + schema: + type: object + example: |- + { + "preset": "public_chat", + "room_alias_name": "thepub", + "name": "The Grand Duke Pub", + "topic": "All about happy hour", + "creation_content": { + "m.federate": false + } + } + properties: + visibility: + type: string + enum: ["public", "private"] + description: |- + A ``public`` visibility indicates that the room will be shown + in the published room list. A ``private`` visibility will hide + the room from the published room list. Rooms default to + ``private`` visibility if this key is not included. NB: This + should not be confused with ``join_rules`` which also uses the + word ``public``. + room_alias_name: + type: string + description: |- + The desired room alias **local part**. If this is included, a + room alias will be created and mapped to the newly created + room. The alias will belong on the *same* home server which + created the room. For example, if this was set to "foo" and + sent to the homeserver "example.com" the complete room alias + would be ``#foo:example.com``. + name: + type: string + description: |- + If this is included, an ``m.room.name`` event will be sent + into the room to indicate the name of the room. See Room + Events for more information on ``m.room.name``. + topic: + type: string + description: |- + If this is included, an ``m.room.topic`` event will be sent + into the room to indicate the topic for the room. See Room + Events for more information on ``m.room.topic``. + invite: + type: array + description: |- + A list of user IDs to invite to the room. This will tell the + server to invite everyone in the list to the newly created room. + items: + type: string + creation_content: + title: CreationContent + type: object + description: |- + Extra keys to be added to the content of the ``m.room.create``. + The server will clober the following keys: ``creator``. Future + versions of the specification may allow the server to clobber + other keys. + initial_state: + type: array + description: |- + A list of state events to set in the new room. This allows + the user to override the default state events set in the new + room. The expected format of the state events are an object + with type, state_key and content keys set. + Takes precedence over events set by ``presets``, but gets + overriden by ``name`` and ``topic`` keys. + items: + type: object + title: StateEvent + properties: + type: + type: string + state_key: + type: string + content: + type: string + preset: + type: string + enum: ["private_chat", "public_chat", "trusted_private_chat"] + description: |- + Convenience parameter for setting various default state events + based on a preset. Must be either: + + ``private_chat`` => + ``join_rules`` is set to ``invite``. + ``history_visibility`` is set to ``shared``. + + ``trusted_private_chat`` => + ``join_rules`` is set to ``invite``. + ``history_visibility`` is set to ``shared``. + All invitees are given the same power level as the room creator. + + ``public_chat``: => + ``join_rules`` is set to ``public``. + ``history_visibility`` is set to ``shared``. + + responses: + 200: + description: Information about the newly created room. + schema: + type: object + description: Information about the newly created room. + properties: + room_id: + type: string + description: |- + The created room's ID. + examples: + application/json: |- + { + "room_id": "!sefiuhWgwghwWgh:example.com" + } + 400: + description: > + The request body is malformed or the room alias specified is already taken. \ No newline at end of file diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index d8b9299a..f27e4d42 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -665,140 +665,19 @@ Rooms Creation ~~~~~~~~ -To create a room, a client has to use the |createRoom|_ API. There are various -options which can be set when creating a room: - -``visibility`` - Type: - String - Optional: - Yes - Value: - Either ``public`` or ``private``. - Description: - A ``public`` visibility indicates that the room will be shown in the public - room list. A ``private`` visibility will hide the room from the public room - list. Rooms default to ``private`` visibility if this key is not included. - -``room_alias_name`` - Type: - String - Optional: - Yes - Value: - The room alias localpart. - Description: - If this is included, a room alias will be created and mapped to the newly - created room. The alias will belong on the same home server which created - the room, e.g. ``!qadnasoi:domain.com >>> #room_alias_name:domain.com`` - -``name`` - Type: - String - Optional: - Yes - Value: - The ``name`` value for the ``m.room.name`` state event. - Description: - If this is included, an ``m.room.name`` event will be sent into the room to - indicate the name of the room. See `Room Events`_ for more information on - ``m.room.name``. - -``topic`` - Type: - String - Optional: - Yes - Value: - The ``topic`` value for the ``m.room.topic`` state event. - Description: - If this is included, an ``m.room.topic`` event will be sent into the room - to indicate the topic for the room. See `Room Events`_ for more information - on ``m.room.topic``. - -``invite`` - Type: - List - Optional: - Yes - Value: - A list of user ids to invite. - Description: - This will tell the server to invite everyone in the list to the newly - created room. - -``creation_content`` - Type: - Object - Optional: - Yes - Value: - Extra keys to be added to the content of the ``m.room.create``. The server - will clober the following keys: ``creator``. Future versions of this - spec may allow the server to clobber other keys if required. - Description: - Allows clients to add keys to the content of ``m.room.create``. - -``preset`` - Type: - String - Optional: - Yes - Value: - ``private_chat``, ``trusted_private_chat`` or ``public_chat`` - Description: - Convenience parameter for setting various default state events based on a - preset. - - Three presets are defined: - - - ``private_chat``: Sets the ``join_rules`` to ``invite`` and - ``history_visibility`` to ``shared`` - - ``trusted_private_chat``: Set the ``join_rules`` to ``invite``, - ``history_visibility`` to ``shared`` and gives all invitees the same - power level as the creator. - - ``public_chat``: Sets the ``join_rules`` to ``public`` and - ``history_visibility`` to ``shared`` - -``initial_state`` - Type: - List - Optional: - Yes - Value: - A list of state events to set in the new room. - Description: - Allows the user to override the default state events set in the new room. - - The expected format of the state events are an object with ``type``, - ``state_key`` and ``content`` keys set. - - Takes precedence over events set by ``presets``, but gets overriden by - ``name`` and ``topic`` keys. - -Example:: - - { - "preset": "public_chat", - "room_alias_name": "thepub", - "name": "The Grand Duke Pub", - "topic": "All about happy hour", - "creation_content": { - "m.federate": false - } - } - -The home server will create a ``m.room.create`` event when the room is created, -which serves as the root of the PDU graph for this room. This event also has a +The home server will create a ``m.room.create`` event when a room is created, +which serves as the root of the event graph for this room. This event also has a ``creator`` key which contains the user ID of the room creator. It will also generate several other events in order to manage permissions in this room. This includes: - - ``m.room.power_levels`` : Sets the power levels of users and required power - levels. + - ``m.room.power_levels`` : Sets the power levels of users and required power levels. - ``m.room.join_rules`` : Whether the room is "invite-only" or not. -See `Room Events`_ for more information on these events. +See `Room Events`_ for more information on these events. To create a room, a +client has to use the the following API. + +{{create_room_http_api}} Room aliases ~~~~~~~~~~~~ From 34bd8edec5395a8accde330be4b04dbca41f770b Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 13:26:58 +0100 Subject: [PATCH 13/41] Move related auth sections together --- specification/0-events.rst | 4 +- specification/1-client_server_api.rst | 274 +++++++++++++------------- 2 files changed, 138 insertions(+), 140 deletions(-) diff --git a/specification/0-events.rst b/specification/0-events.rst index 16948462..64007568 100644 --- a/specification/0-events.rst +++ b/specification/0-events.rst @@ -1,5 +1,5 @@ -Events -====== +Event Structure +=============== All communication in Matrix is expressed in the form of data objects called Events. These are the fundamental building blocks common to the client-server, diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index f27e4d42..319fed91 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -340,6 +340,141 @@ This MUST return an HTML page which can perform this authentication stage. This page must attempt to call the JavaScript function ``window.onAuthDone`` when the authentication has been completed. +Registration +~~~~~~~~~~~~ +This section refers to API Version 2. These API calls currently use the prefix +``/_matrix/client/v2_alpha``. + +Registering for a user account is done using the request:: + + POST $V2PREFIX/register + +This API endpoint uses the User-Interactive Authentication API. +This API endpoint does not require an access token. + +The body of the POST request is a JSON object containing: + +username + Optional. This is the local part of the desired Matrix ID. If omitted, the + Home Server must generate a Matrix ID local part. +password + Required. The desired password for the account. +bind_email + Optional. If ``true``, the server binds the email used for authentication to + the Matrix ID with the ID Server. + +On success, this returns a JSON object with keys: + +user_id + The fully-qualified Matrix ID that has been registered. +access_token + An access token for the new account. +home_server + The hostname of the Home Server on which the account has been registered. + +This endpoint may also return the following error codes: + +M_USER_IN_USE + If the Matrix ID is already in use +M_EXCLUSIVE + If the requested Matrix ID is in the exclusive namespace of an application + service. + +Home Servers MUST perform the relevant checks and return these codes before +performing User-Interactive Authentication, although they may also return +them after authentication is completed if, for example, the requested user ID +was registered whilst the client was performing authentication. + +Old V1 API docs: |register|_ + +{{login_http_api}} + +Changing Password ++++++++++++++++++ +This section refers to API Version 2. These API calls currently use the prefix +``/_matrix/client/v2_alpha``. + +Request:: + + POST $V2PREFIX/account/password + +This API endpoint uses the User-Interactive Authentication API. An access token +should be submitted to this endpoint if the client has an active session. The +Home Server may change the flows available depending on whether a valid access +token is provided. + +The body of the POST request is a JSON object containing: + +new_password + The new password for the account. + +On success, an empty JSON object is returned. + +The error code M_NOT_FOUND is returned if the user authenticated with a third +party identifier but the Home Server could not find a matching account in its +database. + +Adding a Third Party Identifier ++++++++++++++++++++++++++++++++ +This section refers to API Version 2. These API calls currently use the prefix +``/_matrix/client/v2_alpha``. + +Request:: + + POST $V2PREFIX/account/3pid + +Used to add a third party identifier to the user's account. + +The body of the POST request is a JSON object containing: + +threePidCreds + An object containing third party identifier credentials. +bind + Optional. A boolean indicating whether the Home Server should also bind this + third party identifier to the account's matrix ID with the Identity Server. If + supplied and true, the Home Server must bind the 3pid accordingly. + +The third party identifier credentials object comprises: + +id_server + The colon-separated hostname and port of the Identity Server used to + authenticate the third party identifier. If the port is the default, it and the + colon should be omitted. +sid + The session ID given by the Identity Server +client_secret + The client secret used in the session with the Identity Server. + +On success, the empty JSON object is returned. + +May also return error codes: + +M_THREEPID_AUTH_FAILED + If the credentials provided could not be verified with the ID Server. + +Fetching Currently Associated Third Party Identifiers ++++++++++++++++++++++++++++++++++++++++++++++++++++++ +This section refers to API Version 2. These API calls currently use the prefix +``/_matrix/client/v2_alpha``. + +Request:: + + GET $V2PREFIX/account/3pid + +This returns a list of third party identifiers that the Home Server has +associated with the user's account. This is *not* the same as the list of third +party identifiers bound to the user's Matrix ID in Identity Servers. Identifiers +in this list may be used by the Home Server as, for example, identifiers that it +will accept to reset the user's account password. + +Returns a JSON object with the key ``threepids`` whose contents is an array of +objects with the following keys: + +medium + The medium of the 3pid (eg, ``email``) +address + The textual address of the 3pid, eg. the email address + Events ------ @@ -742,7 +877,7 @@ certain operations such as kicking, banning and sending state events. See `m.room.power_levels`_ for more information. Joining rooms -------------- +~~~~~~~~~~~~~ Users need to be a member of a room in order to send and receive events in that room. There are several states in which a user may be, in relation to a room: @@ -821,143 +956,6 @@ member's state, by making a request to "membership": "ban" } - -Registration ------------- -This section refers to API Version 2. These API calls currently use the prefix -``/_matrix/client/v2_alpha``. - -Registering for a user account is done using the request:: - - POST $V2PREFIX/register - -This API endpoint uses the User-Interactive Authentication API. -This API endpoint does not require an access token. - -The body of the POST request is a JSON object containing: - -username - Optional. This is the local part of the desired Matrix ID. If omitted, the - Home Server must generate a Matrix ID local part. -password - Required. The desired password for the account. -bind_email - Optional. If ``true``, the server binds the email used for authentication to - the Matrix ID with the ID Server. - -On success, this returns a JSON object with keys: - -user_id - The fully-qualified Matrix ID that has been registered. -access_token - An access token for the new account. -home_server - The hostname of the Home Server on which the account has been registered. - -This endpoint may also return the following error codes: - -M_USER_IN_USE - If the Matrix ID is already in use -M_EXCLUSIVE - If the requested Matrix ID is in the exclusive namespace of an application - service. - -Home Servers MUST perform the relevant checks and return these codes before -performing User-Interactive Authentication, although they may also return -them after authentication is completed if, for example, the requested user ID -was registered whilst the client was performing authentication. - -Old V1 API docs: |register|_ - -{{login_http_api}} - -Changing Password -~~~~~~~~~~~~~~~~~ -This section refers to API Version 2. These API calls currently use the prefix -``/_matrix/client/v2_alpha``. - -Request:: - - POST $V2PREFIX/account/password - -This API endpoint uses the User-Interactive Authentication API. An access token -should be submitted to this endpoint if the client has an active session. The -Home Server may change the flows available depending on whether a valid access -token is provided. - -The body of the POST request is a JSON object containing: - -new_password - The new password for the account. - -On success, an empty JSON object is returned. - -The error code M_NOT_FOUND is returned if the user authenticated with a third -party identifier but the Home Server could not find a matching account in its -database. - -Adding a Third Party Identifier -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This section refers to API Version 2. These API calls currently use the prefix -``/_matrix/client/v2_alpha``. - -Request:: - - POST $V2PREFIX/account/3pid - -Used to add a third party identifier to the user's account. - -The body of the POST request is a JSON object containing: - -threePidCreds - An object containing third party identifier credentials. -bind - Optional. A boolean indicating whether the Home Server should also bind this - third party identifier to the account's matrix ID with the Identity Server. If - supplied and true, the Home Server must bind the 3pid accordingly. - -The third party identifier credentials object comprises: - -id_server - The colon-separated hostname and port of the Identity Server used to - authenticate the third party identifier. If the port is the default, it and the - colon should be omitted. -sid - The session ID given by the Identity Server -client_secret - The client secret used in the session with the Identity Server. - -On success, the empty JSON object is returned. - -May also return error codes: - -M_THREEPID_AUTH_FAILED - If the credentials provided could not be verified with the ID Server. - -Fetching Currently Associated Third Party Identifiers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This section refers to API Version 2. These API calls currently use the prefix -``/_matrix/client/v2_alpha``. - -Request:: - - GET $V2PREFIX/account/3pid - -This returns a list of third party identifiers that the Home Server has -associated with the user's account. This is *not* the same as the list of third -party identifiers bound to the user's Matrix ID in Identity Servers. Identifiers -in this list may be used by the Home Server as, for example, identifiers that it -will accept to reset the user's account password. - -Returns a JSON object with the key ``threepids`` whose contents is an array of -objects with the following keys: - -medium - The medium of the 3pid (eg, ``email``) -address - The textual address of the 3pid, eg. the email address - - Profiles -------- From 4a9db39277a6f61bb4efc27bb6a14815a89acd05 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 14 Oct 2015 13:38:28 +0100 Subject: [PATCH 14/41] Clean up some untruths in the login api doc add "type", and "username" -> "user" --- api/client-server/v1/login.yaml | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml index 3d415c29..bbab46df 100644 --- a/api/client-server/v1/login.yaml +++ b/api/client-server/v1/login.yaml @@ -33,17 +33,21 @@ paths: type: object example: |- { - "username": "cheeky_monkey", + "type": "m.login.pasword", + "user": "cheeky_monkey", "password": "ilovebananas" } properties: - username: + type: + type: string + description: The login type being used. Currently only "m.login.password" is supported. + user: type: string description: The fully qualified user ID or just local part of the user ID, to log in. password: type: string description: The user's password. - required: ["username", "password"] + required: ["type", "user", "password"] responses: 200: description: The user has been authenticated. @@ -78,6 +82,15 @@ paths: home_server: type: string description: The hostname of the Home Server on which the account has been registered. + 400: + description: |- + Part of the request was invalid. For example, the login type may not be recognised. + examples: + application/json: |- + { + "errcode": "M_UNKNOWN", + "error": "Bad login type." + } 403: description: |- The login attempt failed. For example, the password may have been incorrect. From 270e1d7aa81b8692dbffe4604df81c96de99d6cb Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 14 Oct 2015 14:29:36 +0100 Subject: [PATCH 15/41] Add richvdh to list of trusted pushers for spectulator richvdh is awesome, he should have all the permissions. --- scripts/speculator/main.go | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go index d641d7da..dab624df 100644 --- a/scripts/speculator/main.go +++ b/scripts/speculator/main.go @@ -364,6 +364,7 @@ func main() { "illicitonion": true, "Kegsay": true, "NegativeMjark": true, + "richvdh": true, } rand.Seed(time.Now().Unix()) masterCloneDir, err := gitClone(matrixDocCloneURL, false) From 013b6db8958b7b02000044ee6829ccbdbd24a08f Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 14 Oct 2015 14:42:52 +0100 Subject: [PATCH 16/41] Add a note on building the speculator ... because the wiki says you need to read the README on how to build it. --- scripts/speculator/README | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/speculator/README b/scripts/speculator/README index 0a9f53fd..c54e0f2b 100644 --- a/scripts/speculator/README +++ b/scripts/speculator/README @@ -8,3 +8,7 @@ It serves the following HTTP endpoints: To run it, you must install the `go` tool, and run: `go run main.go` + +To build the binary (which is necessary for deployment to the matrix.org +servers), you must again install `go`, and then run: + `go build` From 3608f3a282d9d9eae775dafb30eb22b813d319a6 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 14:44:09 +0100 Subject: [PATCH 17/41] SPEC-144 : Use https examples not http. --- specification/modules/push.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specification/modules/push.rst b/specification/modules/push.rst index 6e9c8536..525e730d 100644 --- a/specification/modules/push.rst +++ b/specification/modules/push.rst @@ -353,14 +353,14 @@ Examples To create a rule that suppresses notifications for the room with ID ``!dj234r78wl45Gh4D:matrix.org``:: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" -d \ '{ "actions" : ["dont_notify"] }' To suppress notifications for the user ``@spambot:matrix.org``:: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" -d \ '{ "actions" : ["dont_notify"] }' @@ -368,7 +368,7 @@ To suppress notifications for the user ``@spambot:matrix.org``:: To always notify for messages that contain the work 'cake' and set a specific sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``):: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" -d \ '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] @@ -377,7 +377,7 @@ sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``):: To add a rule suppressing notifications for messages starting with 'cake' but ending with 'lie', superseding the previous rule:: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" -d \ '{ "pattern": "cake*lie", "actions" : ["notify"] @@ -387,7 +387,7 @@ To add a custom sound for notifications messages containing the word 'beer' in any rooms with 10 members or fewer (with greater importance than the room, sender and content rules):: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456" -d \ '{ "conditions": [ {"kind": "event_match", "key": "content.body", "pattern": "beer" }, From cd34d4e19b334c3a8165bb48deaa5adc7085fecd Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 15:12:15 +0100 Subject: [PATCH 18/41] SPEC-182: Make it clearer how ASes are supposed to create users by linkifying the right text. --- specification/0-feature_profiles.rst | 2 +- specification/3-application_service_api.rst | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index b9f12b74..6c11dd2f 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -1,7 +1,7 @@ Feature Profiles ================ -.. sect:feature-profiles: +.. _sect:feature-profiles: Matrix supports many different kinds of clients: from embedded IoT devices to desktop clients. Not all clients can provide the same feature sets as other diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst index a6e82137..d1799158 100644 --- a/specification/3-application_service_api.rst +++ b/specification/3-application_service_api.rst @@ -94,7 +94,7 @@ API called when: - HS receives an event for an unknown user ID in the AS's namespace, e.g. an invite event to a room. Notes: - - When the AS receives this request, if the user exists, it must create the user via + - When the AS receives this request, if the user exists, it must `create the user`_ via the CS API. - It can also set arbitrary information about the user (e.g. display name, join rooms, etc) using the CS API. @@ -226,6 +226,9 @@ Ordering notes: ] } + +.. _create the user: `sect:asapi-permissions`_ + Client-Server v2 API Extensions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -291,6 +294,9 @@ Notes: Server admin style permissions ++++++++++++++++++++++++++++++ + +.. _sect:asapi-permissions: + The home server needs to give the application service *full control* over its namespace, both for users and for room aliases. This means that the AS should be able to create/edit/delete any room alias in its namespace, as well as From 45b1d08c7e338586d1a716e447e3f025a960aaa6 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 15:16:12 +0100 Subject: [PATCH 19/41] SPEC-207: Add a warning about needing AS tokens to be unique and why --- specification/3-application_service_api.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst index a6e82137..557e16aa 100644 --- a/specification/3-application_service_api.rst +++ b/specification/3-application_service_api.rst @@ -54,6 +54,11 @@ An example HS configuration required to pass traffic to the AS is: aliases: [] # Namespaces of room aliases which should be delegated to the AS rooms: [] # Namespaces of room ids which should be delegated to the AS +.. WARNING:: + If the homeserver in question has multiple application services, each + ``as_token`` MUST be unique per application service as this token is used to + identify the application service. + - An application service can state whether they should be the only ones who can manage a specified namespace. This is referred to as an "exclusive" namespace. An exclusive namespace prevents humans and other application From 44b19a8b0e16a25950899c4f5d90b2ea99b4de69 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 15:27:56 +0100 Subject: [PATCH 20/41] SPEC-165: Outline directory traversal attacks on MXC URIs --- specification/modules/content_repo.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst index 9ac5e199..52937ead 100644 --- a/specification/modules/content_repo.rst +++ b/specification/modules/content_repo.rst @@ -52,7 +52,17 @@ The HTTP GET endpoint does not require any authentication. Knowing the URL of the content is sufficient to retrieve the content, even if the entity isn't in the room. -Homeservers have additional concerns: +MXC URIs are vulnerable to directory traversal attacks such as +``mxc://127.0.0.1/../../../some_service/etc/passwd``. This would cause the target +homeserver to try to access and return this file. As such, homeservers MUST +sanitise MXC URIs by allowing only alphanumeric (``A-Za-z0-9``), ``_`` +and ``-`` characters in the ``server-name`` and ``media-id`` values. This set +of whitelisted characters allows URL-safe base64 encodings specified in RFC 4648. +Applying this character whitelist is preferable to blacklisting ``.`` and ``/`` +as there are techniques around blacklisted characters (percent-encoded characters, +UTF-8 encoded traversals, etc). + +Homeservers have additional content-specific concerns: - Clients may try to upload very large files. Homeservers should not store files that are too large and should not serve them to clients. From f0d80529517eddac9432daab9984c38784528a76 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 15:42:44 +0100 Subject: [PATCH 21/41] SPEC-205: Warn about password strengths in m.login.password section --- specification/1-client_server_api.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 59e6b68e..a63c114a 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -215,6 +215,12 @@ To respond to this type, reply with an auth dict as follows:: "password": "" } +.. WARNING:: + Clients SHOULD enforce that the password provided is suitably complex. The + password SHOULD include a lower-case letter, an upper-case letter, a number + and a symbol and be at a minimum 8 characters in length. Servers MAY reject + weak passwords with an error code ``M_WEAK_PASSWORD``. + Google ReCaptcha ~~~~~~~~~~~~~~~~ :Type: From 6a0595bc46c3751090d5d24e2ab94647c9573915 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 16:15:55 +0100 Subject: [PATCH 22/41] Add in size limits as per SPEC-222 --- .../v1/core-event-schema/state_event.json | 2 +- event-schemas/schema/v1/m.room.name | 2 +- specification/0-events.rst | 17 +++++++++++++++++ specification/0-intro.rst | 2 +- 4 files changed, 20 insertions(+), 3 deletions(-) diff --git a/event-schemas/schema/v1/core-event-schema/state_event.json b/event-schemas/schema/v1/core-event-schema/state_event.json index f70118f4..88b4900a 100644 --- a/event-schemas/schema/v1/core-event-schema/state_event.json +++ b/event-schemas/schema/v1/core-event-schema/state_event.json @@ -8,7 +8,7 @@ "properties": { "state_key": { "type": "string", - "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event." + "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event. The key MUST NOT start with '_'." }, "prev_content": { "type": "object", diff --git a/event-schemas/schema/v1/m.room.name b/event-schemas/schema/v1/m.room.name index b43f02cc..0e0b25be 100644 --- a/event-schemas/schema/v1/m.room.name +++ b/event-schemas/schema/v1/m.room.name @@ -11,7 +11,7 @@ "properties": { "name": { "type": "string", - "description": "The name of the room." + "description": "The name of the room. This MUST NOT exceed 255 bytes." } }, "required": ["name"] diff --git a/specification/0-events.rst b/specification/0-events.rst index 16948462..ae5a698a 100644 --- a/specification/0-events.rst +++ b/specification/0-events.rst @@ -12,6 +12,23 @@ server-server and application-service APIs, and are described below. {{common_state_event_fields}} +Size limits +----------- + +The total size of any event MUST NOT exceed 65 KB. There are additional +restrictions on sizes per key: + + - ``user_id`` MUST NOT exceed 255 bytes (including domain). + - ``room_id`` MUST NOT exceed 255 bytes. + - ``state_key`` MUST NOT exceed 255 bytes. + - ``type`` MUST NOT exceed 255 bytes. + - ``event_id`` MUST NOT exceed 255 bytes. + - ``user_id`` MUST NOT exceed 255 bytes. + +Some event types have additional sizes restrictions which are specified in +the description of the event. Additional keys have no limit other than that +implied by the total 65 KB limit on events. + Room Events ----------- .. NOTE:: diff --git a/specification/0-intro.rst b/specification/0-intro.rst index 445b32ef..e5cf1ac8 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -296,7 +296,7 @@ be obtained by visiting the domain specified. They are case-insensitive. Note that the mapping from a room alias to a room ID is not fixed, and may change over time to point to a different room ID. For this reason, Clients SHOULD resolve the room alias to a room ID once and then use that ID on subsequent -requests. +requests. Room aliases MUST NOT exceed 255 bytes. When resolving a room alias the server will also respond with a list of servers that are in the room that can be used to join via. From 70704240dcc16ab8e4c4a982ab5b73a0cc047d44 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 14 Oct 2015 16:29:18 +0100 Subject: [PATCH 23/41] gendoc should just add newlines rather than complaining about the lack thereof --- scripts/gendoc.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 8db604c4..74a6c1da 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -128,11 +128,8 @@ def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles): ) else: rst = f.read() - if rst[-2:] != "\n\n": - raise Exception( - ("File %s should end with TWO new-line characters to ensure " + - "file concatenation works correctly.") % (file_info,) - ) + + rst += "\n\n" return rst # dicts look like {0: filepath, 1: filepath} where the key is the title level elif isinstance(file_info, dict): From 84af5776d7487473aae3d8679120e8d3992b4d4e Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 17:02:39 +0100 Subject: [PATCH 24/41] Newlines --- api/client-server/v1/create_room.yaml | 2 +- api/client-server/v1/message_pagination.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/api/client-server/v1/create_room.yaml b/api/client-server/v1/create_room.yaml index 5bb4e17c..051c4b84 100644 --- a/api/client-server/v1/create_room.yaml +++ b/api/client-server/v1/create_room.yaml @@ -145,4 +145,4 @@ paths: } 400: description: > - The request body is malformed or the room alias specified is already taken. \ No newline at end of file + The request body is malformed or the room alias specified is already taken. diff --git a/api/client-server/v1/message_pagination.yaml b/api/client-server/v1/message_pagination.yaml index b3317c13..d2bc0554 100644 --- a/api/client-server/v1/message_pagination.yaml +++ b/api/client-server/v1/message_pagination.yaml @@ -128,4 +128,4 @@ paths: } 403: description: > - You aren't a member of the room. \ No newline at end of file + You aren't a member of the room. From 1f2f14dc08a8b4dbed83e37dfc667a0777e4accb Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 17:07:25 +0100 Subject: [PATCH 25/41] YAML tweaks --- api/client-server/v1/sync.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml index 398b0773..50af8546 100644 --- a/api/client-server/v1/sync.yaml +++ b/api/client-server/v1/sync.yaml @@ -31,7 +31,7 @@ paths: type: string name: from description: |- - The token to stream from. This token is either from the a previous + The token to stream from. This token is either from a previous request to this API or from the initial sync API. required: false x-example: "s3456_9_0" @@ -45,10 +45,10 @@ paths: type: boolean name: archived description: |- - Whether to include rooms that the user has left. If absent then + Whether to include rooms that the user has left. If ``false`` then only rooms that the user has been invited to or has joined are - included. If set to "true" then rooms that the user has left are - included as well. + included. If set to ``true`` then rooms that the user has left are + included as well. By default this is ``false``. required: false x-example: "true" responses: From e6eb19c46115f1e324fa3c474341c627b4219cf9 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 17:34:25 +0100 Subject: [PATCH 26/41] Review comments --- specification/1-client_server_api.rst | 37 +++++++++++++++++---------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 319fed91..ecd5a3e2 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -29,6 +29,9 @@ return with a status of 401 and the error code, ``M_MISSING_TOKEN`` or User-Interactive Authentication API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. _sect:auth-api: + This section refers to API Version 2. Some API endpoints such as ``login`` or ``register`` require authentication that @@ -349,9 +352,11 @@ Registering for a user account is done using the request:: POST $V2PREFIX/register -This API endpoint uses the User-Interactive Authentication API. +This API endpoint uses the `User-Interactive Authentication API`_. This API endpoint does not require an access token. +.. _User-Interactive Authentication API: `sect:auth-api`_ + The body of the POST request is a JSON object containing: username @@ -371,6 +376,9 @@ access_token An access token for the new account. home_server The hostname of the Home Server on which the account has been registered. +refresh_token + A token that may be exchanged for a new ``access_token`` using the + ``/tokenrefresh`` API endpoint. This endpoint may also return the following error codes: @@ -381,10 +389,12 @@ M_EXCLUSIVE service. Home Servers MUST perform the relevant checks and return these codes before -performing User-Interactive Authentication, although they may also return +performing `User-Interactive Authentication`_, although they may also return them after authentication is completed if, for example, the requested user ID was registered whilst the client was performing authentication. +.. _User-Interactive Authentication: `sect:auth-api`_ + Old V1 API docs: |register|_ {{login_http_api}} @@ -414,8 +424,8 @@ The error code M_NOT_FOUND is returned if the user authenticated with a third party identifier but the Home Server could not find a matching account in its database. -Adding a Third Party Identifier -+++++++++++++++++++++++++++++++ +Adding Account Administrative Contact Information ++++++++++++++++++++++++++++++++++++++++++++++++++ This section refers to API Version 2. These API calls currently use the prefix ``/_matrix/client/v2_alpha``. @@ -423,18 +433,18 @@ Request:: POST $V2PREFIX/account/3pid -Used to add a third party identifier to the user's account. +Used to add contact information to the user's account. The body of the POST request is a JSON object containing: threePidCreds - An object containing third party identifier credentials. + An object containing contact information. bind Optional. A boolean indicating whether the Home Server should also bind this third party identifier to the account's matrix ID with the Identity Server. If supplied and true, the Home Server must bind the 3pid accordingly. -The third party identifier credentials object comprises: +The contact information object comprises: id_server The colon-separated hostname and port of the Identity Server used to @@ -452,8 +462,8 @@ May also return error codes: M_THREEPID_AUTH_FAILED If the credentials provided could not be verified with the ID Server. -Fetching Currently Associated Third Party Identifiers -+++++++++++++++++++++++++++++++++++++++++++++++++++++ +Fetching Currently Associated Contact Information ++++++++++++++++++++++++++++++++++++++++++++++++++ This section refers to API Version 2. These API calls currently use the prefix ``/_matrix/client/v2_alpha``. @@ -706,8 +716,8 @@ Pagination Pagination is the process of dividing a dataset into multiple discrete pages. Matrix makes use of pagination to allow clients to view extremely large datasets. These datasets are not limited to events in a room (for example clients may want -to paginate rooms in addition to events within those rooms). Regardless of *what* -is being paginated, there is a common underlying API which is used to +to paginate a list of rooms in addition to events within those rooms). Regardless +of *what* is being paginated, there is a common underlying API which is used to to give clients a consistent way of selecting subsets of a potentially changing dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters which describe where to read from the stream. ``from`` and ``to`` are opaque @@ -800,13 +810,14 @@ Rooms Creation ~~~~~~~~ -The home server will create a ``m.room.create`` event when a room is created, +The home server will create an ``m.room.create`` event when a room is created, which serves as the root of the event graph for this room. This event also has a ``creator`` key which contains the user ID of the room creator. It will also generate several other events in order to manage permissions in this room. This includes: - - ``m.room.power_levels`` : Sets the power levels of users and required power levels. + - ``m.room.power_levels`` : Sets the power levels of users and required power + levels for various actions within the room such as sending events. - ``m.room.join_rules`` : Whether the room is "invite-only" or not. See `Room Events`_ for more information on these events. To create a room, a From c4d1b568435aca0216762e1331a6d42f49bf5f68 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 09:57:21 +0100 Subject: [PATCH 27/41] Add a comment about enforcement. --- specification/3-application_service_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst index 557e16aa..8d0efdef 100644 --- a/specification/3-application_service_api.rst +++ b/specification/3-application_service_api.rst @@ -57,7 +57,7 @@ An example HS configuration required to pass traffic to the AS is: .. WARNING:: If the homeserver in question has multiple application services, each ``as_token`` MUST be unique per application service as this token is used to - identify the application service. + identify the application service. The homeserver MUST enforce this. - An application service can state whether they should be the only ones who can manage a specified namespace. This is referred to as an "exclusive" From c82c07ddd26055cb832d00e8df920a571ead9074 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 09:58:39 +0100 Subject: [PATCH 28/41] Review comments --- specification/0-events.rst | 3 +-- specification/0-intro.rst | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/specification/0-events.rst b/specification/0-events.rst index ae5a698a..f24ad984 100644 --- a/specification/0-events.rst +++ b/specification/0-events.rst @@ -23,9 +23,8 @@ restrictions on sizes per key: - ``state_key`` MUST NOT exceed 255 bytes. - ``type`` MUST NOT exceed 255 bytes. - ``event_id`` MUST NOT exceed 255 bytes. - - ``user_id`` MUST NOT exceed 255 bytes. -Some event types have additional sizes restrictions which are specified in +Some event types have additional size restrictions which are specified in the description of the event. Additional keys have no limit other than that implied by the total 65 KB limit on events. diff --git a/specification/0-intro.rst b/specification/0-intro.rst index e5cf1ac8..90ec94ef 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -296,7 +296,7 @@ be obtained by visiting the domain specified. They are case-insensitive. Note that the mapping from a room alias to a room ID is not fixed, and may change over time to point to a different room ID. For this reason, Clients SHOULD resolve the room alias to a room ID once and then use that ID on subsequent -requests. Room aliases MUST NOT exceed 255 bytes. +requests. Room aliases MUST NOT exceed 255 bytes (including the domain). When resolving a room alias the server will also respond with a list of servers that are in the room that can be used to join via. From f51ee706330be5904077b85b595d209794f290d3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 10:19:51 +0100 Subject: [PATCH 29/41] Review comments round 2 --- specification/modules/instant_messaging.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 487240de..0b4e3e64 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -84,6 +84,9 @@ Local echo Messages SHOULD appear immediately in the message view when a user presses the "send" button. This should occur even if the message is still sending. This is referred to as "local echo". Clients SHOULD implement "local echo" of messages. +Clients MAY display messages in a different format to indicate that the server +has not processed the message. This format should be removed when the server +responds. Clients need to be able to match the message they are sending with the same message which they receive from the event stream. The echo of the same message @@ -91,7 +94,7 @@ from the event stream is referred to as "remote echo". Both echoes need to be identified as the same message in order to prevent duplicate messages being displayed. Ideally this pairing would occur transparently to the user: the UI would not flicker as it transitions from local to remote. Flickering cannot be -fully avoided in version 1 of the client-server API. Two scenarios need to be +fully avoided in the current client-server API. Two scenarios need to be considered: - The client sends a message and the remote echo arrives on the event stream @@ -106,8 +109,8 @@ arrives before the client has obtained an event ID. This makes it impossible to identify it as a duplicate event. This results in the client displaying the message twice for a fraction of a second before the the original request to send the message completes. Once it completes, the client can take remedial actions -to remove the duplicate event by looking for duplicate event IDs. Version 2 of -the client-server API resolves this by attaching the transaction ID of the +to remove the duplicate event by looking for duplicate event IDs. A future version +of the client-server API will resolve this by attaching the transaction ID of the sending request to the event itself. @@ -132,7 +135,7 @@ Server behaviour ---------------- Homeservers SHOULD reject ``m.room.message`` events which don't have a -``msgtype`` key, or who don't have a textual ``body`` key, with an HTTP status +``msgtype`` key, or which don't have a textual ``body`` key, with an HTTP status code of 400. Security considerations @@ -141,7 +144,7 @@ Security considerations Messages sent using this module are not encrypted. Messages can be encrypted using the `E2E module`_. -Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site +Clients should sanitise **all displayed keys** for unsafe HTML to prevent Cross-Site Scripting (XSS) attacks. This includes room names and topics. .. _`E2E module`: `module:e2e`_ From 507cb29e39c0dec1f0ad51095f350d4e5f37e4e7 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 10:37:15 +0100 Subject: [PATCH 30/41] Review comments round deux --- specification/modules/history_visibility.rst | 30 +++++++++++--------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 12ea7b22..0405b3f1 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -4,21 +4,25 @@ Room History Visibility .. _module:history-visibility: This module adds support for controlling the visibility of previous events in a -room. Whether a member of a room can see the events that happened in a room from -before they joined the room is controlled by the ``m.room.history_visibility`` -event outlined below. In all cases, the member still needs to be joined to the -room to receive events for that room. The visibility option simply determines -which subset of events in the room are presented to the client. Visibility can -take the form of one of three options: - -- ``shared`` - Previous events are always shown to newly joined members. All - events in the room are shown, even those sent when the member was not a part +room. + +In all cases, a user needs to join a room to view events in that room. Once they +have joined a room, they will gain access to a subset of events in the room. How +this subset is chosen is controlled by the ``m.room.history_visibility`` event +outlined below. After a user has left a room, they may seen any events which they +were allowed to see before they left the room, but no events received after they +left. + +The three options for this event are: + +- ``shared`` - Previous events are always accessible to newly joined members. All + events in the room are accessible, even those sent when the member was not a part of the room. -- ``invited`` - Previous events are shown to newly joined members from the point - they were invited onwards. Events stop being shown when the member's state +- ``invited`` - Previous events are accessible to newly joined members from the point + they were invited onwards. Events stop being accessible when the member's state changes to something other than ``invite`` or ``join``. -- ``joined`` - Previous events are shown to newly joined members from the point - they joined the room onwards. Events stop being shown when the members state +- ``joined`` - Previous events are accessible to newly joined members from the point + they joined the room onwards. Events stop being accessible when the members state changes to something other than ``join``. .. WARNING:: From 5aad67f74d7fe611981aa75c3b42d7c17726b4de Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 11:01:11 +0100 Subject: [PATCH 31/41] Review comments round III --- specification/modules/history_visibility.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 0405b3f1..f26d1b59 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -13,7 +13,7 @@ outlined below. After a user has left a room, they may seen any events which the were allowed to see before they left the room, but no events received after they left. -The three options for this event are: +The three options for the ``m.room.history_visibility`` event are: - ``shared`` - Previous events are always accessible to newly joined members. All events in the room are accessible, even those sent when the member was not a part @@ -22,7 +22,7 @@ The three options for this event are: they were invited onwards. Events stop being accessible when the member's state changes to something other than ``invite`` or ``join``. - ``joined`` - Previous events are accessible to newly joined members from the point - they joined the room onwards. Events stop being accessible when the members state + they joined the room onwards. Events stop being accessible when the member's state changes to something other than ``join``. .. WARNING:: From 6b72ddfb8cbf8628f2ea2efe24d05a609588a211 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 13:36:43 +0100 Subject: [PATCH 32/41] Spelling and typos --- specification/3-application_service_api.rst | 19 ++++++++++--------- specification/4-server_server_api.rst | 2 +- specification/6-appendices.rst | 6 +++--- 3 files changed, 14 insertions(+), 13 deletions(-) diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst index 8d0efdef..cf4942af 100644 --- a/specification/3-application_service_api.rst +++ b/specification/3-application_service_api.rst @@ -322,7 +322,7 @@ including the AS token on a ``/register`` request, along with a login type of Application services which attempt to create users or aliases *outside* of their defined namespaces will receive an error code ``M_EXCLUSIVE``. Similarly, -normal users who attempt to create users or alises *inside* an application +normal users who attempt to create users or aliases *inside* an application service-defined namespace will receive the same ``M_EXCLUSIVE`` error code, but only if the application service has defined the namespace as ``exclusive``. @@ -375,9 +375,10 @@ an API is exposed. Room Aliases ++++++++++++ We may want to expose some 3P network rooms so Matrix users can join them directly, -e.g. IRC rooms. We don't want to expose every 3P network room though, e.g. mailto, -tel. Rooms which are publicly accessible (e.g. IRC rooms) can be exposed as an alias by -the application service. Private rooms (e.g. sending an email to someone) should not +e.g. IRC rooms. We don't want to expose every 3P network room though, e.g. +``mailto``, ``tel``. Rooms which are publicly accessible (e.g. IRC rooms) can be +exposed as an alias by the application service. Private rooms +(e.g. sending an email to someone) should not be exposed in this way, but should instead operate using normal invite/join semantics. Therefore, the ID conventions discussed below are only valid for public rooms which expose room aliases. @@ -397,9 +398,9 @@ SHOULD be mapped in the same way as "user" URIs. Event fields ++++++++++++ -We recommend that any gatewayed events should include an ``external_url`` field -in their content to provide a way for Matrix clients to link into the 'native' -client from which the event originated. For instance, this could contain the -message-ID for emails/nntp posts, or a link to a blog comment when gatewaying -blog comment traffic in & out of matrix +We recommend that any events that originated from a remote network should +include an ``external_url`` field in their content to provide a way for Matrix +clients to link into the 'native' client from which the event originated. +For instance, this could contain the message-ID for emails/nntp posts, or a link +to a blog comment when bridging blog comment traffic in & out of Matrix. diff --git a/specification/4-server_server_api.rst b/specification/4-server_server_api.rst index c5ff2b87..66367cb0 100644 --- a/specification/4-server_server_api.rst +++ b/specification/4-server_server_api.rst @@ -630,7 +630,7 @@ because HTTP services like Matrix are often deployed behind load balancers that handle the TLS and these load balancers make it difficult to check TLS client certificates. -A home server may provide a TLS client certficate and the receiving home server +A home server may provide a TLS client certificate and the receiving home server may check that the client certificate matches the certificate of the origin home server. diff --git a/specification/6-appendices.rst b/specification/6-appendices.rst index de1ac290..c45aa0a5 100644 --- a/specification/6-appendices.rst +++ b/specification/6-appendices.rst @@ -24,7 +24,7 @@ Threat: Unrecoverable Consistency Violations ++++++++++++++++++++++++++++++++++++++++++++ An attacker could send messages which created an unrecoverable "split-brain" -state in the cluster such that the victim's servers could no longer dervive a +state in the cluster such that the victim's servers could no longer derive a consistent view of the chatroom state. Threat: Bad History @@ -63,7 +63,7 @@ Spoofing An attacker could try to send a message claiming to be from the victim without the victim having sent the message in order to: -* Impersonate the victim while performing illict activity. +* Impersonate the victim while performing illicit activity. * Obtain privileges of the victim. Threat: Altering Message Contents @@ -81,7 +81,7 @@ with a phony "origin" field. Spamming ~~~~~~~~ -The attacker could try to send a high volume of solicicted or unsolicted +The attacker could try to send a high volume of solicited or unsolicited messages to the victim in order to: * Find victims for scams. From 3d087df5387f59c1d18f3d0020d0aa26914d2208 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 13:40:05 +0100 Subject: [PATCH 33/41] Kill number prefixes for good --- .../{6-appendices.rst => appendices.rst} | 0 ...ce_api.rst => application_service_api.rst} | 0 ...t_server_api.rst => client_server_api.rst} | 0 ...{0-event_signing.rst => event_signing.rst} | 0 specification/{0-events.rst => events.rst} | 0 ...ture_profiles.rst => feature_profiles.rst} | 0 ...ntity_servers.rst => identity_servers.rst} | 0 specification/{0-intro.rst => intro.rst} | 0 specification/{2-modules.rst => modules.rst} | 0 ...r_server_api.rst => server_server_api.rst} | 0 specification/targets.yaml | 20 +++++++++---------- 11 files changed, 10 insertions(+), 10 deletions(-) rename specification/{6-appendices.rst => appendices.rst} (100%) rename specification/{3-application_service_api.rst => application_service_api.rst} (100%) rename specification/{1-client_server_api.rst => client_server_api.rst} (100%) rename specification/{0-event_signing.rst => event_signing.rst} (100%) rename specification/{0-events.rst => events.rst} (100%) rename specification/{0-feature_profiles.rst => feature_profiles.rst} (100%) rename specification/{5-identity_servers.rst => identity_servers.rst} (100%) rename specification/{0-intro.rst => intro.rst} (100%) rename specification/{2-modules.rst => modules.rst} (100%) rename specification/{4-server_server_api.rst => server_server_api.rst} (100%) diff --git a/specification/6-appendices.rst b/specification/appendices.rst similarity index 100% rename from specification/6-appendices.rst rename to specification/appendices.rst diff --git a/specification/3-application_service_api.rst b/specification/application_service_api.rst similarity index 100% rename from specification/3-application_service_api.rst rename to specification/application_service_api.rst diff --git a/specification/1-client_server_api.rst b/specification/client_server_api.rst similarity index 100% rename from specification/1-client_server_api.rst rename to specification/client_server_api.rst diff --git a/specification/0-event_signing.rst b/specification/event_signing.rst similarity index 100% rename from specification/0-event_signing.rst rename to specification/event_signing.rst diff --git a/specification/0-events.rst b/specification/events.rst similarity index 100% rename from specification/0-events.rst rename to specification/events.rst diff --git a/specification/0-feature_profiles.rst b/specification/feature_profiles.rst similarity index 100% rename from specification/0-feature_profiles.rst rename to specification/feature_profiles.rst diff --git a/specification/5-identity_servers.rst b/specification/identity_servers.rst similarity index 100% rename from specification/5-identity_servers.rst rename to specification/identity_servers.rst diff --git a/specification/0-intro.rst b/specification/intro.rst similarity index 100% rename from specification/0-intro.rst rename to specification/intro.rst diff --git a/specification/2-modules.rst b/specification/modules.rst similarity index 100% rename from specification/2-modules.rst rename to specification/modules.rst diff --git a/specification/4-server_server_api.rst b/specification/server_server_api.rst similarity index 100% rename from specification/4-server_server_api.rst rename to specification/server_server_api.rst diff --git a/specification/targets.yaml b/specification/targets.yaml index c2b6eeda..4ac34ae3 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -1,17 +1,17 @@ targets: main: # arbitrary name to identify this build target files: # the sort order of files to cat - - 0-intro.rst - - 1-client_server_api.rst - - { 1: 0-events.rst } - - { 1: 0-event_signing.rst } - - 2-modules.rst - - { 1: 0-feature_profiles.rst } + - intro.rst + - client_server_api.rst + - { 1: events.rst } + - { 1: event_signing.rst } + - modules.rst + - { 1: feature_profiles.rst } - { 1: "group:modules" } # reference a group of files - - 3-application_service_api.rst - - 4-server_server_api.rst - - 5-identity_servers.rst - - 6-appendices.rst + - application_service_api.rst + - server_server_api.rst + - identity_servers.rst + - appendices.rst groups: # reusable blobs of files when prefixed with 'group:' modules: - modules/instant_messaging.rst From eca98af89693ec09eb68c8ab7f230d9852606367 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 16:38:12 +0100 Subject: [PATCH 34/41] Swaggerify /register endpoint Need to move registration/login/auth sections around once #94 lands. --- api/client-server/v2_alpha/registration.yaml | 95 ++++++++++++++++++++ specification/1-client_server_api.rst | 49 ++-------- 2 files changed, 102 insertions(+), 42 deletions(-) create mode 100644 api/client-server/v2_alpha/registration.yaml diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml new file mode 100644 index 00000000..ad591206 --- /dev/null +++ b/api/client-server/v2_alpha/registration.yaml @@ -0,0 +1,95 @@ +swagger: '2.0' +info: + title: "Matrix Client-Server v2 Registration API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/api/v2_alpha +consumes: + - application/json +produces: + - application/json +paths: + "/register": + post: + summary: Register for an account on this homeserver. + description: |- + Register for an account on this homeserver. + parameters: + - in: body + name: body + schema: + type: object + example: |- + { + "username": "cheeky_monkey", + "password": "ilovebananas", + "bind_email": false + } + properties: + bind_email: + type: boolean + description: |- + If true, the server binds the email used for authentication to + the Matrix ID with the ID Server. + username: + type: string + description: |- + The local part of the desired Matrix ID. If omitted, + the homeserver MUST generate a Matrix ID local part. + password: + type: string + description: The desired password for the account. + required: ["password"] + responses: + 200: + description: The account has been registered. + examples: + application/json: |- + { + "user_id": "@cheeky_monkey:matrix.org", + "access_token": "abc123", + "home_server": "matrix.org", + "refresh_token": "def456" + } + schema: + type: object + properties: + user_id: + type: string + description: The fully-qualified Matrix ID that has been registered. + access_token: + type: string + description: |- + An access token for the account. + This access token can then be used to authorize other requests. + The access token may expire at some point, and if so, it SHOULD come with a ``refresh_token``. + There is no specific error message to indicate that a request has failed because + an access token has expired; instead, if a client has reason to believe its + access token is valid, and it receives an auth error, they should attempt to + refresh for a new token on failure, and retry the request with the new token. + refresh_token: + type: string + # TODO: Work out how to linkify /tokenrefresh + description: |- + (optional) A ``refresh_token`` may be exchanged for a new ``access_token`` using the /tokenrefresh API endpoint. + home_server: + type: string + description: The hostname of the Home Server on which the account has been registered. + 400: + description: |- + Part of the request was invalid. For example, this user ID may be taken. This may + be returned at any stage of the registration process, including after authentication + if the requested user ID was registered whilst the client was performing authentication. + examples: + application/json: |- + { + "errcode": "M_USER_IN_USE", + "error": "Desired user ID is already taken." + } + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index a63c114a..84de17da 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -1001,52 +1001,17 @@ member's state, by making a request to } -Registration ------------- -This section refers to API Version 2. These API calls currently use the prefix -``/_matrix/client/v2_alpha``. - -Registering for a user account is done using the request:: - - POST $V2PREFIX/register +Account operations +------------------ +Registration +~~~~~~~~~~~~ This API endpoint uses the User-Interactive Authentication API. -This API endpoint does not require an access token. -The body of the POST request is a JSON object containing: +{{v2_registration_http_api}} -username - Optional. This is the local part of the desired Matrix ID. If omitted, the - Home Server must generate a Matrix ID local part. -password - Required. The desired password for the account. -bind_email - Optional. If ``true``, the server binds the email used for authentication to - the Matrix ID with the ID Server. - -On success, this returns a JSON object with keys: - -user_id - The fully-qualified Matrix ID that has been registered. -access_token - An access token for the new account. -home_server - The hostname of the Home Server on which the account has been registered. - -This endpoint may also return the following error codes: - -M_USER_IN_USE - If the Matrix ID is already in use -M_EXCLUSIVE - If the requested Matrix ID is in the exclusive namespace of an application - service. - -Home Servers MUST perform the relevant checks and return these codes before -performing User-Interactive Authentication, although they may also return -them after authentication is completed if, for example, the requested user ID -was registered whilst the client was performing authentication. - -Old V1 API docs: |register|_ +Login +~~~~~ {{login_http_api}} From 20b11281ea822f672f7f24d139a7e170efb4cbf3 Mon Sep 17 00:00:00 2001 From: Daniel Wagner-Hall Date: Fri, 16 Oct 2015 13:10:19 +0100 Subject: [PATCH 35/41] Remove lies about OAuth which doesn't exist --- specification/3-application_service_api.rst | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst index 8d0efdef..23fc126f 100644 --- a/specification/3-application_service_api.rst +++ b/specification/3-application_service_api.rst @@ -249,11 +249,8 @@ additional permissions granting the AS permission to masquerade as a matrix user Inputs: - Application service token (``access_token``) + - User ID in the AS namespace to act as. - Either: - - User ID in the AS namespace to act as. - Or: - - OAuth2 token of real user (which may end up being an access token) Notes: - This will apply on all aspects of the CS API, except for Account Management. - The ``as_token`` is inserted into ``access_token`` which is usually where the @@ -268,12 +265,6 @@ Notes: access_token: The application service token user_id: The desired user ID to act as. - /path?access_token=$token&user_token=$token - - Query Parameters: - access_token: The application service token - user_token: The token granted to the AS by the real user - Timestamp massaging +++++++++++++++++++ The application service may want to inject events at a certain time (reflecting From e0fe3c42c8cb58c994396310279a18b9fa8b9f01 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 16 Oct 2015 14:19:00 +0100 Subject: [PATCH 36/41] Review comments --- api/client-server/v2_alpha/registration.yaml | 12 +++++++++--- specification/1-client_server_api.rst | 2 +- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml index ad591206..9973d0b4 100644 --- a/api/client-server/v2_alpha/registration.yaml +++ b/api/client-server/v2_alpha/registration.yaml @@ -80,9 +80,15 @@ paths: description: The hostname of the Home Server on which the account has been registered. 400: description: |- - Part of the request was invalid. For example, this user ID may be taken. This may - be returned at any stage of the registration process, including after authentication - if the requested user ID was registered whilst the client was performing authentication. + Part of the request was invalid. This may include one of the following error codes: + + * ``M_USER_IN_USE`` : The desired user ID is already taken. + * ``M_EXCLUSIVE`` : The desired user ID is in the exclusive namespace + claimed by an application service. + + This errors may be returned at any stage of the registration process, + including after authentication if the requested user ID was registered + whilst the client was performing authentication. examples: application/json: |- { diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 84de17da..331f951a 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -1006,7 +1006,7 @@ Account operations Registration ~~~~~~~~~~~~ -This API endpoint uses the User-Interactive Authentication API. +This API endpoint uses the `User-Interactive Authentication API`_. {{v2_registration_http_api}} From 4bb042daeb3b6bce59f75c308f059910517bd9b0 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 16 Oct 2015 15:22:50 +0100 Subject: [PATCH 37/41] Review comments round II --- api/client-server/v1/sync.yaml | 30 ++-- specification/1-client_server_api.rst | 210 +++++++++++++------------- 2 files changed, 122 insertions(+), 118 deletions(-) diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml index 50af8546..d07e9399 100644 --- a/api/client-server/v1/sync.yaml +++ b/api/client-server/v1/sync.yaml @@ -41,16 +41,6 @@ paths: description: The maximum time in milliseconds to wait for an event. required: false x-example: "35000" - - in: query - type: boolean - name: archived - description: |- - Whether to include rooms that the user has left. If ``false`` then - only rooms that the user has been invited to or has joined are - included. If set to ``true`` then rooms that the user has left are - included as well. By default this is ``false``. - required: false - x-example: "true" responses: 200: description: "The events received, which may be none." @@ -80,19 +70,19 @@ paths: start: type: string description: |- - A token which correlates to the first value in ``chunk``. Used - for pagination. + A token which correlates to the first value in ``chunk``. This + is usually the same token supplied to ``from=``. end: type: string description: |- - A token which correlates to the last value in ``chunk``. Used - for pagination. + A token which correlates to the last value in ``chunk``. This + token should be used in the next request to ``/events``. chunk: type: array description: "An array of events." items: type: object - title: RoomEvent + title: Event allOf: - "$ref": "core-event-schema/room_event.json" 400: @@ -112,6 +102,16 @@ paths: description: The maximum number of messages to return for each room. required: false x-example: "2" + - in: query + type: boolean + name: archived + description: |- + Whether to include rooms that the user has left. If ``false`` then + only rooms that the user has been invited to or has joined are + included. If set to ``true`` then rooms that the user has left are + included as well. By default this is ``false``. + required: false + x-example: "true" responses: 200: description: The user's current state. diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index b96b4bb3..96f62c9b 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -491,6 +491,100 @@ medium address The textual address of the 3pid, eg. the email address +Pagination +---------- + +.. NOTE:: + The paths referred to in this section are not actual endpoints. They only + serve as examples to explain how pagination functions. + +Pagination is the process of dividing a dataset into multiple discrete pages. +Matrix makes use of pagination to allow clients to view extremely large datasets. +These datasets are not limited to events in a room (for example clients may want +to paginate a list of rooms in addition to events within those rooms). Regardless +of *what* is being paginated, there is a common underlying API which is used to +to give clients a consistent way of selecting subsets of a potentially changing +dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters +which describe where to read from the stream. ``from`` and ``to`` are opaque +textual 'stream tokens' which describe the current position in the dataset. +The ``dir`` parameter is an enum representing the direction of events to return: +either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and +``end`` stream token values which can then be passed to subsequent requests to +continue pagination. Not all endpoints will make use of all the parameters +outlined here: see the specific endpoint in question for more information. + +Pagination Request Query Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Query parameters: + from: + $streamtoken - The opaque token to start streaming from. + to: + $streamtoken - The opaque token to end streaming at. Typically, + clients will not know the item of data to end at, so this will usually be + omitted. + limit: + integer - An integer representing the maximum number of items to + return. + dir: + f|b - The direction to return events in. Typically this is ``b`` to paginate + backwards in time. + +'START' and 'END' are placeholder values used in these examples to describe the +start and end of the dataset respectively. + +Unless specified, the default pagination parameters are ``from=START``, +``to=END``, without a limit set. + +For example, if an endpoint had events E1 -> E15. The client wants the last 5 +events and doesn't know any previous events:: + + S E + |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| + | | | + | _____| <--backwards-- | + |__________________ | | ________| + | | | | + GET /somepath?to=START&limit=5&dir=b&from=END + Returns: + E15,E14,E13,E12,E11 + + +Another example: a public room list has rooms R1 -> R17. The client is showing 5 +rooms at a time on screen, and is on page 2. They want to +now show page 3 (rooms R11 -> 15):: + + S E + | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | stream token + |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room + |____________| |________________| + | | + Currently | + viewing | + | + GET /roomslist?from=9&to=END&limit=5 + Returns: R11,R12,R13,R14,R15 + +Note that tokens are treated in an *exclusive*, not inclusive, manner. The end +token from the initial request was '9' which corresponded to R10. When the 2nd +request was made, R10 did not appear again, even though from=9 was specified. If +you know the token, you already have the data. + +Pagination Response +~~~~~~~~~~~~~~~~~~~ + +Responses to pagination requests MUST follow the format:: + + { + "chunk": [ ... , Responses , ... ], + "start" : $streamtoken, + "end" : $streamtoken + } + +Where $streamtoken is an opaque token which can be used in another query to +get the next set of results. The "start" and "end" keys can only be omitted if +the complete dataset is provided in "chunk". + Events ------ @@ -563,8 +657,14 @@ returning early if an event occurs. Only the events API supports long-polling. All events which are visible to the client will appear in the events API. When the request returns, an ``end`` token is included in the response. This token can be used in the next request to continue where the -last request left off. Multiple events can be returned per long-poll. All events -must be de-duplicated based on their event ID. +last request left off. Multiple events can be returned per long-poll. + +.. Warning:: + Events are ordered in this API according to the arrival time of the event on + the homeserver. This can conflict with other APIs which order events based on + their partial ordering in the event graph. This can result in duplicate events + being received (once per API call). Clients SHOULD de-duplicate events based + on the event ID when this happens. .. TODO is deduplication actually a hard requirement in CS v2? @@ -573,8 +673,8 @@ must be de-duplicated based on their event ID. Do we ever support streaming requests? Why not websockets? When the client first logs in, they will need to initially synchronise with -their home server. This is achieved via the initial sync API. This API also -returns an ``end`` token which can be used with the event stream. +their home server. This is achieved via the initial sync API described below. +This API also returns an ``end`` token which can be used with the event stream. {{sync_http_api}} @@ -667,6 +767,9 @@ There are several APIs provided to ``GET`` events for a room: {{rooms_http_api}} + +{{message_pagination_http_api}} + Redactions ~~~~~~~~~~ Since events are extensible it is possible for malicious users and/or servers @@ -712,105 +815,6 @@ The redaction event should be added under the key ``redacted_because``. When a client receives a redaction event it should change the redacted event in the same way a server does. -Pagination ----------- - -.. NOTE:: - The paths referred to in this section are not actual endpoints. They only - serve as examples to explain how pagination functions. - -Pagination is the process of dividing a dataset into multiple discrete pages. -Matrix makes use of pagination to allow clients to view extremely large datasets. -These datasets are not limited to events in a room (for example clients may want -to paginate a list of rooms in addition to events within those rooms). Regardless -of *what* is being paginated, there is a common underlying API which is used to -to give clients a consistent way of selecting subsets of a potentially changing -dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters -which describe where to read from the stream. ``from`` and ``to`` are opaque -textual 'stream tokens' which describe the current position in the dataset. -The ``dir`` parameter is an enum representing the direction of events to return: -either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and -``end`` stream token values which can then be passed to subsequent requests to -continue pagination. Not all endpoints will make use of all the parameters -outlined here: see the specific endpoint in question for more information. - -Pagination Request Query Parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Query parameters: - from: - $streamtoken - The opaque token to start streaming from. - to: - $streamtoken - The opaque token to end streaming at. Typically, - clients will not know the item of data to end at, so this will usually be - omitted. - limit: - integer - An integer representing the maximum number of items to - return. - dir: - f|b - The direction to return events in. Typically this is ``b`` to paginate - backwards in time. - -'START' and 'END' are placeholder values used in these examples to describe the -start and end of the dataset respectively. - -Unless specified, the default pagination parameters are ``from=START``, -``to=END``, without a limit set. - -For example, if an endpoint had events E1 -> E15. The client wants the last 5 -events and doesn't know any previous events:: - - S E - |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| - | | | - | _____| <--backwards-- | - |__________________ | | ________| - | | | | - GET /somepath?to=START&limit=5&dir=b&from=END - Returns: - E15,E14,E13,E12,E11 - - -Another example: a public room list has rooms R1 -> R17. The client is showing 5 -rooms at a time on screen, and is on page 2. They want to -now show page 3 (rooms R11 -> 15):: - - S E - | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | stream token - |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room - |____________| |________________| - | | - Currently | - viewing | - | - GET /roomslist?from=9&to=END&limit=5 - Returns: R11,R12,R13,R14,R15 - -Note that tokens are treated in an *exclusive*, not inclusive, manner. The end -token from the initial request was '9' which corresponded to R10. When the 2nd -request was made, R10 did not appear again, even though from=9 was specified. If -you know the token, you already have the data. - -Pagination Response -~~~~~~~~~~~~~~~~~~~ - -Responses to pagination requests MUST follow the format:: - - { - "chunk": [ ... , Responses , ... ], - "start" : $streamtoken, - "end" : $streamtoken - } - -Where $streamtoken is an opaque token which can be used in another query to -get the next set of results. The "start" and "end" keys can only be omitted if -the complete dataset is provided in "chunk". - -Pagination APIs -~~~~~~~~~~~~~~~ - -{{message_pagination_http_api}} - Rooms ----- From 62d53b4f3321d27cb0b3bc95e5b7625c1efe454d Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 16 Oct 2015 15:58:11 +0100 Subject: [PATCH 38/41] Review comments round III --- specification/client_server_api.rst | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 96f62c9b..ee70d9d3 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -663,11 +663,8 @@ last request left off. Multiple events can be returned per long-poll. Events are ordered in this API according to the arrival time of the event on the homeserver. This can conflict with other APIs which order events based on their partial ordering in the event graph. This can result in duplicate events - being received (once per API call). Clients SHOULD de-duplicate events based - on the event ID when this happens. - -.. TODO - is deduplication actually a hard requirement in CS v2? + being received (once per distinct API called). Clients SHOULD de-duplicate + events based on the event ID when this happens. .. TODO-spec Do we ever support streaming requests? Why not websockets? From ae90d15b3c6955779356466a6edb6e7e6d48c105 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 19 Oct 2015 10:46:07 +0100 Subject: [PATCH 39/41] Add m.room.avatar --- event-schemas/examples/v1/m.room.avatar | 18 ++++++ event-schemas/schema/v1/m.room.avatar | 64 +++++++++++++++++++++ specification/modules/instant_messaging.rst | 2 + 3 files changed, 84 insertions(+) create mode 100644 event-schemas/examples/v1/m.room.avatar create mode 100644 event-schemas/schema/v1/m.room.avatar diff --git a/event-schemas/examples/v1/m.room.avatar b/event-schemas/examples/v1/m.room.avatar new file mode 100644 index 00000000..9fb1189c --- /dev/null +++ b/event-schemas/examples/v1/m.room.avatar @@ -0,0 +1,18 @@ +{ + "age": 242352, + "content": { + "info": { + "h": 398, + "w": 394, + "mimetype": "image/jpeg", + "size": 31037 + }, + "url": "mxc://localhost/JWEIFJgwEIhweiWJE" + }, + "origin_server_ts": 1431961217939, + "event_id": "$WLGTSEFSEF:localhost", + "type": "m.room.avatar", + "state_key": "", + "room_id": "!Cuyf34gef24t:localhost", + "user_id": "@example:localhost" +} diff --git a/event-schemas/schema/v1/m.room.avatar b/event-schemas/schema/v1/m.room.avatar new file mode 100644 index 00000000..276c975c --- /dev/null +++ b/event-schemas/schema/v1/m.room.avatar @@ -0,0 +1,64 @@ +{ + "title": "RoomAvatar", + "description": "A picture that is associated with the room. This can be displayed alongside the room information.", + "type": "object", + "allOf": [{ + "$ref": "core-event-schema/state_event.json" + }], + "properties": { + "content": { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "The URL to the image." + }, + "thumbnail_url": { + "type": "string", + "description": "The URL to the thumbnail of the image." + }, + "thumbnail_info": { + "type": "object", + "title": "ImageInfo", + "description": "Metadata about the image referred to in ``thumbnail_url``.", + "allOf": [{ + "$ref": "core-event-schema/msgtype_infos/image_info.json" + }] + }, + "info": { + "type": "object", + "title": "ImageInfo", + "description": "Metadata about the image referred to in ``url``.", + "properties": { + "size": { + "type": "integer", + "description": "Size of the image in bytes." + }, + "w": { + "type": "integer", + "description": "The width of the image in pixels." + }, + "h": { + "type": "integer", + "description": "The height of the image in pixels." + }, + "mimetype": { + "type": "string", + "description": "The mimetype of the image, e.g. ``image/jpeg``." + } + } + } + }, + "required": ["url"] + }, + "state_key": { + "type": "string", + "description": "A zero-length string.", + "pattern": "^$" + }, + "type": { + "type": "string", + "enum": ["m.room.avatar"] + } + } +} diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 0b4e3e64..09fcb843 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -30,6 +30,8 @@ Usage of this event is discouraged for several reasons: {{m_room_topic_event}} +{{m_room_avatar_event}} + m.room.message msgtypes ~~~~~~~~~~~~~~~~~~~~~~~ From b2aae762fc5740a6e471745e85d1f0287ddabb8b Mon Sep 17 00:00:00 2001 From: Daniel Wagner-Hall Date: Mon, 19 Oct 2015 12:36:49 +0100 Subject: [PATCH 40/41] Give useful error if z-schema is missing --- event-schemas/check.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/event-schemas/check.sh b/event-schemas/check.sh index fac36cf8..a6d03b5a 100755 --- a/event-schemas/check.sh +++ b/event-schemas/check.sh @@ -1,5 +1,11 @@ #!/bin/bash -e # Runs z-schema over all of the schema files (looking for matching examples) + +if ! which z-schema; then + echo >&2 "Need to install z-schema; run: sudo npm install -g z-schema" + exit 1 +fi + find schema/v1/m.* | while read line do split_path=(${line///// }) From f450fc1db6adb9cb76925e90aff8f3782081bb07 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 19 Oct 2015 13:32:43 +0100 Subject: [PATCH 41/41] Typo --- api/client-server/v2_alpha/registration.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml index 9973d0b4..5222b3f3 100644 --- a/api/client-server/v2_alpha/registration.yaml +++ b/api/client-server/v2_alpha/registration.yaml @@ -86,7 +86,7 @@ paths: * ``M_EXCLUSIVE`` : The desired user ID is in the exclusive namespace claimed by an application service. - This errors may be returned at any stage of the registration process, + These errors may be returned at any stage of the registration process, including after authentication if the requested user ID was registered whilst the client was performing authentication. examples: