From 458383585f6e58b588171d37584daccba49ff5bb Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 2 Oct 2015 15:03:55 +0100 Subject: [PATCH 1/5] Stub sections --- specification/modules/instant_messaging.rst | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 43a06aa1..619226df 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -3,6 +3,9 @@ Instant Messaging .. _module:im: +This module adds support for sending human-readable messages to a room. It also +adds human-readable information to the room itself such as a room name and topic. + Events ------ @@ -15,7 +18,7 @@ Events {{m_room_topic_event}} m.room.message msgtypes ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ .. TODO-spec How a client should handle unknown message types. @@ -27,3 +30,13 @@ outlined below. {{msgtype_events}} + +Client behaviour +---------------- + +Server behaviour +---------------- + +Security considerations +----------------------- + From 8e5c832ff9fa8d8e674346e3776c08e659c306c9 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 5 Oct 2015 13:45:23 +0100 Subject: [PATCH 2/5] Flesh out more of the IM module --- .../schema/v1/m.room.message.feedback | 2 +- specification/modules/instant_messaging.rst | 59 +++++++++++++++++-- 2 files changed, 54 insertions(+), 7 deletions(-) diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback index 1bbfc1ba..2eaed999 100644 --- a/event-schemas/schema/v1/m.room.message.feedback +++ b/event-schemas/schema/v1/m.room.message.feedback @@ -1,7 +1,7 @@ { "type": "object", "title": "MessageFeedback", - "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.", + "description": "**NB: Usage of this event is discouraged in favour of the** `receipts module`_. **Most clients will not recognise this event.** Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged.", "allOf": [{ "$ref": "core-event-schema/room_event.json" }], diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 619226df..7934b80a 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -4,15 +4,37 @@ Instant Messaging .. _module:im: This module adds support for sending human-readable messages to a room. It also -adds human-readable information to the room itself such as a room name and topic. +adds support for associating human-readable information with the room itself +such as a room name and topic. Events ------ {{m_room_message_event}} + +.. admonition:: Rationale + + Not all clients can display all message types. The most commonly supported + message type is raw text. As a result, we chose to have a textual fallback + display method represented by the ``body`` key. This means that even if the + client cannot display a particular ``msgtype``, they can still display + *something*, even if it is just plain text. + {{m_room_message_feedback_event}} +Usage of this event is discouraged for several reasons: + - The number of feedback events will grow very quickly with the number of users + in the room. This event provides no way to "batch" feedback, unlike the + `receipts module`_. + - Pairing feedback to messages gets complicated when paginating as feedback + arrives before the message it is acknowledging. + - There are no guarantees that the client has seen the event ID being + acknowledged. + + +.. _`receipts module`: `module:receipts`_ + {{m_room_name_event}} {{m_room_topic_event}} @@ -20,13 +42,10 @@ Events m.room.message msgtypes ~~~~~~~~~~~~~~~~~~~~~~~ -.. TODO-spec - How a client should handle unknown message types. - - Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type of message being sent. Each type has their own required and optional keys, as -outlined below. +outlined below. If a client cannot display the given ``msgtype`` then it MUST +display the fallback plain text ``body`` key instead. {{msgtype_events}} @@ -34,9 +53,37 @@ outlined below. Client behaviour ---------------- +Events which have attachments (e.g. ``m.image``, ``m.file``) are advised to be +uploaded using the `content repository module`_ where available. The +resulting ``mxc://`` URI can then be used in the ``url`` key. The +attachment SHOULD be uploaded *prior* to sending the event in order to stop a +race condition where the recipient receives a link to a non-existent attachment. + +.. _`content repository module`: `module:content`_ + +Recommendations when sending messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- Advise using idempotent PUTs to send messages (and why) +- Retries (exp backoff, giveup eventually allowing manual retry) +- Queueing (bucket per room) + +Implementing local echo +~~~~~~~~~~~~~~~~~~~~~~~ +- Local echo (document bug with races) - sending state. Pairing returned event ID. + +Displaying membership information with messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Member name linking (incl. pagination aka historical display names) + Server behaviour ---------------- +- SHOULD enforce the body/msgtype keys are present (can 400 them) + Security considerations ----------------------- +- Not encrypted, link to E2E module. +- XSS: Should sanitise ALL KEYS before injecting as unsafe HTML (name/topic/body/etc) + From 91ca36509b3112a83680a97a0d672a428b4d7fef Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 7 Oct 2015 11:51:49 +0100 Subject: [PATCH 3/5] Flesh out IM module --- specification/0-intro.rst | 2 + specification/modules/instant_messaging.rst | 89 ++++++++++++++++++--- 2 files changed, 80 insertions(+), 11 deletions(-) diff --git a/specification/0-intro.rst b/specification/0-intro.rst index 445b32ef..7c27e0d7 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -434,6 +434,8 @@ Some requests have unique error codes: :``M_LOGIN_EMAIL_URL_NOT_YET``: Encountered when polling for an email link which has not been clicked yet. +.. _sect:txn_ids: + The C-S API typically uses ``HTTP POST`` to submit requests. This means these requests are not idempotent. The C-S API also allows ``HTTP PUT`` to make requests idempotent. In order to use a ``PUT``, paths should be suffixed with diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 7934b80a..426189b5 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -53,7 +53,14 @@ display the fallback plain text ``body`` key instead. Client behaviour ---------------- -Events which have attachments (e.g. ``m.image``, ``m.file``) are advised to be +Clients SHOULD verify the structure of incoming events to ensure that the +expected keys exist and that they are of the right type. Clients can discard +malformed events or display a placeholder message to the user. Redacted +``m.room.message`` events MUST be removed from the client. This can either be +replaced with placeholder text (e.g. "[REDACTED]") or the redacted message can +be removed entirely from the messages view. + +Events which have attachments (e.g. ``m.image``, ``m.file``) SHOULD be uploaded using the `content repository module`_ where available. The resulting ``mxc://`` URI can then be used in the ``url`` key. The attachment SHOULD be uploaded *prior* to sending the event in order to stop a @@ -63,27 +70,87 @@ race condition where the recipient receives a link to a non-existent attachment. Recommendations when sending messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Advise using idempotent PUTs to send messages (and why) -- Retries (exp backoff, giveup eventually allowing manual retry) -- Queueing (bucket per room) -Implementing local echo -~~~~~~~~~~~~~~~~~~~~~~~ -- Local echo (document bug with races) - sending state. Pairing returned event ID. +Clients can send messages using ``POST`` or ``PUT`` requests. Clients SHOULD use +``PUT`` requests with `transaction IDs`_ to make requests idempotent. This +ensures that messages are sent exactly once even under poor network conditions. +Clients SHOULD retry requests using an exponential-backoff algorithm for a +certain amount of time T. It is recommended that T is no longer than 5 minutes. +After this time, the client should stop retrying and mark the message as "unsent". +Users should be able to manually resend unsent messages. + +Users may type several messages at once and send them all in quick succession. +Clients SHOULD preserve the order in which they were sent by the user. This +means that clients should wait for the response to the previous request before +sending the next request. This can lead to head-of-line blocking. In order to +reduce the impact of head-of-line blocking, clients should use a queue per room +rather than a global queue, as ordering is only relevant within a single room +rather than between rooms. + +.. _`transaction IDs`: `sect:txn_ids`_ + +Local echo +~~~~~~~~~~ + +Messages SHOULD appear immediately in the message view when a user presses the +"send" button. This should occur even if the message is still sending. This is +referred to as "local echo". Clients SHOULD implement "local echo" of messages. + +Clients need to be able to pair up the "remote echo" from the server with the +"local echo" to prevent duplicate messages being displayed. Ideally this pairing +would occur transparently to the user: the UI would not flicker as it transitions +from local to remote. Flickering cannot be fully avoided in version 1 of the +client-server API. Two scenarios need to be considered: + +- The client sends a message and the remote echo arrives on the event stream + *after* the request to send the message completes. +- The client sends a message and the remote echo arrives on the event stream + *before* the request to send the message completes. + +In the first scenario, the client will receive an event ID when the request to +send the message completes. This ID can be used to identify the duplicate event +when it arrives on the event stream. However, in the second scenario, the event +arrives before the client has obtained an event ID. This makes it impossible to +identify it as a duplicate event. This results in the client displaying the +message twice for a fraction of a second before the the original request to send +the message completes. Once it completes, the client can take remedial actions +to remove the duplicate event by looking for duplicate event IDs. Version 2 of +the client-server API resolves this by attaching the transaction ID of the +sending request to the event itself. + Displaying membership information with messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Member name linking (incl. pagination aka historical display names) +Clients may wish to show the display name and avatar URL of the room member who +sent a message. This can be achieved by inspecting the ``m.room.member`` event +for that user ID. + +When a user paginates the message history, clients may wish to show the +**historical** display name and avatar URL for a room member. This is possible +because older ``m.room.member`` events are returned when paginating. This can +be implemented efficiently by keeping two sets of room state: old and current. +As new events arrive and/or the user paginates back in time, these two sets of +state diverge from each other. New events update the current state and paginated +events update the old state. When paginated events are processed sequentially, +the old state represents the state of the room *at the time the event was sent*. +This can then be used to set the historical display name and avatar URL. Server behaviour ---------------- -- SHOULD enforce the body/msgtype keys are present (can 400 them) +Homeservers SHOULD enforce that ``m.room.message`` events have textual ``body`` +and ``msgtype`` keys by 400ing the request to send a message. Security considerations ----------------------- -- Not encrypted, link to E2E module. -- XSS: Should sanitise ALL KEYS before injecting as unsafe HTML (name/topic/body/etc) +Messages sent using this module are not encrypted. Encryption can be layered +over the top of this module: where the plaintext format is an ``m.room.message`` +conforming to this module. This can be achieved using the `E2E module`_. + +Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site +Scripting (XSS) attacks. This includes room names and topics. + +.. _`E2E module`: `module:e2e`_ From 4170dbd5cf065f7e1c7fde32f4c5a6e622c98055 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 13 Oct 2015 11:29:54 +0100 Subject: [PATCH 4/5] Review comments --- event-schemas/schema/v1/m.room.message | 2 +- specification/modules/instant_messaging.rst | 44 +++++++++------------ 2 files changed, 19 insertions(+), 27 deletions(-) diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message index 27b7e925..91c04b7f 100644 --- a/event-schemas/schema/v1/m.room.message +++ b/event-schemas/schema/v1/m.room.message @@ -1,7 +1,7 @@ { "type": "object", "title": "Message", - "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.", + "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message. This allows clients to display *something* even if it is just plain text.", "allOf": [{ "$ref": "core-event-schema/room_event.json" }], diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 426189b5..487240de 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -12,15 +12,6 @@ Events {{m_room_message_event}} - -.. admonition:: Rationale - - Not all clients can display all message types. The most commonly supported - message type is raw text. As a result, we chose to have a textual fallback - display method represented by the ``body`` key. This means that even if the - client cannot display a particular ``msgtype``, they can still display - *something*, even if it is just plain text. - {{m_room_message_feedback_event}} Usage of this event is discouraged for several reasons: @@ -44,7 +35,7 @@ m.room.message msgtypes Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type of message being sent. Each type has their own required and optional keys, as -outlined below. If a client cannot display the given ``msgtype`` then it MUST +outlined below. If a client cannot display the given ``msgtype`` then it SHOULD display the fallback plain text ``body`` key instead. {{msgtype_events}} @@ -62,9 +53,7 @@ be removed entirely from the messages view. Events which have attachments (e.g. ``m.image``, ``m.file``) SHOULD be uploaded using the `content repository module`_ where available. The -resulting ``mxc://`` URI can then be used in the ``url`` key. The -attachment SHOULD be uploaded *prior* to sending the event in order to stop a -race condition where the recipient receives a link to a non-existent attachment. +resulting ``mxc://`` URI can then be used in the ``url`` key. .. _`content repository module`: `module:content`_ @@ -95,12 +84,15 @@ Local echo Messages SHOULD appear immediately in the message view when a user presses the "send" button. This should occur even if the message is still sending. This is referred to as "local echo". Clients SHOULD implement "local echo" of messages. - -Clients need to be able to pair up the "remote echo" from the server with the -"local echo" to prevent duplicate messages being displayed. Ideally this pairing -would occur transparently to the user: the UI would not flicker as it transitions -from local to remote. Flickering cannot be fully avoided in version 1 of the -client-server API. Two scenarios need to be considered: + +Clients need to be able to match the message they are sending with the same +message which they receive from the event stream. The echo of the same message +from the event stream is referred to as "remote echo". Both echoes need to be +identified as the same message in order to prevent duplicate messages being +displayed. Ideally this pairing would occur transparently to the user: the UI +would not flicker as it transitions from local to remote. Flickering cannot be +fully avoided in version 1 of the client-server API. Two scenarios need to be +considered: - The client sends a message and the remote echo arrives on the event stream *after* the request to send the message completes. @@ -123,8 +115,8 @@ Displaying membership information with messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Clients may wish to show the display name and avatar URL of the room member who -sent a message. This can be achieved by inspecting the ``m.room.member`` event -for that user ID. +sent a message. This can be achieved by inspecting the ``m.room.member`` state +event for that user ID. When a user paginates the message history, clients may wish to show the **historical** display name and avatar URL for a room member. This is possible @@ -139,15 +131,15 @@ This can then be used to set the historical display name and avatar URL. Server behaviour ---------------- -Homeservers SHOULD enforce that ``m.room.message`` events have textual ``body`` -and ``msgtype`` keys by 400ing the request to send a message. +Homeservers SHOULD reject ``m.room.message`` events which don't have a +``msgtype`` key, or who don't have a textual ``body`` key, with an HTTP status +code of 400. Security considerations ----------------------- -Messages sent using this module are not encrypted. Encryption can be layered -over the top of this module: where the plaintext format is an ``m.room.message`` -conforming to this module. This can be achieved using the `E2E module`_. +Messages sent using this module are not encrypted. Messages can be encrypted +using the `E2E module`_. Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site Scripting (XSS) attacks. This includes room names and topics. From f51ee706330be5904077b85b595d209794f290d3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 10:19:51 +0100 Subject: [PATCH 5/5] Review comments round 2 --- specification/modules/instant_messaging.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 487240de..0b4e3e64 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -84,6 +84,9 @@ Local echo Messages SHOULD appear immediately in the message view when a user presses the "send" button. This should occur even if the message is still sending. This is referred to as "local echo". Clients SHOULD implement "local echo" of messages. +Clients MAY display messages in a different format to indicate that the server +has not processed the message. This format should be removed when the server +responds. Clients need to be able to match the message they are sending with the same message which they receive from the event stream. The echo of the same message @@ -91,7 +94,7 @@ from the event stream is referred to as "remote echo". Both echoes need to be identified as the same message in order to prevent duplicate messages being displayed. Ideally this pairing would occur transparently to the user: the UI would not flicker as it transitions from local to remote. Flickering cannot be -fully avoided in version 1 of the client-server API. Two scenarios need to be +fully avoided in the current client-server API. Two scenarios need to be considered: - The client sends a message and the remote echo arrives on the event stream @@ -106,8 +109,8 @@ arrives before the client has obtained an event ID. This makes it impossible to identify it as a duplicate event. This results in the client displaying the message twice for a fraction of a second before the the original request to send the message completes. Once it completes, the client can take remedial actions -to remove the duplicate event by looking for duplicate event IDs. Version 2 of -the client-server API resolves this by attaching the transaction ID of the +to remove the duplicate event by looking for duplicate event IDs. A future version +of the client-server API will resolve this by attaching the transaction ID of the sending request to the event itself. @@ -132,7 +135,7 @@ Server behaviour ---------------- Homeservers SHOULD reject ``m.room.message`` events which don't have a -``msgtype`` key, or who don't have a textual ``body`` key, with an HTTP status +``msgtype`` key, or which don't have a textual ``body`` key, with an HTTP status code of 400. Security considerations @@ -141,7 +144,7 @@ Security considerations Messages sent using this module are not encrypted. Messages can be encrypted using the `E2E module`_. -Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site +Clients should sanitise **all displayed keys** for unsafe HTML to prevent Cross-Site Scripting (XSS) attacks. This includes room names and topics. .. _`E2E module`: `module:e2e`_