From 078dd0165ffe045229b0efee92ec8f425cfd28df Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 25 Sep 2015 11:58:47 +0100 Subject: [PATCH 01/54] Update the room creation API spec to include new keys: 'preset' and 'initial_state' --- specification/1-client_server_api.rst | 33 ++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 93e3cb907..e3dbde5a3 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -750,10 +750,41 @@ options which can be set when creating a room: This will tell the server to invite everyone in the list to the newly created room. +``preset`` + Type: + String + Optional: + Yes + Value: + ``private_chat`` or ``public_chat`` + Description: + Convenience parameter for setting various default state events based on a + preset. + + Two presets are defined: + + - ``private_chat``: Sets the ``join_rules`` to ``invite`` and + ``history_visibility`` to ``shared`` + - ``public_chat``: Sets the ``join_rules`` to ``public`` and + ``history_visibility`` to ``shared`` + +``initial_state`` + Type: + List + Optional: + Yes + Value: + A list of state events to set in the new room. + Description: + Allows the user to override the default state events set in the new room. + + The expected format of the state events are an object with ``type``, + ``state_key`` and ``content`` keys set. + Example:: { - "visibility": "public", + "preset": "public_chat", "room_alias_name": "thepub", "name": "The Grand Duke Pub", "topic": "All about happy hour" From d7d9f406a67419306d992bf906f01232aad59e47 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 25 Sep 2015 14:21:50 +0100 Subject: [PATCH 02/54] Bundle some state into invites --- api/client-server/v1/sync.yaml | 5 +++++ event-schemas/schema/v1/m.room.member | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml index 833c425ab..c7f781877 100644 --- a/api/client-server/v1/sync.yaml +++ b/api/client-server/v1/sync.yaml @@ -267,6 +267,11 @@ paths: type: string description: "The user's membership state in this room." enum: ["invite", "join", "leave", "ban"] + invite: + type: object + description: "The invite event if `membership` is `invite`" + allOf: + - "$ref": "room.room.member" messages: type: object title: PaginationChunk diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member index 49b9f5b83..32302b85f 100644 --- a/event-schemas/schema/v1/m.room.member +++ b/event-schemas/schema/v1/m.room.member @@ -32,6 +32,25 @@ "type": { "type": "string", "enum": ["m.room.member"] + }, + "invite_room_state": { + "type": "array", + "description": "A subset of the state of the room at the time of the invite, if ``membership`` is ``invite``", + "items": { + "type": "object", + "title": "StateEvent", + "properties": { + "type": { + "type": "string" + }, + "state_key": { + "type": "string" + }, + "content": { + "type": "object" + } + } + } } } } From 37ccddb3087283067617a94d83dc968489bae20f Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 25 Sep 2015 14:25:07 +0100 Subject: [PATCH 03/54] Typo --- api/client-server/v1/sync.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml index c7f781877..d82ee9e4e 100644 --- a/api/client-server/v1/sync.yaml +++ b/api/client-server/v1/sync.yaml @@ -271,7 +271,7 @@ paths: type: object description: "The invite event if `membership` is `invite`" allOf: - - "$ref": "room.room.member" + - "$ref": "m.room.member" messages: type: object title: PaginationChunk From 18dc7784df09c3ad6c3f4d844b87d046538bd5a5 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 25 Sep 2015 14:34:06 +0100 Subject: [PATCH 04/54] Mention precedence --- specification/1-client_server_api.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index e3dbde5a3..c5453931a 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -781,6 +781,9 @@ options which can be set when creating a room: The expected format of the state events are an object with ``type``, ``state_key`` and ``content`` keys set. + Takes precedence over events set by ``presets``, but gets overriden by + ``name`` and ``topic`` keys. + Example:: { From c6375ed3d1743eb931e37d12b082c91a2b20d7b4 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 25 Sep 2015 15:09:15 +0100 Subject: [PATCH 05/54] Flesh out feature profiles section Add table detailing the profiles. Add anchors to link through to each module following a well-defined format (rather than the name of the module section). Allow UTF-8 in the spec. --- specification/0-feature_profiles.rst | 35 +++++++++++++++++++ specification/modules/content_repo.rst | 2 ++ .../modules/end_to_end_encryption.rst | 2 ++ specification/modules/instant_messaging.rst | 2 ++ specification/modules/presence.rst | 2 ++ specification/modules/push_overview.rst | 2 ++ specification/modules/receipts.rst | 2 ++ .../modules/typing_notifications.rst | 2 ++ specification/modules/voip_events.rst | 3 ++ templating/build.py | 4 +-- 10 files changed, 54 insertions(+), 2 deletions(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index 234e14db0..dcf7f6426 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -1,3 +1,38 @@ Feature Profiles ================ +Matrix supports many different kinds of clients: from embedded IoT devices to +desktop clients. Not all clients can provide the same feature sets as other +clients e.g. due to lack of physical hardware such as not having a screen. +Clients can fall into one of several profiles and each profile contains a set +of features that the client MUST support. This section details a set of +"feature profiles". Clients are expected to implement a profile in its entirety +in order for it to be classified as that profile. + +Summary +------- + +============================ ===== =========== ======== ========= ===== ===== + Module / Profile Web Embed-Web Mobile Desktop CLI IoT +============================ ===== =========== ======== ========= ===== ===== + `End-to-End Encryption`_ YES YES YES YES + `Instant Messaging`_ YES YES YES YES YES YES + `Presence`_ YES YES YES YES + `Push Notifications`_ YES + `Receipts`_ YES YES YES YES + `Typing Notifications`_ YES YES YES YES + `VoIP`_ YES YES YES + `Content Repository`_ YES YES YES YES +============================ ===== =========== ======== ========= ===== ===== + +*Please see each module for more details on what clients need to implement.* + +.. _End-to-End Encryption: `module:e2e`_ +.. _Instant Messaging: `module:im`_ +.. _Presence: `module:presence`_ +.. _Push Notifications: `module:push`_ +.. _Receipts: `module:receipts`_ +.. _Typing Notifications: `module:typing`_ +.. _VoIP: `module:voip`_ +.. _Content Repository: `module:content`_ + diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst index 2c45ced74..c2cf3505b 100644 --- a/specification/modules/content_repo.rst +++ b/specification/modules/content_repo.rst @@ -1,6 +1,8 @@ Content repository ================== +.. _module:content: + HTTP API -------- diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 023881527..e3a526136 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -1,6 +1,8 @@ End-to-End Encryption ===================== +.. _module:e2e: + .. TODO-doc - Why is this needed. - Overview of process diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 7f582ca4b..43a06aa13 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -1,6 +1,8 @@ Instant Messaging ================= +.. _module:im: + Events ------ diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst index ddd2adff2..cb71107f3 100644 --- a/specification/modules/presence.rst +++ b/specification/modules/presence.rst @@ -1,5 +1,7 @@ Presence ======== + +.. _module:presence: Each user has the concept of presence information. This encodes the "availability" of that user, suitable for display on other user's clients. diff --git a/specification/modules/push_overview.rst b/specification/modules/push_overview.rst index 972a8eea1..46028283a 100644 --- a/specification/modules/push_overview.rst +++ b/specification/modules/push_overview.rst @@ -1,6 +1,8 @@ Push Notifications ================== +.. _module:push: + Overview -------- diff --git a/specification/modules/receipts.rst b/specification/modules/receipts.rst index e2f83eea8..9787682f0 100644 --- a/specification/modules/receipts.rst +++ b/specification/modules/receipts.rst @@ -1,6 +1,8 @@ Receipts -------- +.. _module:receipts: + Receipts are used to publish which events in a room the user or their devices have interacted with. For example, which events the user has read. For efficiency this is done as "up to" markers, i.e. marking a particular event diff --git a/specification/modules/typing_notifications.rst b/specification/modules/typing_notifications.rst index 25b714ab4..614f6af72 100644 --- a/specification/modules/typing_notifications.rst +++ b/specification/modules/typing_notifications.rst @@ -1,6 +1,8 @@ Typing Notifications -------------------- +.. _module:typing: + Client APIs ~~~~~~~~~~~ diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst index a54682379..60d610788 100644 --- a/specification/modules/voip_events.rst +++ b/specification/modules/voip_events.rst @@ -1,5 +1,8 @@ Voice over IP ------------- + +.. _module:voip: + Matrix can also be used to set up VoIP calls. This is part of the core specification, although is at a relatively early stage. Voice (and video) over Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like diff --git a/templating/build.py b/templating/build.py index 013248f4e..6f465607b 100755 --- a/templating/build.py +++ b/templating/build.py @@ -122,13 +122,13 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False): # check the input files and substitute in sections where required log("Parsing input template: %s" % file_stream.name) - temp = Template(file_stream.read()) + temp = Template(file_stream.read().decode("utf-8")) log("Creating output for: %s" % file_stream.name) output = create_from_template(temp, sections) with open( os.path.join(out_dir, os.path.basename(file_stream.name)), "w" ) as f: - f.write(output) + f.write(output.encode("utf-8")) log("Output file for: %s" % file_stream.name) check_unaccessed("units", units) From 9fac152d32cea2dff81ed5094bb47dcb793ac640 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 25 Sep 2015 15:26:58 +0100 Subject: [PATCH 06/54] Explain what the clients are and the column reference --- specification/0-feature_profiles.rst | 43 ++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index dcf7f6426..4961c2163 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -36,3 +36,46 @@ Summary .. _VoIP: `module:voip`_ .. _Content Repository: `module:content`_ +Clients +------- + +Stand-alone web client (``Web``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is a web page which heavily uses Matrix for communication. Single-page web +apps would be classified as a stand-alone web client, as would multi-page web +apps which use Matrix on nearly every page. + +Embedded web client (``EmbedWeb``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is a Matrix client which is embedded in another website, e.g. using +iframes. These embedded clients are typically for a single purpose +related to the website in question, and are not intended to be fully-fledged +communication apps. + +Mobile client (``Mobile``) +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is a Matrix client specifically designed for consumption on mobile devices. +This is typically a mobile app but need not be so provided the feature set can +be reached (e.g. if a mobile site could display push notifications it could be +classified as a mobile client). + +Desktop client (``Desktop``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is a native application which can run in its own environment outside a +browser. + +Command Line Interface client (``CLI``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is a client which is used via a text-based terminal. + +Internet of Things client (``IoT``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is a client which is typically running on an embedded device such as a +kettle, fridge or car. + From 510553ee009049616288a262aec3a01705e14429 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 25 Sep 2015 15:29:33 +0100 Subject: [PATCH 07/54] Remove smurf suffixes. Add anchor for feature profiles. --- specification/0-feature_profiles.rst | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index 4961c2163..ab9771625 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -1,6 +1,8 @@ Feature Profiles ================ +.. sect:feature-profiles: + Matrix supports many different kinds of clients: from embedded IoT devices to desktop clients. Not all clients can provide the same feature sets as other clients e.g. due to lack of physical hardware such as not having a screen. @@ -39,42 +41,42 @@ Summary Clients ------- -Stand-alone web client (``Web``) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Stand-alone web (``Web``) +~~~~~~~~~~~~~~~~~~~~~~~~~ This is a web page which heavily uses Matrix for communication. Single-page web apps would be classified as a stand-alone web client, as would multi-page web apps which use Matrix on nearly every page. -Embedded web client (``EmbedWeb``) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Embedded web (``EmbedWeb``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is a Matrix client which is embedded in another website, e.g. using iframes. These embedded clients are typically for a single purpose related to the website in question, and are not intended to be fully-fledged communication apps. -Mobile client (``Mobile``) -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Mobile (``Mobile``) +~~~~~~~~~~~~~~~~~~~ This is a Matrix client specifically designed for consumption on mobile devices. This is typically a mobile app but need not be so provided the feature set can be reached (e.g. if a mobile site could display push notifications it could be classified as a mobile client). -Desktop client (``Desktop``) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Desktop (``Desktop``) +~~~~~~~~~~~~~~~~~~~~~ This is a native application which can run in its own environment outside a browser. -Command Line Interface client (``CLI``) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Command Line Interface (``CLI``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is a client which is used via a text-based terminal. -Internet of Things client (``IoT``) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Internet of Things (``IoT``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is a client which is typically running on an embedded device such as a kettle, fridge or car. From 8c22b715cabdb283de39a0f79bdef71847366323 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 28 Sep 2015 09:29:07 +0100 Subject: [PATCH 08/54] Add title --- api/client-server/v1/sync.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml index d82ee9e4e..dbfa39b80 100644 --- a/api/client-server/v1/sync.yaml +++ b/api/client-server/v1/sync.yaml @@ -269,6 +269,7 @@ paths: enum: ["invite", "join", "leave", "ban"] invite: type: object + title: "InviteEvent" description: "The invite event if `membership` is `invite`" allOf: - "$ref": "m.room.member" From 317c2f20d31345a656c553dc239563cf9f2de78d Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 28 Sep 2015 09:29:35 +0100 Subject: [PATCH 09/54] Draft login token spec --- specification/1-client_server_api.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 93e3cb907..7f9673379 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -197,6 +197,7 @@ This specification defines the following login types: - ``m.login.recaptcha`` - ``m.login.oauth2`` - ``m.login.email.identity`` + - ``m.login.token`` - ``m.login.dummy`` Password-based @@ -228,6 +229,29 @@ To respond to this type, reply with an auth dict as follows:: "response": "" } +Token-based +~~~~~~~~~~~ +:Type: + ``m.login.token`` +:Description: + The client submits a username and token that was generated by the server. + +To respond to this type, reply with an auth dict as follows:: + + { + "type": "m.login.token", + "user": "", + "token": "", + "nonce": "" + } + +The ``nonce`` should be a random string generated by the client for the +request. The same ``nonce`` should be used if retrying the request. + +The ``token`` may be discovered from e.g. an email or dynamically generated QR +code. + + OAuth2-based ~~~~~~~~~~~~ :Type: From 4c9f524cabbbc74a837743aaa043e977b67ae7d2 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 28 Sep 2015 09:32:03 +0100 Subject: [PATCH 10/54] Neaten things up --- specification/1-client_server_api.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 7f9673379..e3817046a 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -234,7 +234,7 @@ Token-based :Type: ``m.login.token`` :Description: - The client submits a username and token that was generated by the server. + The client submits a username and token. To respond to this type, reply with an auth dict as follows:: @@ -248,9 +248,8 @@ To respond to this type, reply with an auth dict as follows:: The ``nonce`` should be a random string generated by the client for the request. The same ``nonce`` should be used if retrying the request. -The ``token`` may be discovered from e.g. an email or dynamically generated QR -code. - +There are many ways a client may receive a ``token``, including via an email or +from an existing logged in device. OAuth2-based ~~~~~~~~~~~~ From 97154cc6b2088ea30d80b0c2eff3196a40b9a484 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 28 Sep 2015 13:09:54 +0100 Subject: [PATCH 11/54] s/nonce/txn_id/ --- specification/1-client_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index e3817046a..988f7bc44 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -242,7 +242,7 @@ To respond to this type, reply with an auth dict as follows:: "type": "m.login.token", "user": "", "token": "", - "nonce": "" + "txn_id": "" } The ``nonce`` should be a random string generated by the client for the From db8f3c0d5969a6bc731a1e43e23359ea0dd160fe Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 28 Sep 2015 13:11:34 +0100 Subject: [PATCH 12/54] Desktop clients should have a GUI --- specification/0-feature_profiles.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index ab9771625..86f8f3d8a 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -67,7 +67,7 @@ classified as a mobile client). Desktop (``Desktop``) ~~~~~~~~~~~~~~~~~~~~~ -This is a native application which can run in its own environment outside a +This is a native GUI application which can run in its own environment outside a browser. Command Line Interface (``CLI``) From c2fc1a2fb19c6a46cc2bde736222f855f0f8ba3d Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 28 Sep 2015 13:41:31 +0100 Subject: [PATCH 13/54] Add basic module template. This has the core sections for events/server/client/security. --- specification/modules/_template.rst | 47 +++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 specification/modules/_template.rst diff --git a/specification/modules/_template.rst b/specification/modules/_template.rst new file mode 100644 index 000000000..6ea7aaa35 --- /dev/null +++ b/specification/modules/_template.rst @@ -0,0 +1,47 @@ +Module Heading +============== + +.. _module:short-name: + +A short summary of the module. What features does this module provide? An anchor +should be specified at the top of the module using the format ``module:name``. + +Complicated modules may wish to have architecture diagrams or event flows +(e.g. VoIP call flows) here. Custom subsections can be included but they should +be used *sparingly* to reduce the risk of putting client or server behaviour +information in these custom sections. + +Events +------ +List the new event types introduced by this module, if any. If there are no +new events, this section can be omitted. Event types should be done as +subsections. The section is intended to document the "common shared event +structure" between client and server. Deviations from this shared structure +should be documented in the relevant behaviour section. + +example.event.type +~~~~~~~~~~~~~~~~~~ +There should be JSON Schema docs for this event. You can insert a template like +so: + +{{example_event_type_event}} + +Client behaviour +---------------- +List any new HTTP endpoints. List the steps the client needs to take to +correctly process this module. Listing what data structures the client should be +storing to aid implementation is recommended. + +Server behaviour +---------------- +Does the server need to handle any of the new events in a special way (e.g. +typing timeouts, presence). Advice on how to persist events and/or requests are +recommended to aid implementation. + +Security considerations +----------------------- +This includes privacy leaks: for example leaking presence info. How do +misbehaving clients or servers impact this module? This section should always be +included, if only to say "we've thought about it but there isn't anything to do +here". + From 643468e9141902499550ace65da1bf8eb7e53bc5 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 28 Sep 2015 14:01:54 +0100 Subject: [PATCH 14/54] Mention swagger. Clarify how event type template vars are formed. --- specification/modules/_template.rst | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/specification/modules/_template.rst b/specification/modules/_template.rst index 6ea7aaa35..967892a54 100644 --- a/specification/modules/_template.rst +++ b/specification/modules/_template.rst @@ -19,18 +19,24 @@ subsections. The section is intended to document the "common shared event structure" between client and server. Deviations from this shared structure should be documented in the relevant behaviour section. -example.event.type -~~~~~~~~~~~~~~~~~~ -There should be JSON Schema docs for this event. You can insert a template like -so: +m.example.event.type +~~~~~~~~~~~~~~~~~~~~ +There should be JSON Schema docs for this event. Once there is JSON schema, +there will be a template variable with dots in the event type replaced with +underscores. You can insert a template like so: -{{example_event_type_event}} +{{m_example_event_type_event}} Client behaviour ---------------- -List any new HTTP endpoints. List the steps the client needs to take to +List any new HTTP endpoints. These endpoints should be docced using Swagger. You +can insert a template for swagger docs like so: + +{{name-of-yaml-file-without-file-ext_http_api}} + +List the steps the client needs to take to correctly process this module. Listing what data structures the client should be -storing to aid implementation is recommended. +storing to aid implementation is recommended. Server behaviour ---------------- From 70518ae655eb9fae6ac49a3a1ddea700c7defc44 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 28 Sep 2015 14:03:54 +0100 Subject: [PATCH 15/54] Clarify template suffixes --- specification/modules/_template.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/specification/modules/_template.rst b/specification/modules/_template.rst index 967892a54..bb662c051 100644 --- a/specification/modules/_template.rst +++ b/specification/modules/_template.rst @@ -23,14 +23,16 @@ m.example.event.type ~~~~~~~~~~~~~~~~~~~~ There should be JSON Schema docs for this event. Once there is JSON schema, there will be a template variable with dots in the event type replaced with -underscores. You can insert a template like so: +underscores and the suffix ``_event``. You can insert a template like so: {{m_example_event_type_event}} Client behaviour ---------------- -List any new HTTP endpoints. These endpoints should be docced using Swagger. You -can insert a template for swagger docs like so: +List any new HTTP endpoints. These endpoints should be docced using Swagger. +Once there is Swagger, there will be a template variable based on the name of +the YAML file with the suffix ``_http_api``. You can insert a template for +swagger docs like so: {{name-of-yaml-file-without-file-ext_http_api}} From 5abea1f2bc2d36789164a4682cb8a668fb684845 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 28 Sep 2015 14:07:34 +0100 Subject: [PATCH 16/54] Minor tweaks --- specification/modules/_template.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/specification/modules/_template.rst b/specification/modules/_template.rst index bb662c051..9eee98439 100644 --- a/specification/modules/_template.rst +++ b/specification/modules/_template.rst @@ -15,12 +15,12 @@ Events ------ List the new event types introduced by this module, if any. If there are no new events, this section can be omitted. Event types should be done as -subsections. The section is intended to document the "common shared event +subsections. This section is intended to document the "common shared event structure" between client and server. Deviations from this shared structure should be documented in the relevant behaviour section. -m.example.event.type -~~~~~~~~~~~~~~~~~~~~ +``m.example.event.type`` +~~~~~~~~~~~~~~~~~~~~~~~~ There should be JSON Schema docs for this event. Once there is JSON schema, there will be a template variable with dots in the event type replaced with underscores and the suffix ``_event``. You can insert a template like so: @@ -29,7 +29,7 @@ underscores and the suffix ``_event``. You can insert a template like so: Client behaviour ---------------- -List any new HTTP endpoints. These endpoints should be docced using Swagger. +List any new HTTP endpoints. These endpoints should be documented using Swagger. Once there is Swagger, there will be a template variable based on the name of the YAML file with the suffix ``_http_api``. You can insert a template for swagger docs like so: @@ -37,14 +37,15 @@ swagger docs like so: {{name-of-yaml-file-without-file-ext_http_api}} List the steps the client needs to take to -correctly process this module. Listing what data structures the client should be -storing to aid implementation is recommended. +correctly process this module. List what data structures the client should be +storing in order to aid implementation. Server behaviour ---------------- Does the server need to handle any of the new events in a special way (e.g. typing timeouts, presence). Advice on how to persist events and/or requests are -recommended to aid implementation. +recommended to aid implementation. Federation-specific logic should be included +here. Security considerations ----------------------- From ce53a171800d765b9bcc86e019ec11d4583ed4ea Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 28 Sep 2015 14:51:10 +0100 Subject: [PATCH 17/54] Add txn_id rationale --- specification/1-client_server_api.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 988f7bc44..89698c663 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -251,6 +251,13 @@ request. The same ``nonce`` should be used if retrying the request. There are many ways a client may receive a ``token``, including via an email or from an existing logged in device. +The ``txn_id`` may be used by the server to disallow other devices from using +the token, thus providing "single use" tokens while still allowing the device +to retry the request. This would be done by tying the token to the ``txn_id`` +server side, as well as potentially invalidating the token completely once the +device has successfully logged in (e.g. when we receive a request from the +newly provisioned access_token). + OAuth2-based ~~~~~~~~~~~~ :Type: From 6e6bc8a5a9745627eb172dd98c77c140e89f1a95 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 28 Sep 2015 14:51:44 +0100 Subject: [PATCH 18/54] Mandate macaroon --- specification/1-client_server_api.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 89698c663..893aec734 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -258,6 +258,8 @@ server side, as well as potentially invalidating the token completely once the device has successfully logged in (e.g. when we receive a request from the newly provisioned access_token). +The ``token`` must be a macaroon. + OAuth2-based ~~~~~~~~~~~~ :Type: From 50e1b4c3a7147cb2a03fab56ff0403b998ef559e Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 29 Sep 2015 09:17:33 +0100 Subject: [PATCH 19/54] Fix up rst --- api/client-server/v1/sync.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml index dbfa39b80..8fe30056d 100644 --- a/api/client-server/v1/sync.yaml +++ b/api/client-server/v1/sync.yaml @@ -270,9 +270,9 @@ paths: invite: type: object title: "InviteEvent" - description: "The invite event if `membership` is `invite`" + description: "The invite event if ``membership`` is ``invite``" allOf: - - "$ref": "m.room.member" + - "$ref": "v1-event-schema/m.room.member" messages: type: object title: PaginationChunk From 083a76096f0edf6bf7810bd4f617d532499367ae Mon Sep 17 00:00:00 2001 From: Daniel Wagner-Hall Date: Tue, 29 Sep 2015 12:41:07 +0100 Subject: [PATCH 20/54] Support viewing the spec at head --- scripts/speculator/main.go | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go index b4fe2aef0..2609fa7ec 100644 --- a/scripts/speculator/main.go +++ b/scripts/speculator/main.go @@ -136,20 +136,30 @@ func generateAt(repo, sha string) (dst string, err error) { } func serveSpec(w http.ResponseWriter, req *http.Request) { - pr, err := lookupPullRequest(*req.URL, "/spec") - if err != nil { - writeError(w, 400, err) - return - } + var cloneURL string + var sha string + + if strings.ToLower(req.URL.Path) == "/spec/head" { + cloneURL = "https://github.com/matrix-org/matrix-doc.git" + sha = "HEAD" + } else { + pr, err := lookupPullRequest(*req.URL, "/spec") + if err != nil { + writeError(w, 400, err) + return + } - // We're going to run whatever Python is specified in the pull request, which - // may do bad things, so only trust people we trust. - if err := checkAuth(pr); err != nil { - writeError(w, 403, err) - return + // We're going to run whatever Python is specified in the pull request, which + // may do bad things, so only trust people we trust. + if err := checkAuth(pr); err != nil { + writeError(w, 403, err) + return + } + cloneURL = pr.Head.Repo.CloneURL + sha = pr.Head.SHA } - dst, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA) + dst, err := generateAt(cloneURL, sha) defer os.RemoveAll(dst) if err != nil { writeError(w, 500, err) @@ -287,7 +297,7 @@ func listPulls(w http.ResponseWriter, req *http.Request) { s += fmt.Sprintf(`
  • %d: %s: %s: spec spec diff rst diff
    • `, pull.Number, pull.User.HTMLURL, pull.User.Login, pull.HTMLURL, pull.Title, pull.Number, pull.Number, pull.Number) } - s += "" + s += `` io.WriteString(w, s) } From 82c27884baed11d5dd61bb84786ce05d5c5e47c3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 29 Sep 2015 16:29:16 +0100 Subject: [PATCH 21/54] Make E2E optional --- specification/0-feature_profiles.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index 86f8f3d8a..01997dd53 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -17,7 +17,7 @@ Summary ============================ ===== =========== ======== ========= ===== ===== Module / Profile Web Embed-Web Mobile Desktop CLI IoT ============================ ===== =========== ======== ========= ===== ===== - `End-to-End Encryption`_ YES YES YES YES + `End-to-End Encryption`_ `Instant Messaging`_ YES YES YES YES YES YES `Presence`_ YES YES YES YES `Push Notifications`_ YES From cdf9f011e9eb85cde0840548b1b9a1e5b7885b4e Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 29 Sep 2015 16:33:34 +0100 Subject: [PATCH 22/54] Add room history visibility as a module. --- specification/0-feature_profiles.rst | 2 ++ specification/modules/history_visibility.rst | 2 ++ 2 files changed, 4 insertions(+) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index 01997dd53..59e015e9f 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -25,6 +25,7 @@ Summary `Typing Notifications`_ YES YES YES YES `VoIP`_ YES YES YES `Content Repository`_ YES YES YES YES + `History Visibility`_ YES YES YES YES ============================ ===== =========== ======== ========= ===== ===== *Please see each module for more details on what clients need to implement.* @@ -37,6 +38,7 @@ Summary .. _Typing Notifications: `module:typing`_ .. _VoIP: `module:voip`_ .. _Content Repository: `module:content`_ +.. _History Visibility: `module:history-visibility`_ Clients ------- diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 01c2e419a..371282bd8 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -1,6 +1,8 @@ Room History Visibility ----------------------- +.. _module:history-visibility: + Whether a member of a room can see the events that happened in a room from before they joined the room is controlled by the ``history_visibility`` key of the ``m.room.history_visibility`` state event. The valid values for From 0a04672d764e070fc0199c958b8c006498314fbf Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 29 Sep 2015 17:57:44 +0100 Subject: [PATCH 23/54] Start converting the presence module. Add Rationale admonition. --- scripts/nature.css | 6 ++ specification/1-client_server_api.rst | 21 +++++ specification/modules/presence.rst | 114 ++++++++++++++++---------- 3 files changed, 97 insertions(+), 44 deletions(-) diff --git a/scripts/nature.css b/scripts/nature.css index 9d67f689b..f1b4edece 100644 --- a/scripts/nature.css +++ b/scripts/nature.css @@ -282,3 +282,9 @@ td[colspan]:not([colspan="1"]) { thead { background: #eeeeee; } + +div.admonition-rationale { + background-color: #efe; + border: 1px solid #ccc; +} + diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 893aec734..4403e0d3e 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -1092,6 +1092,27 @@ Profiles {{profile_http_api}} +Events on Change of Profile Information +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Because the profile display name and avatar information are likely to be used in +many places of a client's display, changes to these fields cause an automatic +propagation event to occur, informing likely-interested parties of the new +values. This change is conveyed using two separate mechanisms: + + - a ``m.room.member`` event is sent to every room the user is a member of, + to update the ``displayname`` and ``avatar_url``. + - a ``m.presence`` presence status update is sent, again containing the new + values of the ``displayname`` and ``avatar_url`` keys, in addition to the + required ``presence`` key containing the current presence state of the user. + +Both of these should be done automatically by the home server when a user +successfully changes their display name or avatar URL fields. + +Additionally, when home servers emit room membership events for their own +users, they should include the display name and avatar URL fields in these +events so that clients already have these details to hand, and do not have to +perform extra round trips to query it. + Security -------- diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst index ddd2adff2..4c84359ad 100644 --- a/specification/modules/presence.rst +++ b/specification/modules/presence.rst @@ -1,63 +1,89 @@ Presence ======== -Each user has the concept of presence information. This encodes the -"availability" of that user, suitable for display on other user's clients. +Each user has presence information associated with them. This encodes the +"availability" of that user, suitable for display on other clients. This is transmitted as an ``m.presence`` event and is one of the few events -which are sent *outside the context of a room*. The basic piece of presence -information is represented by the ``presence`` key, which is an enum of one -of the following: +which are sent *outside the context of a room*. Their presence state is +represented by the ``presence`` key, which is an enum of one of the following: - ``online`` : The default state when the user is connected to an event stream. - - ``unavailable`` : The user is not reachable at this time. - - ``offline`` : The user is not connected to an event stream. + - ``unavailable`` : The user is not reachable at this time e.g. they are + idle. + - ``offline`` : The user is not connected to an event stream or is + explicitly suppressing their profile information from being sent. - ``free_for_chat`` : The user is generally willing to receive messages moreso than default. - - ``hidden`` : Behaves as offline, but allows the user to see the client - state anyway and generally interact with client features. (Not yet - implemented in synapse). - -In addition, the server maintains a timestamp of the last time it saw a -pro-active event from the user; either sending a message to a room, or -changing presence state from a lower to a higher level of availability -(thus: changing state from ``unavailable`` to ``online`` counts as a -proactive event, whereas in the other direction it will not). This timestamp -is presented via a key called ``last_active_ago``, which gives the relative -number of milliseconds since the message is generated/emitted that the user -was last seen active. Events ------ {{presence_events}} -Presence HTTP API ------------------ -.. TODO-spec - - Define how users receive presence invites, and how they accept/decline them +Client behaviour +---------------- + +Clients can manually set/get their presence using the HTTP APIs listed below. {{presence_http_api}} - -Events on Change of Profile Information ---------------------------------------- -Because the profile displayname and avatar information are likely to be used in -many places of a client's display, changes to these fields cause an automatic -propagation event to occur, informing likely-interested parties of the new -values. This change is conveyed using two separate mechanisms: - - - a ``m.room.member`` event is sent to every room the user is a member of, - to update the ``displayname`` and ``avatar_url``. - - a ``m.presence`` presence status update is sent, again containing the new values of the - ``displayname`` and ``avatar_url`` keys, in addition to the required - ``presence`` key containing the current presence state of the user. - -Both of these should be done automatically by the home server when a user -successfully changes their displayname or avatar URL fields. - -Additionally, when home servers emit room membership events for their own -users, they should include the displayname and avatar URL fields in these -events so that clients already have these details to hand, and do not have to -perform extra roundtrips to query it. +Idle timeout +~~~~~~~~~~~~ + +Clients SHOULD implement an "idle timeout". This is a timer which fires after +a period of inactivity on the client. The definition of inactivity varies +depending on the client. For example, web implementations may determine +inactivity to be not moving the mouse for a certain period of time. When this +timer fires it should set the presence state to ``unavailable``. When the user +becomes active again (e.g. by moving the mouse) the client should set the +presence state to ``online``. A timeout value between 1 and 5 minutes is +recommended. + +Server behaviour +---------------- + +Propagating profile information +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Because the profile display name and avatar information are likely to be used in +many places of a client's display, changes to these fields SHOULD cause an +automatic propagation event to occur, informing likely-interested parties of the +new values. One of these change mechanisms SHOULD be via ``m.presence`` events. +These events should set ``displayname`` and ``avatar_url`` to the new values +along with the presence-specific keys. This SHOULD be done automatically by the +home server when a user successfully changes their display name or avatar URL. + +.. admonition:: Rationale + + The intention for sending this information in ``m.presence`` is so that any + "user list" can display the *current* name/presence for a user ID outside the + scope of a room (e.g. a user page which has a "start conversation" button). + This is bundled into a single event to avoid "flickering" on this page which + can occur if you received presence first and then display name later (the + user's name would flicker from their user ID to the display name). + + +Last active ago +~~~~~~~~~~~~~~~ +The server maintains a timestamp of the last time it saw a +pro-active event from the user. A pro-active event may be sending a message to a +room or changing presence state to a higher level of availability. Levels of +availability are defined from low to high as follows: + + - ``offline`` + - ``unavailable`` + - ``online`` + - ``free_for_chat`` + +Based on this list, changing state from ``unavailable`` to ``online`` counts as +a pro-active event, whereas ``online`` to ``unavailable`` does not. This +timestamp is presented via a key called ``last_active_ago`` which gives the +relative number of milliseconds since the pro-active event. + +Security considerations +----------------------- + +Presence information is shared with all users who share a room with the target +user. In large public rooms this could be undesirable. From 73b4090f52f324c598db699899b22eed72503a3d Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 30 Sep 2015 10:22:12 +0100 Subject: [PATCH 24/54] Add private_chat_shared_power --- specification/1-client_server_api.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index c5453931a..329a87ddb 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -756,15 +756,18 @@ options which can be set when creating a room: Optional: Yes Value: - ``private_chat`` or ``public_chat`` + ``private_chat``, ``private_chat_shared_power`` or ``public_chat`` Description: Convenience parameter for setting various default state events based on a preset. - Two presets are defined: + Three presets are defined: - ``private_chat``: Sets the ``join_rules`` to ``invite`` and ``history_visibility`` to ``shared`` + - ``private_chat_shared_power``: Set the ``join_rules`` to + ``invite``, ``history_visibility`` to ``shared`` and gives all invitees + the same power level as the creator. - ``public_chat``: Sets the ``join_rules`` to ``public`` and ``history_visibility`` to ``shared`` From 069e4e39f4d326ef9e4188159767a1194db1ae76 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 10:22:51 +0100 Subject: [PATCH 25/54] Move presence specific sections from intro to presence module --- specification/0-intro.rst | 43 ------------------------------ specification/modules/presence.rst | 35 ++++++++++++++++++++---- 2 files changed, 30 insertions(+), 48 deletions(-) diff --git a/specification/0-intro.rst b/specification/0-intro.rst index a5196e990..1b4e8672f 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -338,49 +338,6 @@ Usage of an IS is not required in order for a client application to be part of the Matrix ecosystem. However, without one clients will not be able to look up user IDs using 3PIDs. -Presence -~~~~~~~~ - -Each user has the concept of presence information. This encodes: - - * Whether the user is currently online - * How recently the user was last active (as seen by the server) - * Whether a given client considers the user to be currently idle - * Arbitrary information about the user's current status (e.g. "in a meeting"). - -This information is collated from both per-device (online; idle; last_active) and -per-user (status) data, aggregated by the user's homeserver and transmitted as -an ``m.presence`` event. This is one of the few events which are sent *outside -the context of a room*. Presence events are sent to all users who subscribe to -this user's presence through a presence list or by sharing membership of a room. - -.. TODO - How do we let users hide their presence information? - -.. TODO - The last_active specifics should be moved to the detailed presence event section - -Last activity is tracked by the server maintaining a timestamp of the last time -it saw a pro-active event from the user. Any event which could be triggered by a -human using the application is considered pro-active (e.g. sending an event to a -room). An example of a non-proactive client activity would be a client setting -'idle' presence status, or polling for events. This timestamp is presented via a -key called ``last_active_ago``, which gives the relative number of milliseconds -since the message is generated/emitted that the user was last seen active. - -N.B. in v1 API, status/online/idle state are muxed into a single 'presence' -field on the ``m.presence`` event. - -Presence Lists -~~~~~~~~~~~~~~ - -Each user's home server stores a "presence list". This stores a list of user IDs -whose presence the user wants to follow. - -To be added to this list, the user being added must be invited by the list owner -and accept the invitation. Once accepted, both user's HSes track the -subscription. - Profiles ~~~~~~~~ diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst index 4c84359ad..1a359e5a6 100644 --- a/specification/modules/presence.rst +++ b/specification/modules/presence.rst @@ -1,5 +1,23 @@ Presence ======== + +Each user has the concept of presence information. This encodes: + + * Whether the user is currently online + * How recently the user was last active (as seen by the server) + * Whether a given client considers the user to be currently idle + * Arbitrary information about the user's current status (e.g. "in a meeting"). + +This information is collated from both per-device (``online``, ``idle``, +``last_active``) and per-user (status) data, aggregated by the user's homeserver +and transmitted as an ``m.presence`` event. This is one of the few events which +are sent *outside the context of a room*. Presence events are sent to all users +who subscribe to this user's presence through a presence list or by sharing +membership of a room. + +A presence list is a list of user IDs whose presence the user wants to follow. +To be added to this list, the user being added must be invited by the list owner +who must accept the invitation. Each user has presence information associated with them. This encodes the "availability" of that user, suitable for display on other clients. @@ -15,6 +33,7 @@ represented by the ``presence`` key, which is an enum of one of the following: explicitly suppressing their profile information from being sent. - ``free_for_chat`` : The user is generally willing to receive messages moreso than default. + Events ------ @@ -24,7 +43,8 @@ Events Client behaviour ---------------- -Clients can manually set/get their presence using the HTTP APIs listed below. +Clients can manually set/get their presence/presence list using the HTTP APIs +listed below. {{presence_http_api}} @@ -43,6 +63,9 @@ recommended. Server behaviour ---------------- +Each user's home server stores a "presence list" per user. Once a user accepts +a presence list, both user's HSes must track the subscription. + Propagating profile information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -58,10 +81,12 @@ home server when a user successfully changes their display name or avatar URL. The intention for sending this information in ``m.presence`` is so that any "user list" can display the *current* name/presence for a user ID outside the - scope of a room (e.g. a user page which has a "start conversation" button). - This is bundled into a single event to avoid "flickering" on this page which - can occur if you received presence first and then display name later (the - user's name would flicker from their user ID to the display name). + scope of a room e.g. for a user page. This is bundled into a single event for + several reasons. The user's display name can change per room. This + event provides the "canonical" name for the user. In addition, the name is + bundled into a single event for the ease of client implementations. If this + was not done, the client would need to search all rooms for their own + membership event to pull out the display name. Last active ago From 4e1e82f995f899d397ac5a21194a48973e262948 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 10:43:02 +0100 Subject: [PATCH 26/54] Fix paragraph CSS and adjust where paragraphs are in the RST The CSS for `nature.css` was such that it was preventing `p` tags from having sufficient vertical whitespace. This meant that you couldn't insert any kind of spacing between lengthy sections (they just appeared as new lines). This PR fixes this so you can actually have some whitespace between paragraphs. As a result of this change, some parts of the spec appeared to have too much whitespace. These were often sections which shouldn't have begun a new paragraph anyway (e.g. a single sentence being an entire paragraph, `TODO` blocks resulting in new paragraphs). This PR fixes the most offending areas where we shouldn't have been inserting new paragraphs. --- scripts/nature.css | 2 +- specification/0-intro.rst | 69 +++++++++------------ specification/1-client_server_api.rst | 34 ++++------ specification/3-application_service_api.rst | 18 +++--- specification/4-server_server_api.rst | 12 ++-- specification/modules/content_repo.rst | 24 +++---- specification/modules/push_cs_api.rst | 17 +++-- specification/modules/voip_events.rst | 9 ++- 8 files changed, 77 insertions(+), 108 deletions(-) diff --git a/scripts/nature.css b/scripts/nature.css index 9d67f689b..ac171923b 100644 --- a/scripts/nature.css +++ b/scripts/nature.css @@ -245,7 +245,7 @@ div.viewcode-block:target { } p { - margin: 0; + /* margin: 0; - setting this leads to no spacing between paragraphs which looks ugly */ } ul li dd { diff --git a/specification/0-intro.rst b/specification/0-intro.rst index a5196e990..93458d97d 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -24,7 +24,6 @@ Introduction The Matrix specification is still evolving: the APIs are not yet frozen and this document is in places a work in progress or stale. We have made every effort to clearly flag areas which are still being finalised. - We're publishing it at this point because it's complete enough to be more than useful and provide a canonical reference to how Matrix is evolving. Our end goal is to mirror WHATWG's `Living Standard @@ -34,10 +33,9 @@ Matrix is a set of open APIs for open-federated Instant Messaging (IM), Voice over IP (VoIP) and Internet of Things (IoT) communication, designed to create and support a new global real-time communication ecosystem. The intention is to provide an open decentralised pubsub layer for the internet for securely -persisting and publishing/subscribing JSON objects. - -This specification is the ongoing result of standardising the APIs used by the -various components of the Matrix ecosystem to communicate with one another. +persisting and publishing/subscribing JSON objects. This specification is the +ongoing result of standardising the APIs used by the various components of the +Matrix ecosystem to communicate with one another. The principles that Matrix attempts to follow are: @@ -214,10 +212,8 @@ which have the form:: There is exactly one room ID for each room. Whilst the room ID does contain a domain, it is simply for globally namespacing room IDs. The room does NOT reside on the domain specified. Room IDs are not meant to be human readable. -They are case-sensitive. - -The following conceptual diagram shows an ``m.room.message`` event being sent to -the room ``!qporfwt:matrix.org``:: +They are case-sensitive. The following conceptual diagram shows an +``m.room.message`` event being sent to the room ``!qporfwt:matrix.org``:: { @alice:matrix.org } { @bob:domain.com } | ^ @@ -258,28 +254,28 @@ the room ``!qporfwt:matrix.org``:: Federation maintains *shared data structures* per-room between multiple home servers. The data is split into ``message events`` and ``state events``. -``Message events`` describe transient 'once-off' activity in a room such as an -instant messages, VoIP call setups, file transfers, etc. They generally describe -communication activity. +Message events: + These describe transient 'once-off' activity in a room such as an + instant messages, VoIP call setups, file transfers, etc. They generally + describe communication activity. -``State events`` describe updates to a given piece of persistent information -('state') related to a room, such as the room's name, topic, membership, -participating servers, etc. State is modelled as a lookup table of key/value -pairs per room, with each key being a tuple of ``state_key`` and ``event type``. -Each state event updates the value of a given key. +State events: + These describe updates to a given piece of persistent information + ('state') related to a room, such as the room's name, topic, membership, + participating servers, etc. State is modelled as a lookup table of key/value + pairs per room, with each key being a tuple of ``state_key`` and ``event type``. + Each state event updates the value of a given key. The state of the room at a given point is calculated by considering all events preceding and including a given event in the graph. Where events describe the same state, a merge conflict algorithm is applied. The state resolution algorithm is transitive and does not depend on server state, as it must consistently select the same event irrespective of the server or the order the -events were received in. - -Events are signed by the originating server (the signature includes the parent -relations, type, depth and payload hash) and are pushed over federation to the -participating servers in a room, currently using full mesh topology. Servers may -also request backfill of events over federation from the other servers -participating in a room. +events were received in. Events are signed by the originating server (the +signature includes the parent relations, type, depth and payload hash) and are +pushed over federation to the participating servers in a room, currently using +full mesh topology. Servers may also request backfill of events over federation +from the other servers participating in a room. Room Aliases @@ -324,12 +320,10 @@ Users in Matrix are identified via their matrix user ID (MXID). However, existing 3rd party ID namespaces can also be used in order to identify Matrix users. A Matrix "Identity" describes both the user ID and any other existing IDs from third party namespaces *linked* to their account. - Matrix users can *link* third-party IDs (3PIDs) such as email addresses, social network accounts and phone numbers to their user ID. Linking 3PIDs creates a mapping from a 3PID to a user ID. This mapping can then be used by Matrix users in order to discover the MXIDs of their contacts. - In order to ensure that the mapping from 3PID to user ID is genuine, a globally federated cluster of trusted "Identity Servers" (IS) are used to verify the 3PID and persist and replicate the mappings. @@ -410,6 +404,10 @@ dedicated API. The API is symmetrical to managing Profile data. API Standards ------------- +.. TODO + Need to specify any HMAC or access_token lifetime/ratcheting tricks + We need to specify capability negotiation for extensible transports + The mandatory baseline for communication in Matrix is exchanging JSON objects over HTTP APIs. HTTPS is mandated as the baseline for server-server (federation) communication. HTTPS is recommended for client-server @@ -417,20 +415,11 @@ communication, although HTTP may be supported as a fallback to support basic HTTP clients. More efficient optional transports for client-server communication will in future be supported as optional extensions - e.g. a packed binary encoding over stream-cipher encrypted TCP socket for -low-bandwidth/low-roundtrip mobile usage. - -.. TODO - We need to specify capability negotiation for extensible transports - -For the default HTTP transport, all API calls use a Content-Type of -``application/json``. In addition, all strings MUST be encoded as UTF-8. - -Clients are authenticated using opaque ``access_token`` strings (see -`Client Authentication`_ for details), passed as a query string parameter on -all requests. - -.. TODO - Need to specify any HMAC or access_token lifetime/ratcheting tricks +low-bandwidth/low-roundtrip mobile usage. For the default HTTP transport, all +API calls use a Content-Type of ``application/json``. In addition, all strings +MUST be encoded as UTF-8. Clients are authenticated using opaque +``access_token`` strings (see `Client Authentication`_ for details), passed as a +query string parameter on all requests. Any errors which occur at the Matrix API level MUST return a "standard error response". This is a JSON object which looks like:: diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 893aec734..7d8d57e40 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -672,13 +672,10 @@ to add keys that are, for example offensive or illegal. Since some events cannot be simply deleted, e.g. membership events, we instead 'redact' events. This involves removing all keys from an event that are not required by the protocol. This stripped down event is thereafter returned anytime a client or -remote server requests it. - -Events that have been redacted include a ``redacted_because`` key whose value -is the event that caused it to be redacted, which may include a reason. - -Redacting an event cannot be undone, allowing server owners to delete the -offending content from the databases. +remote server requests it. Redacting an event cannot be undone, allowing server +owners to delete the offending content from the databases. Events that have been +redacted include a ``redacted_because`` key whose value is the event that caused +it to be redacted, which may include a reason. .. TODO Currently, only room admins can redact events by sending a ``m.room.redaction`` @@ -709,12 +706,10 @@ one of the following event types: .. TODO Need to update m.room.power_levels to reflect new power levels formatting -The redaction event should be added under the key ``redacted_because``. - -When a client receives a redaction event it should change the redacted event +The redaction event should be added under the key ``redacted_because``. When a +client receives a redaction event it should change the redacted event in the same way a server does. - Rooms ----- @@ -853,18 +848,14 @@ Permissions Permissions for rooms are done via the concept of power levels - to do any action in a room a user must have a suitable power level. Power levels are -stored as state events in a given room. - -The power levels required for operations and the power levels for users are -defined in ``m.room.power_levels``, where both a default and specific users' -power levels can be set. - +stored as state events in a given room. The power levels required for operations +and the power levels for users are defined in ``m.room.power_levels``, where +both a default and specific users' power levels can be set. By default all users have a power level of 0, other than the room creator whose power level defaults to 100. Users can grant other users increased power levels up to their own power level. For example, user A with a power level of 50 could increase the power level of user B to a maximum of level 50. Power levels for users are tracked per-room even if the user is not present in the room. - The keys contained in ``m.room.power_levels`` determine the levels required for certain operations such as kicking, banning and sending state events. See `m.room.power_levels`_ for more information. @@ -881,10 +872,9 @@ room. There are several states in which a user may be, in relation to a room: - Banned (the user is not allowed to join the room) Some rooms require that users be invited to it before they can join; others -allow anyone to join. - -Whether a given room is an "invite-only" room is determined by the room config -key ``m.room.join_rules``. It can have one of the following values: +allow anyone to join. Whether a given room is an "invite-only" room is +determined by the room config key ``m.room.join_rules``. It can have one of the +following values: ``public`` This room is free for anyone to join without an invite. diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst index e982390b4..bdde77894 100644 --- a/specification/3-application_service_api.rst +++ b/specification/3-application_service_api.rst @@ -4,11 +4,9 @@ Application Service API The Matrix client-server API and server-server APIs provide the means to implement a consistent self-contained federated messaging fabric. However, they provide limited means of implementing custom server-side behaviour in Matrix -(e.g. gateways, filters, extensible hooks etc). - -The Application Service API defines a standard API to allow such extensible -functionality to be implemented irrespective of the underlying homeserver -implementation. +(e.g. gateways, filters, extensible hooks etc). The Application Service API +defines a standard API to allow such extensible functionality to be implemented +irrespective of the underlying homeserver implementation. .. TODO-spec Add in Client-Server services? Overview of bots? Seems weird to be in the spec @@ -18,12 +16,10 @@ Passive Application Services ---------------------------- "Passive" application services can only observe events from a given home server. They cannot prevent events from being sent, nor can they modify the content of -the event being sent. - -In order to observe events from a homeserver, the homeserver needs to be -configured to pass certain types of traffic to the application service. This -is achieved by manually configuring the homeserver with information about the -AS.. +the event being sent. In order to observe events from a homeserver, the +homeserver needs to be configured to pass certain types of traffic to the +application service. This is achieved by manually configuring the homeserver +with information about the AS. .. NOTE:: Previously, application services could register with a homeserver via HTTP diff --git a/specification/4-server_server_api.rst b/specification/4-server_server_api.rst index 8d1f8898f..c5ff2b877 100644 --- a/specification/4-server_server_api.rst +++ b/specification/4-server_server_api.rst @@ -59,13 +59,11 @@ and an optional TLS port. .. ** If the port is present then the server is discovered by looking up an AAAA or -A record for the DNS name and connecting to the specified TLS port. - -If the port is absent then the server is discovered by looking up a -``_matrix._tcp`` SRV record for the DNS name. If this record does not exist -then the server is discovered by looking up an AAAA or A record on the DNS -name and taking the default fallback port number of 8448. - +A record for the DNS name and connecting to the specified TLS port. If the port +is absent then the server is discovered by looking up a ``_matrix._tcp`` SRV +record for the DNS name. If this record does not exist then the server is +discovered by looking up an AAAA or A record on the DNS name and taking the +default fallback port number of 8448. Home servers may use SRV records to load balance requests between multiple TLS endpoints or to failover to another endpoint if an endpoint fails. diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst index 2c45ced74..5a3cf9b93 100644 --- a/specification/modules/content_repo.rst +++ b/specification/modules/content_repo.rst @@ -56,21 +56,21 @@ Homeservers must never upscale images. Security considerations ----------------------- -Clients may try to upload very large files. Homeservers should not store files -that are too large and should not serve them to clients. + - Clients may try to upload very large files. Homeservers should not store files + that are too large and should not serve them to clients. -Clients may try to upload very large images. Homeservers should not attempt to -generate thumbnails for images that are too large. + - Clients may try to upload very large images. Homeservers should not attempt to + generate thumbnails for images that are too large. -Remote homeservers may host very large files or images. Homeserver should not -proxy or thumbnail large files or images from remote homeservers. + - Remote homeservers may host very large files or images. Homeserver should not + proxy or thumbnail large files or images from remote homeservers. -Clients may try to upload a large number of files. Homeservers should limit the -number and total size of media that can be uploaded by clients. + - Clients may try to upload a large number of files. Homeservers should limit the + number and total size of media that can be uploaded by clients. -Clients may try to access a large number of remote files through a homeserver. -Homeservers should restrict the number and size of remote files that it caches. + - Clients may try to access a large number of remote files through a homeserver. + Homeservers should restrict the number and size of remote files that it caches. -Clients or remote homeservers may try to upload malicious files targeting -vulnerabilities in either the homeserver thumbnailing or the client decoders. + - Clients or remote homeservers may try to upload malicious files targeting + vulnerabilities in either the homeserver thumbnailing or the client decoders. diff --git a/specification/modules/push_cs_api.rst b/specification/modules/push_cs_api.rst index b78b44fde..c63019262 100644 --- a/specification/modules/push_cs_api.rst +++ b/specification/modules/push_cs_api.rst @@ -99,16 +99,13 @@ be redundant. Actions for the highest priority rule and only that rule apply (for example, a set_tweak action in a lower priority rule will not apply if a higher priority rule matches, even if that rule does not specify any tweaks). -Rules also have an identifier, rule_id, which is a string. The rule_id is -unique within the kind of rule and scope: rule_ids need not be unique between -rules of the same kind on different devices. - -A home server may also have server default rules of each kind and in each scope. -Server default rules are lower priority than user-defined rules in each scope. -Server default rules (and only server default rules) begin with a dot ('.') -character. - -In addition, all rules may be enabled or disabled. Disabled rules never match. +Rules also have an identifier, ``rule_id``, which is a string. The ``rule_id`` +is unique within the kind of rule and scope: ``rule_ids`` need not be unique +between rules of the same kind on different devices. A home server may also have +server default rules of each kind and in each scope. Server default rules are +lower priority than user-defined rules in each scope. Server default rules (and +only server default rules) begin with a dot ('.') character. In addition, all +rules may be enabled or disabled. Disabled rules never match. If no rules match an event, the Home Server should not notify for the message (that is to say, the default action is "dont-notify"). Events that the user sent diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst index a54682379..33998cd95 100644 --- a/specification/modules/voip_events.rst +++ b/specification/modules/voip_events.rst @@ -44,11 +44,10 @@ Glare This specification aims to address the problem of two users calling each other at roughly the same time and their invites crossing on the wire. It is a far better experience for the users if their calls are connected if it is clear -that their intention is to set up a call with one another. - -In Matrix, calls are to rooms rather than users (even if those rooms may only -contain one other user) so we consider calls which are to the same room. The -rules for dealing with such a situation are as follows: +that their intention is to set up a call with one another. In Matrix, calls are +to rooms rather than users (even if those rooms may only contain one other user) +so we consider calls which are to the same room. The rules for dealing with such +a situation are as follows: - If an invite to a room is received whilst the client is preparing to send an invite to the same room, the client should cancel its outgoing call and From a49d5f67f1ee8e06c2c7a3e0baaba5f120f7a4e3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 10:54:06 +0100 Subject: [PATCH 27/54] Delete rather than comment out the offending CSS --- scripts/nature.css | 4 ---- 1 file changed, 4 deletions(-) diff --git a/scripts/nature.css b/scripts/nature.css index ac171923b..7c4af305f 100644 --- a/scripts/nature.css +++ b/scripts/nature.css @@ -244,10 +244,6 @@ div.viewcode-block:target { border-bottom: 1px solid #ac9; } -p { - /* margin: 0; - setting this leads to no spacing between paragraphs which looks ugly */ -} - ul li dd { margin-top: 0; } From ede43fbe9fc59b0f4eff83d7e1e8cc0d69c180a6 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 14:31:26 +0100 Subject: [PATCH 28/54] Modify how speculator uses git repos Modified so it has a "master" repo and local A/B repos which pull from the "master". This saves an extra git clone operation per HTTP request. --- scripts/speculator/main.go | 72 +++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 21 deletions(-) diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go index 2609fa7ec..6ee80150d 100644 --- a/scripts/speculator/main.go +++ b/scripts/speculator/main.go @@ -24,6 +24,7 @@ import ( "strconv" "strings" "syscall" + "time" ) type PullRequest struct { @@ -58,16 +59,24 @@ func (u *User) IsTrusted() bool { return allowedMembers[u.Login] } -const pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls" +const ( + pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls" + matrixDocCloneURL = "https://github.com/matrix-org/matrix-doc.git" +) + + +func gitClone(url string, shared bool) (string, error) { + directory := path.Join("/tmp/matrix-doc", strconv.FormatInt(rand.Int63(), 10)) + cmd := exec.Command("git", "clone", url, directory) + if shared { + cmd.Args = append(cmd.Args, "--shared") + } -func gitClone(url string) (string, error) { - dst := path.Join("/tmp/matrix-doc", strconv.FormatInt(rand.Int63(), 10)) - cmd := exec.Command("git", "clone", url, dst) err := cmd.Run() if err != nil { return "", fmt.Errorf("error cloning repo: %v", err) } - return dst, nil + return directory, nil } func gitCheckout(path, sha string) error { @@ -80,6 +89,16 @@ func gitCheckout(path, sha string) error { return nil } +func gitFetch(path string) error { + cmd := exec.Command("git", "fetch") + cmd.Dir = path + err := cmd.Run() + if err != nil { + return fmt.Errorf("error fetching repo: %v", err) + } + return nil +} + func lookupPullRequest(url url.URL, pathPrefix string) (*PullRequest, error) { if !strings.HasPrefix(url.Path, pathPrefix+"/") { return nil, fmt.Errorf("invalid path passed: %s expect %s/123", url.Path, pathPrefix) @@ -119,10 +138,18 @@ func writeError(w http.ResponseWriter, code int, err error) { io.WriteString(w, fmt.Sprintf("%v\n", err)) } +type server struct { + matrixDocCloneURL string +} + // generateAt generates spec from repo at sha. // Returns the path where the generation was done. -func generateAt(repo, sha string) (dst string, err error) { - dst, err = gitClone(repo) +func (s *server) generateAt(sha string) (dst string, err error) { + err = gitFetch(s.matrixDocCloneURL) + if err != nil { + return + } + dst, err = gitClone(s.matrixDocCloneURL, true) if err != nil { return } @@ -135,12 +162,10 @@ func generateAt(repo, sha string) (dst string, err error) { return } -func serveSpec(w http.ResponseWriter, req *http.Request) { - var cloneURL string +func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) { var sha string if strings.ToLower(req.URL.Path) == "/spec/head" { - cloneURL = "https://github.com/matrix-org/matrix-doc.git" sha = "HEAD" } else { pr, err := lookupPullRequest(*req.URL, "/spec") @@ -155,11 +180,10 @@ func serveSpec(w http.ResponseWriter, req *http.Request) { writeError(w, 403, err) return } - cloneURL = pr.Head.Repo.CloneURL sha = pr.Head.SHA } - dst, err := generateAt(cloneURL, sha) + dst, err := s.generateAt(sha) defer os.RemoveAll(dst) if err != nil { writeError(w, 500, err) @@ -181,7 +205,7 @@ func checkAuth(pr *PullRequest) error { return nil } -func serveRSTDiff(w http.ResponseWriter, req *http.Request) { +func (s *server) serveRSTDiff(w http.ResponseWriter, req *http.Request) { pr, err := lookupPullRequest(*req.URL, "/diff/rst") if err != nil { writeError(w, 400, err) @@ -195,14 +219,14 @@ func serveRSTDiff(w http.ResponseWriter, req *http.Request) { return } - base, err := generateAt(pr.Base.Repo.CloneURL, pr.Base.SHA) + base, err := s.generateAt(pr.Base.SHA) defer os.RemoveAll(base) if err != nil { writeError(w, 500, err) return } - head, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA) + head, err := s.generateAt(pr.Head.SHA) defer os.RemoveAll(head) if err != nil { writeError(w, 500, err) @@ -219,7 +243,7 @@ func serveRSTDiff(w http.ResponseWriter, req *http.Request) { w.Write(diff.Bytes()) } -func serveHTMLDiff(w http.ResponseWriter, req *http.Request) { +func (s *server) serveHTMLDiff(w http.ResponseWriter, req *http.Request) { pr, err := lookupPullRequest(*req.URL, "/diff/html") if err != nil { writeError(w, 400, err) @@ -233,14 +257,14 @@ func serveHTMLDiff(w http.ResponseWriter, req *http.Request) { return } - base, err := generateAt(pr.Base.Repo.CloneURL, pr.Base.SHA) + base, err := s.generateAt(pr.Base.SHA) defer os.RemoveAll(base) if err != nil { writeError(w, 500, err) return } - head, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA) + head, err := s.generateAt(pr.Head.SHA) defer os.RemoveAll(head) if err != nil { writeError(w, 500, err) @@ -327,9 +351,15 @@ func main() { "Kegsay": true, "NegativeMjark": true, } - http.HandleFunc("/spec/", serveSpec) - http.HandleFunc("/diff/rst/", serveRSTDiff) - http.HandleFunc("/diff/html/", serveHTMLDiff) + rand.Seed(time.Now().Unix()) + masterCloneDir, err := gitClone(matrixDocCloneURL, false) + if err != nil { + log.Fatal(err) + } + s := server{masterCloneDir} + http.HandleFunc("/spec/", s.serveSpec) + http.HandleFunc("/diff/rst/", s.serveRSTDiff) + http.HandleFunc("/diff/html/", s.serveHTMLDiff) http.HandleFunc("/healthz", serveText("ok")) http.HandleFunc("/", listPulls) log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil)) From b6f0b67da66d4d9d286ad6efb641f79583456cbc Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 15:21:32 +0100 Subject: [PATCH 29/54] Speed up continuserv Ignore .git directory as that shouldn't affect spec generation. Also, when we receive writes from the OS, wait a bit before re-generating the spec to clump together multiple writes rather than re-generating one after another and waiting for no more writes before serving the request. --- scripts/continuserv/main.go | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go index 658ae0fb6..573c2c955 100644 --- a/scripts/continuserv/main.go +++ b/scripts/continuserv/main.go @@ -17,6 +17,7 @@ import ( "strings" "sync" "sync/atomic" + "time" fsnotify "gopkg.in/fsnotify.v1" ) @@ -67,7 +68,6 @@ func watchFS(ch chan struct{}, w *fsnotify.Watcher) { select { case e := <-w.Events: if filter(e) { - wg.Add(1) fmt.Printf("Noticed change to %s, re-generating spec\n", e.Name) ch <- struct{}{} } @@ -98,6 +98,11 @@ func filter(e fsnotify.Event) bool { return false } + // Ignore the .git directory - It's very noisy + if strings.Contains(e.Name, "/.git/") { + return false + } + // Avoid infinite cycles being caused by writing actual output if strings.Contains(e.Name, "/tmp/") || strings.Contains(e.Name, "/gen/") { return false @@ -133,8 +138,20 @@ func populateOnce(dir string) { } func doPopulate(ch chan struct{}, dir string) { - for _ = range ch { - populateOnce(dir) + var pending int + for { + select { + case <-ch: + if pending == 0 { + wg.Add(1) + } + pending++ + case <-time.After(10 * time.Millisecond): + if pending > 0 { + pending = 0 + populateOnce(dir) + } + } } } From 2b7e02c0805b81d1c2ad72245f4dc13bdb56a445 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 11:26:02 +0100 Subject: [PATCH 30/54] Add sections for typing. Add swagger, JSON schema and example m.typing event --- api/client-server/v1/typing.yaml | 77 +++++++++++++++++++ event-schemas/examples/v1/m.typing | 7 ++ event-schemas/schema/v1/m.typing | 28 +++++++ .../modules/typing_notifications.rst | 47 +++++------ 4 files changed, 131 insertions(+), 28 deletions(-) create mode 100644 api/client-server/v1/typing.yaml create mode 100644 event-schemas/examples/v1/m.typing create mode 100644 event-schemas/schema/v1/m.typing diff --git a/api/client-server/v1/typing.yaml b/api/client-server/v1/typing.yaml new file mode 100644 index 000000000..737c69284 --- /dev/null +++ b/api/client-server/v1/typing.yaml @@ -0,0 +1,77 @@ +swagger: '2.0' +info: + title: "Matrix Client-Server v1 Typing API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/api/v1 +consumes: + - application/json +produces: + - application/json +securityDefinitions: + accessToken: + type: apiKey + description: The user_id or application service access_token + name: access_token + in: query +paths: + "/rooms/{roomId}/typing/{userId}": + put: + summary: Informs the server that the user has started or stopped typing. + description: |- + This tells the server that the user is typing for the next N + milliseconds where N is the value specified in the ``timeout`` key. + Alternatively, if ``typing`` is ``false``, it tells the server that the + user has stopped typing. + security: + - accessToken: [] + parameters: + - in: path + type: string + name: userId + description: The user who has started to type. + required: true + x-example: "@alice:example.com" + - in: path + type: string + name: roomId + description: The room in which the user is typing. + required: true + x-example: "!wefh3sfukhs:example.com" + - in: body + name: typingState + description: The current typing state. + required: true + schema: + type: object + example: |- + { + "typing": true, + "timeout": 30000 + } + properties: + typing: + type: boolean + description: |- + Whether the user is typing or not. If ``false``, the ``timeout`` + key can be omitted. + timeout: + type: integer + description: The length of time in milliseconds to mark this user as typing. + required: ["typing"] + responses: + 200: + description: The new typing state was set. + examples: + application/json: |- + {} + schema: + type: object # empty json object + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" + diff --git a/event-schemas/examples/v1/m.typing b/event-schemas/examples/v1/m.typing new file mode 100644 index 000000000..bd53f6fb6 --- /dev/null +++ b/event-schemas/examples/v1/m.typing @@ -0,0 +1,7 @@ +{ + "type": "m.typing", + "room_id": "!z0mnsuiwhifuhwwfw:matrix.org", + "content": { + "user_ids": ["@alice:matrix.org", "@bob:example.com"] + } +} \ No newline at end of file diff --git a/event-schemas/schema/v1/m.typing b/event-schemas/schema/v1/m.typing new file mode 100644 index 000000000..b712f6ecb --- /dev/null +++ b/event-schemas/schema/v1/m.typing @@ -0,0 +1,28 @@ +{ + "type": "object", + "title": "Typing Event", + "description": "Informs the client of the list of users currently typing.", + "properties": { + "content": { + "type": "object", + "properties": { + "user_ids": { + "type": "array", + "items": { + "type": "string" + }, + "description": "The list of user IDs typing in this room, if any." + } + }, + "required": ["user_ids"] + }, + "type": { + "type": "string", + "enum": ["m.typing"] + }, + "room_id": { + "type": "string" + } + }, + "required": ["type", "room_id", "content"] +} diff --git a/specification/modules/typing_notifications.rst b/specification/modules/typing_notifications.rst index 25b714ab4..b32e3411e 100644 --- a/specification/modules/typing_notifications.rst +++ b/specification/modules/typing_notifications.rst @@ -1,45 +1,30 @@ Typing Notifications --------------------- +==================== -Client APIs -~~~~~~~~~~~ +Events +------ -To set "I am typing for the next N msec":: +{{m_typing_event}} - PUT .../rooms//typing/ - Content: { "typing": true, "timeout": N } - # timeout is in milliseconds; suggested no more than 20 or 30 seconds +Client behaviour +---------------- + + - suggested no more than 20-30 seconds This should be re-sent by the client to continue informing the server the user is still typing; a safety margin of 5 seconds before the expected timeout runs out is recommended. Just keep declaring a new timeout, it will replace the old one. -To set "I am no longer typing":: - - PUT ../rooms//typing/ - Content: { "typing": false } - -Client Events -~~~~~~~~~~~~~ - -All room members will receive an event on the event stream:: - - { - "type": "m.typing", - "room_id": "!room-id-here:matrix.org", - "content": { - "user_ids": ["list of", "every user", "who is", "currently typing"] - } - } - -The client must use this list to *REPLACE* its knowledge of every user who is +Event: The client must use this list to *REPLACE* its knowledge of every user who is currently typing. The reason for this is that the server DOES NOT remember users who are not currently typing, as that list gets big quickly. The client should mark as not typing, any user ID who is not in that list. -Server APIs -~~~~~~~~~~~ +{{typing_http_api}} + +Server behaviour +---------------- Servers will emit EDUs in the following form:: @@ -59,3 +44,9 @@ originating HSes to ensure they eventually send "stop" notifications. ((This will eventually need addressing, as part of the wider typing/presence timer addition work)) +Security considerations +----------------------- + +Clients may not wish to inform everyone in a room that they are typing and +instead only specific users in the room. + From a82f2ad4ac0b622899fcc039b05d2cd14c26e37c Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 11:55:34 +0100 Subject: [PATCH 31/54] Flesh out typing module --- specification/0-intro.rst | 2 + .../modules/typing_notifications.rst | 39 ++++++++++++------- 2 files changed, 27 insertions(+), 14 deletions(-) diff --git a/specification/0-intro.rst b/specification/0-intro.rst index 93458d97d..5fec59c5c 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -180,6 +180,8 @@ of a "Room". Event Graphs ~~~~~~~~~~~~ +.. _sect:event-graph: + Events exchanged in the context of a room are stored in a directed acyclic graph (DAG) called an ``event graph``. The partial ordering of this graph gives the chronological ordering of events within the room. Each event in the graph has a diff --git a/specification/modules/typing_notifications.rst b/specification/modules/typing_notifications.rst index b32e3411e..110cd1348 100644 --- a/specification/modules/typing_notifications.rst +++ b/specification/modules/typing_notifications.rst @@ -1,6 +1,13 @@ Typing Notifications ==================== +Users often desire to see when another user is typing. This can be achieved +using typing notifications. These are ephemeral events scoped to a ``room_id``. +This means they do not form part of the `Event Graph`_ but still have a +``room_id`` key. + +.. _Event Graph: `sect:event-graph`_ + Events ------ @@ -9,24 +16,28 @@ Events Client behaviour ---------------- - - suggested no more than 20-30 seconds - -This should be re-sent by the client to continue informing the server the user -is still typing; a safety margin of 5 seconds before the expected -timeout runs out is recommended. Just keep declaring a new timeout, it will -replace the old one. - -Event: The client must use this list to *REPLACE* its knowledge of every user who is -currently typing. The reason for this is that the server DOES NOT remember -users who are not currently typing, as that list gets big quickly. The client -should mark as not typing, any user ID who is not in that list. +When a client receives an ``m.typing`` event, it MUST use the user ID list to +**REPLACE** its knowledge of every user who is currently typing. The reason for +this is that the server *does not remember* users who are not currently typing +as that list gets big quickly. The client should mark as not typing any user ID +who is not in that list. + +It is recommended that clients store a ``boolean`` indicating whether the user +is typing or not. Whilst this value is ``true`` a timer should fire periodically +every N seconds to send a typing HTTP request. The value of N is recommended to +be no more than 20-30 seconds. This request should be re-sent by the client to +continue informing the server the user is still typing. As subsequent +requests will replace older requests, a safety margin of 5 seconds before the +expected timeout runs out is recommended. When the user stops typing, the +state change of the ``boolean`` to ``false`` should trigger another HTTP request +to inform the server that the user has stopped typing. {{typing_http_api}} Server behaviour ---------------- -Servers will emit EDUs in the following form:: +Servers MUST emit typing EDUs in the following form:: { "type": "m.typing", @@ -37,8 +48,8 @@ Servers will emit EDUs in the following form:: } } -Server EDUs don't (currently) contain timing information; it is up to -originating HSes to ensure they eventually send "stop" notifications. +This does not contain timing information so it is up to originating homeservers +to ensure they eventually send "stop" notifications. .. TODO ((This will eventually need addressing, as part of the wider typing/presence From 097dc501805891c594dc218d6751fb529d2dd6a0 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 15:45:29 +0100 Subject: [PATCH 32/54] Minor tweaks --- specification/modules/typing_notifications.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/specification/modules/typing_notifications.rst b/specification/modules/typing_notifications.rst index 110cd1348..74bd76bcd 100644 --- a/specification/modules/typing_notifications.rst +++ b/specification/modules/typing_notifications.rst @@ -1,10 +1,10 @@ Typing Notifications ==================== -Users often desire to see when another user is typing. This can be achieved -using typing notifications. These are ephemeral events scoped to a ``room_id``. -This means they do not form part of the `Event Graph`_ but still have a -``room_id`` key. +Users may wish to be informed when another user is typing in a room. This can be +achieved using typing notifications. These are ephemeral events scoped to a +``room_id``. This means they do not form part of the `Event Graph`_ but still +have a ``room_id`` key. .. _Event Graph: `sect:event-graph`_ @@ -37,7 +37,8 @@ to inform the server that the user has stopped typing. Server behaviour ---------------- -Servers MUST emit typing EDUs in the following form:: +Servers MUST emit typing EDUs in a different form to ``m.typing`` events which +are shown to clients. This form looks like:: { "type": "m.typing", From 9964dd1401ac0127a8476b14106274c9d84d0fbd Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 30 Sep 2015 16:11:31 +0100 Subject: [PATCH 33/54] Make explicit the state event only includes some keys --- event-schemas/schema/v1/m.room.member | 1 + 1 file changed, 1 insertion(+) diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member index 32302b85f..c0fb103c4 100644 --- a/event-schemas/schema/v1/m.room.member +++ b/event-schemas/schema/v1/m.room.member @@ -39,6 +39,7 @@ "items": { "type": "object", "title": "StateEvent", + "description": "A stripped down state event, with only the ``type``, ``state_key`` and ``content`` keys.", "properties": { "type": { "type": "string" From 0320e8cef32d59ac84131f445937ddc2c05523a3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 16:41:47 +0100 Subject: [PATCH 34/54] Table tweaks from PR --- specification/0-feature_profiles.rst | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index 59e015e9f..1d88311b1 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -14,19 +14,19 @@ in order for it to be classified as that profile. Summary ------- -============================ ===== =========== ======== ========= ===== ===== - Module / Profile Web Embed-Web Mobile Desktop CLI IoT -============================ ===== =========== ======== ========= ===== ===== - `End-to-End Encryption`_ - `Instant Messaging`_ YES YES YES YES YES YES - `Presence`_ YES YES YES YES - `Push Notifications`_ YES - `Receipts`_ YES YES YES YES - `Typing Notifications`_ YES YES YES YES - `VoIP`_ YES YES YES - `Content Repository`_ YES YES YES YES - `History Visibility`_ YES YES YES YES -============================ ===== =========== ======== ========= ===== ===== +============================ ========== ========== ========== ========== ========== + Module / Profile Web Mobile Desktop CLI Embedded +============================ ========== ========== ========== ========== ========== + `Instant Messaging`_ Required Required Required Required Optional + `Presence`_ Required Required Required Required Optional + `Push Notifications`_ Optional Required Optional Optional Optional + `Receipts`_ Required Required Required Required Optional + `Typing Notifications`_ Required Required Required Required Optional + `VoIP`_ Required Required Required Optional Optional + `Content Repository`_ Required Required Required Optional Optional + `History Visibility`_ Required Required Required Required Optional + `End-to-End Encryption`_ Optional Optional Optional Optional Optional +============================ ========== ========== ========== ========== ========== *Please see each module for more details on what clients need to implement.* From be9402b66fbb49c4c23c57e7aec93d1f1efc03f3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 16:43:09 +0100 Subject: [PATCH 35/54] Move feature profiles section to modules as a sub-section --- specification/0-feature_profiles.rst | 2 +- specification/targets.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index 1d88311b1..0580c18a6 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -25,7 +25,7 @@ Summary `VoIP`_ Required Required Required Optional Optional `Content Repository`_ Required Required Required Optional Optional `History Visibility`_ Required Required Required Required Optional - `End-to-End Encryption`_ Optional Optional Optional Optional Optional + `End-to-End Encryption`_ Optional Optional Optional Optional Optional ============================ ========== ========== ========== ========== ========== *Please see each module for more details on what clients need to implement.* diff --git a/specification/targets.yaml b/specification/targets.yaml index 62585c698..d77bf8b59 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -2,11 +2,11 @@ targets: main: # arbitrary name to identify this build target files: # the sort order of files to cat - 0-intro.rst - - { 1: 0-feature_profiles.rst } - 1-client_server_api.rst - { 1: 0-events.rst } - { 1: 0-event_signing.rst } - 2-modules.rst + - { 1: 0-feature_profiles.rst } - { 1: "group:modules" } # reference a group of files - 3-application_service_api.rst - 4-server_server_api.rst From 91b6347f74c4e01ab62b56ee784b735c97d60df0 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 16:48:47 +0100 Subject: [PATCH 36/54] Explain what 'embedded' clients are --- specification/0-feature_profiles.rst | 30 ++++++++++++++++++---------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index 0580c18a6..5a7e5513c 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -50,14 +50,6 @@ This is a web page which heavily uses Matrix for communication. Single-page web apps would be classified as a stand-alone web client, as would multi-page web apps which use Matrix on nearly every page. -Embedded web (``EmbedWeb``) -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This is a Matrix client which is embedded in another website, e.g. using -iframes. These embedded clients are typically for a single purpose -related to the website in question, and are not intended to be fully-fledged -communication apps. - Mobile (``Mobile``) ~~~~~~~~~~~~~~~~~~~ @@ -77,9 +69,25 @@ Command Line Interface (``CLI``) This is a client which is used via a text-based terminal. -Internet of Things (``IoT``) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Embedded (``Embedded``) +~~~~~~~~~~~~~~~~~~~~~~~ + +This is a client which is embedded into another application or an embedded +device. + +Application ++++++++++++ + +This is a Matrix client which is embedded in another website, e.g. using +iframes. These embedded clients are typically for a single purpose +related to the website in question, and are not intended to be fully-fledged +communication apps. + +Device +++++++ This is a client which is typically running on an embedded device such as a -kettle, fridge or car. +kettle, fridge or car. These clients tend to perform a few operations and run +in a resource constrained environment. Like embedded applications, they are +not intended to be fully-fledged communication systems. From 6c3e70d2721554827d14d27ff07e892976ffcd24 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 30 Sep 2015 17:32:44 +0100 Subject: [PATCH 37/54] Start fleshing out voip module --- specification/1-client_server_api.rst | 2 + specification/modules/voip_events.rst | 84 ++++++++++++++++++--------- 2 files changed, 59 insertions(+), 27 deletions(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 7d8d57e40..db88bbd6b 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -427,6 +427,8 @@ the complete dataset is provided in "chunk". Events ------ +.. _sect:events: + Overview ~~~~~~~~ diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst index 33998cd95..f1b8ae058 100644 --- a/specification/modules/voip_events.rst +++ b/specification/modules/voip_events.rst @@ -1,17 +1,18 @@ Voice over IP -------------- -Matrix can also be used to set up VoIP calls. This is part of the core -specification, although is at a relatively early stage. Voice (and video) over -Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like -any other event. This means that clients must only send call events to rooms -with exactly two participants as currently the WebRTC standard is based around -two-party communication. +============= -{{voip_events}} +This module outlines how two users in a room can set up a Voice over IP (VoIP) +call to each other. Voice and video calls are built upon the WebRTC 1.0 standard. +Call signalling is achieved by sending `message events`_ to the room. As a result, +this means that clients MUST only send call events to rooms with exactly two +participants as currently the WebRTC standard is based around two-party +communication. + +.. _message events: `sect:events`_ Message Exchange -~~~~~~~~~~~~~~~~ -A call is set up with messages exchanged as follows: +---------------- +A call is set up with message events exchanged as follows: :: @@ -38,28 +39,57 @@ Or a rejected call: Calls are negotiated according to the WebRTC specification. +Events +------ + +{{voip_events}} + +Client behaviour +---------------- Glare ~~~~~ -This specification aims to address the problem of two users calling each other -at roughly the same time and their invites crossing on the wire. It is a far -better experience for the users if their calls are connected if it is clear -that their intention is to set up a call with one another. In Matrix, calls are -to rooms rather than users (even if those rooms may only contain one other user) -so we consider calls which are to the same room. The rules for dealing with such -a situation are as follows: - - - If an invite to a room is received whilst the client is preparing to send an - invite to the same room, the client should cancel its outgoing call and - instead automatically accept the incoming call on behalf of the user. - - If an invite to a room is received after the client has sent an invite to - the same room and is waiting for a response, the client should perform a - lexicographical comparison of the call IDs of the two calls and use the - lesser of the two calls, aborting the greater. If the incoming call is the - lesser, the client should accept this call on behalf of the user. + +"Glare" is a problem which occurs when two users call each other at roughly the +same time. This results in the call failing to set up as there already is an +incoming/outgoing call. A glare resolution algorithm can be used to determine +which call to hangup and which call to answer. If both clients implement the +same algorithm then they will both select the same call and the call will be +successfully connected. + + +As calls are "placed" to rooms rather than users, the glare resolution algorithm +outlined below is only considered for calls which are to the same room. The +algorithm is as follows: + + - If an invite to a room is received whilst the client is **preparing to send** + an invite to the same room: + + * the client should cancel its outgoing call and instead + automatically accept the incoming call on behalf of the user. + + - If an invite to a room is received **after the client has sent** an invite to + the same room and is waiting for a response: + + * the client should perform a lexicographical comparison of the call IDs of + the two calls and use the **lesser** of the two calls, aborting the + greater. If the incoming call is the lesser, the client should accept + this call on behalf of the user. + The call setup should appear seamless to the user as if they had simply placed -a call and the other party had accepted. Thusly, any media stream that had been +a call and the other party had accepted. This means any media stream that had been setup for use on a call should be transferred and used for the call that replaces it. +Server behaviour +---------------- + +TURN Servers +~~~~~~~~~~~~ + +Security considerations +----------------------- + + + From d092b22848e4ec63aee7c157e2bc2203d6d48d87 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 1 Oct 2015 09:23:08 +0100 Subject: [PATCH 38/54] Rename to 'Managing history visibility' --- specification/0-feature_profiles.rst | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst index 5a7e5513c..b9f12b74b 100644 --- a/specification/0-feature_profiles.rst +++ b/specification/0-feature_profiles.rst @@ -14,19 +14,19 @@ in order for it to be classified as that profile. Summary ------- -============================ ========== ========== ========== ========== ========== - Module / Profile Web Mobile Desktop CLI Embedded -============================ ========== ========== ========== ========== ========== - `Instant Messaging`_ Required Required Required Required Optional - `Presence`_ Required Required Required Required Optional - `Push Notifications`_ Optional Required Optional Optional Optional - `Receipts`_ Required Required Required Required Optional - `Typing Notifications`_ Required Required Required Required Optional - `VoIP`_ Required Required Required Optional Optional - `Content Repository`_ Required Required Required Optional Optional - `History Visibility`_ Required Required Required Required Optional - `End-to-End Encryption`_ Optional Optional Optional Optional Optional -============================ ========== ========== ========== ========== ========== +===================================== ========== ========== ========== ========== ========== + Module / Profile Web Mobile Desktop CLI Embedded +===================================== ========== ========== ========== ========== ========== + `Instant Messaging`_ Required Required Required Required Optional + `Presence`_ Required Required Required Required Optional + `Push Notifications`_ Optional Required Optional Optional Optional + `Receipts`_ Required Required Required Required Optional + `Typing Notifications`_ Required Required Required Required Optional + `VoIP`_ Required Required Required Optional Optional + `Content Repository`_ Required Required Required Optional Optional + `Managing History Visibility`_ Required Required Required Required Optional + `End-to-End Encryption`_ Optional Optional Optional Optional Optional +===================================== ========== ========== ========== ========== ========== *Please see each module for more details on what clients need to implement.* @@ -38,7 +38,7 @@ Summary .. _Typing Notifications: `module:typing`_ .. _VoIP: `module:voip`_ .. _Content Repository: `module:content`_ -.. _History Visibility: `module:history-visibility`_ +.. _Managing History Visibility: `module:history-visibility`_ Clients ------- From e82661413e3bf51dd248778b0f3dd9a83b29efb9 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 1 Oct 2015 11:04:42 +0100 Subject: [PATCH 39/54] Add /turnServer endpoint --- api/client-server/v1/voip.yaml | 68 +++++++++++++++++++++++++++ specification/modules/voip_events.rst | 30 +++++++----- 2 files changed, 85 insertions(+), 13 deletions(-) create mode 100644 api/client-server/v1/voip.yaml diff --git a/api/client-server/v1/voip.yaml b/api/client-server/v1/voip.yaml new file mode 100644 index 000000000..5fdf1ca7a --- /dev/null +++ b/api/client-server/v1/voip.yaml @@ -0,0 +1,68 @@ +swagger: '2.0' +info: + title: "Matrix Client-Server v1 Voice over IP API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/api/v1 +consumes: + - application/json +produces: + - application/json +securityDefinitions: + accessToken: + type: apiKey + description: The user_id or application service access_token + name: access_token + in: query +paths: + "/turnServer": + get: + summary: Obtain TURN server credentials. + description: |- + This API provides credentials for the client to use when initiating + calls. + security: + - accessToken: [] + responses: + 200: + description: The TURN server credentials. + examples: + application/json: |- + { + "username":"1443779631:@user:example.com", + "password":"JlKfBy1QwLrO20385QyAtEyIv0=", + "uris":[ + "turn:turn.example.com:3478?transport=udp", + "turn:10.20.30.40:3478?transport=tcp", + "turns:10.20.30.40:443?transport=tcp" + ], + "ttl":86400 + } + schema: + type: object + properties: + username: + type: string + description: |- + The username to use. + password: + type: string + description: |- + The password to use. + uris: + type: array + items: + type: string + description: A list of TURN URIs + ttl: + type: integer + description: The time-to-live in seconds + required: ["username", "password", "uris", "ttl"] + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" + diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst index f1b8ae058..1bf1ea1d7 100644 --- a/specification/modules/voip_events.rst +++ b/specification/modules/voip_events.rst @@ -10,8 +10,14 @@ communication. .. _message events: `sect:events`_ -Message Exchange +Events +------ + +{{voip_events}} + +Client behaviour ---------------- + A call is set up with message events exchanged as follows: :: @@ -39,14 +45,6 @@ Or a rejected call: Calls are negotiated according to the WebRTC specification. -Events ------- - -{{voip_events}} - -Client behaviour ----------------- - Glare ~~~~~ @@ -72,7 +70,7 @@ algorithm is as follows: the same room and is waiting for a response: * the client should perform a lexicographical comparison of the call IDs of - the two calls and use the **lesser** of the two calls, aborting the + the two calls and use the *lesser* of the two calls, aborting the greater. If the incoming call is the lesser, the client should accept this call on behalf of the user. @@ -85,11 +83,17 @@ replaces it. Server behaviour ---------------- -TURN Servers -~~~~~~~~~~~~ +The server MAY provide a TURN server which clients can use to contact the +remote party. This server should be accessible via the HTTP endpoint listed +below. + +{{voip_http_api}} + Security considerations ----------------------- - +Calls should only be placed to rooms with one other user in them. If they are +placed to group chat rooms it is possible that another user will intercept and +answer the call. From 3b73b07babbcd86dbb8318f4813b27322a689848 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 1 Oct 2015 11:11:08 +0100 Subject: [PATCH 40/54] Clarifications that room invites are m.call.invites not actual invites --- specification/modules/voip_events.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst index 1bf1ea1d7..9d27c23b5 100644 --- a/specification/modules/voip_events.rst +++ b/specification/modules/voip_events.rst @@ -60,14 +60,14 @@ As calls are "placed" to rooms rather than users, the glare resolution algorithm outlined below is only considered for calls which are to the same room. The algorithm is as follows: - - If an invite to a room is received whilst the client is **preparing to send** - an invite to the same room: + - If an ``m.call.invite`` to a room is received whilst the client is + **preparing to send** an ``m.call.invite`` to the same room: * the client should cancel its outgoing call and instead automatically accept the incoming call on behalf of the user. - - If an invite to a room is received **after the client has sent** an invite to - the same room and is waiting for a response: + - If an ``m.call.invite`` to a room is received **after the client has sent** + an ``m.call.invite`` to the same room and is waiting for a response: * the client should perform a lexicographical comparison of the call IDs of the two calls and use the *lesser* of the two calls, aborting the @@ -83,9 +83,9 @@ replaces it. Server behaviour ---------------- -The server MAY provide a TURN server which clients can use to contact the -remote party. This server should be accessible via the HTTP endpoint listed -below. +The homeserver MAY provide a TURN server which clients can use to contact the +remote party. The following HTTP API endpoints will be used by clients in order +to get information about the TURN server. {{voip_http_api}} From 365a9076b93500cd87e85e68129e6ba7729f7625 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 1 Oct 2015 12:11:26 +0100 Subject: [PATCH 41/54] Add nested dict template support; Add x-pattern For cases where event schema specify `patternProperties` it would be nice to give that pattern a "human-readable" form rather than a raw regex. This is now supported by specifying `x-pattern` in the value part of the specified pattern e.g. `patternProperties:{ "^.*":{ x-pattern: "$THING", ... } }` Templating had limited record type descriptions limited to value primitives e.g. `{string: integer}`. It now supports inspecting the values recursively if the value is `object`. Updated `m.receipt` to take both these points into account to make it read better. Tweak receipt module text. --- event-schemas/schema/v1/m.receipt | 9 ++++--- specification/modules/receipts.rst | 37 ++++++++++++---------------- templating/matrix_templates/units.py | 26 +++++++++++++++---- 3 files changed, 43 insertions(+), 29 deletions(-) diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt index 0f365eed5..5be232ad4 100644 --- a/event-schemas/schema/v1/m.receipt +++ b/event-schemas/schema/v1/m.receipt @@ -5,17 +5,20 @@ "properties": { "content": { "type": "object", - "description": "The event ids which the receipts relate to.", "patternProperties": { "^\\$": { "type": "object", - "description": "The types of the receipts.", + "x-pattern": "$EVENT_ID", + "description": "The mapping of event ID to receipt type. The event ID is the ID which the receipts relate to and *not* an ID for the receipt itself. The key in the object is an enum which must be ``read``.", "additionalProperties": { "type": "object", - "description": "User ids of the receipts", + "title": "Users", "patternProperties": { "^@": { "type": "object", + "title": "Receipt", + "description": "The mapping of user ID to receipt. The user ID is the entity who sent this receipt.", + "x-pattern": "$USER_ID", "properties": { "ts": { "type": "number", diff --git a/specification/modules/receipts.rst b/specification/modules/receipts.rst index e2f83eea8..f32c12dbf 100644 --- a/specification/modules/receipts.rst +++ b/specification/modules/receipts.rst @@ -6,8 +6,16 @@ have interacted with. For example, which events the user has read. For efficiency this is done as "up to" markers, i.e. marking a particular event as, say, ``read`` indicates the user has read all events *up to* that event. -Client-Server API -~~~~~~~~~~~~~~~~~ +Events +------ + +{{m_receipt_event}} + +Client behaviour +---------------- + + - When clients should send receipts + - What clients should do when they receive these receipts Clients will receive receipts in the following format:: @@ -25,22 +33,6 @@ Clients will receive receipts in the following format:: } } -For example:: - - { - "type": "m.receipt", - "room_id": "!KpjVgQyZpzBwvMBsnT:matrix.org", - "content": { - "$1435641916114394fHBLK:matrix.org": { - "read": { - "@erikj:jki.re": { "ts": 1436451550453 }, - ... - } - }, - ... - } - } - For efficiency, receipts are batched into one event per room. In the initialSync and v2 sync APIs the receipts are listed in a separate top level ``receipts`` key. Each ``user_id``, ``receipt_type`` pair must be associated with only a @@ -56,9 +48,8 @@ A client can update the markers for its user by issuing a request:: Where the contents of the ``POST`` will be included in the content sent to other users. The server will automatically set the ``ts`` field. - -Server-Server API -~~~~~~~~~~~~~~~~~ +Server behaviour +---------------- Receipts are sent across federation as EDUs with type ``m.receipt``. The format of the EDUs are:: @@ -75,3 +66,7 @@ format of the EDUs are:: These are always sent as deltas to previously sent receipts. +Security considerations +----------------------- + + diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py index 50fa784e6..473fdd820 100644 --- a/templating/matrix_templates/units.py +++ b/templating/matrix_templates/units.py @@ -49,8 +49,17 @@ def get_json_schema_object_fields(obj, enforce_title=False): } tables = [fields] - props = obj.get("properties", obj.get("patternProperties")) parents = obj.get("allOf") + props = obj.get("properties") + if not props: + props = obj.get("patternProperties") + if props: + # try to replace horrible regex key names with pretty x-pattern ones + for key_name in props.keys(): + pretty_key = props[key_name].get("x-pattern") + if pretty_key: + props[pretty_key] = props[key_name] + del props[key_name] if not props and not parents: raise Exception( "Object %s has no properties or parents." % obj @@ -70,10 +79,17 @@ def get_json_schema_object_fields(obj, enforce_title=False): if props[key_name]["type"] == "object": if props[key_name].get("additionalProperties"): # not "really" an object, just a KV store - value_type = ( - "{string: %s}" % - props[key_name]["additionalProperties"]["type"] - ) + prop_val = props[key_name]["additionalProperties"]["type"] + if prop_val == "object": + nested_object = get_json_schema_object_fields( + props[key_name]["additionalProperties"], + enforce_title=True + ) + value_type = "{string: %s}" % nested_object[0]["title"] + if not nested_object[0].get("no-table"): + tables += nested_object + else: + value_type = "{string: %s}" % prop_val else: nested_object = get_json_schema_object_fields( props[key_name], From c972dad8b383a8345864d32ba6997b5da1fd57fa Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 1 Oct 2015 15:41:54 +0100 Subject: [PATCH 42/54] Flesh out receipts module. Add receipts swagger Add templating support for v2 apis. --- api/client-server/v2_alpha/receipts.yaml | 68 +++++++++++++++++++++ event-schemas/examples/v1/m.receipt | 2 +- event-schemas/schema/v1/m.receipt | 35 ++++++----- specification/modules/receipts.rst | 76 ++++++++++++++---------- templating/matrix_templates/units.py | 30 ++++++---- 5 files changed, 151 insertions(+), 60 deletions(-) create mode 100644 api/client-server/v2_alpha/receipts.yaml diff --git a/api/client-server/v2_alpha/receipts.yaml b/api/client-server/v2_alpha/receipts.yaml new file mode 100644 index 000000000..4ef435b02 --- /dev/null +++ b/api/client-server/v2_alpha/receipts.yaml @@ -0,0 +1,68 @@ +swagger: '2.0' +info: + title: "Matrix Client-Server v2 Receipts API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/v2_alpha +consumes: + - application/json +produces: + - application/json +securityDefinitions: + accessToken: + type: apiKey + description: The user_id or application service access_token + name: access_token + in: query +paths: + "/rooms/{roomId}/receipt/{receiptType}/{eventId}": + post: + summary: Send a receipt for the given event ID. + description: |- + This API updates the marker for the given receipt type to the event ID + specified. + security: + - accessToken: [] + parameters: + - in: path + type: string + name: roomId + description: The room in which to send the event. + required: true + x-example: "!wefuh21ffskfuh345:example.com" + - in: path + type: string + name: receiptType + description: The type of receipt to send. + required: true + x-example: "m.read" + enum: ["m.read"] + - in: path + type: string + name: eventId + description: The event ID to acknowledge up to. + required: true + x-example: "$1924376522eioj:example.com" + - in: body + description: |- + Extra receipt information to attach to ``content`` if any. The + server will automatically set the ``ts`` field. + schema: + type: object + example: |- + {} + responses: + 200: + description: The receipt was sent. + examples: + application/json: |- + {} + schema: + type: object # empty json object + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" \ No newline at end of file diff --git a/event-schemas/examples/v1/m.receipt b/event-schemas/examples/v1/m.receipt index 83515317a..bd0b726c3 100644 --- a/event-schemas/examples/v1/m.receipt +++ b/event-schemas/examples/v1/m.receipt @@ -3,7 +3,7 @@ "room_id": "!KpjVgQyZpzBwvMBsnT:matrix.org", "content": { "$1435641916114394fHBLK:matrix.org": { - "read": { + "m.read": { "@rikj:jki.re": { "ts": 1436451550453 } diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt index 5be232ad4..d0f79ac4a 100644 --- a/event-schemas/schema/v1/m.receipt +++ b/event-schemas/schema/v1/m.receipt @@ -9,25 +9,28 @@ "^\\$": { "type": "object", "x-pattern": "$EVENT_ID", - "description": "The mapping of event ID to receipt type. The event ID is the ID which the receipts relate to and *not* an ID for the receipt itself. The key in the object is an enum which must be ``read``.", - "additionalProperties": { - "type": "object", - "title": "Users", - "patternProperties": { - "^@": { - "type": "object", - "title": "Receipt", - "description": "The mapping of user ID to receipt. The user ID is the entity who sent this receipt.", - "x-pattern": "$USER_ID", - "properties": { - "ts": { - "type": "number", - "description": "The timestamp the receipt was sent at" + "title": "Receipts", + "description": "The mapping of event ID to a collection of receipts for this event ID. The event ID is the ID of the event being acknowledged and *not* an ID for the receipt itself.", + "properties": { + "m.read": { + "type": "object", + "title": "Users", + "description": "A collection of users who have sent ``m.read`` receipts for this event.", + "patternProperties": { + "^@": { + "type": "object", + "title": "Receipt", + "description": "The mapping of user ID to receipt. The user ID is the entity who sent this receipt.", + "x-pattern": "$USER_ID", + "properties": { + "ts": { + "type": "number", + "description": "The timestamp the receipt was sent at." + } } } } - }, - "additionalProperties": false + } } } }, diff --git a/specification/modules/receipts.rst b/specification/modules/receipts.rst index f32c12dbf..9dabba30e 100644 --- a/specification/modules/receipts.rst +++ b/specification/modules/receipts.rst @@ -1,56 +1,63 @@ Receipts --------- +======== -Receipts are used to publish which events in a room the user or their devices -have interacted with. For example, which events the user has read. For -efficiency this is done as "up to" markers, i.e. marking a particular event -as, say, ``read`` indicates the user has read all events *up to* that event. +This module adds in support for receipts. These receipts are a form of +acknowledgement of an event. This module defines a single acknowledgement: +``m.read`` which indicates that the user has read up to a given event. + +Sending a receipt for each event can result in sending large amounts of traffic +to a homeserver. To prevent this from becoming a problem, receipts are implemented +using "up to" markers. This marker indicates that the acknowledgement applies +to all events "up to and including" the event specified. For example, marking +an event as "read" would indicate that the user had read all events *up to* the +referenced event. Events ------ +Each ``user_id``, ``receipt_type`` pair must be associated with only a +single ``event_id``. {{m_receipt_event}} Client behaviour ---------------- - - When clients should send receipts - - What clients should do when they receive these receipts - -Clients will receive receipts in the following format:: +In v1 ``/initialSync``, receipts are listed in a separate top level ``receipts`` +key. In v2 ``/sync``, receipts are contained in the ``ephemeral`` block for a +room. New receipts that come down the event streams are deltas which update +existing mappings. Clients should replace older receipt acknowledgements based +on ``user_id`` and ``receipt_type`` pairs. For example:: - { - "type": "m.receipt", - "room_id": , - "content": { - : { - : { - : { "ts": , ... }, - ... - } - }, - ... - } - } + Client receives m.receipt: + user = @alice:example.com + receipt_type = m.read + event_id = $aaa:example.com -For efficiency, receipts are batched into one event per room. In the initialSync -and v2 sync APIs the receipts are listed in a separate top level ``receipts`` -key. Each ``user_id``, ``receipt_type`` pair must be associated with only a -single ``event_id``. New receipts that come down the event streams are deltas. -Deltas update existing mappings, clobbering based on ``user_id``, -``receipt_type`` pairs. + Client receives another m.receipt: + user = @alice:example.com + receipt_type = m.read + event_id = $bbb:example.com + The client should replace the older acknowledgement for $aaa:example.com with + this one for $bbb:example.com -A client can update the markers for its user by issuing a request:: +Clients should send read receipts when there is some certainty that the event in +question has been **displayed** to the user. Simply receiving an event does not +provide enough certainty that the user has seen the event. The user SHOULD need +to *take some action* such as viewing the room that the event was sent to or +dismissing a notification in order for the event to count as "read". - POST /_matrix/client/v2_alpha/rooms//receipt/read/ +A client can update the markers for its user by interacting with the following +HTTP APIs. -Where the contents of the ``POST`` will be included in the content sent to -other users. The server will automatically set the ``ts`` field. +{{v2_receipts_http_api}} Server behaviour ---------------- +For efficiency, receipts SHOULD be batched into one event per room before +delivering them to clients. + Receipts are sent across federation as EDUs with type ``m.receipt``. The format of the EDUs are:: @@ -64,9 +71,12 @@ format of the EDUs are:: ... } -These are always sent as deltas to previously sent receipts. +These are always sent as deltas to previously sent receipts. Currently only a +single ```` should be used: ``m.read``. Security considerations ----------------------- +As receipts are sent outside the context of the event graph, there are no +integrity checks performed on the contents of ``m.receipt`` events. diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py index 473fdd820..eca52acb1 100644 --- a/templating/matrix_templates/units.py +++ b/templating/matrix_templates/units.py @@ -19,6 +19,7 @@ import yaml V1_CLIENT_API = "../api/client-server/v1" V1_EVENT_EXAMPLES = "../event-schemas/examples/v1" V1_EVENT_SCHEMA = "../event-schemas/schema/v1" +V2_CLIENT_API = "../api/client-server/v2_alpha" CORE_EVENT_SCHEMA = "../event-schemas/schema/v1/core-event-schema" CHANGELOG = "../CHANGELOG.rst" TARGETS = "../specification/targets.yaml" @@ -336,18 +337,27 @@ class MatrixUnits(Units): } def load_swagger_apis(self): - path = V1_CLIENT_API + paths = [ + V1_CLIENT_API, V2_CLIENT_API + ] apis = {} - for filename in os.listdir(path): - if not filename.endswith(".yaml"): + for path in paths: + is_v2 = (path == V2_CLIENT_API) + if not os.path.exists(V2_CLIENT_API): + self.log("Skipping v2 apis: %s does not exist.", V2_CLIENT_API) continue - self.log("Reading swagger API: %s" % filename) - with open(os.path.join(path, filename), "r") as f: - # strip .yaml - group_name = filename[:-5] - api = yaml.load(f.read()) - api["__meta"] = self._load_swagger_meta(api, group_name) - apis[group_name] = api + for filename in os.listdir(path): + if not filename.endswith(".yaml"): + continue + self.log("Reading swagger API: %s" % filename) + with open(os.path.join(path, filename), "r") as f: + # strip .yaml + group_name = filename[:-5] + if is_v2: + group_name = "v2_" + group_name + api = yaml.load(f.read()) + api["__meta"] = self._load_swagger_meta(api, group_name) + apis[group_name] = api return apis def load_common_event_fields(self): From 560cd7a58fe6ecdb713dd3cd30b58b3e98a04d3d Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 1 Oct 2015 15:54:45 +0100 Subject: [PATCH 43/54] This isn't javascript. s/,/%/ --- templating/matrix_templates/units.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py index eca52acb1..71b6acc6d 100644 --- a/templating/matrix_templates/units.py +++ b/templating/matrix_templates/units.py @@ -344,7 +344,7 @@ class MatrixUnits(Units): for path in paths: is_v2 = (path == V2_CLIENT_API) if not os.path.exists(V2_CLIENT_API): - self.log("Skipping v2 apis: %s does not exist.", V2_CLIENT_API) + self.log("Skipping v2 apis: %s does not exist." % V2_CLIENT_API) continue for filename in os.listdir(path): if not filename.endswith(".yaml"): From 87b6dd845e0e6c685800bbc13f9459a7d21ad59e Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 1 Oct 2015 17:55:16 +0100 Subject: [PATCH 44/54] Flesh out content repo; modify templating to support headers Edit content-repo.yaml to include examples and headers. Restructure content module to conform to the module template. Adjust the HTTP API template to give 1 more char to the response param to fit "Content-Disposition" correctly. Edit the templating system to support displaying enums for swagger APIs (before it was just JSON schema). Also add support for introspecting headers from swagger. Finally, replace - with _ when forming the {{ template_var }} else things whine. --- api/client-server/v1/content-repo.yaml | 49 +++++++++++++++++-- specification/modules/content_repo.rst | 30 ++++++++++-- .../matrix_templates/templates/http-api.tmpl | 12 ++--- templating/matrix_templates/units.py | 22 +++++++-- 4 files changed, 96 insertions(+), 17 deletions(-) diff --git a/api/client-server/v1/content-repo.yaml b/api/client-server/v1/content-repo.yaml index fe3d1dc37..aa47dc184 100644 --- a/api/client-server/v1/content-repo.yaml +++ b/api/client-server/v1/content-repo.yaml @@ -15,16 +15,22 @@ paths: summary: Upload some content to the content repository. produces: ["application/json"] parameters: + - in: header + name: Content-Type + type: string + description: The content type of the file being uploaded + x-example: "Content-Type: audio/mpeg" - in: body - name: content + name: "" description: The content to be uploaded. required: true schema: type: string + example: "" format: byte responses: 200: - description: Information about the uploaded content. + description: The MXC URI for the uploaded content. schema: type: object required: ["content_uri"] @@ -32,6 +38,11 @@ paths: content_uri: type: string description: "The MXC URI to the uploaded content." + examples: + "application/json": |- + { + "content_uri": "mxc://example.com/AQwafuaFswefuhsfAFAgsw" + } "/download/{serverName}/{mediaId}": get: summary: "Download content from the content repository." @@ -40,20 +51,32 @@ paths: - in: path type: string name: serverName + x-example: matrix.org required: true description: | The server name from the ``mxc://`` URI (the authoritory component) - in: path type: string name: mediaId + x-example: ascERGshawAWawugaAcauga required: true description: | The media ID from the ``mxc://`` URI (the path component) responses: 200: - description: "The content downloaded." + description: "The content that was previously uploaded." + headers: + Content-Type: + description: "The content type of the file that was previously uploaded." + x-example: "audio/mpeg" + type: "string" + Content-Disposition: + description: "The name of the file that was previously uploaded, if set." + x-example: "attachment;filename=03-cool.mp3" + type: "string" schema: type: file + name: "" "/thumbnail/{serverName}/{mediaId}": get: summary: "Download a thumbnail of the content from the content repository." @@ -63,31 +86,47 @@ paths: type: string name: serverName required: true + x-example: matrix.org description: | The server name from the ``mxc://`` URI (the authoritory component) - in: path type: string name: mediaId + x-example: ascERGshawAWawugaAcauga required: true description: | The media ID from the ``mxc://`` URI (the path component) - in: query type: integer + x-example: 64 name: width - description: The desired width of the thumbnail. + description: |- + The *desired* width of the thumbnail. The actual thumbnail may not + match the size specified. - in: query type: integer + x-example: 64 name: height - description: The desired height of the thumbnail. + description: |- + The *desired* height of the thumbnail. The actual thumbnail may not + match the size specified. - in: query type: string enum: ["crop", "scale"] name: method + x-example: "scale" description: The desired resizing method. responses: 200: description: "A thumbnail of the requested content." + headers: + Content-Type: + description: "The content type of the thumbnail." + x-example: "image/jpeg" + type: "string" + enum: ["image/jpeg", "image/png"] schema: type: file + name: "" diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst index 83333f379..de464e203 100644 --- a/specification/modules/content_repo.rst +++ b/specification/modules/content_repo.rst @@ -3,8 +3,23 @@ Content repository .. _module:content: -HTTP API --------- +This module allows users to upload content to their homeserver which is +retrievable from other homeservers. Its' purpose is to allow users to share +attachments in a room. Content locations are represented as Matrix Content (MXC) +URIs. They look like:: + + mxc:/// + + : The name of the homeserver where this content can be found, e.g. matrix.org + : An opaque ID which identifies the content. + +Client behaviour +---------------- + +Clients can upload and download content using the following HTTP APIs. + +{{content_repo_http_api}} + Uploads are POSTed to a resource which returns a token which is used to GET the download. Uploads are POSTed to the sender's local homeserver, but are @@ -49,6 +64,9 @@ width and height are close to the requested size and the aspect matches the requested size. The client should scale the image if it needs to fit within a given rectangle. +Server behaviour +---------------- + Homeservers may generate thumbnails for content uploaded to remote homeservers themselves or may rely on the remote homeserver to thumbnail the content. Homeservers may return thumbnails of a different size to that @@ -58,13 +76,19 @@ Homeservers must never upscale images. Security considerations ----------------------- +The HTTP GET endpoint does not require any authentication. Knowing the URL of +the content is sufficient to retrieve the content, even if the entity isn't in +the room. + +Homeservers have additional concerns: + - Clients may try to upload very large files. Homeservers should not store files that are too large and should not serve them to clients. - Clients may try to upload very large images. Homeservers should not attempt to generate thumbnails for images that are too large. - - Remote homeservers may host very large files or images. Homeserver should not + - Remote homeservers may host very large files or images. Homeservers should not proxy or thumbnail large files or images from remote homeservers. - Clients may try to upload a large number of files. Homeservers should limit the diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl index eb3f3e64e..472c9d7ae 100644 --- a/templating/matrix_templates/templates/http-api.tmpl +++ b/templating/matrix_templates/templates/http-api.tmpl @@ -31,18 +31,18 @@ Response format: {% for table in endpoint.res_tables -%} {{"``"+table.title+"``" if table.title else "" }} -================== ================= =========================================== +=================== ================= ========================================== Param Type Description -================== ================= =========================================== +=================== ================= ========================================== {% for row in table.rows -%} {# -#} -{# Row type needs to prepend spaces to line up with the type column (19 ch) -#} +{# Row type needs to prepend spaces to line up with the type column (20 ch) -#} {# Desc needs to prepend the required text (maybe) and prepend spaces too -#} -{# It also needs to then wrap inside the desc col (43 ch width) -#} +{# It also needs to then wrap inside the desc col (42 ch width) -#} {# -#} -{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(37)}} +{{row.key}}{{row.type|indent(20-row.key|length)}}{{row.desc|wrap(42,row.req_str | indent(18 - (row.type|length))) |indent_block(38)}} {% endfor -%} -================== ================= =========================================== +=================== ================= ========================================== {% endfor %} {% endif -%} diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py index 50fa784e6..04f3bae10 100644 --- a/templating/matrix_templates/units.py +++ b/templating/matrix_templates/units.py @@ -151,6 +151,13 @@ class MatrixUnits(Units): # assign value expected for this param val_type = param.get("type") # integer/string + + if param.get("enum"): + val_type = "enum" + desc += ( + " One of: %s" % json.dumps(param.get("enum")) + ) + refType = Units.prop(param, "schema/$ref/") # Error,Event schemaFmt = Units.prop(param, "schema/format") # bytes e.g. uploads if not val_type and refType: @@ -255,7 +262,7 @@ class MatrixUnits(Units): res_type = Units.prop(good_response, "schema/type") if res_type and res_type not in ["object", "array"]: # response is a raw string or something like that - endpoint["res_tables"].append({ + good_table = { "title": None, "rows": [{ "key": good_response["schema"].get("name", ""), @@ -263,7 +270,16 @@ class MatrixUnits(Units): "desc": res.get("description", ""), "req_str": "" }] - }) + } + if good_response.get("headers"): + for (header_name, header) in good_response.get("headers").iteritems(): + good_table["rows"].append({ + "key": header_name, + "type": "Header<" + header["type"] + ">", + "desc": header["description"], + "req_str": "" + }) + endpoint["res_tables"].append(good_table) elif res_type and Units.prop(good_response, "schema/properties"): # response is an object: schema = good_response["schema"] @@ -328,7 +344,7 @@ class MatrixUnits(Units): self.log("Reading swagger API: %s" % filename) with open(os.path.join(path, filename), "r") as f: # strip .yaml - group_name = filename[:-5] + group_name = filename[:-5].replace("-", "_") api = yaml.load(f.read()) api["__meta"] = self._load_swagger_meta(api, group_name) apis[group_name] = api From 8c4d7f50510edb2a8e0932248efed1949bf1715a Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 1 Oct 2015 18:03:34 +0100 Subject: [PATCH 45/54] Do not try to parse non-json request examples as json --- api/check_examples.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/check_examples.py b/api/check_examples.py index a0cd0658b..f08b2dc1a 100755 --- a/api/check_examples.py +++ b/api/check_examples.py @@ -34,7 +34,7 @@ def check_parameter(filepath, request, parameter): example = None try: example_json = schema.get('example') - if example_json: + if example_json and not schema.get("format") == "byte": example = json.loads(example_json) except Exception as e: raise ValueError("Error parsing JSON example request for %r" % ( From 30232f20aa571cbd402ff56eaa970bbc0fc8d34c Mon Sep 17 00:00:00 2001 From: Daniel Wagner-Hall Date: Thu, 1 Oct 2015 19:13:09 -0500 Subject: [PATCH 46/54] speculator: Merge after fetching, so that /spec/head works --- scripts/speculator/main.go | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go index 6ee80150d..442400dfe 100644 --- a/scripts/speculator/main.go +++ b/scripts/speculator/main.go @@ -60,11 +60,10 @@ func (u *User) IsTrusted() bool { } const ( - pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls" + pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls" matrixDocCloneURL = "https://github.com/matrix-org/matrix-doc.git" ) - func gitClone(url string, shared bool) (string, error) { directory := path.Join("/tmp/matrix-doc", strconv.FormatInt(rand.Int63(), 10)) cmd := exec.Command("git", "clone", url, directory) @@ -80,21 +79,22 @@ func gitClone(url string, shared bool) (string, error) { } func gitCheckout(path, sha string) error { - cmd := exec.Command("git", "checkout", sha) - cmd.Dir = path - err := cmd.Run() - if err != nil { - return fmt.Errorf("error checking out repo: %v", err) + return runGitCommand(path, []string{"checkout", sha}) +} + +func gitFetchAndMerge(path string) error { + if err := runGitCommand(path, []string{"fetch"}); err != nil { + return err } - return nil + return runGitCommand(path, []string{"merge"}) } -func gitFetch(path string) error { - cmd := exec.Command("git", "fetch") +func runGitCommand(path string, args []string) error { + cmd := exec.Command("git", args...) cmd.Dir = path err := cmd.Run() if err != nil { - return fmt.Errorf("error fetching repo: %v", err) + return fmt.Errorf("error running %s: %v", strings.Join(cmd.Args, " "), err) } return nil } @@ -145,7 +145,7 @@ type server struct { // generateAt generates spec from repo at sha. // Returns the path where the generation was done. func (s *server) generateAt(sha string) (dst string, err error) { - err = gitFetch(s.matrixDocCloneURL) + err = gitFetchAndMerge(s.matrixDocCloneURL) if err != nil { return } From a69e03f57778b9ef743cb03430db472e1b0a28a9 Mon Sep 17 00:00:00 2001 From: Daniel Wagner-Hall Date: Thu, 1 Oct 2015 19:15:30 -0500 Subject: [PATCH 47/54] speculator: Report listening port --- scripts/speculator/main.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go index 6ee80150d..ece653726 100644 --- a/scripts/speculator/main.go +++ b/scripts/speculator/main.go @@ -362,6 +362,8 @@ func main() { http.HandleFunc("/diff/html/", s.serveHTMLDiff) http.HandleFunc("/healthz", serveText("ok")) http.HandleFunc("/", listPulls) + + fmt.Printf("Listening on port %d\n", *port) log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil)) } From 3d9dbe42e691f446830e348eb680297728dadede Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 2 Oct 2015 10:21:48 +0100 Subject: [PATCH 48/54] Bump to swagger-parser 3.2.1 - remove x- keys on headers Removed x- keys due to https://github.com/BigstickCarpet/swagger-parser/issues/23 --- api/client-server/v1/content-repo.yaml | 5 ----- api/package.json | 2 +- api/validator.js | 10 +++++----- 3 files changed, 6 insertions(+), 11 deletions(-) diff --git a/api/client-server/v1/content-repo.yaml b/api/client-server/v1/content-repo.yaml index aa47dc184..8e6e8d1a4 100644 --- a/api/client-server/v1/content-repo.yaml +++ b/api/client-server/v1/content-repo.yaml @@ -68,15 +68,12 @@ paths: headers: Content-Type: description: "The content type of the file that was previously uploaded." - x-example: "audio/mpeg" type: "string" Content-Disposition: description: "The name of the file that was previously uploaded, if set." - x-example: "attachment;filename=03-cool.mp3" type: "string" schema: type: file - name: "" "/thumbnail/{serverName}/{mediaId}": get: summary: "Download a thumbnail of the content from the content repository." @@ -122,11 +119,9 @@ paths: headers: Content-Type: description: "The content type of the thumbnail." - x-example: "image/jpeg" type: "string" enum: ["image/jpeg", "image/png"] schema: type: file - name: "" diff --git a/api/package.json b/api/package.json index 151934935..84b9dd7b5 100644 --- a/api/package.json +++ b/api/package.json @@ -10,6 +10,6 @@ "license": "ISC", "dependencies": { "nopt": "^3.0.2", - "swagger-parser": "^2.4.1" + "swagger-parser": "^3.2.1" } } diff --git a/api/validator.js b/api/validator.js index 3b89a5a36..0d76c09d3 100644 --- a/api/validator.js +++ b/api/validator.js @@ -26,11 +26,10 @@ if (!opts.schema) { } -var errFn = function(err, api, metadata) { +var errFn = function(err, api) { if (!err) { return; } - console.log(metadata); console.error(err); process.exit(1); }; @@ -46,11 +45,12 @@ if (isDir) { files.forEach(function(f) { var suffix = ".yaml"; if (f.indexOf(suffix, f.length - suffix.length) > 0) { - parser.parse(path.join(opts.schema, f), function(err, api, metadata) { + parser.validate(path.join(opts.schema, f), function(err, api, metadata) { if (!err) { console.log("%s is valid.", f); } else { + console.error("%s is not valid.", f); errFn(err, api, metadata); } }); @@ -59,12 +59,12 @@ if (isDir) { }); } else{ - parser.parse(opts.schema, function(err, api, metadata) { + parser.validate(opts.schema, function(err, api) { if (!err) { console.log("%s is valid", opts.schema); } else { - errFn(err, api, metadata); + errFn(err, api); } }); }; From dbc72c43aca270d86c128caba1cde434642bcb8a Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 2 Oct 2015 10:28:29 +0100 Subject: [PATCH 49/54] s/private_chat_shared_power/trusted_private_chat/ --- specification/1-client_server_api.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 329a87ddb..1c64f0218 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -756,7 +756,7 @@ options which can be set when creating a room: Optional: Yes Value: - ``private_chat``, ``private_chat_shared_power`` or ``public_chat`` + ``private_chat``, ``trusted_private_chat`` or ``public_chat`` Description: Convenience parameter for setting various default state events based on a preset. @@ -765,9 +765,9 @@ options which can be set when creating a room: - ``private_chat``: Sets the ``join_rules`` to ``invite`` and ``history_visibility`` to ``shared`` - - ``private_chat_shared_power``: Set the ``join_rules`` to - ``invite``, ``history_visibility`` to ``shared`` and gives all invitees - the same power level as the creator. + - ``trusted_private_chat``: Set the ``join_rules`` to ``invite``, + ``history_visibility`` to ``shared`` and gives all invitees the same + power level as the creator. - ``public_chat``: Sets the ``join_rules`` to ``public`` and ``history_visibility`` to ``shared`` From 4dabcd112ef787c2f09ddbd61415e145e1b146e3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 2 Oct 2015 10:44:50 +0100 Subject: [PATCH 50/54] Remove redundant info now we have the http api template. Minor tweaks to display of schema with no names but a type --- specification/modules/content_repo.rst | 46 +++++--------------------- templating/matrix_templates/units.py | 3 +- 2 files changed, 11 insertions(+), 38 deletions(-) diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst index de464e203..9ac5e1991 100644 --- a/specification/modules/content_repo.rst +++ b/specification/modules/content_repo.rst @@ -10,9 +10,15 @@ URIs. They look like:: mxc:/// - : The name of the homeserver where this content can be found, e.g. matrix.org + : The name of the homeserver where this content originated, e.g. matrix.org : An opaque ID which identifies the content. +Uploads are POSTed to a resource on the user's local homeserver which returns a +token which is used to GET the download. Content is downloaded from the +recipient's local homeserver, which must first transfer the content from the +origin homeserver using the same API (unless the origin and destination +homeservers are the same). + Client behaviour ---------------- @@ -20,42 +26,8 @@ Clients can upload and download content using the following HTTP APIs. {{content_repo_http_api}} - -Uploads are POSTed to a resource which returns a token which is used to GET -the download. Uploads are POSTed to the sender's local homeserver, but are -downloaded from the recipient's local homeserver, which must thus first transfer -the content from the origin homeserver using the same API (unless the origin -and destination homeservers are the same). The upload/download API is:: - - => POST /_matrix/media/v1/upload HTTP/1.1 - Content-Type: - - - - <= HTTP/1.1 200 OK - Content-Type: application/json - - { "content-uri": "mxc:///" } - - => GET /_matrix/media/v1/download// HTTP/1.1 - - <= HTTP/1.1 200 OK - Content-Type: - Content-Disposition: attachment;filename= - - - -Clients can get thumbnails by supplying a desired width and height and -thumbnailing method:: - - => GET /_matrix/media/v1/thumbnail/ - /?width=&height=&method= HTTP/1.1 - - <= HTTP/1.1 200 OK - Content-Type: image/jpeg or image/png - - - +Thumbnails +~~~~~~~~~~ The thumbnail methods are "crop" and "scale". "scale" tries to return an image where either the width or the height is smaller than the requested size. The client should then scale and letterbox the image if it needs to diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py index 04f3bae10..1e449b5d6 100644 --- a/templating/matrix_templates/units.py +++ b/templating/matrix_templates/units.py @@ -260,12 +260,13 @@ class MatrixUnits(Units): if good_response: self.log("Found a 200 response for this API") res_type = Units.prop(good_response, "schema/type") + res_name = Units.prop(good_response, "schema/name") if res_type and res_type not in ["object", "array"]: # response is a raw string or something like that good_table = { "title": None, "rows": [{ - "key": good_response["schema"].get("name", ""), + "key": "<" + res_type + ">" if not res_name else res_name, "type": res_type, "desc": res.get("description", ""), "req_str": "" From 0e8f1b5475f70d8436ec3a1d19886c2fe71ca677 Mon Sep 17 00:00:00 2001 From: Daniel Wagner-Hall Date: Fri, 2 Oct 2015 07:33:26 -0500 Subject: [PATCH 51/54] Quote args --- scripts/speculator/main.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go index 442400dfe..07ffa44ef 100644 --- a/scripts/speculator/main.go +++ b/scripts/speculator/main.go @@ -94,7 +94,7 @@ func runGitCommand(path string, args []string) error { cmd.Dir = path err := cmd.Run() if err != nil { - return fmt.Errorf("error running %s: %v", strings.Join(cmd.Args, " "), err) + return fmt.Errorf("error running %q: %v", strings.Join(cmd.Args, " "), err) } return nil } From c8ddf1af09581ca5d54a1e21561d9adce2b3d99d Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 2 Oct 2015 16:00:15 +0100 Subject: [PATCH 52/54] Add changelog --- CHANGELOG.rst | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 78aebddc9..66f3d0a80 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -6,6 +6,29 @@ .. in Jenkins. Comments like this are ignored by both RST and the templating .. system. Add the newest release notes beneath this comment. +Specification changes in v0.2.0 (2015-10-02) +============================================ + +This update restructures the specification and begins to aggressively standardise +on using Swagger and JSON Schema to document HTTP endpoints and Events +respectively. It also introduces a number of new concepts to Matrix. + +Additions: + - New section: Feature Profiles. + - New section: Receipts. + - New section: Room history visibility. + - New event: ``m.receipt``. + - New event: ``m.room.canonical_alias`` + - New event: ``m.room.history_visibility`` + - New keys: ``/createRoom`` - allows room "presets" using ``preset`` and + ``initial_state`` keys. + - New endpoint: ``/tokenrefresh`` - Related to refreshing access tokens. + +Modifications: + - Convert most of the older HTTP APIs to Swagger documentation. + - Convert most of the older event formats to JSON Schema. + - Move selected client-server sections to be "Modules". + Specification changes in v0.1.0 (2015-06-01) ============================================ - First numbered release. From 417c5b53c4c6264f6b91a4bfbaaa4049da6934a0 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 2 Oct 2015 16:24:33 +0100 Subject: [PATCH 53/54] Remove duplicate sentences from merge conflicts --- specification/modules/presence.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst index cff4490be..79151c4fe 100644 --- a/specification/modules/presence.rst +++ b/specification/modules/presence.rst @@ -21,11 +21,8 @@ A presence list is a list of user IDs whose presence the user wants to follow. To be added to this list, the user being added must be invited by the list owner who must accept the invitation. -Each user has presence information associated with them. This encodes the -"availability" of that user, suitable for display on other clients. -This is transmitted as an ``m.presence`` event and is one of the few events -which are sent *outside the context of a room*. Their presence state is -represented by the ``presence`` key, which is an enum of one of the following: +User's presence state is represented by the ``presence`` key, which is an enum +of one of the following: - ``online`` : The default state when the user is connected to an event stream. @@ -35,7 +32,6 @@ represented by the ``presence`` key, which is an enum of one of the following: explicitly suppressing their profile information from being sent. - ``free_for_chat`` : The user is generally willing to receive messages moreso than default. - Events ------ From 28fd1aa205b28adadee838547aed62e9f2a0a115 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 2 Oct 2015 16:34:06 +0100 Subject: [PATCH 54/54] Go into a bit more detail about feature profiles --- CHANGELOG.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 66f3d0a80..6e3198ef9 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -9,9 +9,14 @@ Specification changes in v0.2.0 (2015-10-02) ============================================ -This update restructures the specification and begins to aggressively standardise -on using Swagger and JSON Schema to document HTTP endpoints and Events -respectively. It also introduces a number of new concepts to Matrix. +This update fundamentally restructures the specification. The specification has +been split into more digestible "modules" which each describe a particular +function (e.g. typing). This was done in order make the specification easier to +maintain and help define which modules are mandatory for certain types +of clients. Types of clients along with the mandatory modules can be found in a +new "Feature Profiles" section. This update also begins to aggressively +standardise on using Swagger and JSON Schema to document HTTP endpoints and +Events respectively. It also introduces a number of new concepts to Matrix. Additions: - New section: Feature Profiles.