From 598e9b3113204a0b6728c3e5c2b64ee6e2a4764a Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 16 Mar 2015 00:14:03 -0400 Subject: [PATCH] big reshuffle: * try to make 00_basis slightly more accessible and less terse * move CS API before Events, as it's meaningless to dive into specific events if you haven't been introduced to how the CS API works * Try to improve the beginning of the CS API by merging back in some of kegan's original v1 content (which has subsequently been deleted somewhere along the line?) * document m.presence event consistently * move typing from drafts to spec * move push stuff from drafts to spec ...and lots of other stuff move push & typing from drafts into main spec. --- drafts/erik-model.rst | 5 +- drafts/general_api.rst | 6 +- drafts/media_repository.rst | 77 -- drafts/object_model.rst | 5 +- scripts/gendoc.py | 3 +- specification/00_basis.rst | 323 ++--- ...erver_api.rst => 10_client_server_api.rst} | 1188 +++++++++-------- .../{10_events.rst => 20_events.rst} | 53 +- ...api.rst => 25_application_service_api.rst} | 0 ...event_signing.rst => 31_event_signing.rst} | 0 .../42_push_overview.rst | 5 +- .../43_push_cs_api.rst | 6 +- .../44_push_push_gw_api.rst | 1 + .../45_typing_notifications.rst | 1 + 14 files changed, 861 insertions(+), 812 deletions(-) delete mode 100644 drafts/media_repository.rst rename specification/{20_client_server_api.rst => 10_client_server_api.rst} (81%) rename specification/{10_events.rst => 20_events.rst} (89%) rename specification/{40_application_service_api.rst => 25_application_service_api.rst} (100%) rename specification/{11_event_signing.rst => 31_event_signing.rst} (100%) rename drafts/push_overview.rst => specification/42_push_overview.rst (98%) rename drafts/push_csapi.rst => specification/43_push_cs_api.rst (99%) rename drafts/push_pgwapi.rst => specification/44_push_push_gw_api.rst (99%) rename drafts/typing_notifications.rst => specification/45_typing_notifications.rst (99%) diff --git a/drafts/erik-model.rst b/drafts/erik-model.rst index c21c558d..b2f9908d 100644 --- a/drafts/erik-model.rst +++ b/drafts/erik-model.rst @@ -1,4 +1,7 @@ -This is a standalone description of the data architecture of Synapse. There is a lot of overlap with the currennt specification, so it has been split out here for posterity. Hopefully all the important bits have been merged into the relevant places in the main spec. +This is a standalone description of the data architecture of Synapse. There is a +lot of overlap with the current specification, so it has been split out here for +posterity. Hopefully all the important bits have been merged into the relevant +places in the main spec. Model diff --git a/drafts/general_api.rst b/drafts/general_api.rst index 2b42b8b0..6c87c0f6 100644 --- a/drafts/general_api.rst +++ b/drafts/general_api.rst @@ -555,7 +555,7 @@ signature. Requesting the "raw" federation event will have to return these keys. Account Management API ``[Draft]`` ---------------------------------- -The registration and login APIs in v2 do not support specifying device IDs. In v2, +The registration and login APIs in v1 do not support specifying device IDs. In v2, this will become *mandatory* when sending your initial request. Access tokens will be scoped per device, so using the same device ID twice when logging in will clobber the old access token. @@ -810,6 +810,10 @@ Notes: Presence API ``[Draft]`` ------------------------ + +FIXME: this seems to be ignoring activity timers entirely, which were present on +the planning etherpad and are present in the actual HTTP API. Needs attention. + The goals of presence are to: - Let other users know if someone is "online". diff --git a/drafts/media_repository.rst b/drafts/media_repository.rst deleted file mode 100644 index 879e1b85..00000000 --- a/drafts/media_repository.rst +++ /dev/null @@ -1,77 +0,0 @@ -Media Repository -================ - -File uploading and downloading. - -HTTP API --------- - -Uploads are POSTed to a resource which returns a token which is used to GET -the download. Uploads are POSTed to the sender's local homeserver, but are -downloaded from the recipient's local homeserver, which must thus first transfer -the content from the origin homeserver using the same API (unless the origin -and destination homeservers are the same). The upload/download API is:: - - => POST /_matrix/media/v1/upload HTTP/1.1 - Content-Type: - - - - <= HTTP/1.1 200 OK - Content-Type: application/json - - { "content-uri": "mxc:///" } - - => GET /_matrix/media/v1/download// HTTP/1.1 - - <= HTTP/1.1 200 OK - Content-Type: - Content-Disposition: attachment;filename= - - - -Clients can get thumbnails by supplying a desired width and height and -thumbnailing method:: - - => GET /_matrix/media/v1/thumbnail/ - /?width=&height=&method= HTTP/1.1 - - <= HTTP/1.1 200 OK - Content-Type: image/jpeg or image/png - - - -The thumbnail methods are "crop" and "scale". "scale" trys to return an -image where either the width or the height is smaller than the requested -size. The client should then scale and letterbox the image if it needs to -fit within a given rectangle. "crop" trys to return an image where the -width and height are close to the requested size and the aspect matches -the requested size. The client should scale the image if it needs to fit -within a given rectangle. - -Homeservers may generate thumbnails for content uploaded to remote -homeservers themselves or may rely on the remote homeserver to thumbnail -the content. Homeservers may return thumbnails of a different size to that -requested. However homeservers should provide exact matches where reasonable. -Homeservers must never upscale images. - -Security --------- - -Clients may try to upload very large files. Homeservers should not store files -that are too large and should not serve them to clients. - -Clients may try to upload very large images. Homeservers should not attempt to -generate thumbnails for images that are too large. - -Remote homeservers may host very large files or images. Homeserver should not -proxy or thumbnail large files or images from remote homeservers. - -Clients may try to upload a large number of files. Homeservers should limit the -number and total size of media that can be uploaded by clients. - -Clients may try to access a large number of remote files through a homeserver. -Homeservers should restrict the number and size of remote files that it caches. - -Clients or remote homeservers may try to upload malicious files targeting -vulnerabilities in either the homeserver thumbnailing or the client decoders. diff --git a/drafts/object_model.rst b/drafts/object_model.rst index 1ae60b2c..0341d6a5 100644 --- a/drafts/object_model.rst +++ b/drafts/object_model.rst @@ -1,6 +1,5 @@ - - - +..TODO + What are the start & end tokens doing here?! :: diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 1b008685..431b1710 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -101,7 +101,8 @@ def prepare_env(): pass def cleanup_env(): - shutil.rmtree("./tmp") + pass + #shutil.rmtree("./tmp") def main(): prepare_env() diff --git a/specification/00_basis.rst b/specification/00_basis.rst index e97d3406..37d1b9ef 100644 --- a/specification/00_basis.rst +++ b/specification/00_basis.rst @@ -1,8 +1,8 @@ Matrix Specification ==================== -.. NOTE:: - The git version of this document is ``$GIT_VERSION`` +Version: ``$GIT_VERSION`` +------------------------------------------- Table of Contents ================= @@ -13,11 +13,14 @@ Table of Contents Introduction ============ -Matrix is a new set of open APIs for open-federated Instant Messaging and VoIP -functionality, designed to create and support a new global real-time -communication ecosystem on the internet. This specification is the ongoing -result of standardising the APIs used by the various components of the Matrix -ecosystem to communicate with one another. +Matrix is a set of open APIs for open-federated Instant Messaging, VoIP and +Internet of Things communication, designed to create and support a new global +real-time communication ecosystem. The intention is to provide an open +decentralised pubsub fabric for the internet for securely persisting and +publishing/subscribing JSON objects. + +This specification is the ongoing result of standardising the APIs used by the +various components of the Matrix ecosystem to communicate with one another. .. WARNING:: The Matrix specification is still evolving: the APIs are not yet frozen @@ -97,8 +100,34 @@ Overview Architecture ------------ -Clients transmit data to other clients through home servers (HSes). Clients do -not communicate with each other directly. +Matrix defines APIs for synchronising extensible JSON objects known as +`events` between compatible clients, servers and services. Clients are +typically messaging/VoIP applications or IoT devices/hubs and communicate by +synchronising communication history with their `homeserver` using the +`Client-Server API`. Each homeserver stores the communication history and +account information for all of its clients, and shares data with the wider +Matrix ecosystem by synchronising communication history with other homeservers +and their clients. + +Clients typically communicate with each other by emitting events in the +context of a virtual `room`. Room data is replicated across *all of the +homeservers* whose users are participating in a given room. As such, *no +single homeserver has control or ownership over a given room*. Homeservers +model communication history as a partially ordered graph of events known as +the room's `event graph`, which is synchronised with eventual consistency +between the participating servers using the Server-Server API. Matrix optimises +for the the Availability and Partitioned properties of CAP theorem at the +expense of Consistency. + +For example, for client A to send a message to client B, client A performs an +HTTP PUT of the required JSON event on its homeserver (HS) using the +client-server API. A's HS appends this event to its copy of the room's event +graph, signing the message in the context of the graph for integrity. A's HS +then replicates the message to B's HS by performing an HTTP PUT using the +server-server API. B's HS authenticates the request, validates the event's +signature, authorises the event's contents and then adds it to its copy of the +room's event graph. Client B then receives the message from his homeserver via +a long-lived GET request. :: @@ -107,21 +136,22 @@ not communicate with each other directly. { Matrix client A } { Matrix client B } ^ | ^ | - | events | | events | + | events | Client-Server API | events | | V | V +------------------+ +------------------+ | |---------( HTTPS )--------->| | | Home Server | | Home Server | | |<--------( HTTPS )----------| | - +------------------+ Federation +------------------+ + +------------------+ Server-Server API +------------------+ + History Synchronisation + -A "Client" typically represents a human using a web application or mobile app. -Clients use the "Client-to-Server" (C-S) API to communicate with their home -server, which stores their profile data and their record of the conversations -in which they participate. Each client is associated with a user account (and -may optionally support multiple user accounts). A user account is represented -by a unique "User ID". This ID is namespaced to the home server which allocated -the account and looks like:: +Users +~~~~~ + +Each client is associated with a user account, which is identified in Matrix +using a unique "User ID". This ID is namespaced to the home server which +allocated the account and has the form:: @localpart:domain @@ -131,19 +161,14 @@ this user. They are case-insensitive. .. TODO-spec - Need to specify precise grammar for Matrix IDs -A "Home Server" is a server which provides C-S APIs and has the ability to -federate with other HSes. It is typically responsible for multiple clients. -"Federation" is the term used to describe the sharing of data between two or -more home servers. - Events ~~~~~~ -Data in Matrix is encapsulated in an "event". An event is an action within the -system. Typically each action (e.g. sending a message) correlates with exactly -one event. Each event has a ``type`` which is used to differentiate different -kinds of data. ``type`` values MUST be uniquely globally namespaced following -Java's `package naming conventions +All data exchanged over Matrix is expressed as an "event". Typically each client +action (e.g. sending a message) correlates with exactly one event. Each event +has a ``type`` which is used to differentiate different kinds of data. ``type`` +values MUST be uniquely globally namespaced following Java's `package naming +conventions `, e.g. ``com.example.myapp.event``. The special top-level namespace ``m.`` is reserved for events defined in the Matrix specification - for instance ``m.room.message`` @@ -153,36 +178,39 @@ of a "Room". Event Graphs ~~~~~~~~~~~~ -Each event has a list of zero or more `parent` events. These relations form -directed acyclic graphs of events called `event graphs`. Every event graph has a single root event, and each event graph forms the -basis of the history of a matrix room. - -Event graphs give a partial ordering of events, i.e. given two events one may -be considered to have come before the other if one is an ancestor of the other. -Since two events may be on separate branches, not all events can be compared in -this manner. - -Every event has a metadata `depth` field that is a positive integer that is -strictly greater than the depths of any of its parents. The root event should -have a depth of 1. - -[Note: if one event is before another, then it must have a strictly smaller -depth] +Events exchanged in the context of a room are stored in a directed acyclic graph +(DAG) called an `event graph`. The partial ordering of this graph gives the +chronological ordering of events within the room. Each event in the graph has a +list of zero or more `parent` events, which refer to any preceeding events which +have no chronological successor from the perspective of the homeserver which +created the event. + +Typically an event has a single parent: the most recent message in the room at +the point it was sent. However, homeservers may legitimately race with each +other when sending messages, resulting in a single event having multiple +successors. The next event added to the graph thus will have multiple parents. +Every event graph has a single root event with no parent. + +To order and ease chronological comparison between the events within the graph, +homeservers maintain a `depth` metadata field on each event. An event's `depth` +is a positive integer that is strictly greater than the depths of any of its +parents. The root event should have a depth of 1. Thus if one event is before +another, then it must have a strictly smaller depth. Room structure ~~~~~~~~~~~~~~ -A room is a conceptual place where users can send and receive events. -Events are sent to a room, and all participants in -that room with sufficient access will receive the event. Rooms are uniquely -identified internally via a "Room ID", which look like:: +A room is a conceptual place where users can send and receive events. Events are +sent to a room, and all participants in that room with sufficient access will +receive the event. Rooms are uniquely identified internally via "Room IDs", +which have the form:: !opaque_id:domain There is exactly one room ID for each room. Whilst the room ID does contain a domain, it is simply for globally namespacing room IDs. The room does NOT reside on the domain specified. Room IDs are not meant to be human readable. -They ARE case-sensitive. +They are case-sensitive. The following conceptual diagram shows an ``m.room.message`` event being sent to the room ``!qporfwt:matrix.org``:: @@ -190,6 +218,7 @@ the room ``!qporfwt:matrix.org``:: { @alice:matrix.org } { @bob:domain.com } | ^ | | + [HTTP POST] [HTTP GET] Room ID: !qporfwt:matrix.org Room ID: !qporfwt:matrix.org Event type: m.room.message Event type: m.room.message Content: { JSON object } Content: { JSON object } @@ -200,7 +229,7 @@ the room ``!qporfwt:matrix.org``:: | matrix.org | | domain.com | +------------------+ +------------------+ | ^ - | | + | [HTTP POST] | | Room ID: !qporfwt:matrix.org | | Event type: m.room.message | | Content: { JSON object } | @@ -222,7 +251,7 @@ the room ``!qporfwt:matrix.org``:: | Content: { JSON object } | |...................................| -Federation maintains shared data structures per-room between multiple home +Federation maintains *shared data structures* per-room between multiple home servers. The data is split into ``message events`` and ``state events``. ``Message events`` describe transient 'once-off' activity in a room such as an @@ -252,7 +281,7 @@ participating in a room. Room Aliases ++++++++++++ -Each room can also have multiple "Room Aliases", which looks like:: +Each room can also have multiple "Room Aliases", which look like:: #room_alias:domain @@ -272,7 +301,7 @@ that are in the room that can be used to join via. :: - GET + HTTP GET #matrix:domain.com !aaabaa:matrix.org | ^ | | @@ -285,7 +314,7 @@ that are in the room that can be used to join via. |________________________________| Identity -++++++++ +~~~~~~~~ Users in Matrix are identified via their matrix user ID (MXID). However, existing 3rd party ID namespaces can also be used in order to identify Matrix @@ -306,47 +335,37 @@ the Matrix ecosystem. However, without one clients will not be able to look up user IDs using 3PIDs. Presence -++++++++ - -Each user has the concept of presence information. This encodes the -"availability" of that user, suitable for display on other user's clients. This -is transmitted as an ``m.presence`` event and is one of the few events which -are sent *outside the context of a room*. The basic piece of presence -information is represented by the ``presence`` key, which is an enum of one of -the following: - - - ``online`` : The default state when the user is connected to an event - stream. - - ``unavailable`` : The user is not reachable at this time. - - ``offline`` : The user is not connected to an event stream. - - ``free_for_chat`` : The user is generally willing to receive messages - moreso than default. - - ``hidden`` : Behaves as offline, but allows the user to see the client - state anyway and generally interact with client features. (Not yet - implemented in synapse). - -.. TODO-spec - This seems like a very constrained list of states - surely presence states - should be extensible, with us providing a baseline, and possibly a scale of - availability? For instance, do-not-disturb is missing here, as well as a - distinction between 'away' and 'busy'. +~~~~~~~~ -This basic ``presence`` field applies to the user as a whole, regardless of how -many client devices they have connected. The presence state is pushed by the homeserver to all connected clients for a user to ensure a consistent experience for the user. +Each user has the concept of presence information. This encodes: -.. TODO-spec - We need per-device presence in order to handle push notification semantics and similar. + * Whether the user is currently online + * How recently the user was last active (as seen by the server) + * Whether a given client considers the user to be currently idle + * Arbitrary information about the user's current status (e.g. "in a meeting"). -In addition, the server maintains a timestamp of the last time it saw a -pro-active event from the user; either sending a message to a room, or changing -presence state from a lower to a higher level of availability (thus: changing -state from ``unavailable`` to ``online`` counts as a proactive event, whereas in -the other direction it will not). This timestamp is presented via a key called -``last_active_ago``, which gives the relative number of milliseconds since the -message is generated/emitted that the user was last seen active. +This information is collated from both per-device (online; idle; last_active) and +per-user (status) data, aggregated by the user's homeserver and transmitted as +an ``m.presence`` event. This is one of the few events which are sent *outside +the context of a room*. Presence events are sent to all users who subscribe to +this user's presence through a presence list or by sharing membership of a room. -Presence List -~~~~~~~~~~~~~ +.. TODO + How do we let users hide their presence information? + +.. TODO + The last_active specifics should be moved to the detailed presence event section + +Last activity is tracked by the server maintaining a timestamp of the last time +it saw a pro-active event from the user; either sending a message to a room, +coming online or back from idle, etc. This timestamp is presented via a key +called ``last_active_ago``, which gives the relative number of milliseconds +since the message is generated/emitted that the user was last seen active. + +N.B. in v1 API, status/online/idle state are muxed into a single 'presence' field on the m.presence event. + +Presence Lists +~~~~~~~~~~~~~~ Each user's home server stores a "presence list". This stores a list of user IDs whose presence the user wants to follow. @@ -355,38 +374,31 @@ To be added to this list, the user being added must be invited by the list owner and accept the invitation. Once accepted, both user's HSes track the subscription. -Presence and Permissions -~~~~~~~~~~~~~~~~~~~~~~~~ -For a viewing user to be allowed to see the presence information of a target -user, either: - - - The target user has allowed the viewing user to add them to their presence - list, or - - The two users share at least one room in common - -In the latter case, this allows for clients to display some minimal sense of -presence information in a user list for a room. +Profiles +~~~~~~~~ +Users may publish arbitrary key/value data associated with their account - such +as a human readable ``display name``, a profile photo URL, contact information +(email address, phone nubers, website URLs etc). -Profiles -++++++++ +Profile data is typed using namespaced keys for interoperability, much like +events - e.g. ``m.profile.display_name``. -.. TODO-spec - - Metadata extensibility +..TODO + Actually specify the different types of data - e.g. what format are display + names allowed to be? -Internally within Matrix users are referred to by their user ID, which is -typically a compact unique identifier. Profiles grant users the ability to see -human-readable names for other users that are in some way meaningful to them. -Additionally, profiles can publish additional information, such as the user's -age or location. +Private User Data +~~~~~~~~~~~~~~~~~ -A Profile consists of a display name, an avatar picture, and a set of other -metadata fields that the user may wish to publish (email address, phone -numbers, website URLs, etc...). This specification puts no requirements on the -display name other than it being a valid unicode string. Avatar images are not -stored directly; instead the home server stores an ``http``-scheme URL from which clients may fetch the image. +Users may also store arbitrary private key/value data in their account - such as +client preferences, or server configuration settings which lack any other +dedicated API. The API is symmetrical to managing Profile data. +..TODO + Would it really be overengineered to use the same API for both profile & + private user data, but with different ACLs? API Standards ------------- @@ -430,6 +442,9 @@ For example, if there was a custom namespace ``com.mydomain.here``, and a ``COM.MYDOMAIN.HERE_FORBIDDEN``. There may be additional keys depending on the error, but the keys ``error`` and ``errcode`` MUST always be present. +..TODO + Why the weird mix of underscore and dots? + Some standard error codes are below: :``M_FORBIDDEN``: @@ -502,81 +517,3 @@ In contrast, these are invalid requests:: "key": "This is a put but it is missing a txnId." } -Glossary --------- - -Backfilling: - The process of synchronising historic state from one home server to another, - to backfill the event storage so that scrollback can be presented to the - client(s). Not to be confused with pagination. - -Context: - A single human-level entity of interest (currently, a chat room) - -EDU (Ephemeral Data Unit): - A message that relates directly to a given pair of home servers that are - exchanging it. EDUs are short-lived messages that related only to one single - pair of servers; they are not persisted for a long time and are not forwarded - on to other servers. Because of this, they have no internal ID nor previous - EDUs reference chain. - -Event: - A record of activity that records a single thing that happened on to a context - (currently, a chat room). These are the "chat messages" that Synapse makes - available. - -PDU (Persistent Data Unit): - A message that relates to a single context, irrespective of the server that - is communicating it. PDUs either encode a single Event, or a single State - change. A PDU is referred to by its PDU ID; the pair of its origin server - and local reference from that server. - -PDU ID: - The pair of PDU Origin and PDU Reference, that together globally uniquely - refers to a specific PDU. - -PDU Origin: - The name of the origin server that generated a given PDU. This may not be the - server from which it has been received, due to the way they are copied around - from server to server. The origin always records the original server that - created it. - -PDU Reference: - A local ID used to refer to a specific PDU from a given origin server. These - references are opaque at the protocol level, but may optionally have some - structured meaning within a given origin server or implementation. - -Presence: - The concept of whether a user is currently online, how available they declare - they are, and so on. See also: doc/model/presence - -Profile: - A set of metadata about a user, such as a display name, provided for the - benefit of other users. See also: doc/model/profiles - -Room ID: - An opaque string (of as-yet undecided format) that identifies a particular - room and used in PDUs referring to it. - -Room Alias: - A human-readable string of the form #name:some.domain that users can use as a - pointer to identify a room; a Directory Server will map this to its Room ID - -State: - A set of metadata maintained about a Context, which is replicated among the - servers in addition to the history of Events. - -User ID: - A string of the form @localpart:domain.name that identifies a user for - wire-protocol purposes. The localpart is meaningless outside of a particular - home server. This takes a human-readable form that end-users can use directly - if they so wish, avoiding the 3PIDs. - -Transaction: - A message which relates to the communication between a given pair of servers. - A transaction contains possibly-empty lists of PDUs and EDUs. - -.. TODO - This glossary contradicts the terms used above - especially on State Events v. "State" - and Non-State Events v. "Events". We need better consistent names. - diff --git a/specification/20_client_server_api.rst b/specification/10_client_server_api.rst similarity index 81% rename from specification/20_client_server_api.rst rename to specification/10_client_server_api.rst index af071d6f..ebea7bb9 100644 --- a/specification/20_client_server_api.rst +++ b/specification/10_client_server_api.rst @@ -1,377 +1,437 @@ Client-Server API v1 ==================== -This outlines version 1 of the client-server API. - -Registration and Login ----------------------- - -Clients must register with a home server in order to use Matrix. After -registering, the client will be given an access token which must be used in ALL -requests to that home server as a query parameter 'access_token'. - -If the client has already registered, they need to be able to login to their -account. The home server may provide many different ways of logging in, such as -user/password auth, login via a social network (OAuth2), login by confirming a -token sent to their email address, etc. This specification does not define how -home servers should authorise their users who want to login to their existing -accounts, but instead defines the standard interface which implementations -should follow so that ANY client can login to ANY home server. Clients login -using the |login|_ API. Clients register using the |register|_ API. -Registration follows the same general procedure as login, but the path requests -are sent to and the details contained in them are different. - -In both registration and login cases, the process takes the form of one or more -stages, where at each stage the client submits a set of data for a given stage -type and awaits a response from the server, which will either be a final -success or a request to perform an additional stage. This exchange continues -until the final success. - -In order to determine up-front what the server's requirements are, the client -can request from the server a complete description of all of its acceptable -flows of the registration or login process. It can then inspect the list of -returned flows looking for one for which it believes it can complete all of the -required stages, and perform it. As each home server may have different ways of -logging in, the client needs to know how they should login. All distinct login -stages MUST have a corresponding ``type``. A ``type`` is a namespaced string -which details the mechanism for logging in. - -A client may be able to login via multiple valid login flows, and should choose -a single flow when logging in. A flow is a series of login stages. The home -server MUST respond with all the valid login flows when requested by a simple -``GET`` request directly to the ``/login`` or ``/register`` paths:: - - { - "flows": [ - { - "type": "", - "stages": [ "", "" ] - }, - { - "type": "", - "stages": [ "", "" ] - }, - { - "type": "" - } - ] - } +Overview +-------- -The client can now select which flow it wishes to use, and begin making -``POST`` requests to the ``/login`` or ``/register`` paths with JSON body -content containing the name of the stage as the ``type`` key, along with -whatever additional parameters are required for that login or registration type -(see below). After the flow is completed, the client's fully-qualified user -ID and a new access token MUST be returned:: +The client-server API provides a simple lightweight API to let clients send +messages, control rooms and synchronise conversation history. It is designed to +support both lightweight clients which store no state and lazy-load data from +the server as required - as well as heavyweight clients which maintain a full +local peristent copy of server state. + +This describes v1 of the Client-Server API as featured in the original September +2014 launch of Matrix. Version 2 is currently in development (as of Jan-March +2015) as an incremental but backwards-incompatible refinement of Version 1 and +will be released shortly. + +Pagination +---------- + +Querying large datasets in Matrix always uses the same pagination API pattern to +to give clients a consistent way of selecting subsets of a potentially changing +dataset. Requests pass in `from`, `to` and `limit` parameters which describe +where to read from the stream. `from` and `to` are opaque textual 'stream +tokens' which describe positions in the dataset. The response returns new +`start` and `end` stream token values which can then be passed to subsequent +requests to continue pagination. + +Pagination Request Query Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Query parameters: + from: + $streamtoken - The opaque token to start streaming from. + to: + $streamtoken - The opaque token to end streaming at. Typically, + clients will not know the item of data to end at, so this will usually be + START or END. + limit: + integer - An integer representing the maximum number of items to + return. + +'START' and 'END' are magic token values which specify the start and end of the +dataset respectively. + +Unless specified, the default pagination parameters are from=START, to=END, +without a limit set. This allows you to hit an API like +/events without any query parameters to get everything. + +For example, the event stream has events E1 -> E15. The client wants the last 5 +events and doesn't know any previous events:: + + S E + |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| + | | | + | _____| | + |__________________ | ___________________| + | | | + GET /events?to=START&limit=5&from=END + Returns: + E15,E14,E13,E12,E11 + + +Another example: a public room list has rooms R1 -> R17. The client is showing 5 +rooms at a time on screen, and is on page 2. They want to +now show page 3 (rooms R11 -> 15):: + + S E + | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | stream token + |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room + |____________| |________________| + | | + Currently | + viewing | + | + GET /rooms/list?from=9&to=END&limit=5 + Returns: R11,R12,R13,R14,R15 + +Note that tokens are treated in an *exclusive*, not inclusive, manner. The end +token from the intial request was '9' which corresponded to R10. When the 2nd +request was made, R10 did not appear again, even though from=9 was specified. If +you know the token, you already have the data. + +Pagination Response +~~~~~~~~~~~~~~~~~~~ + +Responses to pagination requests MUST follow the format:: { - "user_id": "@user:matrix.org", - "access_token": "abcdef0123456789" + "chunk": [ ... , Responses , ... ], + "start" : $streamtoken, + "end" : $streamtoken } -The ``user_id`` key is particularly useful if the home server wishes to support -localpart entry of usernames (e.g. "user" rather than "@user:matrix.org"), as -the client may not be able to determine its ``user_id`` in this case. +Where $streamtoken is an opaque token which can be used in another query to +get the next set of results. The "start" and "end" keys can only be omitted if +the complete dataset is provided in "chunk". -If the flow has multiple stages to it, the home server may wish to create a -session to store context between requests. If a home server responds with a -``session`` key to a request, clients MUST submit it in subsequent requests -until the flow is completed:: +If the client wants earlier results, they should use from=$start_streamtoken, +to=START. Likewise, if the client wants later results, they should use +from=$end_streamtoken, to=END. - { - "session": "" - } -This specification defines the following login types: - - ``m.login.password`` - - ``m.login.oauth2`` - - ``m.login.email.code`` - - ``m.login.email.url`` - - ``m.login.email.identity`` +Events +------ -Password-based -~~~~~~~~~~~~~~ -:Type: - ``m.login.password`` -:Description: - Login is supported via a username and password. +Overview +~~~~~~~~ -To respond to this type, reply with:: +The model of conversation history exposed by the client-server API can be +considered as a list of events. The server 'linearises' the +eventually-consistent event graph of events into an 'event stream' at any given +point in time:: + + [E0]->[E1]->[E2]->[E3]->[E4]->[E5]->[E6]->[E7]->[E8]->[E9] + +Clients can add to the stream by POSTing message or state events, and can read +from the stream via the |initialSync|_, |/rooms//initialSync|_, `Event +Stream`_ and |/rooms//messages|_ APIs. + +For reading events, the intended flow of operation is to call +$PREFIX/initialSync, which returns all of the state and the last N events in the +event stream for each room, including `start` and `end` values describing the +pagination of each room's event stream. For instance, +$PREFIX/initialSync?limit=5 might return the events for a room in the +rooms[0].messages.chunk[] array, with tokens describing the start and end of the +range in rooms[0].messages.start as '1-2-3' and rooms[0].messages.end as +'a-b-c'. + +You can visualise the range of events being returned as:: + + [E0]->[E1]->[E2]->[E3]->[E4]->[E5]->[E6]->[E7]->[E8]->[E9] + ^ ^ + | | + start: '1-2-3' end: 'a-b-c' + +Now, to receive future events in realtime on the eventstream, you simply GET +$PREFIX/events with a `from` parameter of 'a-b-c': in other words passing in the +`end` token returned by initialsync. The request blocks until new events are +available or until your specified timeout elapses, and then returns a +new paginatable chunk of events alongside new start and end parameters:: + + [E0]->[E1]->[E2]->[E3]->[E4]->[E5]->[E6]->[E7]->[E8]->[E9]->[E10] + ^ ^ + | | + | end: 'x-y-z' + start: 'a-b-c' + +To resume polling the events stream, you pass in the new `end` token as the +`from` parameter of $PREFIX/events and poll again. + +Similarly, to paginate events backwards in order to lazy-load in previous +history from the room, you simply GET $PREFIX/rooms//messages +specifying the `from` token to paginate backwards from and a limit of the number +of messages to retrieve. For instance, calling this API with a `from` parameter +of '1-2-3' and a limit of 5 would return: + + [E0]->[E1]->[E2]->[E3]->[E4]->[E5]->[E6]->[E7]->[E8]->[E9]->[E10] + ^ ^ + | | + start: 'u-v-w' end: '1-2-3' + +To continue paginating backwards, one calls the /messages API again, supplying +the new `start` value as the `from` parameter. - { - "type": "m.login.password", - "user": "", - "password": "" - } -The home server MUST respond with either new credentials, the next stage of the -login process, or a standard error response. +Receiving live updates on a client +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Captcha-based -~~~~~~~~~~~~~ -:Type: - ``m.login.recaptcha`` -:Description: - Login is supported by responding to a captcha (in the case of the Synapse - implementation, Google's Recaptcha library is used). +Clients receive new events by long-polling the home server via the +$PREFIX/events API, specifying a timeout in milliseconds in the timeout +parameter. This will hold open the HTTP connection for a short period of time +waiting for new events, returning early if an event occurs. This is called the +`Event Stream`_. All events which are visible to the client will appear in the +event stream. When the request returns, an ``end`` token is included in the +response. This token can be used in the next request to continue where the +last request left off. -To respond to this type, reply with:: +All events must be deduplicated based on their event ID (TODO: is this actually +a hard requirement in CS v2?) - { - "type": "m.login.recaptcha", - "challenge": "", - "response": "" - } +.. TODO-spec + Do we ever return multiple events in a single request? + Don't we get lots of request setup RTT latency if we only do one event per request? + Do we ever support streaming requests? Why not websockets? -.. NOTE:: - In Synapse, the Recaptcha parameters can be obtained in Javascript by calling: - Recaptcha.get_challenge(); - Recaptcha.get_response(); +When the client first logs in, they will need to initially synchronise with +their home server. This is achieved via the |initialSync|_ API. This API also +returns an ``end`` token which can be used with the event stream. See the 'Room Sync' section below. -The home server MUST respond with either new credentials, the next stage of the -login process, or a standard error response. +Events in a room +~~~~~~~~~~~~~~~~ -OAuth2-based -~~~~~~~~~~~~ -:Type: - ``m.login.oauth2`` -:Description: - Login is supported via OAuth2 URLs. This login consists of multiple requests. +Room events are split into two categories: -To respond to this type, reply with:: +:Message events: + These are events which describe transient "once-off" activity in a room: + typically communication such as sending an instant messaage or setting up a + VoIP call. These used to be called 'non-state' events. - { - "type": "m.login.oauth2", - "user": "" - } +:State Events: + These are events which update the metadata state of the room (e.g. room topic, + room membership etc). State is keyed by a tuple of event ``type`` and a + ``state_key``. State in the room with the same key-tuple will be overwritten. -The server MUST respond with:: +This specification outlines several events, all with the event type prefix +``m.``. However, applications may wish to add their own type of event, and this +can be achieved using the REST API detailed in the following sections. If new +events are added, the event ``type`` key SHOULD follow the Java package naming +convention, e.g. ``com.example.myapp.event``. This ensures event types are +suitably namespaced for each application and reduces the risk of clashes. - { - "uri": - } +State events +~~~~~~~~~~~~ -The home server acts as a 'confidential' client for the purposes of OAuth2. If -the uri is a ``sevice selection URI``, it MUST point to a webpage which prompts -the user to choose which service to authorize with. On selection of a service, -this MUST link through to an ``Authorization Request URI``. If there is only 1 -service which the home server accepts when logging in, this indirection can be -skipped and the "uri" key can be the ``Authorization Request URI``. +State events can be sent by ``PUT`` ing to +|/rooms//state//|_. These events will be +overwritten if ````, ```` and ```` all match. +If the state event has no ``state_key``, it can be omitted from the path. These +requests **cannot use transaction IDs** like other ``PUT`` paths because they +cannot be differentiated from the ``state_key``. Furthermore, ``POST`` is +unsupported on state paths. Valid requests look like:: -The client then visits the ``Authorization Request URI``, which then shows the -OAuth2 Allow/Deny prompt. Hitting 'Allow' returns the ``redirect URI`` with the -auth code. Home servers can choose any path for the ``redirect URI``. The -client should visit the ``redirect URI``, which will then finish the OAuth2 -login process, granting the home server an access token for the chosen service. -When the home server gets this access token, it verifies that the cilent has -authorised with the 3rd party, and can now complete the login. The OAuth2 -``redirect URI`` (with auth code) MUST respond with either new credentials, the -next stage of the login process, or a standard error response. + PUT /rooms/!roomid:domain/state/m.example.event + { "key" : "without a state key" } -For example, if a home server accepts OAuth2 from Google, it would return the -Authorization Request URI for Google:: + PUT /rooms/!roomid:domain/state/m.another.example.event/foo + { "key" : "with 'foo' as the state key" } - { - "uri": "https://accounts.google.com/o/oauth2/auth?response_type=code& - client_id=CLIENT_ID&redirect_uri=REDIRECT_URI&scope=photos" - } +In contrast, these requests are invalid:: -The client then visits this URI and authorizes the home server. The client then -visits the REDIRECT_URI with the auth code= query parameter which returns:: + POST /rooms/!roomid:domain/state/m.example.event/ + { "key" : "cannot use POST here" } - { - "user_id": "@user:matrix.org", - "access_token": "0123456789abcdef" - } + PUT /rooms/!roomid:domain/state/m.another.example.event/foo/11 + { "key" : "txnIds are not supported" } -Email-based (code) -~~~~~~~~~~~~~~~~~~ -:Type: - ``m.login.email.code`` -:Description: - Login is supported by typing in a code which is sent in an email. This login - consists of multiple requests. +Care should be taken to avoid setting the wrong ``state key``:: -To respond to this type, reply with:: + PUT /rooms/!roomid:domain/state/m.another.example.event/11 + { "key" : "with '11' as the state key, but was probably intended to be a txnId" } - { - "type": "m.login.email.code", - "user": "", - "email": "" - } +The ``state_key`` is often used to store state about individual users, by using +the user ID as the ``state_key`` value. For example:: -After validating the email address, the home server MUST send an email -containing an authentication code and return:: + PUT /rooms/!roomid:domain/state/m.favorite.animal.event/%40my_user%3Adomain.com + { "animal" : "cat", "reason": "fluffy" } - { - "type": "m.login.email.code", - "session": "" - } +In some cases, there may be no need for a ``state_key``, so it can be omitted:: -The second request in this login stage involves sending this authentication -code:: + PUT /rooms/!roomid:domain/state/m.room.bgd.color + { "color": "red", "hex": "#ff0000" } - { - "type": "m.login.email.code", - "session": "", - "code": "" - } +See `Room Events`_ for the ``m.`` event specification. -The home server MUST respond to this with either new credentials, the next -stage of the login process, or a standard error response. +Message events +~~~~~~~~~~~~~~ -Email-based (url) -~~~~~~~~~~~~~~~~~ -:Type: - ``m.login.email.url`` -:Description: - Login is supported by clicking on a URL in an email. This login consists of - multiple requests. +Message events can be sent by sending a request to +|/rooms//send/|_. These requests *can* use transaction +IDs and ``PUT``/``POST`` methods. Message events allow access to historical +events and pagination, making it best suited for sending messages. For +example:: -To respond to this type, reply with:: + POST /rooms/!roomid:domain/send/m.custom.example.message + { "text": "Hello world!" } - { - "type": "m.login.email.url", - "user": "", - "email": "" - } + PUT /rooms/!roomid:domain/send/m.custom.example.message/11 + { "text": "Goodbye world!" } -After validating the email address, the home server MUST send an email -containing an authentication URL and return:: +See `Room Events`_ for the ``m.`` event specification. - { - "type": "m.login.email.url", - "session": "" - } +Syncing rooms +~~~~~~~~~~~~~ -The email contains a URL which must be clicked. After it has been clicked, the -client should perform another request:: +.. NOTE:: + This section is a work in progress. - { - "type": "m.login.email.url", - "session": "" - } +When a client logs in, they may have a list of rooms which they have already +joined. These rooms may also have a list of events associated with them. The +purpose of 'syncing' is to present the current room and event information in a +convenient, compact manner. The events returned are not limited to room events; +presence events will also be returned. A single syncing API is provided: -The home server MUST respond to this with either new credentials, the next -stage of the login process, or a standard error response. + - |initialSync|_ : A global sync which will present room and event information + for all rooms the user has joined. -A common client implementation will be to periodically poll until the link is -clicked. If the link has not been visited yet, a standard error response with -an errcode of ``M_LOGIN_EMAIL_URL_NOT_YET`` should be returned. +.. TODO-spec room-scoped initial sync + - |/rooms//initialSync|_ : A sync scoped to a single room. Presents + room and event information for this room only. + - Room-scoped initial sync is Very Tricky because typically people would + want to sync the room then listen for any new content from that point + onwards. The event stream cannot do this for a single room currently. + As a result, commenting room-scoped initial sync at this time. +The |initialSync|_ API contains the following keys: -Email-based (identity server) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -:Type: - ``m.login.email.identity`` -:Description: - Login is supported by authorising an email address with an identity server. +``presence`` + Description: + Contains a list of presence information for users the client is interested + in. + Format: + A JSON array of ``m.presence`` events. -Prior to submitting this, the client should authenticate with an identity -server. After authenticating, the session information should be submitted to -the home server. +``end`` + Description: + Contains an event stream token which can be used with the `Event Stream`_. + Format: + A string containing the event stream token. -To respond to this type, reply with:: +``rooms`` + Description: + Contains a list of room information for all rooms the client has joined, + and limited room information on rooms the client has been invited to. + Format: + A JSON array containing Room Information JSON objects. - { - "type": "m.login.email.identity", - "threepidCreds": [ - { - "sid": "", - "clientSecret": "", - "idServer": "" - } - ] - } +Room Information: + Description: + Contains all state events for the room, along with a limited amount of + the most recent events, configured via the ``limit`` query + parameter. Also contains additional keys with room metadata, such as the + ``room_id`` and the client's ``membership`` to the room. + Format: + A JSON object with the following keys: + ``room_id`` + A string containing the ID of the room being described. + ``membership`` + A string representing the client's membership status in this room. + ``messages`` + An event stream JSON object containing a ``chunk`` of recent + events (both state events and non-state events), along with an ``end`` token. + ``state`` + A JSON array containing all the current state events for this room. +Getting events for a room +~~~~~~~~~~~~~~~~~~~~~~~~~ +There are several APIs provided to ``GET`` events for a room: -N-Factor Authentication -~~~~~~~~~~~~~~~~~~~~~~~ -Multiple login stages can be combined to create N-factor authentication during -login. +``/rooms//state//`` + Description: + Get the state event identified. + Response format: + A JSON object representing the state event **content**. + Example: + ``/rooms/!room:domain.com/state/m.room.name`` returns ``{ "name": "Room name" }`` -This can be achieved by responding with the ``next`` login type on completion -of a previous login stage:: +|/rooms//state|_ + Description: + Get all state events for a room. + Response format: + ``[ { state event }, { state event }, ... ]`` + Example: + TODO-doc - { - "next": "" - } +|/rooms//members|_ + Description: + Get all ``m.room.member`` state events. + Response format: + ``{ "start": "", "end": "", "chunk": [ { m.room.member event }, ... ] }`` + Example: + TODO-doc -If a home server implements N-factor authentication, it MUST respond with all -``stages`` when initially queried for their login requirements:: +|/rooms//messages|_ + Description: + Get all ``m.room.message`` and ``m.room.member`` events. This API supports + pagination using ``from`` and ``to`` query parameters, coupled with the + ``start`` and ``end`` tokens from an |initialSync|_ API. + + XXX: Is this accurate? Doesn't it return all events - not just m.room.message/member? + + Response format: + ``{ "start": "", "end": "" }`` + Example: + TODO-doc - { - "type": "<1st login type>", - "stages": [ <1st login type>, <2nd login type>, ... , ] - } +|/rooms//initialSync|_ + Description: + Get all relevant events for a room. This includes state events, paginated + non-state events and presence events. + Response format: + `` { TODO-doc } `` + Example: + TODO-doc -This can be represented conceptually as:: +Redactions +~~~~~~~~~~ +Since events are extensible it is possible for malicious users and/or servers +to add keys that are, for example offensive or illegal. Since some events +cannot be simply deleted, e.g. membership events, we instead 'redact' events. +This involves removing all keys from an event that are not required by the +protocol. This stripped down event is thereafter returned anytime a client or +remote server requests it. - _______________________ - | Login Stage 1 | - | type: "" | - | ___________________ | - | |_Request_1_________| | <-- Returns "session" key which is used throughout. - | ___________________ | - | |_Request_2_________| | <-- Returns a "next" value of "login type2" - |_______________________| - | - | - _________V_____________ - | Login Stage 2 | - | type: "" | - | ___________________ | - | |_Request_1_________| | - | ___________________ | - | |_Request_2_________| | - | ___________________ | - | |_Request_3_________| | <-- Returns a "next" value of "login type3" - |_______________________| - | - | - _________V_____________ - | Login Stage 3 | - | type: "" | - | ___________________ | - | |_Request_1_________| | <-- Returns user credentials - |_______________________| +Events that have been redacted include a ``redacted_because`` key whose value +is the event that caused it to be redacted, which may include a reason. -Fallback -~~~~~~~~ -Clients cannot be expected to be able to know how to process every single login -type. If a client determines it does not know how to handle a given login type, -it should request a login fallback page:: +Redacting an event cannot be undone, allowing server owners to delete the +offending content from the databases. - GET matrix/client/api/v1/login/fallback +Currently, only room admins can redact events by sending a ``m.room.redaction`` +event, but server admins also need to be able to redact events by a similar +mechanism. -This MUST return an HTML page which can perform the entire login process. +Upon receipt of a redaction event, the server should strip off any keys not in +the following list: -Events ------- + - ``event_id`` + - ``type`` + - ``room_id`` + - ``user_id`` + - ``state_key`` + - ``prev_state`` + - ``content`` -Receiving live updates on a client -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The content object should also be stripped of all keys, unless it is one of +one of the following event types: -Clients can receive new events by long-polling the home server. This will hold -open the HTTP connection for a short period of time waiting for new events, -returning early if an event occurs. This is called the `Event Stream`_. All -events which are visible to the client will appear in the event stream. When -the request returns, an ``end`` token is included in the response. This token -can be used in the next request to continue where the client left off. + - ``m.room.member`` allows key ``membership`` + - ``m.room.create`` allows key ``creator`` + - ``m.room.join_rules`` allows key ``join_rule`` + - ``m.room.power_levels`` allows keys that are user ids or ``default`` + - ``m.room.add_state_level`` allows key ``level`` + - ``m.room.send_event_level`` allows key ``level`` + - ``m.room.ops_levels`` allows keys ``kick_level``, ``ban_level`` + and ``redact_level`` + - ``m.room.aliases`` allows key ``aliases`` -All events must be deduplicated based on their event ID (TODO: is this actually a -hard requirement in CS v2?) +The redaction event should be added under the key ``redacted_because``. -.. TODO-spec - How do we filter the event stream? - Do we ever return multiple events in a single request? Don't we get lots of request - setup RTT latency if we only do one event per request? Do we ever support streaming - requests? Why not websockets? -When the client first logs in, they will need to initially synchronise with -their home server. This is achieved via the |initialSync|_ API. This API also -returns an ``end`` token which can be used with the event stream. +When a client receives a redaction event it should change the redacted event +in the same way a server does. Rooms @@ -687,247 +747,351 @@ member's state, by making a request to "membership": "ban" } -Events in a room -~~~~~~~~~~~~~~~~ -Room events can be split into two categories: -:State Events: - These are events which replace events that came before it, depending on a set - of unique keys. These keys are the event ``type`` and a ``state_key``. - Events with the same set of keys will be overwritten. Typically, state events - are used to store state, hence their name. - -:Non-state events: - These are events which cannot be overwritten after sending. The list of - events continues to grow as more events are sent. As this list grows, it - becomes necessary to provide a mechanism for navigating this list. Pagination - APIs are used to view the list of historical non-state events. Typically, - non-state events are used to send messages. +Registration and Login +---------------------- -This specification outlines several events, all with the event type prefix -``m.``. However, applications may wish to add their own type of event, and this -can be achieved using the REST API detailed in the following sections. If new -events are added, the event ``type`` key SHOULD follow the Java package naming -convention, e.g. ``com.example.myapp.event``. This ensures event types are -suitably namespaced for each application and reduces the risk of clashes. +Clients must register with a home server in order to use Matrix. After +registering, the client will be given an access token which must be used in ALL +requests to that home server as a query parameter 'access_token'. -State events +If the client has already registered, they need to be able to login to their +account. The home server may provide many different ways of logging in, such as +user/password auth, login via a social network (OAuth2), login by confirming a +token sent to their email address, etc. This specification does not define how +home servers should authorise their users who want to login to their existing +accounts, but instead defines the standard interface which implementations +should follow so that ANY client can login to ANY home server. Clients login +using the |login|_ API. Clients register using the |register|_ API. +Registration follows the same general procedure as login, but the path requests +are sent to and the details contained in them are different. + +In both registration and login cases, the process takes the form of one or more +stages, where at each stage the client submits a set of data for a given stage +type and awaits a response from the server, which will either be a final +success or a request to perform an additional stage. This exchange continues +until the final success. + +In order to determine up-front what the server's requirements are, the client +can request from the server a complete description of all of its acceptable +flows of the registration or login process. It can then inspect the list of +returned flows looking for one for which it believes it can complete all of the +required stages, and perform it. As each home server may have different ways of +logging in, the client needs to know how they should login. All distinct login +stages MUST have a corresponding ``type``. A ``type`` is a namespaced string +which details the mechanism for logging in. + +A client may be able to login via multiple valid login flows, and should choose +a single flow when logging in. A flow is a series of login stages. The home +server MUST respond with all the valid login flows when requested by a simple +``GET`` request directly to the ``/login`` or ``/register`` paths:: + + { + "flows": [ + { + "type": "", + "stages": [ "", "" ] + }, + { + "type": "", + "stages": [ "", "" ] + }, + { + "type": "" + } + ] + } + +The client can now select which flow it wishes to use, and begin making +``POST`` requests to the ``/login`` or ``/register`` paths with JSON body +content containing the name of the stage as the ``type`` key, along with +whatever additional parameters are required for that login or registration type +(see below). After the flow is completed, the client's fully-qualified user +ID and a new access token MUST be returned:: + + { + "user_id": "@user:matrix.org", + "access_token": "abcdef0123456789" + } + +The ``user_id`` key is particularly useful if the home server wishes to support +localpart entry of usernames (e.g. "user" rather than "@user:matrix.org"), as +the client may not be able to determine its ``user_id`` in this case. + +If the flow has multiple stages to it, the home server may wish to create a +session to store context between requests. If a home server responds with a +``session`` key to a request, clients MUST submit it in subsequent requests +until the flow is completed:: + + { + "session": "" + } + +This specification defines the following login types: + - ``m.login.password`` + - ``m.login.oauth2`` + - ``m.login.email.code`` + - ``m.login.email.url`` + - ``m.login.email.identity`` + +Password-based +~~~~~~~~~~~~~~ +:Type: + ``m.login.password`` +:Description: + Login is supported via a username and password. + +To respond to this type, reply with:: + + { + "type": "m.login.password", + "user": "", + "password": "" + } + +The home server MUST respond with either new credentials, the next stage of the +login process, or a standard error response. + +Captcha-based +~~~~~~~~~~~~~ +:Type: + ``m.login.recaptcha`` +:Description: + Login is supported by responding to a captcha (in the case of the Synapse + implementation, Google's Recaptcha library is used). + +To respond to this type, reply with:: + + { + "type": "m.login.recaptcha", + "challenge": "", + "response": "" + } + +.. NOTE:: + In Synapse, the Recaptcha parameters can be obtained in Javascript by calling: + Recaptcha.get_challenge(); + Recaptcha.get_response(); + +The home server MUST respond with either new credentials, the next stage of the +login process, or a standard error response. + +OAuth2-based ~~~~~~~~~~~~ -State events can be sent by ``PUT`` ing to -|/rooms//state//|_. These events will be -overwritten if ````, ```` and ```` all match. -If the state event has no ``state_key``, it can be omitted from the path. These -requests **cannot use transaction IDs** like other ``PUT`` paths because they -cannot be differentiated from the ``state_key``. Furthermore, ``POST`` is -unsupported on state paths. Valid requests look like:: +:Type: + ``m.login.oauth2`` +:Description: + Login is supported via OAuth2 URLs. This login consists of multiple requests. - PUT /rooms/!roomid:domain/state/m.example.event - { "key" : "without a state key" } +To respond to this type, reply with:: - PUT /rooms/!roomid:domain/state/m.another.example.event/foo - { "key" : "with 'foo' as the state key" } + { + "type": "m.login.oauth2", + "user": "" + } -In contrast, these requests are invalid:: +The server MUST respond with:: - POST /rooms/!roomid:domain/state/m.example.event/ - { "key" : "cannot use POST here" } + { + "uri": + } - PUT /rooms/!roomid:domain/state/m.another.example.event/foo/11 - { "key" : "txnIds are not supported" } +The home server acts as a 'confidential' client for the purposes of OAuth2. If +the uri is a ``sevice selection URI``, it MUST point to a webpage which prompts +the user to choose which service to authorize with. On selection of a service, +this MUST link through to an ``Authorization Request URI``. If there is only 1 +service which the home server accepts when logging in, this indirection can be +skipped and the "uri" key can be the ``Authorization Request URI``. -Care should be taken to avoid setting the wrong ``state key``:: +The client then visits the ``Authorization Request URI``, which then shows the +OAuth2 Allow/Deny prompt. Hitting 'Allow' returns the ``redirect URI`` with the +auth code. Home servers can choose any path for the ``redirect URI``. The +client should visit the ``redirect URI``, which will then finish the OAuth2 +login process, granting the home server an access token for the chosen service. +When the home server gets this access token, it verifies that the cilent has +authorised with the 3rd party, and can now complete the login. The OAuth2 +``redirect URI`` (with auth code) MUST respond with either new credentials, the +next stage of the login process, or a standard error response. - PUT /rooms/!roomid:domain/state/m.another.example.event/11 - { "key" : "with '11' as the state key, but was probably intended to be a txnId" } +For example, if a home server accepts OAuth2 from Google, it would return the +Authorization Request URI for Google:: -The ``state_key`` is often used to store state about individual users, by using -the user ID as the ``state_key`` value. For example:: + { + "uri": "https://accounts.google.com/o/oauth2/auth?response_type=code& + client_id=CLIENT_ID&redirect_uri=REDIRECT_URI&scope=photos" + } - PUT /rooms/!roomid:domain/state/m.favorite.animal.event/%40my_user%3Adomain.com - { "animal" : "cat", "reason": "fluffy" } +The client then visits this URI and authorizes the home server. The client then +visits the REDIRECT_URI with the auth code= query parameter which returns:: -In some cases, there may be no need for a ``state_key``, so it can be omitted:: + { + "user_id": "@user:matrix.org", + "access_token": "0123456789abcdef" + } - PUT /rooms/!roomid:domain/state/m.room.bgd.color - { "color": "red", "hex": "#ff0000" } +Email-based (code) +~~~~~~~~~~~~~~~~~~ +:Type: + ``m.login.email.code`` +:Description: + Login is supported by typing in a code which is sent in an email. This login + consists of multiple requests. -See `Room Events`_ for the ``m.`` event specification. +To respond to this type, reply with:: -Non-state events -~~~~~~~~~~~~~~~~ -Non-state events can be sent by sending a request to -|/rooms//send/|_. These requests *can* use transaction -IDs and ``PUT``/``POST`` methods. Non-state events allow access to historical -events and pagination, making it best suited for sending messages. For -example:: + { + "type": "m.login.email.code", + "user": "", + "email": "" + } - POST /rooms/!roomid:domain/send/m.custom.example.message - { "text": "Hello world!" } +After validating the email address, the home server MUST send an email +containing an authentication code and return:: + + { + "type": "m.login.email.code", + "session": "" + } - PUT /rooms/!roomid:domain/send/m.custom.example.message/11 - { "text": "Goodbye world!" } +The second request in this login stage involves sending this authentication +code:: -See `Room Events`_ for the ``m.`` event specification. + { + "type": "m.login.email.code", + "session": "", + "code": "" + } -Syncing rooms -~~~~~~~~~~~~~ -.. NOTE:: - This section is a work in progress. +The home server MUST respond to this with either new credentials, the next +stage of the login process, or a standard error response. -When a client logs in, they may have a list of rooms which they have already -joined. These rooms may also have a list of events associated with them. The -purpose of 'syncing' is to present the current room and event information in a -convenient, compact manner. The events returned are not limited to room events; -presence events will also be returned. A single syncing API is provided: +Email-based (url) +~~~~~~~~~~~~~~~~~ +:Type: + ``m.login.email.url`` +:Description: + Login is supported by clicking on a URL in an email. This login consists of + multiple requests. - - |initialSync|_ : A global sync which will present room and event information - for all rooms the user has joined. +To respond to this type, reply with:: -.. TODO-spec room-scoped initial sync - - |/rooms//initialSync|_ : A sync scoped to a single room. Presents - room and event information for this room only. - - Room-scoped initial sync is Very Tricky because typically people would - want to sync the room then listen for any new content from that point - onwards. The event stream cannot do this for a single room currently. - As a result, commenting room-scoped initial sync at this time. + { + "type": "m.login.email.url", + "user": "", + "email": "" + } -The |initialSync|_ API contains the following keys: +After validating the email address, the home server MUST send an email +containing an authentication URL and return:: -``presence`` - Description: - Contains a list of presence information for users the client is interested - in. - Format: - A JSON array of ``m.presence`` events. + { + "type": "m.login.email.url", + "session": "" + } -``end`` - Description: - Contains an event stream token which can be used with the `Event Stream`_. - Format: - A string containing the event stream token. +The email contains a URL which must be clicked. After it has been clicked, the +client should perform another request:: -``rooms`` - Description: - Contains a list of room information for all rooms the client has joined, - and limited room information on rooms the client has been invited to. - Format: - A JSON array containing Room Information JSON objects. + { + "type": "m.login.email.url", + "session": "" + } -Room Information: - Description: - Contains all state events for the room, along with a limited amount of - the most recent non-state events, configured via the ``limit`` query - parameter. Also contains additional keys with room metadata, such as the - ``room_id`` and the client's ``membership`` to the room. - Format: - A JSON object with the following keys: - ``room_id`` - A string containing the ID of the room being described. - ``membership`` - A string representing the client's membership status in this room. - ``messages`` - An event stream JSON object containing a ``chunk`` of recent non-state - events, along with an ``end`` token. *NB: The name of this key will be - changed in a later version.* - ``state`` - A JSON array containing all the current state events for this room. +The home server MUST respond to this with either new credentials, the next +stage of the login process, or a standard error response. -Getting events for a room -~~~~~~~~~~~~~~~~~~~~~~~~~ -There are several APIs provided to ``GET`` events for a room: +A common client implementation will be to periodically poll until the link is +clicked. If the link has not been visited yet, a standard error response with +an errcode of ``M_LOGIN_EMAIL_URL_NOT_YET`` should be returned. -``/rooms//state//`` - Description: - Get the state event identified. - Response format: - A JSON object representing the state event **content**. - Example: - ``/rooms/!room:domain.com/state/m.room.name`` returns ``{ "name": "Room name" }`` -|/rooms//state|_ - Description: - Get all state events for a room. - Response format: - ``[ { state event }, { state event }, ... ]`` - Example: - TODO-doc +Email-based (identity server) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:Type: + ``m.login.email.identity`` +:Description: + Login is supported by authorising an email address with an identity server. +Prior to submitting this, the client should authenticate with an identity +server. After authenticating, the session information should be submitted to +the home server. -|/rooms//members|_ - Description: - Get all ``m.room.member`` state events. - Response format: - ``{ "start": "", "end": "", "chunk": [ { m.room.member event }, ... ] }`` - Example: - TODO-doc +To respond to this type, reply with:: -|/rooms//messages|_ - Description: - Get all ``m.room.message`` and ``m.room.member`` events. This API supports - pagination using ``from`` and ``to`` query parameters, coupled with the - ``start`` and ``end`` tokens from an |initialSync|_ API. - Response format: - ``{ "start": "", "end": "" }`` - Example: - TODO-doc + { + "type": "m.login.email.identity", + "threepidCreds": [ + { + "sid": "", + "clientSecret": "", + "idServer": "" + } + ] + } -|/rooms//initialSync|_ - Description: - Get all relevant events for a room. This includes state events, paginated - non-state events and presence events. - Response format: - `` { TODO-doc } `` - Example: - TODO-doc -Redactions -~~~~~~~~~~ -Since events are extensible it is possible for malicious users and/or servers -to add keys that are, for example offensive or illegal. Since some events -cannot be simply deleted, e.g. membership events, we instead 'redact' events. -This involves removing all keys from an event that are not required by the -protocol. This stripped down event is thereafter returned anytime a client or -remote server requests it. -Events that have been redacted include a ``redacted_because`` key whose value -is the event that caused it to be redacted, which may include a reason. +N-Factor Authentication +~~~~~~~~~~~~~~~~~~~~~~~ +Multiple login stages can be combined to create N-factor authentication during +login. -Redacting an event cannot be undone, allowing server owners to delete the -offending content from the databases. +This can be achieved by responding with the ``next`` login type on completion +of a previous login stage:: -Currently, only room admins can redact events by sending a ``m.room.redaction`` -event, but server admins also need to be able to redact events by a similar -mechanism. + { + "next": "" + } -Upon receipt of a redaction event, the server should strip off any keys not in -the following list: +If a home server implements N-factor authentication, it MUST respond with all +``stages`` when initially queried for their login requirements:: - - ``event_id`` - - ``type`` - - ``room_id`` - - ``user_id`` - - ``state_key`` - - ``prev_state`` - - ``content`` + { + "type": "<1st login type>", + "stages": [ <1st login type>, <2nd login type>, ... , ] + } -The content object should also be stripped of all keys, unless it is one of -one of the following event types: +This can be represented conceptually as:: - - ``m.room.member`` allows key ``membership`` - - ``m.room.create`` allows key ``creator`` - - ``m.room.join_rules`` allows key ``join_rule`` - - ``m.room.power_levels`` allows keys that are user ids or ``default`` - - ``m.room.add_state_level`` allows key ``level`` - - ``m.room.send_event_level`` allows key ``level`` - - ``m.room.ops_levels`` allows keys ``kick_level``, ``ban_level`` - and ``redact_level`` - - ``m.room.aliases`` allows key ``aliases`` + _______________________ + | Login Stage 1 | + | type: "" | + | ___________________ | + | |_Request_1_________| | <-- Returns "session" key which is used throughout. + | ___________________ | + | |_Request_2_________| | <-- Returns a "next" value of "login type2" + |_______________________| + | + | + _________V_____________ + | Login Stage 2 | + | type: "" | + | ___________________ | + | |_Request_1_________| | + | ___________________ | + | |_Request_2_________| | + | ___________________ | + | |_Request_3_________| | <-- Returns a "next" value of "login type3" + |_______________________| + | + | + _________V_____________ + | Login Stage 3 | + | type: "" | + | ___________________ | + | |_Request_1_________| | <-- Returns user credentials + |_______________________| -The redaction event should be added under the key ``redacted_because``. +Fallback +~~~~~~~~ +Clients cannot be expected to be able to know how to process every single login +type. If a client determines it does not know how to handle a given login type, +it should request a login fallback page:: + GET matrix/client/api/v1/login/fallback + +This MUST return an HTML page which can perform the entire login process. -When a client receives a redaction event it should change the redacted event -in the same way a server does. Presence ~~~~~~~~ @@ -1050,36 +1214,6 @@ have to wait in milliseconds before they can try again. homeserver come up with their own idea, causing totally unpredictable performance over federated rooms? -End-to-End Encryption -~~~~~~~~~~~~~~~~~~~~~ - -.. TODO-doc - - Why is this needed. - - Overview of process - - Implementation - -Content repository ------------------- -.. NOTE:: - This section is a work in progress. - -.. TODO-spec - - path to upload - - format for thumbnail paths, mention what it is protecting against. - - content size limit and associated M_ERROR. - - -Address book repository ------------------------ -.. NOTE:: - This section is a work in progress. - -.. TODO-spec - - format: POST(?) wodges of json, some possible processing, then return wodges of json on GET. - - processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of - contacts), etc. - - Standard json format for contacts? Piggy back off vcards? - .. Links through the external API docs are below .. ============================================= diff --git a/specification/10_events.rst b/specification/20_events.rst similarity index 89% rename from specification/10_events.rst rename to specification/20_events.rst index 20dccb6b..84a40a11 100644 --- a/specification/10_events.rst +++ b/specification/20_events.rst @@ -250,7 +250,7 @@ prefixed with ``m.`` Summary: A message. Type: - Non-state event + Message event JSON format: ``{ "msgtype": "string" }`` Example: @@ -266,8 +266,9 @@ prefixed with ``m.`` ``m.room.message.feedback`` Summary: A receipt for a message. + N.B. not implemented in Synapse, and superceded in v2 CS API by the 'relates_to' event field. Type: - Non-state event + Message event JSON format: ``{ "type": "enum [ delivered|read ]", "target_event_id": "string" }`` Example: @@ -283,7 +284,7 @@ prefixed with ``m.`` Summary: Indicates a previous event has been redacted. Type: - Non-state event + Message event JSON format: ``{ "reason": "string" }`` Description: @@ -292,7 +293,7 @@ prefixed with ``m.`` admins to remove offensive or illegal content that may have been attached to any event. This cannot be undone, allowing server owners to physically delete the offending data. There is also a concept of a moderator hiding a - non-state event, which can be undone, but cannot be applied to state + message event, which can be undone, but cannot be applied to state events. The event that has been redacted is specified in the ``redacts`` event level key. @@ -447,6 +448,48 @@ outlined below: .. TODO-spec Make the definitions "inherit" from FileInfo where necessary... +Presence Events (v1) +~~~~~~~~~~~~~~~~~~~~ + +``m.presence`` + Summary: + Informs you of a user's presence state changes. + Type: + Presence event + JSON format:: + { "displayname": "utf-8 string", + "avatar_url": "url", + "presence": "enum [ online|unavailable|offline|free_for_chat|hidden ]", + "last_active_ago": "milliseconds" } + Example: + ``{ "displayname": "Matthew", "avatar_url": "mxc://domain/id", "presence": "online", "last_active_ago": 10000 }`` + Description: + Each user has the concept of presence information. This encodes the + "availability" of that user, suitable for display on other user's clients. + This is transmitted as an ``m.presence`` event and is one of the few events + which are sent *outside the context of a room*. The basic piece of presence + information is represented by the ``presence`` key, which is an enum of one + of the following: + + - ``online`` : The default state when the user is connected to an event + stream. + - ``unavailable`` : The user is not reachable at this time. + - ``offline`` : The user is not connected to an event stream. + - ``free_for_chat`` : The user is generally willing to receive messages + moreso than default. + - ``hidden`` : Behaves as offline, but allows the user to see the client + state anyway and generally interact with client features. (Not yet + implemented in synapse). + + In addition, the server maintains a timestamp of the last time it saw a + pro-active event from the user; either sending a message to a room, or + changing presence state from a lower to a higher level of availability + (thus: changing state from ``unavailable`` to ``online`` counts as a + proactive event, whereas in the other direction it will not). This timestamp + is presented via a key called ``last_active_ago``, which gives the relative + number of milliseconds since the message is generated/emitted that the user + was last seen active. + Events on Change of Profile Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -457,7 +500,7 @@ values. This change is conveyed using two separate mechanisms: - a ``m.room.member`` event is sent to every room the user is a member of, to update the ``displayname`` and ``avatar_url``. - - a presence status update is sent, again containing the new values of the + - a ``m.presence`` presence status update is sent, again containing the new values of the ``displayname`` and ``avatar_url`` keys, in addition to the required ``presence`` key containing the current presence state of the user. diff --git a/specification/40_application_service_api.rst b/specification/25_application_service_api.rst similarity index 100% rename from specification/40_application_service_api.rst rename to specification/25_application_service_api.rst diff --git a/specification/11_event_signing.rst b/specification/31_event_signing.rst similarity index 100% rename from specification/11_event_signing.rst rename to specification/31_event_signing.rst diff --git a/drafts/push_overview.rst b/specification/42_push_overview.rst similarity index 98% rename from drafts/push_overview.rst rename to specification/42_push_overview.rst index c84fc8d2..913b149f 100644 --- a/drafts/push_overview.rst +++ b/specification/42_push_overview.rst @@ -1,5 +1,5 @@ -Push Notifications -================== +Push Notifications Overview +=========================== :: @@ -72,3 +72,4 @@ Push Gateway For information on the client-server API for setting pushers and push rules, see the Client Server API section. For more information on the format of HTTP notifications, see the HTTP Notification Protocol section. + diff --git a/drafts/push_csapi.rst b/specification/43_push_cs_api.rst similarity index 99% rename from drafts/push_csapi.rst rename to specification/43_push_cs_api.rst index c5d9f280..3f235010 100644 --- a/drafts/push_csapi.rst +++ b/specification/43_push_cs_api.rst @@ -1,5 +1,5 @@ -Push Notifications -================== +Push Notifications HTTP API +=========================== Pushers ------- @@ -415,3 +415,5 @@ Rules can be enabled or disabled with a PUT operation to the 'enabled' component beneath the rule's URI with a content of 'true' or 'false':: curl -X PUT -H "Content-Type: application/json" -d 'false' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org/enabled?access_token=123456" + + diff --git a/drafts/push_pgwapi.rst b/specification/44_push_push_gw_api.rst similarity index 99% rename from drafts/push_pgwapi.rst rename to specification/44_push_push_gw_api.rst index d3da9ab2..8890ce5a 100644 --- a/drafts/push_pgwapi.rst +++ b/specification/44_push_push_gw_api.rst @@ -140,3 +140,4 @@ gateway). However, Matrix strongly recommends: * That APNS push gateways do not attempt to wait for errors from the APNS gateway before returning and instead to store failures and return 'rejected' responses next time that pushkey is used. + diff --git a/drafts/typing_notifications.rst b/specification/45_typing_notifications.rst similarity index 99% rename from drafts/typing_notifications.rst rename to specification/45_typing_notifications.rst index 048eba98..4c7d9b72 100644 --- a/drafts/typing_notifications.rst +++ b/specification/45_typing_notifications.rst @@ -55,3 +55,4 @@ originating HSes to ensure they eventually send "stop" notifications. ((This will eventually need addressing, as part of the wider typing/presence timer addition work)) +