From 5f97e28c9bffc7aa268f8c45dd568bae25724a20 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 30 Dec 2014 03:00:37 +0000 Subject: [PATCH 001/526] initial formal proposal for the AS API --- drafts/applications_services.rst | 170 +++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 drafts/applications_services.rst diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst new file mode 100644 index 00000000..cee29212 --- /dev/null +++ b/drafts/applications_services.rst @@ -0,0 +1,170 @@ +Application Services +==================== + +Overview +======== + +Application services provide a way of implementing custom serverside functionality +on top of Matrix without the complexity of implementing the full federation API. +By acting as a trusted service logically located behind an existing homeserver, +Application services are decoupled from: + +* Signing or validating federated traffic or conversation history +* Validating authorisation constraints on federated traffic +* Managing routing or retry schemes to the rest of the Matrix federation + +As such, developers can focus entirely on implementing application logic rather +than being concerned with the details of managing Matrix federation. + +Features available to application services include: + +* Privileged subscription to any events available to the homeserver +* Synthesising virtual users +* Synthesising virtual rooms +* Injecting message history for virtual rooms + +Features not provided by application services include: + +* Intercepting and filtering/modifying message or behaviour within a room + (this is a job for a Policy Server, as it requires a single logical focal + point for messages in order to consistently apply the custom business logic) + +Example use cases for application services include: + +* Exposing existing communication services in Matrix + + * Gateways to/from standards-based protocols (SIP, XMPP, IRC, RCS (MSRP), + SIMPLE, Lync, etc) + * Gateways to/from closed services (e.g. WhatsApp) + * Gateways could be architected as: + + * Act as a virtual client on the non-Matrix network + (e.g. connect as multiple virtual clients to an IRC or XMPP server) + * Act as a server on the non-Matrix network + (e.g. speak s2s XMPP federation, or IRC link protocol) + * Act as an application service on the non-Matrix network + (e.g. link up as IRC services, or an XMPP component) + * Exposing a non-Matrix client interface listener from the AS + (e.g. listen on port 6667 for IRC clients, or port 5222 for XMPP clients) + + * Bridging existing APIs into Matrix + + * e.g. SMS/MMS aggregator APIs + * Domain-specific APIs such as SABRE + + * Integrating more exotic content into Matrix + + * e.g. MIDI<->Matrix gateway/bridge + * 3D world <-> Matrix bridge + + * Application services: + + * VoIP Conference services + * Text-to-speech and Speech-to-text services + * Signal processing + * IVR + * Server-machine translation + * Censorship service + * Multi-User Gaming (Dungeons etc) + * Other "constrained worlds" (e.g. 3D geometry representations) + + * applying physics to a 3D world on the serverside + + * (applying gravity and friction and air resistance... collision detection) + * domain-specific merge conflict resolution of events + + * Payment style transactional usecases with transactional guarantees + +Architecture Outline +==================== + +The application service registers with its host homeserver to offer its services. + +In the registration process, the AS provides: + + * Credentials to identify itself as an approved application service for that HS + * Details of the namespaces of users and rooms the AS is acting on behalf of and + "subscribing to" + * A URL base for receiving requests from the HS (as the AS is a server, + implementers expect to receive data via inbound requests rather than + long-poll outbound requests) + +On HS handling events to unknown users: + + * If the HS receives an event for an unknown user who is in the namespace delegated to + the AS, then the HS queries the AS for the profile of that user. If the AS + confirms the existence of that user (from its perspective), then the HS + creates an account to represent the virtual user. + * The namespace of virtual user accounts could conform to a structure like + @.irc.freenode.Arathorn:matrix.org or similar. + * The AS can preprovision virtual users using the existing CS API rather than + lazy-loading them in this manner. + +On HS handling events to unknown rooms: + + * If the HS receives an invite to an unknown room which is in the namespace + delegated to the AS, then the HS queries the AS for the existence of that room. + If the AS confirms its existence (from its perspective), then the HS creates + the room. + * The initial state of the room may be populated by the AS by querying an + initialSync API (probably a subset of the CS initialSync API, to reuse the + same pattern for the equivalent function). As messages have to be signed + from the point of m.room.create, we will not be able to back-populate + arbitrary history for rooms which are lazy-created in this manner, and instead + have to chose the amount of history to be synchronised into the AS as a one-off. + * If exposing arbitrary history is required, then either the room history must be + preemptively provisioned in the HS by the AS via the CS API (TODO: meaning the + CS API needs to support massaged timestamps), or the HS must delegate conversation + storage entirely to the AS using a Storage API (not defined here) which allows + the existing conversation store to back the HS, complete with all necessary + Matrix metadata (e.g. hashes, signatures, federation DAG, etc). + +On HS handling events to existing users and rooms: + + * If the HS receives an event for a user or room that already exist (either + provisioned by the AS or by normal client interactions), then the message + is handled as normal. + * Events in the namespaces of rooms and users that the AS has subscribed to + are pushed to the AS using the same pattern as the federation API (without + any of the encryption or federation metadata). TODO: are they linearised? + +On AS relaying events from unknown-to-HS users: + + * AS injects the event to the HS using the CS API, irrespective of whether the + target user or room is known to the HS or not. If the HS doesn't recognise + the target it goes through the same lazy-load provisioning as per above. + * The reason for not using a subset of the federation API here is because it + allows AS developers to reuse existing CS SDKs and benefit from the more + meaningful error handling of the CS API. The sending user ID must be + explicitly specified, as it cannot be inferred from the access_token, which + will be the same for all AS requests. + + * TODO: or do we maintain a separate access_token mapping? It seems like + unnecessary overhead for the AS developer; easier to just use a single + privileged access_token and just track which userid is emitting events? + +On AS relaying events in unknown-to-HS rooms: + + * See above. + +On AS publishing aliases for virtual rooms: + + * AS uses the normal alias management API to preemptively create/delete public + directory entries for aliases for virtual rooms provided by the AS. + * In order to create these aliases, the underlying room ID must also exist, so + at least the m.room.create of that room must also be prepopulated. It seems + sensible to prepopulate the required initial state and history of the room to + avoid a two-phase prepopulation process. + +On unregistering the AS from the HS: + + * An AS must tell the HS when it is going offline in order to stop receiving + requests from the HS. It does this by hitting an API on the HS. + +Extensions to CS API +==================== + + * Ability to assert the identity of the virtual user for all methods. + * Ability to massage timestamps when prepopulating historical state and + messages of virtual rooms. + * Ability to delete aliases (including from the directory) as well as create them. \ No newline at end of file From f91f569f3007007ea2c07718f8cde7f562f0d868 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 30 Dec 2014 03:10:11 +0000 Subject: [PATCH 002/526] explain why the @.irc.freenode style prefixes are a good idae --- drafts/applications_services.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst index cee29212..ca012a81 100644 --- a/drafts/applications_services.rst +++ b/drafts/applications_services.rst @@ -95,8 +95,10 @@ On HS handling events to unknown users: the AS, then the HS queries the AS for the profile of that user. If the AS confirms the existence of that user (from its perspective), then the HS creates an account to represent the virtual user. - * The namespace of virtual user accounts could conform to a structure like - @.irc.freenode.Arathorn:matrix.org or similar. + * The namespace of virtual user accounts should conform to a structure like + @.irc.freenode.Arathorn:matrix.org. This lets Matrix users communicate with + foreign users who are not yet mapped into Matrix via 3PID mappings or through + an existing non-virtual Matrix user by trying to talk to them via a gateway. * The AS can preprovision virtual users using the existing CS API rather than lazy-loading them in this manner. From 01bc9eb4a1dba776d14f5048e339ecfc85805f07 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 5 Jan 2015 15:16:10 +0000 Subject: [PATCH 003/526] some forgotten AS use cases --- drafts/applications_services.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst index ca012a81..0333a89a 100644 --- a/drafts/applications_services.rst +++ b/drafts/applications_services.rst @@ -59,6 +59,8 @@ Example use cases for application services include: * Application services: + * Search engines (e.g. elasticsearch search indices) + * Notification systems (e.g. send custom pushes for various hooks) * VoIP Conference services * Text-to-speech and Speech-to-text services * Signal processing From deb24aecf40cb7e6e7fdd4fad8320820ae117c4d Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 5 Jan 2015 18:42:22 +0000 Subject: [PATCH 004/526] some thoughts on invisibility --- drafts/applications_services.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst index 0333a89a..45bb8276 100644 --- a/drafts/applications_services.rst +++ b/drafts/applications_services.rst @@ -159,12 +159,28 @@ On AS publishing aliases for virtual rooms: at least the m.room.create of that room must also be prepopulated. It seems sensible to prepopulate the required initial state and history of the room to avoid a two-phase prepopulation process. - + On unregistering the AS from the HS: * An AS must tell the HS when it is going offline in order to stop receiving requests from the HS. It does this by hitting an API on the HS. +AS Visibility: + + * If an AS needs to sniff events in a room in order to operate on them (e.g. + to act as a search engine) but not inject traffic into the room, it should + do so by subscribing to the relevant events without actually joining the room. + * If the AS needs to participate in the room as a virtual user (e.g. an IVR + service, or a bot, or a gatewayed virtual user), it should join the room + normally. + * There are rare instances where an AS may wish to participate in a room + (including inserting messages), but be hidden from the room list - e.g. a + conferencing server focus bot may wish to join many rooms as the focus and + both listen to VoIP setups and inject its own VoIP answers, without ever + being physically seen in the room. In this scenario, the user should set + its presence to 'invisible', a state that HSes should only allow AS-authed + users to set. + Extensions to CS API ==================== From 259cdd253a001f582a7029a68730155bfcfade16 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 6 Jan 2015 11:54:02 +0000 Subject: [PATCH 005/526] fix plural --- drafts/applications_services.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst index 45bb8276..07a70ce2 100644 --- a/drafts/applications_services.rst +++ b/drafts/applications_services.rst @@ -125,7 +125,7 @@ On HS handling events to unknown rooms: On HS handling events to existing users and rooms: - * If the HS receives an event for a user or room that already exist (either + * If the HS receives an event for a user or room that already exists (either provisioned by the AS or by normal client interactions), then the message is handled as normal. * Events in the namespaces of rooms and users that the AS has subscribed to From b557e71121a420270d5f79ab124596d0d627223b Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 13 Jan 2015 16:25:28 +0000 Subject: [PATCH 006/526] update AS API to merge in feedback from the pull request --- ..._services.rst => application_services.rst} | 63 ++++++++++++++++--- 1 file changed, 53 insertions(+), 10 deletions(-) rename drafts/{applications_services.rst => application_services.rst} (70%) diff --git a/drafts/applications_services.rst b/drafts/application_services.rst similarity index 70% rename from drafts/applications_services.rst rename to drafts/application_services.rst index 07a70ce2..9ec271ec 100644 --- a/drafts/applications_services.rst +++ b/drafts/application_services.rst @@ -87,6 +87,11 @@ In the registration process, the AS provides: * Credentials to identify itself as an approved application service for that HS * Details of the namespaces of users and rooms the AS is acting on behalf of and "subscribing to" + * Namespaces are defined as a list of regexps against which to match room aliases, + room IDs, and user IDs. + * There is overlap between selecting events via the csv2 Filter API and subscribing + to events here - perhaps subscription involves passing a filter token into the + registration API. * A URL base for receiving requests from the HS (as the AS is a server, implementers expect to receive data via inbound requests rather than long-poll outbound requests) @@ -101,8 +106,13 @@ On HS handling events to unknown users: @.irc.freenode.Arathorn:matrix.org. This lets Matrix users communicate with foreign users who are not yet mapped into Matrix via 3PID mappings or through an existing non-virtual Matrix user by trying to talk to them via a gateway. - * The AS can preprovision virtual users using the existing CS API rather than - lazy-loading them in this manner. + * The AS can alternatively preprovision virtual users using the existing CS API + rather than lazy-loading them in this manner. + * The AS may want to link the matrix ID of the sender through to their 3PID in + the remote ecosystem. E.g. a message sent from @matthew:matrix.org may wish + to originate from Arathorn on irc.freenode.net in the case of an IRC bridge. + It's left as an AS implementation detail as to how the user should authorise + the AS to act on its behalf. On HS handling events to unknown rooms: @@ -116,12 +126,19 @@ On HS handling events to unknown rooms: from the point of m.room.create, we will not be able to back-populate arbitrary history for rooms which are lazy-created in this manner, and instead have to chose the amount of history to be synchronised into the AS as a one-off. - * If exposing arbitrary history is required, then either the room history must be - preemptively provisioned in the HS by the AS via the CS API (TODO: meaning the - CS API needs to support massaged timestamps), or the HS must delegate conversation - storage entirely to the AS using a Storage API (not defined here) which allows - the existing conversation store to back the HS, complete with all necessary - Matrix metadata (e.g. hashes, signatures, federation DAG, etc). + * If exposing arbitrary history is required, then: + + * either: the room history must be preemptively provisioned in the HS by the AS via + the CS API (TODO: meaning the CS API needs to support massaged + timestamps), resulting in conversation history being replicated between + the HS and the source store. + * or: the HS must delegate conversation storage entirely to the + AS using a Storage API (not defined here) which allows the existing + conversation store to back the HS, complete with all necessary Matrix + metadata (e.g. hashes, signatures, federation DAG, etc). This obviously + increases the burden of implementing an AS considerably, but is the only + option if the implementer wants to avoid duplicating conversation history + between the external data source and the HS. On HS handling events to existing users and rooms: @@ -130,7 +147,20 @@ On HS handling events to existing users and rooms: is handled as normal. * Events in the namespaces of rooms and users that the AS has subscribed to are pushed to the AS using the same pattern as the federation API (without - any of the encryption or federation metadata). TODO: are they linearised? + any of the encryption or federation metadata). This serves precisely the + same purpose as the CS event stream and has the same data flow semantics + (and indeed an AS implementer could chose to use the CS event stream instead) + + * Events are linearised to avoid the AS having to handle the complexity of + linearisation, and because if linearisation is good enough for CS, it + should be good enough for AS. Should the AS require non-linearised events + from Matrix, it should implement the federation API rather than the AS API + instead. + * HS->AS event pushes are retried for reliability with sequence numbers + (or logical timestamping?) to presereve the linearisation order and ensure + a reliable event stream. + * Clustered HSes must linearise just as they do for the CS API. Clustered + ASes must loadbalance the inbound stream across the cluster as required. On AS relaying events from unknown-to-HS users: @@ -146,6 +176,12 @@ On AS relaying events from unknown-to-HS users: * TODO: or do we maintain a separate access_token mapping? It seems like unnecessary overhead for the AS developer; easier to just use a single privileged access_token and just track which userid is emitting events? + * If the AS is spoofing the identity of a real (not virtual) matrix user, + we should actually let them log themselves in via OAuth2 to give permission + to the AS to act on their behalf. + * We can't auth gatewayed virtual users from 3rd party systems who are being + relayed into Matrix, as the relaying is happening whether the user likes it + or not. Therefore we do need to be able to spoof sender ID for virtual users. On AS relaying events in unknown-to-HS rooms: @@ -180,11 +216,18 @@ AS Visibility: being physically seen in the room. In this scenario, the user should set its presence to 'invisible', a state that HSes should only allow AS-authed users to set. + +E2E Encryption + + * The AS obviously has no visibility to E2E encrypted messages, unless it is + explicitly added to an encrypted room and participates in the group chat + itself. Extensions to CS API ==================== * Ability to assert the identity of the virtual user for all methods. * Ability to massage timestamps when prepopulating historical state and - messages of virtual rooms. + messages of virtual rooms (either by overriding origin_server_ts (preferred) or + adding an as_ts which we expect clients to honour) * Ability to delete aliases (including from the directory) as well as create them. \ No newline at end of file From 540b3a41ff69cbd0c7a7ca49432ef12e274f01f4 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 14 Jan 2015 15:15:40 +0000 Subject: [PATCH 007/526] RSTify; typos --- drafts/application_services.rst | 267 ++++++++++++++++---------------- 1 file changed, 132 insertions(+), 135 deletions(-) diff --git a/drafts/application_services.rst b/drafts/application_services.rst index 9ec271ec..2c73107f 100644 --- a/drafts/application_services.rst +++ b/drafts/application_services.rst @@ -33,32 +33,29 @@ Example use cases for application services include: * Exposing existing communication services in Matrix - * Gateways to/from standards-based protocols (SIP, XMPP, IRC, RCS (MSRP), - SIMPLE, Lync, etc) - * Gateways to/from closed services (e.g. WhatsApp) - * Gateways could be architected as: - - * Act as a virtual client on the non-Matrix network - (e.g. connect as multiple virtual clients to an IRC or XMPP server) - * Act as a server on the non-Matrix network - (e.g. speak s2s XMPP federation, or IRC link protocol) - * Act as an application service on the non-Matrix network - (e.g. link up as IRC services, or an XMPP component) - * Exposing a non-Matrix client interface listener from the AS - (e.g. listen on port 6667 for IRC clients, or port 5222 for XMPP clients) - - * Bridging existing APIs into Matrix - + * Gateways to/from standards-based protocols (SIP, XMPP, IRC, RCS (MSRP), SIMPLE, Lync, etc) + * Gateways to/from closed services (e.g. WhatsApp) + * Gateways could be architected as: + + * Act as a virtual client on the non-Matrix network + (e.g. connect as multiple virtual clients to an IRC or XMPP server) + * Act as a server on the non-Matrix network + (e.g. speak s2s XMPP federation, or IRC link protocol) + * Act as an application service on the non-Matrix network + (e.g. link up as IRC services, or an XMPP component) + * Exposing a non-Matrix client interface listener from the AS + (e.g. listen on port 6667 for IRC clients, or port 5222 for XMPP clients) + + +* Bridging existing APIs into Matrix * e.g. SMS/MMS aggregator APIs * Domain-specific APIs such as SABRE - * Integrating more exotic content into Matrix - +* Integrating more exotic content into Matrix * e.g. MIDI<->Matrix gateway/bridge * 3D world <-> Matrix bridge - * Application services: - +* Application services: * Search engines (e.g. elasticsearch search indices) * Notification systems (e.g. send custom pushes for various hooks) * VoIP Conference services @@ -84,150 +81,150 @@ The application service registers with its host homeserver to offer its services In the registration process, the AS provides: - * Credentials to identify itself as an approved application service for that HS - * Details of the namespaces of users and rooms the AS is acting on behalf of and - "subscribing to" - * Namespaces are defined as a list of regexps against which to match room aliases, - room IDs, and user IDs. - * There is overlap between selecting events via the csv2 Filter API and subscribing - to events here - perhaps subscription involves passing a filter token into the - registration API. - * A URL base for receiving requests from the HS (as the AS is a server, - implementers expect to receive data via inbound requests rather than - long-poll outbound requests) +* Credentials to identify itself as an approved application service for that HS +* Details of the namespaces of users and rooms the AS is acting on behalf of and + "subscribing to" +* Namespaces are defined as a list of regexps against which to match room aliases, + room IDs, and user IDs. +* There is overlap between selecting events via the csv2 Filter API and subscribing + to events here - perhaps subscription involves passing a filter token into the + registration API. +* A URL base for receiving requests from the HS (as the AS is a server, + implementers expect to receive data via inbound requests rather than + long-poll outbound requests) On HS handling events to unknown users: - * If the HS receives an event for an unknown user who is in the namespace delegated to - the AS, then the HS queries the AS for the profile of that user. If the AS - confirms the existence of that user (from its perspective), then the HS - creates an account to represent the virtual user. - * The namespace of virtual user accounts should conform to a structure like - @.irc.freenode.Arathorn:matrix.org. This lets Matrix users communicate with - foreign users who are not yet mapped into Matrix via 3PID mappings or through - an existing non-virtual Matrix user by trying to talk to them via a gateway. - * The AS can alternatively preprovision virtual users using the existing CS API - rather than lazy-loading them in this manner. - * The AS may want to link the matrix ID of the sender through to their 3PID in - the remote ecosystem. E.g. a message sent from @matthew:matrix.org may wish - to originate from Arathorn on irc.freenode.net in the case of an IRC bridge. - It's left as an AS implementation detail as to how the user should authorise - the AS to act on its behalf. +* If the HS receives an event for an unknown user who is in the namespace delegated to + the AS, then the HS queries the AS for the profile of that user. If the AS + confirms the existence of that user (from its perspective), then the HS + creates an account to represent the virtual user. +* The namespace of virtual user accounts should conform to a structure like + ``@.irc.freenode.Arathorn:matrix.org``. This lets Matrix users communicate with + foreign users who are not yet mapped into Matrix via 3PID mappings or through + an existing non-virtual Matrix user by trying to talk to them via a gateway. +* The AS can alternatively preprovision virtual users using the existing CS API + rather than lazy-loading them in this manner. +* The AS may want to link the matrix ID of the sender through to their 3PID in + the remote ecosystem. E.g. a message sent from ``@matthew:matrix.org`` may wish + to originate from Arathorn on irc.freenode.net in the case of an IRC bridge. + It's left as an AS implementation detail as to how the user should authorise + the AS to act on its behalf. On HS handling events to unknown rooms: - * If the HS receives an invite to an unknown room which is in the namespace - delegated to the AS, then the HS queries the AS for the existence of that room. - If the AS confirms its existence (from its perspective), then the HS creates - the room. - * The initial state of the room may be populated by the AS by querying an - initialSync API (probably a subset of the CS initialSync API, to reuse the - same pattern for the equivalent function). As messages have to be signed - from the point of m.room.create, we will not be able to back-populate - arbitrary history for rooms which are lazy-created in this manner, and instead - have to chose the amount of history to be synchronised into the AS as a one-off. - * If exposing arbitrary history is required, then: +* If the HS receives an invite to an unknown room which is in the namespace + delegated to the AS, then the HS queries the AS for the existence of that room. + If the AS confirms its existence (from its perspective), then the HS creates + the room. +* The initial state of the room may be populated by the AS by querying an + initialSync API (probably a subset of the CS initialSync API, to reuse the + same pattern for the equivalent function). As messages have to be signed + from the point of ``m.room.create``, we will not be able to back-populate + arbitrary history for rooms which are lazy-created in this manner, and instead + have to chose the amount of history to be synchronised into the AS as a one-off. +* If exposing arbitrary history is required, then: - * either: the room history must be preemptively provisioned in the HS by the AS via - the CS API (TODO: meaning the CS API needs to support massaged - timestamps), resulting in conversation history being replicated between - the HS and the source store. - * or: the HS must delegate conversation storage entirely to the - AS using a Storage API (not defined here) which allows the existing - conversation store to back the HS, complete with all necessary Matrix - metadata (e.g. hashes, signatures, federation DAG, etc). This obviously - increases the burden of implementing an AS considerably, but is the only - option if the implementer wants to avoid duplicating conversation history - between the external data source and the HS. + * either: the room history must be preemptively provisioned in the HS by the AS via + the CS API (TODO: meaning the CS API needs to support massaged + timestamps), resulting in conversation history being replicated between + the HS and the source store. + * or: the HS must delegate conversation storage entirely to the + AS using a Storage API (not defined here) which allows the existing + conversation store to back the HS, complete with all necessary Matrix + metadata (e.g. hashes, signatures, federation DAG, etc). This obviously + increases the burden of implementing an AS considerably, but is the only + option if the implementer wants to avoid duplicating conversation history + between the external data source and the HS. On HS handling events to existing users and rooms: - * If the HS receives an event for a user or room that already exists (either - provisioned by the AS or by normal client interactions), then the message - is handled as normal. - * Events in the namespaces of rooms and users that the AS has subscribed to - are pushed to the AS using the same pattern as the federation API (without - any of the encryption or federation metadata). This serves precisely the - same purpose as the CS event stream and has the same data flow semantics - (and indeed an AS implementer could chose to use the CS event stream instead) - - * Events are linearised to avoid the AS having to handle the complexity of - linearisation, and because if linearisation is good enough for CS, it - should be good enough for AS. Should the AS require non-linearised events - from Matrix, it should implement the federation API rather than the AS API - instead. - * HS->AS event pushes are retried for reliability with sequence numbers - (or logical timestamping?) to presereve the linearisation order and ensure - a reliable event stream. - * Clustered HSes must linearise just as they do for the CS API. Clustered - ASes must loadbalance the inbound stream across the cluster as required. +* If the HS receives an event for a user or room that already exists (either + provisioned by the AS or by normal client interactions), then the message + is handled as normal. +* Events in the namespaces of rooms and users that the AS has subscribed to + are pushed to the AS using the same pattern as the federation API (without + any of the encryption or federation metadata). This serves precisely the + same purpose as the CS event stream and has the same data flow semantics + (and indeed an AS implementer could chose to use the CS event stream instead) + + * Events are linearised to avoid the AS having to handle the complexity of + linearisation, and because if linearisation is good enough for CS, it + should be good enough for AS. Should the AS require non-linearised events + from Matrix, it should implement the federation API rather than the AS API + instead. + * HS->AS event pushes are retried for reliability with sequence numbers + (or logical timestamping?) to preserve the linearisation order and ensure + a reliable event stream. + * Clustered HSes must linearise just as they do for the CS API. Clustered + ASes must loadbalance the inbound stream across the cluster as required. On AS relaying events from unknown-to-HS users: - * AS injects the event to the HS using the CS API, irrespective of whether the - target user or room is known to the HS or not. If the HS doesn't recognise - the target it goes through the same lazy-load provisioning as per above. - * The reason for not using a subset of the federation API here is because it - allows AS developers to reuse existing CS SDKs and benefit from the more - meaningful error handling of the CS API. The sending user ID must be - explicitly specified, as it cannot be inferred from the access_token, which - will be the same for all AS requests. - - * TODO: or do we maintain a separate access_token mapping? It seems like - unnecessary overhead for the AS developer; easier to just use a single - privileged access_token and just track which userid is emitting events? - * If the AS is spoofing the identity of a real (not virtual) matrix user, - we should actually let them log themselves in via OAuth2 to give permission - to the AS to act on their behalf. - * We can't auth gatewayed virtual users from 3rd party systems who are being - relayed into Matrix, as the relaying is happening whether the user likes it - or not. Therefore we do need to be able to spoof sender ID for virtual users. +* AS injects the event to the HS using the CS API, irrespective of whether the + target user or room is known to the HS or not. If the HS doesn't recognise + the target it goes through the same lazy-load provisioning as per above. +* The reason for not using a subset of the federation API here is because it + allows AS developers to reuse existing CS SDKs and benefit from the more + meaningful error handling of the CS API. The sending user ID must be + explicitly specified, as it cannot be inferred from the access_token, which + will be the same for all AS requests. + + * TODO: or do we maintain a separate ``access_token`` mapping? It seems like + unnecessary overhead for the AS developer; easier to just use a single + privileged ``access_token`` and just track which ``user_id`` is emitting events? + * If the AS is spoofing the identity of a real (not virtual) matrix user, + we should actually let them log themselves in via OAuth2 to give permission + to the AS to act on their behalf. + * We can't auth gatewayed virtual users from 3rd party systems who are being + relayed into Matrix, as the relaying is happening whether the user likes it + or not. Therefore we do need to be able to spoof sender ID for virtual users. On AS relaying events in unknown-to-HS rooms: - * See above. +* See above. On AS publishing aliases for virtual rooms: - * AS uses the normal alias management API to preemptively create/delete public - directory entries for aliases for virtual rooms provided by the AS. - * In order to create these aliases, the underlying room ID must also exist, so - at least the m.room.create of that room must also be prepopulated. It seems - sensible to prepopulate the required initial state and history of the room to - avoid a two-phase prepopulation process. +* AS uses the normal alias management API to preemptively create/delete public + directory entries for aliases for virtual rooms provided by the AS. +* In order to create these aliases, the underlying room ID must also exist, so + at least the ``m.room.create`` of that room must also be prepopulated. It seems + sensible to prepopulate the required initial state and history of the room to + avoid a two-phase prepopulation process. On unregistering the AS from the HS: - * An AS must tell the HS when it is going offline in order to stop receiving - requests from the HS. It does this by hitting an API on the HS. +* An AS must tell the HS when it is going offline in order to stop receiving + requests from the HS. It does this by hitting an API on the HS. AS Visibility: - * If an AS needs to sniff events in a room in order to operate on them (e.g. - to act as a search engine) but not inject traffic into the room, it should - do so by subscribing to the relevant events without actually joining the room. - * If the AS needs to participate in the room as a virtual user (e.g. an IVR - service, or a bot, or a gatewayed virtual user), it should join the room - normally. - * There are rare instances where an AS may wish to participate in a room - (including inserting messages), but be hidden from the room list - e.g. a - conferencing server focus bot may wish to join many rooms as the focus and - both listen to VoIP setups and inject its own VoIP answers, without ever - being physically seen in the room. In this scenario, the user should set - its presence to 'invisible', a state that HSes should only allow AS-authed - users to set. +* If an AS needs to sniff events in a room in order to operate on them (e.g. + to act as a search engine) but not inject traffic into the room, it should + do so by subscribing to the relevant events without actually joining the room. +* If the AS needs to participate in the room as a virtual user (e.g. an IVR + service, or a bot, or a gatewayed virtual user), it should join the room + normally. +* There are rare instances where an AS may wish to participate in a room + (including inserting messages), but be hidden from the room list - e.g. a + conferencing server focus bot may wish to join many rooms as the focus and + both listen to VoIP setups and inject its own VoIP answers, without ever + being physically seen in the room. In this scenario, the user should set + its presence to 'invisible', a state that HSes should only allow AS-authed + users to set. E2E Encryption - * The AS obviously has no visibility to E2E encrypted messages, unless it is - explicitly added to an encrypted room and participates in the group chat - itself. +* The AS obviously has no visibility to E2E encrypted messages, unless it is + explicitly added to an encrypted room and participates in the group chat + itself. Extensions to CS API ==================== - * Ability to assert the identity of the virtual user for all methods. - * Ability to massage timestamps when prepopulating historical state and - messages of virtual rooms (either by overriding origin_server_ts (preferred) or - adding an as_ts which we expect clients to honour) - * Ability to delete aliases (including from the directory) as well as create them. \ No newline at end of file +* Ability to assert the identity of the virtual user for all methods. +* Ability to massage timestamps when prepopulating historical state and + messages of virtual rooms (either by overriding ``origin_server_ts`` (preferred) or + adding an ``as_ts`` which we expect clients to honour) +* Ability to delete aliases (including from the directory) as well as create them. From 68c774689aa67231421e6d669a6d240bb94de3ac Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Jan 2015 15:34:07 +0000 Subject: [PATCH 008/526] Add stub AS HTTP API doc --- drafts/as-http-api.rst | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 drafts/as-http-api.rst diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst new file mode 100644 index 00000000..e69de29b From ad7d675d3d269de2be9cd7d1b4f7dd1b7772d3cd Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 14 Jan 2015 16:40:50 +0000 Subject: [PATCH 009/526] Draft Registration and User Query APIs --- drafts/as-http-api.rst | 140 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index e69de29b..41afc213 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -0,0 +1,140 @@ +Application Services HTTP API +============================= + +.. contents:: Table of Contents +.. sectnum:: + +Application Service -> Home Server +---------------------------------- +This contains home server APIs which are used by the application service. + +Registration API ``[Draft]`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This API registers the application service with its host homeserver to offer its services. + +Inputs: + - Credentials (e.g. some kind of string token) + - Namespace[users] + - Namespace[room aliases] + - URL base to receive inbound comms +Output: + - The credentials the HS will use to query the AS with in return. (e.g. some kind of string token) + - The complete namespace prefix for each namespace requested. +Side effects: + - The HS will start delivering events to the URL base specified if this 200s. +API called when: + - The application service wants to register with a brand new home server. +Notes: + - Namespaces are represented in JSON, with a well-defined conversion to IDs. This prevents + parsing errors and allows the HS to enforce their own namespacing semantics. They look like:: + users: { + irc: ["freenode", "rizon"] + } + which represents 2 namespaces: ``@.irc.freenode.*`` and ``@.irc.rizon.*``. The leaf nodes + must be an array. Intermediate nodes must be JSON objects with the key as the desired string + segment. A more complicated example:: + rooms: { + irc: { + freenode: ["matrix"], + rizon: ["matrixorg"] + } + } + which represents 2 namespaces: ``#.irc.freenode.matrix.*`` and ``#.irc.rizon.matrixorg.*``. + - By specifying namespaces like this, you allow home servers to namespace application services + sensibly, rather than every IRC AS trying to nab all ``@.irc.*`` users. In order for home + servers to do this, they need to tell the application service the actual namespaces allocated + for them. This is returned in the JSON response, with the same structure as the original request, + but with the strings now representing the namespace prefix allocated, e.g:: + users: { + irc: [".applicationservice_146.irc.freenode", ".applicationservice_146.irc.rizon"] + } + The sigil prefix ``@`` is omitted since it is clear from the ``users`` key that these namespace + prefixes are for users. + - This makes the request/response JSON structure deliciously symmetric. +:: + + POST /register + + Request format + { + url: "https://my.application.service.com/matrix/", + as_token: "some_AS_token", + namespaces: { + users: { + irc: ["freenode", "rizon"] + }, + rooms: { + irc: { + freenode: ["matrix"], + rizon: ["matrixorg"] + } + } + } + } + + + Returns: + 200 : Registration accepted. + 400 : Namespaces do not conform to regex + 401 : Credentials need to be supplied. + 403 : AS credentials rejected. + + + 200 OK response format + + { + hs_token: "foobar", + namespaces: { + users: { + irc: [".applicationservice_146.irc.freenode", ".applicationservice_146.irc.rizon"] + }, + rooms: { + irc: { + freenode: [".irc.freenode.matrix"], + rizon: [".applicationservice_146.this.can.be.any.prefix.you.like"] + } + } + } + } + + +Home Server -> Application Service +---------------------------------- +This contains application service APIs which are used by the home server. + +User Query ``[Draft]`` +~~~~~~~~~~~~~~~~~~~~~~ +This API is called by the HS to query the existence of a user on the Application Service's namespace. + +Inputs: + - User ID + - HS Credentials +Output: + - Profile info +Side effects: + - User is created on the HS if this response 200s. +API called when: + - HS receives an event for an unknown user ID in the AS's namespace. +Notes: + - The created user will have their profile info set based on the output. + +:: + + GET /users/$user_id?access_token=$hs_token + + Returns: + 200 : User is recognised. + 404 : User not found. + 401 : Credentials need to be supplied. + 403 : HS credentials rejected. + + + 200 OK response format + + { + profile: { + display_name: "Foo" + avatar_url: "mxc://foo/bar" + } + } + From d240e1f856584c598bfa7ef4c14f19b8b9f72535 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 14 Jan 2015 17:01:33 +0000 Subject: [PATCH 010/526] Add TODO APIs --- drafts/as-http-api.rst | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 41afc213..4915ab6e 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -96,6 +96,10 @@ Notes: } } } + +Unregister API ``[TODO]`` +~~~~~~~~~~~~~~~~~~~~~~~~~ + Home Server -> Application Service @@ -137,4 +141,35 @@ Notes: avatar_url: "mxc://foo/bar" } } + +Room Query ``[TODO]`` +~~~~~~~~~~~~~~~~~~~~~ +This API is called by the HS to query the existence of a room on the Application Service's namespace. + +Pushing ``[TODO]`` +~~~~~~~~~~~~~~~~~~ +This API is called by the HS when the HS wants to push an event (or batch of events) to the AS. + + - Retry semantics + - Ordering + + +Client -> Application Service +----------------------------- +This contains application service APIs which are used by the client. + +Linking ``[TODO]`` +~~~~~~~~~~~~~~~~~~ +Clients may want to link their matrix user ID to their 3PID (e.g. IRC nick). This +API allows the AS to do this, so messages sent from the AS are sent as the client. + +- Probably OAuth2 + +Client-Server v2 API Extensions +------------------------------- + + - Identity assertion (rather than access token inference) + - timestamp massaging (for inserting messages between other messages) + - alias mastery over the ASes namespace + - user ID mastery over the ASes namespace From cdeb0faf853c40dd587507ca534708c98d5aadb5 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 14 Jan 2015 17:17:18 +0000 Subject: [PATCH 011/526] Remove block quotes of shame --- drafts/as-http-api.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 4915ab6e..826a774d 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -150,8 +150,8 @@ Pushing ``[TODO]`` ~~~~~~~~~~~~~~~~~~ This API is called by the HS when the HS wants to push an event (or batch of events) to the AS. - - Retry semantics - - Ordering +- Retry semantics +- Ordering @@ -169,7 +169,7 @@ API allows the AS to do this, so messages sent from the AS are sent as the clien Client-Server v2 API Extensions ------------------------------- - - Identity assertion (rather than access token inference) - - timestamp massaging (for inserting messages between other messages) - - alias mastery over the ASes namespace - - user ID mastery over the ASes namespace +- Identity assertion (rather than access token inference) +- timestamp massaging (for inserting messages between other messages) +- alias mastery over the ASes namespace +- user ID mastery over the ASes namespace From b19a799a0e1f222d777106e7d924c63bd298c60e Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 15 Jan 2015 01:56:47 +0000 Subject: [PATCH 012/526] notes on how to build pstn gateway ASes --- drafts/pstn_gatewaying.txt | 48 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 drafts/pstn_gatewaying.txt diff --git a/drafts/pstn_gatewaying.txt b/drafts/pstn_gatewaying.txt new file mode 100644 index 00000000..2cb8c649 --- /dev/null +++ b/drafts/pstn_gatewaying.txt @@ -0,0 +1,48 @@ +Gatewaying to the PSTN via Matrix Application Services +====================================================== + +Matrix Application Services (AS) provides a way for PSTN users to interact +with Matrix via an AS acting as a gateway. Each PSTN user is represented as a +virtual user on a specific homeserver maintained by the AS. Typically the AS +is provisioned on a well-known AS-supplier HS (e.g. matrix.openmarket.com) or +is a service provisioned on the user's local HS. + +In either scenario, the AS maintains virtual users of form +@.tel.e164:homeserver. These are lazily created (as per the AS spec) when +matrix users try to contact a user id of form @.tel.*:homeserver, or when the +AS needs to inject traffic into the HS on behalf of the PSTN user. The reason +for these being a visible virtual user rather than an invisible user or an +invisible sniffing AS is because they do represent real physical 3rd party +endpoints in the PSTN, and need to be able to send return messages. + +Communication with an actual PSTN user happens in a normal Matrix room, which +for 1:1 matrix<->pstn contact will typically store all conversation history +with that user. On first contact, the matrix user invites the virtual user +into the room (or vice versa). In the event of switching to another AS-enabled +HS, the matrix user would kick the old AS and invite the new one. In the event +of needing loadbalancing between two SMS gateways (for instance), the user +would set visibility flags (TODO: specify per-message ACLs, or use crypto to +only sign messages so they're visible to certain other users?) to adjust which +virtual AS users could see which messages in the room. + +For group chat, one or more AS virtual users may be invited to a group chat, +where-upon they will relay all the traffic in that group chat through to their +PSTN counterpart (and vice versa). This behaviour requires no additional +functionality beyond that required to support 1:1 chat. + +When contacting a user, Matrix clients should check whether a given E.164 +number is already mapped to a real Matrix user by querying the identity +servers (or subscribing to identity updates for a given E.164 number. TODO: ID +server subscriptions). If the E.164 number has a validated mapping in the ID +server to a Matrix ID, then this target ID should be used instead of +contacting the virtual user. + +It's likely that PSTN gateway ASes will need to charge the end-user for use of +the gateway. The AS must therefore track credit per matrix ID it interacts +with, and stop gatewaying as desired once credit is exhausted. The task of +extracting credit from the end-user and adding it to the AS is not covered by +the Matrix specification. + +For SMS routing, options are: + * Terminate traffic only (from a shared shortcode originator) + * Two-way traffic via a VMN. To save allocating huge numbers of VMNs to Matrix users, the VMN can be allocated from a pool such that each {caller,callee} tuple is unique (but the caller number will only work from that specific callee). \ No newline at end of file From d8cd4a45d4a04cc9de02f7c36bb860bec42720db Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 09:50:38 +0000 Subject: [PATCH 013/526] Modify Register API As per https://github.com/matrix-org/matrix-doc/pull/5#issuecomment-70059200 --- drafts/as-http-api.rst | 65 ++++++++++++------------------------------ 1 file changed, 19 insertions(+), 46 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 826a774d..d7c749f9 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -10,6 +10,9 @@ This contains home server APIs which are used by the application service. Registration API ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. NOTE:: + - Do we really have to use regex for this? Can't we do this a nicer way? + This API registers the application service with its host homeserver to offer its services. Inputs: @@ -19,38 +22,18 @@ Inputs: - URL base to receive inbound comms Output: - The credentials the HS will use to query the AS with in return. (e.g. some kind of string token) - - The complete namespace prefix for each namespace requested. Side effects: - The HS will start delivering events to the URL base specified if this 200s. API called when: - The application service wants to register with a brand new home server. Notes: - - Namespaces are represented in JSON, with a well-defined conversion to IDs. This prevents - parsing errors and allows the HS to enforce their own namespacing semantics. They look like:: - users: { - irc: ["freenode", "rizon"] - } - which represents 2 namespaces: ``@.irc.freenode.*`` and ``@.irc.rizon.*``. The leaf nodes - must be an array. Intermediate nodes must be JSON objects with the key as the desired string - segment. A more complicated example:: - rooms: { - irc: { - freenode: ["matrix"], - rizon: ["matrixorg"] - } - } - which represents 2 namespaces: ``#.irc.freenode.matrix.*`` and ``#.irc.rizon.matrixorg.*``. - - By specifying namespaces like this, you allow home servers to namespace application services - sensibly, rather than every IRC AS trying to nab all ``@.irc.*`` users. In order for home - servers to do this, they need to tell the application service the actual namespaces allocated - for them. This is returned in the JSON response, with the same structure as the original request, - but with the strings now representing the namespace prefix allocated, e.g:: - users: { - irc: [".applicationservice_146.irc.freenode", ".applicationservice_146.irc.rizon"] - } + - Namespaces are represented by POSIX extended regular expressions in JSON. + They look like:: + users: [ + "irc\.freenode\.net/.*", + ] The sigil prefix ``@`` is omitted since it is clear from the ``users`` key that these namespace prefixes are for users. - - This makes the request/response JSON structure deliciously symmetric. :: POST /register @@ -60,15 +43,12 @@ Notes: url: "https://my.application.service.com/matrix/", as_token: "some_AS_token", namespaces: { - users: { - irc: ["freenode", "rizon"] - }, - rooms: { - irc: { - freenode: ["matrix"], - rizon: ["matrixorg"] - } - } + users: [ + "irc\.freenode\.net/.*" + ], + rooms: [ + "irc\.freenode\.net/.*" + ] } } @@ -83,18 +63,7 @@ Notes: 200 OK response format { - hs_token: "foobar", - namespaces: { - users: { - irc: [".applicationservice_146.irc.freenode", ".applicationservice_146.irc.rizon"] - }, - rooms: { - irc: { - freenode: [".irc.freenode.matrix"], - rizon: [".applicationservice_146.this.can.be.any.prefix.you.like"] - } - } - } + hs_token: "foobar" } Unregister API ``[TODO]`` @@ -108,6 +77,10 @@ This contains application service APIs which are used by the home server. User Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~ +.. NOTE:: + - This API may be called a lot by the HS (e.g. incoming events for unknown user IDs, profile + requests, etc. Is this okay? + This API is called by the HS to query the existence of a user on the Application Service's namespace. Inputs: From 9401d9a62e8a792f646d5f89a55d1d90d75e8f07 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 13:48:50 +0000 Subject: [PATCH 014/526] Flesh out the required CS extensions --- drafts/as-http-api.rst | 72 +++++++++++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 12 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index d7c749f9..a66628c3 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -46,7 +46,7 @@ Notes: users: [ "irc\.freenode\.net/.*" ], - rooms: [ + aliases: [ "irc\.freenode\.net/.*" ] } @@ -63,12 +63,30 @@ Notes: 200 OK response format { - hs_token: "foobar" + hs_token: "string" } -Unregister API ``[TODO]`` +Unregister API ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~ +This API unregisters a previously registered AS from the home server. +Inputs: + - AS token +Output: + - None. +Side effects: + - The HS will stop delivering events to the URL base specified for this AS if this 200s. +API called when: + - The application service wants to stop receiving all events from the HS. + +:: + + POST /unregister + + Request format + { + as_token: "string" + } Home Server -> Application Service @@ -77,9 +95,6 @@ This contains application service APIs which are used by the home server. User Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~ -.. NOTE:: - - This API may be called a lot by the HS (e.g. incoming events for unknown user IDs, profile - requests, etc. Is this okay? This API is called by the HS to query the existence of a user on the Application Service's namespace. @@ -110,8 +125,8 @@ Notes: { profile: { - display_name: "Foo" - avatar_url: "mxc://foo/bar" + display_name: "string" + avatar_url: "string(uri)" } } @@ -142,7 +157,40 @@ API allows the AS to do this, so messages sent from the AS are sent as the clien Client-Server v2 API Extensions ------------------------------- -- Identity assertion (rather than access token inference) -- timestamp massaging (for inserting messages between other messages) -- alias mastery over the ASes namespace -- user ID mastery over the ASes namespace +Identity assertion +~~~~~~~~~~~~~~~~~~ +The client-server API infers the user ID from the ``access_token`` provided in every +request. It would be an annoying amount of book-keeping to maintain tokens for every +virtual user. It would be preferable if the application service could use the CS API +with its own ``as_token`` instead, and specify the virtual user they wish to be +acting on behalf of. For real users, this would require additional permissions (see +"C-AS Linking"). + +Inputs: + - Application service token (``as_token``) + Either: + - User ID in the AS namespace to act as. + Or: + - OAuth2 token of real user (which may end up being an access token) +Notes: + - This will apply on all aspects of the CS API, except for Account Management. + + +Timestamp massaging +~~~~~~~~~~~~~~~~~~~ +The application service may want to inject events at a certain time (reflecting +the time on the network they are tracking e.g. irc, xmpp). Application services +need to be able to adjust the ``origin_server_ts`` value to do this. + +Inputs: + - Application service token (``as_token``) + - Desired timestamp +Notes: + - This will only apply when sending events. + +Server admin style permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The home server needs to give the application service *full control* over its +namespace, both for users and for room aliases. This means that the AS should +be able to create/edit/delete any room alias in its namespace, as well as +create/delete any user in its namespace. From d4ea43d2b9126f3004cf948c470010528cd7a655 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 13:56:03 +0000 Subject: [PATCH 015/526] Outline what the CS extensions look like in HTTP --- drafts/as-http-api.rst | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index a66628c3..666d996c 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -175,6 +175,19 @@ Inputs: Notes: - This will apply on all aspects of the CS API, except for Account Management. +:: + + /path?as_token=$token&user_id=$userid + + Query Parameters: + as_token: The application service token + user_id: The desired user ID to act as. + + /path?as_token=$token&user_token=$token + + Query Parameters: + as_token: The application service token + user_token: The token granted to the AS by the real user Timestamp massaging ~~~~~~~~~~~~~~~~~~~ @@ -187,10 +200,19 @@ Inputs: - Desired timestamp Notes: - This will only apply when sending events. + +:: + + /path?as_token=$token&ts=$timestamp + + Query Parameters added to the send event APIs only: + as_token: The application service token + ts: The desired timestamp Server admin style permissions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The home server needs to give the application service *full control* over its namespace, both for users and for room aliases. This means that the AS should be able to create/edit/delete any room alias in its namespace, as well as -create/delete any user in its namespace. +create/delete any user in its namespace. This does not require any additional +public APIs. From 7d56113b4fa1ff3f5d13fa76a3acd005b9ae66da Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 14:27:20 +0000 Subject: [PATCH 016/526] Clarification and tweaks Room ID namespace clarifications and s/as_token/access_token/g for client-server extensions. --- drafts/as-http-api.rst | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 666d996c..5a1ca364 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -33,7 +33,7 @@ Notes: "irc\.freenode\.net/.*", ] The sigil prefix ``@`` is omitted since it is clear from the ``users`` key that these namespace - prefixes are for users. + prefixes are for users. Likewise, ``#`` for ``aliases`` and ``!`` for ``rooms``. :: POST /register @@ -48,6 +48,9 @@ Notes: ], aliases: [ "irc\.freenode\.net/.*" + ], + rooms: [ + "irc\.freenode\.net/.*" ] } } @@ -167,23 +170,26 @@ acting on behalf of. For real users, this would require additional permissions ( "C-AS Linking"). Inputs: - - Application service token (``as_token``) + - Application service token (``access_token``) Either: - User ID in the AS namespace to act as. Or: - OAuth2 token of real user (which may end up being an access token) Notes: - This will apply on all aspects of the CS API, except for Account Management. + - The ``as_token`` is inserted into ``access_token`` which is usually where the client + token is. This is done on purpose to allow application services to reuse client + SDKs. :: - /path?as_token=$token&user_id=$userid + /path?access_token=$token&user_id=$userid Query Parameters: as_token: The application service token user_id: The desired user ID to act as. - /path?as_token=$token&user_token=$token + /path?access_token=$token&user_token=$token Query Parameters: as_token: The application service token From e0e9beebc0923f0fa07c4719f0cf00bfc5e78a11 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 16:48:39 +0000 Subject: [PATCH 017/526] Add draft Linking API Also re-add sigils to regex. --- drafts/as-http-api.rst | 63 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 52 insertions(+), 11 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 5a1ca364..8995ffd8 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -30,10 +30,8 @@ Notes: - Namespaces are represented by POSIX extended regular expressions in JSON. They look like:: users: [ - "irc\.freenode\.net/.*", + "@irc\.freenode\.net/.*", ] - The sigil prefix ``@`` is omitted since it is clear from the ``users`` key that these namespace - prefixes are for users. Likewise, ``#`` for ``aliases`` and ``!`` for ``rooms``. :: POST /register @@ -44,13 +42,13 @@ Notes: as_token: "some_AS_token", namespaces: { users: [ - "irc\.freenode\.net/.*" + "@irc\.freenode\.net/.*" ], aliases: [ - "irc\.freenode\.net/.*" + "#irc\.freenode\.net/.*" ], rooms: [ - "irc\.freenode\.net/.*" + "!irc\.freenode\.net/.*" ] } } @@ -146,16 +144,59 @@ This API is called by the HS when the HS wants to push an event (or batch of eve -Client -> Application Service ------------------------------ -This contains application service APIs which are used by the client. +Client -> Server -> Application Service +--------------------------------------- +This contains home server APIs which are used by the client, to communicate with +the application service. -Linking ``[TODO]`` +Linking ``[Draft]`` ~~~~~~~~~~~~~~~~~~ +.. NOTE:: + - How does the application service know that the matrix user really is the virtual + user they claim to be? If we force an IS lookup, then this would resolve on its + own as anyone who wants to talk to the virtual user will do a lookup before trying + the application service... + Clients may want to link their matrix user ID to their 3PID (e.g. IRC nick). This API allows the AS to do this, so messages sent from the AS are sent as the client. -- Probably OAuth2 +This is not achieved using OAuth2 or similar because the trust relationships are +different. The application service already has a huge amount of trust from the +home server, unlike a random third-party web app. As a result, the data flow is +different because the third-party (the application service) is trusted by the +authorisation entity (the home server). Furthermore, it is desirable to not have +the clients communicate directly with the application service in order to +decrease the complexity of AS design. + +To grant permission for an application service to masquerade as a user: + +Inputs: + - Credentials of user (e.g. ``access_token``) + - The user ID within an application service's namespace to claim. + - Restrictions if any (e.g. only for rooms X,Y,Z. Only for 10 hours. etc) +Output: + - None. +Side effects: + - The home server will generate an ``access_token`` and push it to the + application service along with both user IDs if this response 200s. +Notes: + - Repeated calls to this API should invalidate any existing token for this + user ID / application service combo and provision a new token which is then + pushed. + - The generated access token MUST honour the restrictions laid out by the + client. + +To revoke permission for an application service to masquerade as a user: + +Inputs: + - Credentials of user (e.g. ``access_token``) + - The user ID within an application service's namespace to revoke. If blank, + revokes all user IDs linked to this matrix user ID. +Output: + - None. +Side effects: + - The home server invalidate all access tokens for this user ID / AS combo + and push this invalidation to the application service if this response 200s. Client-Server v2 API Extensions ------------------------------- From 0313397e42591c89f47b65ee35eb40258481f750 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 16:51:20 +0000 Subject: [PATCH 018/526] Clarification on IDs Specifically using the term "virtual user ID" in more places to mean the namespace of AS user IDs. --- drafts/as-http-api.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 8995ffd8..2fb9aed2 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -157,8 +157,9 @@ Linking ``[Draft]`` own as anyone who wants to talk to the virtual user will do a lookup before trying the application service... -Clients may want to link their matrix user ID to their 3PID (e.g. IRC nick). This -API allows the AS to do this, so messages sent from the AS are sent as the client. +Clients may want to link their matrix user ID to their virtual user ID. This +API allows the AS to do this, so messages sent from the AS are sent as the client's +user ID, instead of the virtual user ID. This is not achieved using OAuth2 or similar because the trust relationships are different. The application service already has a huge amount of trust from the @@ -191,7 +192,7 @@ To revoke permission for an application service to masquerade as a user: Inputs: - Credentials of user (e.g. ``access_token``) - The user ID within an application service's namespace to revoke. If blank, - revokes all user IDs linked to this matrix user ID. + revokes all virtual user IDs linked to this matrix user ID. Output: - None. Side effects: From 1b3e27a05a8666e02e031972a7bf0336195d5599 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 17:26:36 +0000 Subject: [PATCH 019/526] Add Linking HTTP API Also add note on masquerading problem. --- drafts/as-http-api.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 2fb9aed2..727a5ff9 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -156,6 +156,8 @@ Linking ``[Draft]`` user they claim to be? If we force an IS lookup, then this would resolve on its own as anyone who wants to talk to the virtual user will do a lookup before trying the application service... + - In other words, what is preventing ``@bob:matrix.org`` masquerading as + ``@.irc.freenode.alice:matrix.org``? Clients may want to link their matrix user ID to their virtual user ID. This API allows the AS to do this, so messages sent from the AS are sent as the client's @@ -186,6 +188,21 @@ Notes: pushed. - The generated access token MUST honour the restrictions laid out by the client. + +:: + + PUT /appservices/$virtual_user_id?access_token=$token + + Request format + { + restrictions: { + expires_in: 3600, + rooms: [ + "!fl3rwfehw:matrix.org", + "!fwet2yugs:matrix.org" + ] + } + } To revoke permission for an application service to masquerade as a user: @@ -198,6 +215,11 @@ Output: Side effects: - The home server invalidate all access tokens for this user ID / AS combo and push this invalidation to the application service if this response 200s. + +:: + + DELETE /appservices/$virtual_user_id?access_token=$token + Client-Server v2 API Extensions ------------------------------- From f33b722e4b006f8ffaee89f38648df0c2b9b2dde Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 16 Jan 2015 08:28:55 +0000 Subject: [PATCH 020/526] More clarifications s/as_token/access_token/ for CS extensions. Clarify when room queries are made. --- drafts/as-http-api.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 727a5ff9..59d5a3e4 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -131,9 +131,11 @@ Notes: } } -Room Query ``[TODO]`` -~~~~~~~~~~~~~~~~~~~~~ -This API is called by the HS to query the existence of a room on the Application Service's namespace. +Room Alias Query ``[TODO]`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This API is called by the HS to query the existence of a room alias on the Application +Service's namespace. + Pushing ``[TODO]`` ~~~~~~~~~~~~~~~~~~ @@ -250,13 +252,13 @@ Notes: /path?access_token=$token&user_id=$userid Query Parameters: - as_token: The application service token + access_token: The application service token user_id: The desired user ID to act as. /path?access_token=$token&user_token=$token Query Parameters: - as_token: The application service token + access_token: The application service token user_token: The token granted to the AS by the real user Timestamp massaging @@ -273,10 +275,10 @@ Notes: :: - /path?as_token=$token&ts=$timestamp + /path?access_token=$token&ts=$timestamp Query Parameters added to the send event APIs only: - as_token: The application service token + access_token: The application service token ts: The desired timestamp Server admin style permissions From 1d45dc08ca07cecb7f9c618591753c4f6540fe06 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 16 Jan 2015 10:56:42 +0000 Subject: [PATCH 021/526] Add room query API. Remove linking API. Also add section on ID conventions which need thinking about. --- drafts/as-http-api.rst | 171 +++++++++++++++++++---------------------- 1 file changed, 81 insertions(+), 90 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 59d5a3e4..fd0b302e 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -13,7 +13,8 @@ Registration API ``[Draft]`` .. NOTE:: - Do we really have to use regex for this? Can't we do this a nicer way? -This API registers the application service with its host homeserver to offer its services. +This API registers the application service with its host homeserver to offer its +services. Inputs: - Credentials (e.g. some kind of string token) @@ -21,7 +22,8 @@ Inputs: - Namespace[room aliases] - URL base to receive inbound comms Output: - - The credentials the HS will use to query the AS with in return. (e.g. some kind of string token) + - The credentials the HS will use to query the AS with in return. (e.g. some + kind of string token) Side effects: - The HS will start delivering events to the URL base specified if this 200s. API called when: @@ -76,7 +78,8 @@ Inputs: Output: - None. Side effects: - - The HS will stop delivering events to the URL base specified for this AS if this 200s. + - The HS will stop delivering events to the URL base specified for this AS if + this 200s. API called when: - The application service wants to stop receiving all events from the HS. @@ -97,7 +100,8 @@ This contains application service APIs which are used by the home server. User Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~ -This API is called by the HS to query the existence of a user on the Application Service's namespace. +This API is called by the HS to query the existence of a user on the Application +Service's namespace. Inputs: - User ID @@ -131,96 +135,71 @@ Notes: } } -Room Alias Query ``[TODO]`` +Room Alias Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This API is called by the HS to query the existence of a room alias on the Application -Service's namespace. - - -Pushing ``[TODO]`` -~~~~~~~~~~~~~~~~~~ -This API is called by the HS when the HS wants to push an event (or batch of events) to the AS. - -- Retry semantics -- Ordering - - - -Client -> Server -> Application Service ---------------------------------------- -This contains home server APIs which are used by the client, to communicate with -the application service. - -Linking ``[Draft]`` -~~~~~~~~~~~~~~~~~~ -.. NOTE:: - - How does the application service know that the matrix user really is the virtual - user they claim to be? If we force an IS lookup, then this would resolve on its - own as anyone who wants to talk to the virtual user will do a lookup before trying - the application service... - - In other words, what is preventing ``@bob:matrix.org`` masquerading as - ``@.irc.freenode.alice:matrix.org``? - -Clients may want to link their matrix user ID to their virtual user ID. This -API allows the AS to do this, so messages sent from the AS are sent as the client's -user ID, instead of the virtual user ID. - -This is not achieved using OAuth2 or similar because the trust relationships are -different. The application service already has a huge amount of trust from the -home server, unlike a random third-party web app. As a result, the data flow is -different because the third-party (the application service) is trusted by the -authorisation entity (the home server). Furthermore, it is desirable to not have -the clients communicate directly with the application service in order to -decrease the complexity of AS design. - -To grant permission for an application service to masquerade as a user: +This API is called by the HS to query the existence of a room alias on the +Application Service's namespace. Inputs: - - Credentials of user (e.g. ``access_token``) - - The user ID within an application service's namespace to claim. - - Restrictions if any (e.g. only for rooms X,Y,Z. Only for 10 hours. etc) + - Room alias + - HS Credentials Output: - - None. + - The current state events for the room if any. + - The message events for the room if any. Side effects: - - The home server will generate an ``access_token`` and push it to the - application service along with both user IDs if this response 200s. + - A new room will be created with the alias input if this response 200s. +API called when: + - HS receives an event to join a room alias in the AS's namespace. Notes: - - Repeated calls to this API should invalidate any existing token for this - user ID / application service combo and provision a new token which is then - pushed. - - The generated access token MUST honour the restrictions laid out by the - client. - + - This can be thought of as an ``initialSync`` but for a 3P networked room, + which is lazily loaded when a matrix user tries to join the room. + :: - PUT /appservices/$virtual_user_id?access_token=$token + GET /rooms/$room_alias?access_token=$hs_token - Request format - { - restrictions: { - expires_in: 3600, - rooms: [ - "!fl3rwfehw:matrix.org", - "!fwet2yugs:matrix.org" + Returns: + 200 : Room is recognised. + 404 : Room not found. + 401 : Credentials need to be supplied. + 403 : HS credentials rejected. + + + 200 OK response format + + { + events: [ + { + content: { + ... + }, + user_id: "string", + origin_server_ts: integer, // massaged timestamps + type: "string" + }, + ... + ], + state: [ + { + content: { + ... + }, + user_id: "string(virtual user id)", + origin_server_ts: integer, + state_key: "string", + type: "string" // e.g. m.room.name + }, + ... ] } - } -To revoke permission for an application service to masquerade as a user: - -Inputs: - - Credentials of user (e.g. ``access_token``) - - The user ID within an application service's namespace to revoke. If blank, - revokes all virtual user IDs linked to this matrix user ID. -Output: - - None. -Side effects: - - The home server invalidate all access tokens for this user ID / AS combo - and push this invalidation to the application service if this response 200s. - -:: +Pushing ``[TODO]`` +~~~~~~~~~~~~~~~~~~ +This API is called by the HS when the HS wants to push an event (or batch of +events) to the AS. - DELETE /appservices/$virtual_user_id?access_token=$token +- Retry semantics +- Ordering Client-Server v2 API Extensions @@ -228,12 +207,12 @@ Client-Server v2 API Extensions Identity assertion ~~~~~~~~~~~~~~~~~~ -The client-server API infers the user ID from the ``access_token`` provided in every -request. It would be an annoying amount of book-keeping to maintain tokens for every -virtual user. It would be preferable if the application service could use the CS API -with its own ``as_token`` instead, and specify the virtual user they wish to be -acting on behalf of. For real users, this would require additional permissions (see -"C-AS Linking"). +The client-server API infers the user ID from the ``access_token`` provided in +every request. It would be an annoying amount of book-keeping to maintain tokens +for every virtual user. It would be preferable if the application service could +use the CS API with its own ``as_token`` instead, and specify the virtual user +they wish to be acting on behalf of. For real users, this would require +additional permissions (see "C-AS Linking"). Inputs: - Application service token (``access_token``) @@ -243,9 +222,9 @@ Inputs: - OAuth2 token of real user (which may end up being an access token) Notes: - This will apply on all aspects of the CS API, except for Account Management. - - The ``as_token`` is inserted into ``access_token`` which is usually where the client - token is. This is done on purpose to allow application services to reuse client - SDKs. + - The ``as_token`` is inserted into ``access_token`` which is usually where the + client token is. This is done on purpose to allow application services to + reuse client SDKs. :: @@ -288,3 +267,15 @@ namespace, both for users and for room aliases. This means that the AS should be able to create/edit/delete any room alias in its namespace, as well as create/delete any user in its namespace. This does not require any additional public APIs. + + +ID conventions +-------------- +This concerns the well-defined conventions for mapping 3P network IDs to matrix +IDs, which we expect clients to be able to do by themselves. + +- What do user IDs look like? Butchered URIs? Can all 3P network IDs be + reasonably expressed as URIs? (e.g. tel, email, irc, xmpp, ...) +- What do room aliases look like? Some cases are clear (e.g. IRC) but others + are a lot more fiddly (e.g. email? You don't want to share a room with + everyone who has ever sent an email to ``bob@gmail.com``)... \ No newline at end of file From bd6fc4b5bfbf1b7e7c9867f1c7f1b48f4b3f4821 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 16 Jan 2015 14:29:53 +0000 Subject: [PATCH 022/526] Add draft Pushing API. Add Examples section. Work through an IRC example; raise issues in the notes section. Add retry notes on HS>AS API calls. --- drafts/as-http-api.rst | 227 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 216 insertions(+), 11 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index fd0b302e..dd2dbf80 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -31,9 +31,11 @@ API called when: Notes: - Namespaces are represented by POSIX extended regular expressions in JSON. They look like:: + users: [ "@irc\.freenode\.net/.*", ] + :: POST /register @@ -70,7 +72,7 @@ Notes: } Unregister API ``[Draft]`` -~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~ This API unregisters a previously registered AS from the home server. Inputs: @@ -111,10 +113,16 @@ Output: Side effects: - User is created on the HS if this response 200s. API called when: - - HS receives an event for an unknown user ID in the AS's namespace. + - HS receives an event for an unknown user ID in the AS's namespace, e.g. an + invite event to a room. Notes: - The created user will have their profile info set based on the output. - +Retry notes: + - The home server cannot respond to the client's request until the response to + this API is obtained from the AS. + - Recommended that home servers try a few times then time out, returning a + 408 Request Timeout to the client. + :: GET /users/$user_id?access_token=$hs_token @@ -136,7 +144,7 @@ Notes: } Room Alias Query ``[Draft]`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This API is called by the HS to query the existence of a room alias on the Application Service's namespace. @@ -153,6 +161,11 @@ API called when: Notes: - This can be thought of as an ``initialSync`` but for a 3P networked room, which is lazily loaded when a matrix user tries to join the room. +Retry notes: + - The home server cannot respond to the client's request until the response to + this API is obtained from the AS. + - Recommended that home servers try a few times then time out, returning a + 408 Request Timeout to the client. :: @@ -193,14 +206,61 @@ Notes: ] } -Pushing ``[TODO]`` -~~~~~~~~~~~~~~~~~~ +Pushing ``[Draft]`` +~~~~~~~~~~~~~~~~~~~ This API is called by the HS when the HS wants to push an event (or batch of events) to the AS. -- Retry semantics -- Ordering +Inputs: + - HS Credentials + - Event(s) to give to the AS + - HS-generated transaction ID +Output: + - None. + +Data flows: + +:: + + Typical + HS ---> AS : Home server sends events with transaction ID T. + <--- : AS sends back 200 OK. + + AS ACK Lost + HS ---> AS : Home server sends events with transaction ID T. + <-/- : AS 200 OK is lost. + HS ---> AS : Home server retries with the same transaction ID of T. + <--- : AS sends back 200 OK. If the AS had processed these events + already, it can NO-OP this request (and it knows if it is the same + events based on the transacton ID). + + +Retry notes: + - If the HS fails to pass on the events to the AS, it must retry the request. + - Since ASes by definition cannot alter the traffic being passed to it (unlike + say, a Policy Server), these requests can be done in parallel to general HS + processing; the HS doesn't need to block whilst doing this. + - Home servers should use exponential backoff as their retry algorithm. + - Home servers MUST NOT alter (e.g. add more) events they were going to + send within that transaction ID on retries, as the AS may have already + processed the events. + +Ordering notes: + - The events sent to the AS should be linearised, as they are from the event + stream. + - The home server will need to maintain a queue of transactions to send to + the AS. +:: + + PUT /transactions/$transaction_id?access_token=$hs_token + + Request format + { + events: [ + ... + ] + } Client-Server v2 API Extensions ------------------------------- @@ -216,6 +276,7 @@ additional permissions (see "C-AS Linking"). Inputs: - Application service token (``access_token``) + Either: - User ID in the AS namespace to act as. Or: @@ -269,8 +330,8 @@ create/delete any user in its namespace. This does not require any additional public APIs. -ID conventions --------------- +ID conventions ``[TODO]`` +------------------------- This concerns the well-defined conventions for mapping 3P network IDs to matrix IDs, which we expect clients to be able to do by themselves. @@ -278,4 +339,148 @@ IDs, which we expect clients to be able to do by themselves. reasonably expressed as URIs? (e.g. tel, email, irc, xmpp, ...) - What do room aliases look like? Some cases are clear (e.g. IRC) but others are a lot more fiddly (e.g. email? You don't want to share a room with - everyone who has ever sent an email to ``bob@gmail.com``)... \ No newline at end of file + everyone who has ever sent an email to ``bob@gmail.com``)... + +Examples +-------- +.. NOTE:: + - User/Alias namespaces are subject to change depending on ID conventions. + - Should home servers by default generate fixed room IDs which match the room + alias? Otherwise, you need to tell the AS that room alias X matches room ID + Y so when the home server pushes events with room ID Y the AS knows which + room that is. + +IRC +~~~ +Pre-conditions: + - Server admin stores the AS token "T_a" on the home server. + - Home server has a token "T_h". + - Home server has the domain "hsdomain.com" + +1. Application service registration +:: + + AS -> HS: Registers itself with the home server + POST /register + { + url: "https://someapp.com/matrix", + as_token: "T_a", + namespaces: { + users: [ + "@irc\.freenode\.net/.*" + ], + aliases: [ + "#irc\.freenode\.net/.*" + ], + rooms: [ + "!irc\.freenode\.net/.*" + ] + } + } + + Returns 200 OK: + { + hs_token: "T_h" + } + +2. IRC user "Bob" says "hello?" on "#matrix" at timestamp 1421416883133: +:: + + - AS stores message as potential scrollback. + - Nothing happens as no Matrix users are in the room. + +3. Matrix user "@alice:hsdomain.com" wants to join "#matrix": +:: + + User -> HS: Request to join "#irc.freenode.net/#matrix:hsdomain.com" + + HS -> AS: Room Query "#irc.freenode.net/#matrix:hsdomain.com" + GET /rooms/%23irc.freenode.net%2F%23matrix%3Ahsdomain.com?access_token=T_h + Returns 200 OK: + { + events: [ + { + content: { + body: "hello?", + msgtype: "m.text" + } + origin_server_ts: 1421416883133, + user_id: "@irc.freenode.net/Bob:hsdomain.com" + type: "m.room.message" + } + ], + state: [ + { + content: { + name: "#matrix" + } + origin_server_ts: 1421416883133, // default this to the first msg? + user_id: "@irc.freenode.net/Bob:hsdomain.com", // see above + state_key: "", + type: "m.room.name" + } + ] + } + + - HS provisions new room with *FIXED* room ID (see notes section) + "!irc.freenode.net/#matrix:hsdomain.com" with normal state events + (e.g. m.room.create). join_rules can be overridden by the AS if supplied in + "state". + - HS injects messages into room. Finds unknown user ID + "@irc.freenode.net/Bob:hsdomain.com" in AS namespace, so queries AS. + + HS -> AS: User Query "@irc.freenode.net/Bob:hsdomain.com" + GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h + Returns 200 OK: + { + profile: { + display_name: "Bob" + } + } + + - HS provisions new user with display name "Bob". + - HS sends room information back to client. + +4. @alice:hsdomain.com says "hi!" in this room: +:: + + User -> HS: Send message "hi!" in room !irc.freenode.net/#matrix:hsdomain.com + + - HS sends message. + - HS sees the room ID is in the AS namespace and pushes it to the AS. + + HS -> AS: Push event + PUT /transactions/1?access_token=T_h + { + events: [ + { + content: { + body: "hi!", + msgtype: "m.text" + }, + origin_server_ts: , + user_id: "@alice:hsdomain.com", + room_id: "!irc.freenode.net/#matrix:hsdomain.com", + type: "m.room.message" + } + ] + } + + - AS passes this through to IRC. + + +5. IRC user "Bob" says "what's up?" on "#matrix" at timestamp 1421418084816: +:: + + IRC -> AS: "what's up?" + AS -> HS: Send message via CS API extension + PUT /rooms/%21irc.freenode.net%2F%23matrix%3Ahsdomain.com/send/m.room.message + ?access_token=T_a + &user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com + &ts=1421418084816 + { + body: "what's up?" + msgtype: "m.text" + } + + - HS modifies the user_id and origin_server_ts on the event and sends it. \ No newline at end of file From 828d6992e563acf78d4a595f446ba1b518844ea4 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 16 Jan 2015 14:35:20 +0000 Subject: [PATCH 023/526] Small thoughts about the IRC example. --- drafts/as-http-api.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index dd2dbf80..1bd247f8 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -348,7 +348,10 @@ Examples - Should home servers by default generate fixed room IDs which match the room alias? Otherwise, you need to tell the AS that room alias X matches room ID Y so when the home server pushes events with room ID Y the AS knows which - room that is. + room that is. Alternatively, do we expect ASes to query the HS via the room + alias APIs just like clients do? This doesn't get us the reverse mapping + though which is what we really want when events are pushed to the AS. Maybe + a special room_id_to_alias event poke is pushed to the AS? IRC ~~~ From 603361bfd68674dae57624122eb4adb5ee6bd689 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Mon, 19 Jan 2015 10:20:39 +0000 Subject: [PATCH 024/526] Update user and room query APIs Expect the AS to create these entities, rather than the HS. --- drafts/as-http-api.rst | 126 ++++++++++++++--------------------------- 1 file changed, 41 insertions(+), 85 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 1bd247f8..931c8a07 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -109,14 +109,21 @@ Inputs: - User ID - HS Credentials Output: - - Profile info + - Whether the user exists. Side effects: - - User is created on the HS if this response 200s. + - User is created on the HS by the AS via CS APIs during the processing of this request. API called when: - HS receives an event for an unknown user ID in the AS's namespace, e.g. an invite event to a room. Notes: - - The created user will have their profile info set based on the output. + - When the AS receives this request, if the user exists, it must create the user via + the CS API. + - It can also set arbitrary information about the user (e.g. display name, join rooms, etc) + using the CS API. + - When this setup is complete, the AS should respond to the HS request. This means the AS + blocks the HS until the user is created. + - This is deemed more flexible than alternative methods (e.g. returning a JSON blob with the + user's display name and get the HS to provision the user). Retry notes: - The home server cannot respond to the client's request until the response to this API is obtained from the AS. @@ -136,12 +143,7 @@ Retry notes: 200 OK response format - { - profile: { - display_name: "string" - avatar_url: "string(uri)" - } - } + {} Room Alias Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -155,12 +157,20 @@ Output: - The current state events for the room if any. - The message events for the room if any. Side effects: - - A new room will be created with the alias input if this response 200s. + - Room is created on the HS by the AS via CS APIs during the processing of + this request. API called when: - HS receives an event to join a room alias in the AS's namespace. Notes: - - This can be thought of as an ``initialSync`` but for a 3P networked room, - which is lazily loaded when a matrix user tries to join the room. + - When the AS receives this request, if the room exists, it must create the room via + the CS API. + - It can also set arbitrary information about the room (e.g. name, topic, etc) + using the CS API. + - When this setup is complete, the AS should respond to the HS request. This means the AS + blocks the HS until the room is created and configured. + - This is deemed more flexible than alternative methods (e.g. returning an initial sync + style JSON blob and get the HS to provision the room). It also means that the AS knows + the room ID -> alias mapping. Retry notes: - The home server cannot respond to the client's request until the response to this API is obtained from the AS. @@ -180,31 +190,7 @@ Retry notes: 200 OK response format - { - events: [ - { - content: { - ... - }, - user_id: "string", - origin_server_ts: integer, // massaged timestamps - type: "string" - }, - ... - ], - state: [ - { - content: { - ... - }, - user_id: "string(virtual user id)", - origin_server_ts: integer, - state_key: "string", - type: "string" // e.g. m.room.name - }, - ... - ] - } + {} Pushing ``[Draft]`` ~~~~~~~~~~~~~~~~~~~ @@ -345,13 +331,6 @@ Examples -------- .. NOTE:: - User/Alias namespaces are subject to change depending on ID conventions. - - Should home servers by default generate fixed room IDs which match the room - alias? Otherwise, you need to tell the AS that room alias X matches room ID - Y so when the home server pushes events with room ID Y the AS knows which - room that is. Alternatively, do we expect ASes to query the HS via the room - alias APIs just like clients do? This doesn't get us the reverse mapping - though which is what we really want when events are pushed to the AS. Maybe - a special room_id_to_alias event poke is pushed to the AS? IRC ~~~ @@ -399,49 +378,26 @@ Pre-conditions: HS -> AS: Room Query "#irc.freenode.net/#matrix:hsdomain.com" GET /rooms/%23irc.freenode.net%2F%23matrix%3Ahsdomain.com?access_token=T_h - Returns 200 OK: - { - events: [ - { - content: { - body: "hello?", - msgtype: "m.text" - } - origin_server_ts: 1421416883133, - user_id: "@irc.freenode.net/Bob:hsdomain.com" - type: "m.room.message" - } - ], - state: [ + [Starts blocking] + AS -> HS: Creates room. Gets room ID "!aasaasasa:matrix.org". + AS -> HS: Sets room name to "#matrix". + AS -> HS: Sends message as ""@irc.freenode.net/Bob:hsdomain.com" + PUT /rooms/%21irc.freenode.net%2F%23matrix%3Ahsdomain.com/send/m.room.message + ?access_token=T_a + &user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com + &ts=1421416883133 { - content: { - name: "#matrix" - } - origin_server_ts: 1421416883133, // default this to the first msg? - user_id: "@irc.freenode.net/Bob:hsdomain.com", // see above - state_key: "", - type: "m.room.name" + body: "hello?" + msgtype: "m.text" } - ] - } - - - HS provisions new room with *FIXED* room ID (see notes section) - "!irc.freenode.net/#matrix:hsdomain.com" with normal state events - (e.g. m.room.create). join_rules can be overridden by the AS if supplied in - "state". - - HS injects messages into room. Finds unknown user ID - "@irc.freenode.net/Bob:hsdomain.com" in AS namespace, so queries AS. - - HS -> AS: User Query "@irc.freenode.net/Bob:hsdomain.com" - GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h - Returns 200 OK: - { - profile: { - display_name: "Bob" - } - } + HS -> AS: User Query "@irc.freenode.net/Bob:hsdomain.com" + GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h + [Starts blocking] + AS -> HS: Creates user using CS API extension. + AS -> HS: Set user display name to "Bob". + [Finishes blocking] + [Finished blocking] - - HS provisions new user with display name "Bob". - HS sends room information back to client. 4. @alice:hsdomain.com says "hi!" in this room: @@ -486,4 +442,4 @@ Pre-conditions: msgtype: "m.text" } - - HS modifies the user_id and origin_server_ts on the event and sends it. \ No newline at end of file + - HS modifies the user_id and origin_server_ts on the event and sends it. From 117b0901927ac814a0bc1e8f17532ed74ce19f73 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Mon, 19 Jan 2015 10:33:08 +0000 Subject: [PATCH 025/526] Add registration API extension. For AS provisioning of users with its namespace. --- drafts/as-http-api.rst | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 931c8a07..11a404b2 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -166,6 +166,7 @@ Notes: the CS API. - It can also set arbitrary information about the room (e.g. name, topic, etc) using the CS API. + - It can send messages as other users in order to populate scrollback. - When this setup is complete, the AS should respond to the HS request. This means the AS blocks the HS until the room is created and configured. - This is deemed more flexible than alternative methods (e.g. returning an initial sync @@ -258,7 +259,7 @@ every request. It would be an annoying amount of book-keeping to maintain tokens for every virtual user. It would be preferable if the application service could use the CS API with its own ``as_token`` instead, and specify the virtual user they wish to be acting on behalf of. For real users, this would require -additional permissions (see "C-AS Linking"). +additional permissions granting the AS permission to masquerade as a matrix user. Inputs: - Application service token (``access_token``) @@ -312,9 +313,26 @@ Server admin style permissions The home server needs to give the application service *full control* over its namespace, both for users and for room aliases. This means that the AS should be able to create/edit/delete any room alias in its namespace, as well as -create/delete any user in its namespace. This does not require any additional -public APIs. +create/delete any user in its namespace. No additional API changes need to be +made in order for control of room aliases to be granted to the AS. Creation of +users needs API changes in order to: +- Work around captchas. +- Have a 'passwordless' user. + +This involves bypassing the registration flows entirely. This is achieved by +including the AS token on a ``/register`` request, along with a login type of +``m.login.application_service`` to set the desired user ID without a password. + +:: + + /register?access_token=$as_token + + Content: + { + type: "m.login.application_service", + user: "" + } ID conventions ``[TODO]`` ------------------------- @@ -394,6 +412,11 @@ Pre-conditions: GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h [Starts blocking] AS -> HS: Creates user using CS API extension. + POST /register?access_token=T_a + { + type: "m.login.application_service", + user: "irc.freenode.net/Bob" + } AS -> HS: Set user display name to "Bob". [Finishes blocking] [Finished blocking] From ae0fb938adf7c333e47720394c0d9f73f9330402 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Mon, 19 Jan 2015 10:45:51 +0000 Subject: [PATCH 026/526] Update IRC example to reflect new room ID --- drafts/as-http-api.rst | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 11a404b2..c8aeca35 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -154,8 +154,7 @@ Inputs: - Room alias - HS Credentials Output: - - The current state events for the room if any. - - The message events for the room if any. + - Whether the room exists. Side effects: - Room is created on the HS by the AS via CS APIs during the processing of this request. @@ -397,10 +396,10 @@ Pre-conditions: HS -> AS: Room Query "#irc.freenode.net/#matrix:hsdomain.com" GET /rooms/%23irc.freenode.net%2F%23matrix%3Ahsdomain.com?access_token=T_h [Starts blocking] - AS -> HS: Creates room. Gets room ID "!aasaasasa:matrix.org". + AS -> HS: Creates room. Gets room ID "!aasaasasa:hsdomain.com". AS -> HS: Sets room name to "#matrix". AS -> HS: Sends message as ""@irc.freenode.net/Bob:hsdomain.com" - PUT /rooms/%21irc.freenode.net%2F%23matrix%3Ahsdomain.com/send/m.room.message + PUT /rooms/%21aasaasasa%3Ahsdomain.com/send/m.room.message ?access_token=T_a &user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com &ts=1421416883133 @@ -426,7 +425,7 @@ Pre-conditions: 4. @alice:hsdomain.com says "hi!" in this room: :: - User -> HS: Send message "hi!" in room !irc.freenode.net/#matrix:hsdomain.com + User -> HS: Send message "hi!" in room !aasaasasa:hsdomain.com - HS sends message. - HS sees the room ID is in the AS namespace and pushes it to the AS. @@ -442,7 +441,7 @@ Pre-conditions: }, origin_server_ts: , user_id: "@alice:hsdomain.com", - room_id: "!irc.freenode.net/#matrix:hsdomain.com", + room_id: "!aasaasasa:hsdomain.com", type: "m.room.message" } ] @@ -456,7 +455,7 @@ Pre-conditions: IRC -> AS: "what's up?" AS -> HS: Send message via CS API extension - PUT /rooms/%21irc.freenode.net%2F%23matrix%3Ahsdomain.com/send/m.room.message + PUT /rooms/%21aasaasasa%3Ahsdomain.com/send/m.room.message ?access_token=T_a &user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com &ts=1421418084816 From 7a2b6abee5ecc713452283034a196d70e5b2bc45 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Tue, 20 Jan 2015 14:22:48 +0000 Subject: [PATCH 027/526] Flesh out ID conventions Going down the API route. --- drafts/as-http-api.rst | 70 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 64 insertions(+), 6 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index c8aeca35..cb64a53c 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -333,16 +333,74 @@ including the AS token on a ``/register`` request, along with a login type of user: "" } -ID conventions ``[TODO]`` +ID conventions ``[Draft]`` ------------------------- +.. NOTE:: + - Giving HSes the freedom to namespace still feels like the Right Thing here. + - Exposing a public API provides the consistency which was the main complaint + against namespacing. + - This may have knock-on effects for the AS registration API. E.g. why don't + we let ASes specify the *URI* regex they want? + This concerns the well-defined conventions for mapping 3P network IDs to matrix IDs, which we expect clients to be able to do by themselves. -- What do user IDs look like? Butchered URIs? Can all 3P network IDs be - reasonably expressed as URIs? (e.g. tel, email, irc, xmpp, ...) -- What do room aliases look like? Some cases are clear (e.g. IRC) but others - are a lot more fiddly (e.g. email? You don't want to share a room with - everyone who has ever sent an email to ``bob@gmail.com``)... +User IDs +~~~~~~~~ +Matrix users may wish to directly contact a virtual user, e.g. to send an email. +The URI format is a well-structured way to represent a number of different ID +types, including: + +- MSISDNs (``tel``) +- Email addresses (``mailto``) +- IRC nicks (``irc`` - https://tools.ietf.org/html/draft-butcher-irc-url-04) +- XMPP (xep-0032) +- SIP URIs (RFC 3261) + +As a result, virtual user IDs SHOULD relate to their URI counterpart. This +mapping from URI to user ID can be expressed in a number of ways: + +- Expose a C-S API on the HS which takes URIs and responds with user IDs. +- Munge the URI with the user ID. + +Exposing an API would allow HSes to internally map user IDs however they like, +at the cost of an extra round trip (of which the response can be cached). +Munging the URI would allow clients to apply the mapping locally, but would force +user X on service Y to always map to the same munged user ID. Considering the +exposed API could just be applying this munging, there is more flexibility if +an API is exposed. + +:: + + GET /_matrix/app/v1/user?uri=$url_encoded_uri + + Returns 200 OK: + { + user_id: + } + +Room Aliases +~~~~~~~~~~~~ +We may want to expose some 3P network rooms so Matrix users can join them directly, +e.g. IRC rooms. We don't want to expose every 3P network room though, e.g. mailto, +tel. Rooms which are publicly accessible (e.g. IRC rooms) can be exposed as an alias by +the application service. Private rooms (e.g. sending an email to someone) should not +be exposed in this way, but should instead operate using normal invite/join semantics. +Therefore, the ID conventions discussed below are only valid for public rooms which +expose room aliases. + +Matrix users may wish to join XMPP rooms (e.g. using XEP-0045) or IRC rooms. In both +cases, these rooms can be expressed as URIs. For consistency, these "room" URIs +SHOULD be mapped in the same way as "user" URIs. + +:: + + GET /_matrix/app/v1/alias?uri=$url_encoded_uri + + Returns 200 OK: + { + alias: + } Examples -------- From 23d3e5081272a8ca53b092e79d1e716641ad8f30 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Tue, 20 Jan 2015 17:48:43 +0000 Subject: [PATCH 028/526] Add use case justification for regex --- drafts/application_services.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drafts/application_services.rst b/drafts/application_services.rst index 2c73107f..c654c010 100644 --- a/drafts/application_services.rst +++ b/drafts/application_services.rst @@ -85,7 +85,8 @@ In the registration process, the AS provides: * Details of the namespaces of users and rooms the AS is acting on behalf of and "subscribing to" * Namespaces are defined as a list of regexps against which to match room aliases, - room IDs, and user IDs. + room IDs, and user IDs. Regexps give the flexibility to say, sub-domain MSISDN + ranges per AS, whereas a blunt prefix string does not. * There is overlap between selecting events via the csv2 Filter API and subscribing to events here - perhaps subscription involves passing a filter token into the registration API. From 90dfee1c275f350773f583d3a87f054e65b4688f Mon Sep 17 00:00:00 2001 From: Kegsay Date: Tue, 20 Jan 2015 17:49:09 +0000 Subject: [PATCH 029/526] Remove TODO note on regex --- drafts/as-http-api.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index cb64a53c..1c1f342f 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -10,8 +10,6 @@ This contains home server APIs which are used by the application service. Registration API ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. NOTE:: - - Do we really have to use regex for this? Can't we do this a nicer way? This API registers the application service with its host homeserver to offer its services. From 0e2a3a892f8cdb79d9d7ed26a30516c6789ce2dc Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 22 Jan 2015 09:46:25 +0000 Subject: [PATCH 030/526] Presence modifications from MatrixPresence notes. - Idle is no longer a flag but a full state like online|offline. - Clarify status examples. - Add device ID to presence state API given it is per device. - Add note on device specificity on event stream presence parameter. - Add global appear-offline API. --- drafts/general_api.rst | 60 +++++++++++++++++++++++++----------------- 1 file changed, 36 insertions(+), 24 deletions(-) diff --git a/drafts/general_api.rst b/drafts/general_api.rst index 5a027d15..682edfe4 100644 --- a/drafts/general_api.rst +++ b/drafts/general_api.rst @@ -816,22 +816,23 @@ The goals of presence are to: - Let other users know specific status information (e.g. "In a Meeting"). "Online" state can be detected by inspecting when the last time the client made -a request to the server. This could be any request, or a specific kind of request. -For connection-orientated protocols, detecting "online" state can be determined by -the state of this connection stream. For HTTP, this can be detected via requests -to the event stream. +a request to the server. This could be any request, or a specific kind of +request. For connection-orientated protocols, detecting "online" state can be +determined by the state of this connection stream. For HTTP, this can be +detected via requests to the event stream. Online state is separate from letting other users know if someone is *likely to -respond* to messages. This introduces the concept of an "idle" flag, which is -set when the user has not done any "interaction" with the app. The definition of -"interaction" varies based on the app, so it is up to the app to set this "idle" -flag. - -Letting users know specific status information can be achieved via the same method -as v1. Status information should be scoped per *user* and not device as determining -a union algorithm between statuses is nonsensical. Passing status information per -device to all other users just redirects the union problem to the client, which -will commonly be presenting this information as an icon alongside the user. +respond* to messages. This introduces the concept of being "idle", which is +when the user has not done any "interaction" with the app for a while. The +definition of "interaction" and "for a while" varies based on the app, so it is +up to the app to set when the user is idle. + +Letting users know specific status information can be achieved via the same +method as v1. Status information should be scoped per *user* and not device as +determining a union algorithm between statuses is nonsensical. Passing status +information per device to all other users just redirects the union problem to +the client, which will commonly be presenting this information as an icon +alongside the user. When a client hits the event stream, the home server can treat the user as "online". This behaviour should be able to be overridden to avoid flicker @@ -840,11 +841,11 @@ appear offline > goes into a tunnel > server times out > device regains connection and hits the event stream forcing the device online before the "appear offline" state can be set). When the client has not hit the event stream for a certain period of time, the home server can treat the user as -"offline". +"offline". The user can also set a global *per-user* appear offline flag. -The user should also be able to set their presence via a direct API, without -having to hit the event stream. The home server will set a timer when the -connection ends, after which it will set that device to offline. +The user should also be able to set their presence state via a direct API, +without having to hit the event stream. The home server will set a timer when +the connection ends, after which it will set that device to offline. As the idle flag and online state is determined per device, there needs to be a union algorithm to merge these into a single state and flag per user, which will @@ -858,22 +859,33 @@ Changing presence status: Inputs: - User ID - - Presence status (online, away, busy, do not disturb, etc) -Outputs: + - Presence status (busy, do not disturb, in a meeting, etc) +Output: - None. -Setting the idle flag: +Setting presence state: Inputs: - User ID - - Is idle -Outputs: + - Device ID + - Presence state (online|idle|offline) +Output: + - None. + +Setting global appear offline: + +Inputs: + - User ID + - Should appear offline (boolean) +Output: - None. Extra parameters associated with the event stream: Inputs: - - Presence state (online, appear offline) + - Presence state (online, idle, offline) +Notes: + - Scoped per device just like the above API, e.g. from the access_token. Typing API ``[Final]`` From 0a07d688184cd0658cd03c25c747706ef4098434 Mon Sep 17 00:00:00 2001 From: David Baker Date: Wed, 28 Jan 2015 18:20:28 +0000 Subject: [PATCH 031/526] Add overview on push. --- drafts/push_overview.rst | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 drafts/push_overview.rst diff --git a/drafts/push_overview.rst b/drafts/push_overview.rst new file mode 100644 index 00000000..6cee68c5 --- /dev/null +++ b/drafts/push_overview.rst @@ -0,0 +1,31 @@ +Push Notifications +================== + +Matrix supports push notifications as a first class citizen. Home Servers send +notifications of user events to user-configured HTTP endpoints. User may also +configure a number of rules that determine what events generate notifications. +These are all stored and managed by the users home server such that settings can +be reused between client apps as appropriate. + +Nomenclature +------------ + +Pusher + A 'pusher' is an activity in the Home Server that manages the sending + of HTTP notifications for a single device of a single user. + +Push Rules + A push rule is a single rule, configured by a matrix user, that gives + instructions to the Home Server about whether an event should be notified + about and how given a set of conditions. Matrix clients allow the user to + configure these. They create and view them via the Client to Server REST API. + +Push Gateway + A push gateway is a server that receives HTTP event notifications from Home + Servers and passes them on to a different protocol such as APNS for iOS + devices or GCM for Android devices. Matrix.org provides a reference push + gateway, 'sygnal'. + +For information on the client-server API for setting pushers and push rules, see +the Client Server API section. For more information on the format of HTTP +notifications, see the HTTP Notification Protocol section. From c27fb8a2620ed952bde6c0032ed25daacab913e3 Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 13:49:18 +0000 Subject: [PATCH 032/526] Add WIP docs for the client-server bit of push. --- drafts/push_csapi.rst | 52 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 drafts/push_csapi.rst diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst new file mode 100644 index 00000000..bc960fe4 --- /dev/null +++ b/drafts/push_csapi.rst @@ -0,0 +1,52 @@ +Push Notifications +================== + +Pushers +------- +To receive any notification pokes at all, it is necessary to configure a +'pusher' on the Home Server that you wish to receive notifications from. There +is a single API endpoint for this:: + +Fetching a user account displayname:: + + POST $PREFIX/pushers/set + +This takes a JSON object with the following keys: + +pushkey + This is a unique identifier for this pusher. The value you should use for this + is the routing or destination address information for the notification, for + example, the APNS token for APNS or the Registration ID for GCM. If your + notification client has no such concept, use any unique identifier. +kind + The kind of pusher to configure. 'http' makes a pusher that sends HTTP pokes. + null deletes the pusher. +instance_handle + This is an identifier for the device which owns the pusher. It may be up to 32 + characters long. It must be unique among all the pushers for a given user + (therefore the device ID may not be used). It is advised that when an app's + data is copied or restored to a different device, this ID remain the same (ie. + be shared by multiple pushers for multiple devices). Client apps should be + aware that this situation can occur and be able to rectify it (eg. by + offerring to reset the instance_hanlde, optionally duplicating all push rules + to new instance handle). +app_id + appId is a reverse-DNS style identifier for the application. It is recommended + that this end with the platform, such that different platform versions get + different app identifiers. Max length, 64 chars. +app_display_name + A string that will allow the user to identify what application owns this + pusher. +device_display_name + A string that will allow the user to identify what device owns this pusher. +lang + The preferred language for receiving notifications (eg, 'en' or 'en-US') +data + A dictionary of information for the pusher implementation itself. For HTTP + pushers, this must contain a 'url' key which is a string of the URL that + should be used to send notifications. + +If the pusher was created successfully, an empty JSON dictionary is returned. + + + From f68fa8d6f887cc58e591cc06c708548b6c2636a7 Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 15:14:45 +0000 Subject: [PATCH 033/526] Push rules c/s spec. --- drafts/push_csapi.rst | 105 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index bc960fe4..89212576 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -49,4 +49,109 @@ data If the pusher was created successfully, an empty JSON dictionary is returned. +Push Rules +---------- +Home Servers have an interface to configure what events trigger notifications. +This behaviour is configured through 'Push Rules'. Push Rules come in a variety +of different kinds and each kind of rule has an associated priority. The +different kinds of rule, in descending order of priority, are: + +Override Rules + The highest priority rules are user-configured overrides. +Content Rules + These configure behaviour for (unencrypted) messages that match certain + patterns. +Room Rules + These change the behaviour of all messages to a given room. The rule_id of a + room rule is always the room that it affects. +Sender + These rules configure notification behaviour for messages from a specific, + named Matrix user ID. The rule_id of Sender rules is always the Matrix user + ID of the user whose messages theyt apply to. +Underride + These are identical to override rules, but have a lower priority than content, + room and sender rules. +Default + These are rules provided by the home server and cannot be changed by the user. + They are the lowest priority rule and establish baseline behaviour. + +In addition, each kind of rule except default may be either global or +device-specific. Device specific rules only affect delivery of notifications via +pushers with a matching instance_handle. All device-specific rules are higher +priority than all global rules. Thusly, the full list of rule kinds, in +descending priority order, is as follows: + + * Device-specific Override + * Device-specific Content + * Device-specific Room + * Device-specific Sender + * Device-specific Underride + * Global Override + * Global Content + * Global Room + * Global Sender + * Global Underride + * Global Default + +For some kinds of rule, rules of the same kind also have an ordering with +respect to one another. The kinds that do not are room and sender rules where +the rules are mutually exclusive by definition and therefore an ordering would +be redundant. Actions for the highest priority rule and only that rule apply +(for example, a set_sound action in a lower priority rule will not apply if a +higher priority rule matches, even if that rule does not specify a sound). + +Push Rules: Actions: +-------------------- +All rules have an associated list of 'actions'. An action affects if and how a +notification is delievered for a matching event. This standard defines the +following actions, although if Home servers wish to support more, they are free +to do so: + +notify + This causes each matching event to generate a notification. +dont_notify + Prevents this event from generating a notification +coalesce + This enables notifications for matching events but activates Home Server + specific behaviour to intelligently coalesce multiple events into a single + notification. Not all Home Servers may support this. Those that do not should + treat it as the 'notify' action. +set_sound + Sets the value 'sound' key that is sent in the notification poke. This has an + associated string which is the value to set the 'sound' key to. + +Actions that have no parameter are represented as a string. Those with a +parameter are represented as a dictionary with a single key/value pair where the +key is the name of the action and the value is the parameter, eg. { "set_sound": +"ping.wav" } + +Push Rules: Conditions: +----------------------- +Override, Underride and Default rules have a list of 'conditions'. All +conditions must hold true for an event in order for a rule to be applied to an +event. Matrix specifies the following conditions, although if Home Servers wish +to support others, they are free to do so: + +event_match + This is a glob pattern match on a field of the event. Parameters: + * 'key': The dot-separated field of the event to match, eg. content.body + * 'pattern': The glob-style pattern to match against. Patterns with no + special glob characters should be treated as having asterisks + prepended and appended when testing the condition. +device + Matches the instance_handle of the device that the notification would be + delivered to. Parameters: + * 'instance_handle': The instance_handle of the device. +contains_display_name + This matches unencrypted messages where content.body contains the owner's + display name in that room. This is a separate rule because display names may + change and as such it would be hard to maintain a rule that matched the user's + display name. This condition has no parameters. +room_member_count + This matches the current number of members in the room. + * 'is': A decimal integer optionally prefixed by one of, '==', '<', '>', + '>=' or '<='. A prefix of '<' matches rooms where the member count is + strictly less than the given number and so forth. If no prefix is present, + this matches rooms where the member count is exactly equal to the given + number (ie. the same as '=='). From b7a6e8f7b52c366cbcb6ed8577e0d12b6e99cf0d Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 16:25:53 +0000 Subject: [PATCH 034/526] Finish non-API doc parts of push rules. --- drafts/push_csapi.rst | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 89212576..bd204389 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -60,7 +60,9 @@ Override Rules The highest priority rules are user-configured overrides. Content Rules These configure behaviour for (unencrypted) messages that match certain - patterns. + patterns. Content rules take one parameter, 'pattern', that gives the pattern + to match against. This is treated in the same way as pattern for event_match + conditions, below. Room Rules These change the behaviour of all messages to a given room. The rule_id of a room rule is always the room that it affects. @@ -100,6 +102,8 @@ be redundant. Actions for the highest priority rule and only that rule apply (for example, a set_sound action in a lower priority rule will not apply if a higher priority rule matches, even if that rule does not specify a sound). +Rules also have an identifier, rule_id, which is a string. + Push Rules: Actions: -------------------- All rules have an associated list of 'actions'. An action affects if and how a @@ -155,3 +159,8 @@ room_member_count this matches rooms where the member count is exactly equal to the given number (ie. the same as '=='). +Room, Sender, User and Content rules do not have conditions in the same way, +but instead have predefined conditions, the behaviour of which can be configured +using parameters named as described above. In the cases of room and sender +rules, the rule_id of the rule determines its behaviour. + From 0c79798f708334c19097f873d24e37e46d75b6da Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 17:10:09 +0000 Subject: [PATCH 035/526] More WIP API docs and remove spurious line. --- drafts/push_csapi.rst | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index bd204389..1c99a859 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -7,8 +7,6 @@ To receive any notification pokes at all, it is necessary to configure a 'pusher' on the Home Server that you wish to receive notifications from. There is a single API endpoint for this:: -Fetching a user account displayname:: - POST $PREFIX/pushers/set This takes a JSON object with the following keys: @@ -164,3 +162,20 @@ but instead have predefined conditions, the behaviour of which can be configured using parameters named as described above. In the cases of room and sender rules, the rule_id of the rule determines its behaviour. +Push Rules: API +--------------- +Rules live under a hierarchy in the REST API that resembles:: + + $PREFIX/pushrules/// + +To add or change a rule, a client performs a PUT request to the appropriate URL. +When adding rules of a type that has an ordering, the client can add parameters +that define the priority of the rule: + +before + Use 'before' with a rule_id as its value to make the new rule the next-less + important rule with respect to the given rule. +after + This makes the new rule the next-most important rule relative to the given + rule. + From 88178e376e6b352dedfe6b98b05c034e3158c94a Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 17:26:58 +0000 Subject: [PATCH 036/526] Oops, before is *more* important. --- drafts/push_csapi.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 1c99a859..115b9216 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -173,9 +173,8 @@ When adding rules of a type that has an ordering, the client can add parameters that define the priority of the rule: before - Use 'before' with a rule_id as its value to make the new rule the next-less + Use 'before' with a rule_id as its value to make the new rule the next-more important rule with respect to the given rule. after - This makes the new rule the next-most important rule relative to the given + This makes the new rule the next-less important rule relative to the given rule. - From c40b97aaf5089140ab29a53529dc9f67b63e820e Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 18:16:36 +0000 Subject: [PATCH 037/526] Rule adding API with examples --- drafts/push_csapi.rst | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 115b9216..3c6edac6 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -168,6 +168,17 @@ Rules live under a hierarchy in the REST API that resembles:: $PREFIX/pushrules/// +The component parts are as follows: + +scope + Either 'global' or 'device/' to specify global rules or + device rules for the given instance_handle. +kind + The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content' or + 'default'. +rule_id + The identifier for the rule. + To add or change a rule, a client performs a PUT request to the appropriate URL. When adding rules of a type that has an ordering, the client can add parameters that define the priority of the rule: @@ -178,3 +189,33 @@ before after This makes the new rule the next-less important rule relative to the given rule. + +All requests to the push rules API also require an access_token as a query +paraemter. + +The content of the PUT request is a JSON object with a list of actions under the +'actions' key and either conditions (under the 'conditions' key) or the +appropriate parameters for the rule (under the appropriate key name). + +Examples: + +To create a rule that suppresses notifications for the room '#spam:matrix.org':: + + curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456" + +To suppress notifications for the user '@spambot:matrix.org':: + + curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/room/%40spambot%3Amatrix.org?access_token=123456" + +To always notify for messages that contain the work 'cake' and set a specific sound (with a rule_id of 'SSByZWFsbHkgbGlrZSBjYWtl'):: + + curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" + +To add a rule suppressing notifications for messages starting with 'cake' but ending with 'lie', superseeding the previous rule:: + + curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake*lie", "actions" : ["notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" + +To add a custom sound for notifications messages containing the word 'beer' in any rooms with 10 members or fewer (with greater importance than the room, sender and content rules):: + + curl -X PUT -H "Content-Type: application/json" -d '{ "conditions": [{"kind": "event_match", "key": "content.body", "pattern": "beer" }, {"kind": "room_member_count", "is": "<=10"}], "actions" : ["notify", {"set_sound":"beeroclock.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456 + From 1a5678786fb34d35110d44ca5a5aefd53509f34a Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 18:36:04 +0000 Subject: [PATCH 038/526] Doc deleting & getting rules. --- drafts/push_csapi.rst | 98 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 3c6edac6..0a3294d4 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -219,3 +219,101 @@ To add a custom sound for notifications messages containing the word 'beer' in a curl -X PUT -H "Content-Type: application/json" -d '{ "conditions": [{"kind": "event_match", "key": "content.body", "pattern": "beer" }, {"kind": "room_member_count", "is": "<=10"}], "actions" : ["notify", {"set_sound":"beeroclock.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456 + +To delete rules, a client would just make a DELETE request to the same URL:: + + curl -X DELETE "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456" + + +Retrieving the current ruleset can be done either by fetching individual rules +using the scheme as specified above. This returns the rule in the same format as +would be given in the PUT API with the addition of a rule_id:: + + curl "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456" + +Returns:: + + { + "actions": [ + "dont_notify" + ], + "rule_id": "#spam:matrix.org" + } + +Clients can also fetch broader sets of rules by removing path components. +Requesting the root level returns a structure as follows:: + + { + "device": { + "exampledevice": { + "content": [], + "default": [], + "override": [], + "room": [ + { + "actions": [ + "dont_notify" + ], + "rule_id": "#spam:matrix.org" + } + ], + "sender": [], + "underride": [] + } + }, + "global": { + "content": [], + "default": [ + { + "actions": [ + "notify", + { + "set_sound": "default" + } + ], + "conditions": [ + { + "key": "content.body", + "kind": "event_match", + "pattern": "*@test:steve*" + } + ] + }, + { + "actions": [ + "notify", + { + "set_sound": "default" + } + ], + "conditions": [ + { + "kind": "contains_display_name" + } + ] + }, + { + "actions": [ + "notify", + { + "set_sound": "default" + } + ], + "conditions": [ + { + "is": "2", + "kind": "room_member_count" + } + ] + } + ], + "override": [], + "room": [], + "sender": [], + "underride": [] + } + } + +Adding patch components to the request drills down into this structure to filter +to only the requested set of rules. + From 067e4bb7771282a71567bc60fc28b99dc75ad5e9 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 3 Feb 2015 14:21:56 +0000 Subject: [PATCH 039/526] Add docs for http notification poke protocol. --- drafts/push_overview.rst | 3 +- drafts/push_pgwapi.rst | 128 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+), 1 deletion(-) create mode 100644 drafts/push_pgwapi.rst diff --git a/drafts/push_overview.rst b/drafts/push_overview.rst index 6cee68c5..216ebdcd 100644 --- a/drafts/push_overview.rst +++ b/drafts/push_overview.rst @@ -24,7 +24,8 @@ Push Gateway A push gateway is a server that receives HTTP event notifications from Home Servers and passes them on to a different protocol such as APNS for iOS devices or GCM for Android devices. Matrix.org provides a reference push - gateway, 'sygnal'. + gateway, 'sygnal'. A client app tells a Home Server what push gateway + to send notifications to when it sets up a pusher. For information on the client-server API for setting pushers and push rules, see the Client Server API section. For more information on the format of HTTP diff --git a/drafts/push_pgwapi.rst b/drafts/push_pgwapi.rst new file mode 100644 index 00000000..bf0a4c37 --- /dev/null +++ b/drafts/push_pgwapi.rst @@ -0,0 +1,128 @@ +Push Notifications: HTTP Notification Protocol +============================================== +This describes the format used by "http" pushers to send notifications of +events. + +Notifications are sent as HTTP POST requests to the URL configured when the +pusher is created. The body of the POST request is a JSON dictionary. The format +is as follows:: + + { + "notification": { + "id": "$3957tyerfgewrf384", + "type": "m.room.message", + "sender": "@exampleuser:matrix.org", + "sender_display_name": "Major Tom", + "room_name": "Mission Control", + "room_alias": "#exampleroom:matrix.org", + "prio": "high", + "content": { + "msgtype": "m.text", + "body": "I'm floating in a most peculiar way." + } + }, + "counts": { + "unread" : 2, + "missed_calls": 1 + } + "devices": [ + { + "app_id": "org.matrix.matrixConsole.ios", + "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/", + "pushkey_ts": 12345678, + "data" : { + }, + "tweaks": { + "sound": "bing.wav" + } + } + ] + } + } + +The contents of this dictionary are defined as follows: + +id + An identifier for this notification that may be used to detect duplicate + notification requests. This is not necessarily the ID of the event that + triggered the notification. +type + The type of the event as in the event's 'type' field. +sender + The sender of the event as in the corresponding event field. +sender_display_name + The current display name of the sender in the room in which the event + occurred. +room_name + The name of the room in which the event occurred. +room_alias + An alias to display for the room in which the event occurred. +prio + The priority of the notification. Acceptable values are 'high' or 'low. If + omitted, 'high' is assumed. This may be used by push gateways to deliver less + time-sensitive notifications in a way that will preserve battery power on + mobile devices. +content + The 'content' field from the event, if present. If the event had no content + field, this field is omitted. +counts + This is a dictionary of the current number of unacknowledged communications + for the recipient user. Counts whose value is zero are omitted. +unread + The number of unread messages a user has accross all of the rooms they are a + member of. +missed_calls + The number of unacknowledged missed calls a user has accross all rooms of + which they are a member. +device + This is an array of devices that the notification should be sent to. +app_id + The app_id given when the pusher was created. +pushkey + The pushkey given when the pusher was created. +pushkey_ts + The unix timestamp (in seconds) when the pushkey was last updated. +data + A dictionary of additional pusher-specific data. For 'http' pushers, this is + the data dictionary passed in at pusher creation minus the 'url' key. +tweaks + A dictionary of customisations made to the way this notification is to be + presented. These are added by push rules. +sound + Sets the sound file that should be played. 'default' means that a default + sound should be played. + +The recipient of an HTTP notification should respond with an HTTP 2xx response +when the notification has been processed. If the endpoint returns an HTTP error +code, the Home Server should retry for a reasonable amount of time with a +reasonable backoff scheme. + +The endpoint should return a JSON dictionary as follows:: + + { + "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ] + } + +Whose keys are: + +rejected + A list of all pushkeys given in the notification request that are not valid. + These could have been rejected by an upstream gateway because they have + expired or have never been valid. Home Servers must cease sending notification + requests for these pushkeys and remove the associated pushers. It may not + necessarily be the notification in the request that failed: it could be that + a previous notification to the same pushkey failed. + +Push: Recommendations for APNS +------------------------------ +For sending APNS notifications, the exact format is flexible and up to the +client app and its push gateway to agree on (since APNS requires that the sender +have a private key owned by the app developer, each app must have its own push +gateway). However, Matrix strongly recommends: + + * That the APNS token be base64 encoded and used as the pushkey. + * That a different app_id be used for apps on the production and sandbox + APS environments. + * That PANS push gateways do not attempt to wait for errors from the APNS + gateway before returning and instead to store failures and return + 'rejected' responses next time that pushkey is used. From 8920d58cc43a6767c042a635ac6109713ce788ac Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 3 Feb 2015 14:35:01 +0000 Subject: [PATCH 040/526] Strongly recommend standard URI scheme for notification pokes. --- drafts/push_pgwapi.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drafts/push_pgwapi.rst b/drafts/push_pgwapi.rst index bf0a4c37..8dd054f6 100644 --- a/drafts/push_pgwapi.rst +++ b/drafts/push_pgwapi.rst @@ -4,7 +4,11 @@ This describes the format used by "http" pushers to send notifications of events. Notifications are sent as HTTP POST requests to the URL configured when the -pusher is created. The body of the POST request is a JSON dictionary. The format +pusher is created, but Matrix strongly recommends that the path should be:: + + /_matrix/push/v1/notify + +The body of the POST request is a JSON dictionary. The format is as follows:: { From 91d12dc9553995fd5dc8907cfaed7b842755c6b5 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 3 Feb 2015 15:38:12 +0000 Subject: [PATCH 041/526] Replace set_sound with more generic set_tweaks. --- drafts/push_csapi.rst | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 0a3294d4..b979331f 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -97,8 +97,8 @@ For some kinds of rule, rules of the same kind also have an ordering with respect to one another. The kinds that do not are room and sender rules where the rules are mutually exclusive by definition and therefore an ordering would be redundant. Actions for the highest priority rule and only that rule apply -(for example, a set_sound action in a lower priority rule will not apply if a -higher priority rule matches, even if that rule does not specify a sound). +(for example, a set_tweak action in a lower priority rule will not apply if a +higher priority rule matches, even if that rule does not specify any tweaks). Rules also have an identifier, rule_id, which is a string. @@ -118,14 +118,28 @@ coalesce specific behaviour to intelligently coalesce multiple events into a single notification. Not all Home Servers may support this. Those that do not should treat it as the 'notify' action. -set_sound - Sets the value 'sound' key that is sent in the notification poke. This has an - associated string which is the value to set the 'sound' key to. - -Actions that have no parameter are represented as a string. Those with a -parameter are represented as a dictionary with a single key/value pair where the -key is the name of the action and the value is the parameter, eg. { "set_sound": -"ping.wav" } +set_tweak + Sets an entry in the 'tweaks' dictionary key that is sent in the notification + poke. This takes the form of a dictionary with a 'set_tweak' key whose value + is the name of the tweak to set. It must also have a 'value' key which is + the value to which it should be set. + +Actions that have no parameters are represented as a string. Otherwise, they are +represented as a dictionary with a key equal to their name and other keys as +their parameters, eg. { "set_tweak": "sound", "value": "default" } + +Push Rule Actions: Tweaks +------------------------- +The 'set_tweak' key action is used to add an entry to the 'tweaks' dictionary +that is sent in the notification poke. The following tweaks are e defined: + +sound + A sound to be played when this notification arrives. 'default' means to + play a default sound. + +Tweaks are passed transparently through the Home Server so client applications +and push gateways may agree on additional tweaks, for example, how to flash the +notification light on a mobile device. Push Rules: Conditions: ----------------------- From f24e951b024c206aa9fa4caee77576d6a1ed12ac Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 3 Feb 2015 16:21:09 +0000 Subject: [PATCH 042/526] s/instance_handle/profile_tag/ --- drafts/push_csapi.rst | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index b979331f..8e1fd99c 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -19,15 +19,13 @@ pushkey kind The kind of pusher to configure. 'http' makes a pusher that sends HTTP pokes. null deletes the pusher. -instance_handle - This is an identifier for the device which owns the pusher. It may be up to 32 - characters long. It must be unique among all the pushers for a given user - (therefore the device ID may not be used). It is advised that when an app's - data is copied or restored to a different device, this ID remain the same (ie. - be shared by multiple pushers for multiple devices). Client apps should be - aware that this situation can occur and be able to rectify it (eg. by - offerring to reset the instance_hanlde, optionally duplicating all push rules - to new instance handle). +profile_tag + This is a string that determines what set of device rules will be matched when + evaluating push rules for this pusher. It is an arbitrary string. Multiple + devices maybe use the same profile_tag. It is advised that when an app's + data is copied or restored to a different device, this value remain the same. + Client apps should offer ways to change the profile_tag, optionally copying + rules from the old profile tag. app_id appId is a reverse-DNS style identifier for the application. It is recommended that this end with the platform, such that different platform versions get @@ -77,7 +75,7 @@ Default In addition, each kind of rule except default may be either global or device-specific. Device specific rules only affect delivery of notifications via -pushers with a matching instance_handle. All device-specific rules are higher +pushers with a matching profile_tag. All device-specific rules are higher priority than all global rules. Thusly, the full list of rule kinds, in descending priority order, is as follows: @@ -154,10 +152,10 @@ event_match * 'pattern': The glob-style pattern to match against. Patterns with no special glob characters should be treated as having asterisks prepended and appended when testing the condition. -device - Matches the instance_handle of the device that the notification would be +profile_tag + Matches the profile_tag of the device that the notification would be delivered to. Parameters: - * 'instance_handle': The instance_handle of the device. + * 'profile_tag': The profile_tag to match with. contains_display_name This matches unencrypted messages where content.body contains the owner's display name in that room. This is a separate rule because display names may @@ -185,8 +183,8 @@ Rules live under a hierarchy in the REST API that resembles:: The component parts are as follows: scope - Either 'global' or 'device/' to specify global rules or - device rules for the given instance_handle. + Either 'global' or 'device/' to specify global rules or + device rules for the given profile_tag. kind The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content' or 'default'. From dbbf9626b4e2d1c0549bae0daee8891553179fa2 Mon Sep 17 00:00:00 2001 From: David Baker Date: Wed, 4 Feb 2015 11:34:32 +0000 Subject: [PATCH 043/526] Fix various incorreect examples in the doc. --- drafts/push_csapi.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 8e1fd99c..9016d0e2 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -61,7 +61,7 @@ Content Rules conditions, below. Room Rules These change the behaviour of all messages to a given room. The rule_id of a - room rule is always the room that it affects. + room rule is always the ID of the room that it affects. Sender These rules configure notification behaviour for messages from a specific, named Matrix user ID. The rule_id of Sender rules is always the Matrix user @@ -211,21 +211,21 @@ appropriate parameters for the rule (under the appropriate key name). Examples: -To create a rule that suppresses notifications for the room '#spam:matrix.org':: +To create a rule that suppresses notifications for the room with ID '!dj234r78wl45Gh4D:matrix.org':: - curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456" + curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" To suppress notifications for the user '@spambot:matrix.org':: - curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/room/%40spambot%3Amatrix.org?access_token=123456" + curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" To always notify for messages that contain the work 'cake' and set a specific sound (with a rule_id of 'SSByZWFsbHkgbGlrZSBjYWtl'):: - curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" + curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" To add a rule suppressing notifications for messages starting with 'cake' but ending with 'lie', superseeding the previous rule:: - curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake*lie", "actions" : ["notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" + curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake*lie", "actions" : ["notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" To add a custom sound for notifications messages containing the word 'beer' in any rooms with 10 members or fewer (with greater importance than the room, sender and content rules):: From 8184855d5ad4dfe03957732b1b3d202c5cce8fdb Mon Sep 17 00:00:00 2001 From: David Baker Date: Wed, 4 Feb 2015 16:30:57 +0000 Subject: [PATCH 044/526] add note about scope of rule_ids. --- drafts/push_csapi.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 9016d0e2..e9abf326 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -98,7 +98,10 @@ be redundant. Actions for the highest priority rule and only that rule apply (for example, a set_tweak action in a lower priority rule will not apply if a higher priority rule matches, even if that rule does not specify any tweaks). -Rules also have an identifier, rule_id, which is a string. +Rules also have an identifier, rule_id, which is a string. The rule_id may +alphanumeric characters only. The rule_id is unique within the kind of rule and +scope: rule_ids need not be unique between rules of the same kind on different +devices. Push Rules: Actions: -------------------- From e6c9e77d350dbc5eceaf6c6a85f3b4dbc19e9a22 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 5 Feb 2015 10:49:48 +0000 Subject: [PATCH 045/526] Change server default rules. --- drafts/push_csapi.rst | 69 ++++++++----------------------------------- 1 file changed, 12 insertions(+), 57 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index e9abf326..ae9591bf 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -69,15 +69,12 @@ Sender Underride These are identical to override rules, but have a lower priority than content, room and sender rules. -Default - These are rules provided by the home server and cannot be changed by the user. - They are the lowest priority rule and establish baseline behaviour. -In addition, each kind of rule except default may be either global or -device-specific. Device specific rules only affect delivery of notifications via -pushers with a matching profile_tag. All device-specific rules are higher -priority than all global rules. Thusly, the full list of rule kinds, in -descending priority order, is as follows: +In addition, each kind of rule may be either global or device-specific. Device +specific rules only affect delivery of notifications via pushers with a matching +profile_tag. All device-specific rules are higher priority than all global +rules. Thusly, the full list of rule kinds, in descending priority order, is as +follows: * Device-specific Override * Device-specific Content @@ -89,7 +86,6 @@ descending priority order, is as follows: * Global Room * Global Sender * Global Underride - * Global Default For some kinds of rule, rules of the same kind also have an ordering with respect to one another. The kinds that do not are room and sender rules where @@ -101,7 +97,12 @@ higher priority rule matches, even if that rule does not specify any tweaks). Rules also have an identifier, rule_id, which is a string. The rule_id may alphanumeric characters only. The rule_id is unique within the kind of rule and scope: rule_ids need not be unique between rules of the same kind on different -devices. +devices. + +A home server may also have server default rules of each kind and in each scope. +Server default rules are lower priority than user-defined rules in eacgh scope. +Server defined rules do not have a rule_id. A rule has a rule_id if and only if +it is a user-defined rule. Push Rules: Actions: -------------------- @@ -189,8 +190,7 @@ scope Either 'global' or 'device/' to specify global rules or device rules for the given profile_tag. kind - The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content' or - 'default'. + The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content'. rule_id The identifier for the rule. @@ -262,7 +262,6 @@ Requesting the root level returns a structure as follows:: "device": { "exampledevice": { "content": [], - "default": [], "override": [], "room": [ { @@ -278,50 +277,6 @@ Requesting the root level returns a structure as follows:: }, "global": { "content": [], - "default": [ - { - "actions": [ - "notify", - { - "set_sound": "default" - } - ], - "conditions": [ - { - "key": "content.body", - "kind": "event_match", - "pattern": "*@test:steve*" - } - ] - }, - { - "actions": [ - "notify", - { - "set_sound": "default" - } - ], - "conditions": [ - { - "kind": "contains_display_name" - } - ] - }, - { - "actions": [ - "notify", - { - "set_sound": "default" - } - ], - "conditions": [ - { - "is": "2", - "kind": "room_member_count" - } - ] - } - ], "override": [], "room": [], "sender": [], From 94af47ffd4eb051282779df1ab6f54f906ae0489 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 5 Feb 2015 13:03:10 +0000 Subject: [PATCH 046/526] Server default riules will have a rule_id when you need it to work out what the rule does. --- drafts/push_csapi.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index ae9591bf..39645ae2 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -101,8 +101,8 @@ devices. A home server may also have server default rules of each kind and in each scope. Server default rules are lower priority than user-defined rules in eacgh scope. -Server defined rules do not have a rule_id. A rule has a rule_id if and only if -it is a user-defined rule. +Server defined rules do not have a rule_id except when it is necessary to derive +the function of the rule (ie. in room and sender rules). Push Rules: Actions: -------------------- From 8360be247b45236410423563298957323265bac7 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 5 Feb 2015 13:15:34 +0000 Subject: [PATCH 047/526] Leave out what characters are allowed in a rule ID as we allow most thigs in matrix usernames currently. --- drafts/push_csapi.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 39645ae2..5f8515f1 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -94,10 +94,9 @@ be redundant. Actions for the highest priority rule and only that rule apply (for example, a set_tweak action in a lower priority rule will not apply if a higher priority rule matches, even if that rule does not specify any tweaks). -Rules also have an identifier, rule_id, which is a string. The rule_id may -alphanumeric characters only. The rule_id is unique within the kind of rule and -scope: rule_ids need not be unique between rules of the same kind on different -devices. +Rules also have an identifier, rule_id, which is a string. The rule_id is +unique within the kind of rule and scope: rule_ids need not be unique between +rules of the same kind on different devices. A home server may also have server default rules of each kind and in each scope. Server default rules are lower priority than user-defined rules in eacgh scope. From 7c0d1c86d8f38f19a6c114961e13f6a83472907b Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 5 Feb 2015 14:56:07 +0000 Subject: [PATCH 048/526] How to tell server default rules from other rules. --- drafts/push_csapi.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 5f8515f1..1cc0131e 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -101,7 +101,8 @@ rules of the same kind on different devices. A home server may also have server default rules of each kind and in each scope. Server default rules are lower priority than user-defined rules in eacgh scope. Server defined rules do not have a rule_id except when it is necessary to derive -the function of the rule (ie. in room and sender rules). +the function of the rule (ie. in room and sender rules). Server default rules +have an attribute, "default" set to true. Push Rules: Actions: -------------------- From e7c522b6068670173a519639b5fa96f831631323 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 5 Feb 2015 16:06:47 +0000 Subject: [PATCH 049/526] Add information on exclusive namespaces --- drafts/application_services.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/drafts/application_services.rst b/drafts/application_services.rst index c654c010..dfec19f7 100644 --- a/drafts/application_services.rst +++ b/drafts/application_services.rst @@ -86,7 +86,13 @@ In the registration process, the AS provides: "subscribing to" * Namespaces are defined as a list of regexps against which to match room aliases, room IDs, and user IDs. Regexps give the flexibility to say, sub-domain MSISDN - ranges per AS, whereas a blunt prefix string does not. + ranges per AS, whereas a blunt prefix string does not. These namespaces are further + configured by setting whether they are ``exclusive`` or not. An exclusive namespace + prevents entities other than the aforementioned AS from creating/editing/deleting + entries within that namespace. This does not affect the visibility/readability of + entries within that namespace (e.g. it doesn't prevent users joining exclusive + aliases, or ASes from listening to exclusive aliases, but does prevent both users + and ASes from creating/editing/deleting aliases within that namespace). * There is overlap between selecting events via the csv2 Filter API and subscribing to events here - perhaps subscription involves passing a filter token into the registration API. From 36395c26111ef7eb09b3cad2d6d2ce4a4bbc40cb Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 6 Feb 2015 11:01:20 +0000 Subject: [PATCH 050/526] Add info on M_EXCLUSIVE --- drafts/as-http-api.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 1c1f342f..0d040868 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -330,6 +330,11 @@ including the AS token on a ``/register`` request, along with a login type of type: "m.login.application_service", user: "" } + +Application services which attempt to create users or aliases *outside* of +their defined namespaces will receive an error code ``M_EXCLUSIVE``. Similarly, +normal users who attempt to create users or alises *inside* an application +service-defined namespace will receive the same ``M_EXCLUSIVE`` error code. ID conventions ``[Draft]`` ------------------------- From 6c5d07a0d14d5ad8a9084bf6b8aa1096cea648d7 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 8 Feb 2015 02:32:06 +0000 Subject: [PATCH 051/526] typoe --- drafts/push_pgwapi.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/push_pgwapi.rst b/drafts/push_pgwapi.rst index 8dd054f6..bb95acaf 100644 --- a/drafts/push_pgwapi.rst +++ b/drafts/push_pgwapi.rst @@ -127,6 +127,6 @@ gateway). However, Matrix strongly recommends: * That the APNS token be base64 encoded and used as the pushkey. * That a different app_id be used for apps on the production and sandbox APS environments. - * That PANS push gateways do not attempt to wait for errors from the APNS + * That APNS push gateways do not attempt to wait for errors from the APNS gateway before returning and instead to store failures and return 'rejected' responses next time that pushkey is used. From 2c8b875cf133b14604f6cb45b55f5eb76511e6ed Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 8 Feb 2015 02:33:53 +0000 Subject: [PATCH 052/526] typoe --- drafts/push_csapi.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 1cc0131e..c23ff0b1 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -133,7 +133,7 @@ their parameters, eg. { "set_tweak": "sound", "value": "default" } Push Rule Actions: Tweaks ------------------------- The 'set_tweak' key action is used to add an entry to the 'tweaks' dictionary -that is sent in the notification poke. The following tweaks are e defined: +that is sent in the notification poke. The following tweaks are defined: sound A sound to be played when this notification arrives. 'default' means to From 7a59d401f64fd6d1fd3c1c97b4c822b2bb2a0442 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 9 Feb 2015 09:17:55 +0000 Subject: [PATCH 053/526] Fix syntax errors --- drafts/as-http-api.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 0d040868..0dfb6ca7 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -337,7 +337,7 @@ normal users who attempt to create users or alises *inside* an application service-defined namespace will receive the same ``M_EXCLUSIVE`` error code. ID conventions ``[Draft]`` -------------------------- +-------------------------- .. NOTE:: - Giving HSes the freedom to namespace still feels like the Right Thing here. - Exposing a public API provides the consistency which was the main complaint @@ -418,6 +418,7 @@ Pre-conditions: - Home server has the domain "hsdomain.com" 1. Application service registration + :: AS -> HS: Registers itself with the home server @@ -444,12 +445,14 @@ Pre-conditions: } 2. IRC user "Bob" says "hello?" on "#matrix" at timestamp 1421416883133: + :: - AS stores message as potential scrollback. - Nothing happens as no Matrix users are in the room. 3. Matrix user "@alice:hsdomain.com" wants to join "#matrix": + :: User -> HS: Request to join "#irc.freenode.net/#matrix:hsdomain.com" @@ -484,6 +487,7 @@ Pre-conditions: - HS sends room information back to client. 4. @alice:hsdomain.com says "hi!" in this room: + :: User -> HS: Send message "hi!" in room !aasaasasa:hsdomain.com @@ -512,6 +516,7 @@ Pre-conditions: 5. IRC user "Bob" says "what's up?" on "#matrix" at timestamp 1421418084816: + :: IRC -> AS: "what's up?" From 56f4467cede56b3014318f914705e8b7f372bdbd Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 9 Feb 2015 09:19:45 +0000 Subject: [PATCH 054/526] Fix syntax error --- drafts/push_csapi.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index c23ff0b1..9c89170d 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -159,6 +159,7 @@ event_match profile_tag Matches the profile_tag of the device that the notification would be delivered to. Parameters: + * 'profile_tag': The profile_tag to match with. contains_display_name This matches unencrypted messages where content.body contains the owner's From 8729cdb067e1382e562694e92064503c43e43a1d Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 12 Feb 2015 14:52:35 +0000 Subject: [PATCH 055/526] Fix broken JSFiddle links --- howtos/client-server.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/howtos/client-server.rst b/howtos/client-server.rst index 37346224..fe1538a6 100644 --- a/howtos/client-server.rst +++ b/howtos/client-server.rst @@ -24,7 +24,7 @@ If you already have an account, you must **login** into it. `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/register_login +.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/register_login Registration ------------ @@ -88,7 +88,7 @@ user and **send a message** to that room. `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/create_room_send_msg +.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/create_room_send_msg Creating a room --------------- @@ -138,7 +138,7 @@ join a room **via a room alias** if one was set up. `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/room_memberships +.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/room_memberships Inviting a user to a room ------------------------- @@ -184,7 +184,7 @@ of getting events, depending on what the client already knows. `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/event_stream +.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/event_stream Getting all state ----------------- @@ -634,4 +634,4 @@ application. `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/example_app +.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/example_app From 246531813bc7e3893596d20ba27aff5f27bd1516 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 12 Feb 2015 20:23:36 +0000 Subject: [PATCH 056/526] typo --- drafts/push_csapi.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 9c89170d..01bbe00a 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -99,7 +99,7 @@ unique within the kind of rule and scope: rule_ids need not be unique between rules of the same kind on different devices. A home server may also have server default rules of each kind and in each scope. -Server default rules are lower priority than user-defined rules in eacgh scope. +Server default rules are lower priority than user-defined rules in each scope. Server defined rules do not have a rule_id except when it is necessary to derive the function of the rule (ie. in room and sender rules). Server default rules have an attribute, "default" set to true. From 7c65e4748e201ec3ea19c04c5ec4b00a98fa290c Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 12 Feb 2015 20:24:14 +0000 Subject: [PATCH 057/526] typo --- drafts/media_repository.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/media_repository.rst b/drafts/media_repository.rst index 14a7c967..25ef9dd6 100644 --- a/drafts/media_repository.rst +++ b/drafts/media_repository.rst @@ -73,4 +73,4 @@ Clients may try to access a large number of remote files through a homeserver. Homeservers should restrict the number and size of remote files that it caches. Clients or remote homeservers may try to upload malicious files targeting -vunerabilities in either the homeserver thumbnailing or the client decoders. +vulnerabilities in either the homeserver thumbnailing or the client decoders. From b02f2aa3641064211d705b643caeed06e8afd302 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 12 Feb 2015 20:31:44 +0000 Subject: [PATCH 058/526] never upscale images --- drafts/media_repository.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drafts/media_repository.rst b/drafts/media_repository.rst index 25ef9dd6..879e1b85 100644 --- a/drafts/media_repository.rst +++ b/drafts/media_repository.rst @@ -52,7 +52,8 @@ within a given rectangle. Homeservers may generate thumbnails for content uploaded to remote homeservers themselves or may rely on the remote homeserver to thumbnail the content. Homeservers may return thumbnails of a different size to that -requested. However homeservers should provide extact matches where reasonable. +requested. However homeservers should provide exact matches where reasonable. +Homeservers must never upscale images. Security -------- From 7a0f7b4b43c5eb34e9fb7e62a270edd2a8e08c90 Mon Sep 17 00:00:00 2001 From: David Baker Date: Fri, 13 Feb 2015 13:40:44 +0000 Subject: [PATCH 059/526] Add that rules with no conditions always match. --- drafts/push_csapi.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 01bbe00a..9cef9004 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -147,8 +147,9 @@ Push Rules: Conditions: ----------------------- Override, Underride and Default rules have a list of 'conditions'. All conditions must hold true for an event in order for a rule to be applied to an -event. Matrix specifies the following conditions, although if Home Servers wish -to support others, they are free to do so: +event. A rule with no conditions always matches. Matrix specifies the following +conditions, although if Home Servers wish to support others, they are free to +do so: event_match This is a glob pattern match on a field of the event. Parameters: From bcba42651a2782998bc731f89f9761577d8a5820 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Mon, 16 Feb 2015 13:10:25 +0000 Subject: [PATCH 060/526] Add support for exclusive namespaces As per https://github.com/matrix-org/matrix-doc/pull/5#issuecomment-73071259 --- drafts/as-http-api.rst | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 0dfb6ca7..d7b076c3 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -27,11 +27,22 @@ Side effects: API called when: - The application service wants to register with a brand new home server. Notes: + - An application service can state whether they should be the only ones who + can manage a specified namespace. This is referred to as an "exclusive" + namespace. An exclusive namespace prevents humans and other application + services from creating/deleting entities in that namespace. Typically, + exclusive namespaces are used when the rooms represent real rooms on + another service (e.g. IRC). Non-exclusive namespaces are used when the + application service is merely augmenting the room itself (e.g. providing + logging or searching facilities). - Namespaces are represented by POSIX extended regular expressions in JSON. They look like:: users: [ - "@irc\.freenode\.net/.*", + { + "exclusive": true, + "regex": "@irc\.freenode\.net/.*" + } ] :: @@ -44,13 +55,22 @@ Notes: as_token: "some_AS_token", namespaces: { users: [ - "@irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "@irc\.freenode\.net/.*" + } ], aliases: [ - "#irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "#irc\.freenode\.net/.*" + } ], rooms: [ - "!irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "!irc\.freenode\.net/.*" + } ] } } @@ -428,13 +448,16 @@ Pre-conditions: as_token: "T_a", namespaces: { users: [ - "@irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "@irc\.freenode\.net/.*" + } ], aliases: [ - "#irc\.freenode\.net/.*" - ], - rooms: [ - "!irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "#irc\.freenode\.net/.*" + } ] } } From f3a498d0bb49a68fcffaadb65afd6ccc2a06e6a3 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 16 Feb 2015 22:09:10 +0000 Subject: [PATCH 061/526] document the concept of a standard `external_url` field for gatewayed events --- drafts/as-http-api.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index d7b076c3..488bfe30 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -425,6 +425,15 @@ SHOULD be mapped in the same way as "user" URIs. alias: } +Event fields +~~~~~~~~~~~~ +We recommend that any gatewayed events should include an `external_url` field in +their content to provide a way for Matrix clients to link into the 'native' +client from which the event originated. For instance, this could contain the +message-ID for emails/nntp posts, or a link to a blog comment when gatewaying +blog comment traffic in & out of matrix + + Examples -------- .. NOTE:: From f0cc0e082c39d3054ba3d381c8022be25ff41879 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 19 Feb 2015 10:13:21 +0000 Subject: [PATCH 062/526] Add room_id in push poke to spec. --- drafts/push_pgwapi.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drafts/push_pgwapi.rst b/drafts/push_pgwapi.rst index bb95acaf..92035581 100644 --- a/drafts/push_pgwapi.rst +++ b/drafts/push_pgwapi.rst @@ -14,6 +14,7 @@ is as follows:: { "notification": { "id": "$3957tyerfgewrf384", + "room_id": "!slw48wfj34rtnrf:example.com", "type": "m.room.message", "sender": "@exampleuser:matrix.org", "sender_display_name": "Major Tom", @@ -50,6 +51,8 @@ id An identifier for this notification that may be used to detect duplicate notification requests. This is not necessarily the ID of the event that triggered the notification. +room_id + The ID of the room in which this event occurred. type The type of the event as in the event's 'type' field. sender From 3ce1e559545b1ecd0e52e63b89be609142f19dbc Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 19 Feb 2015 11:44:10 +0000 Subject: [PATCH 063/526] Move TODO comments into the doc in prep for merging --- drafts/as-http-api.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 488bfe30..3cc8efbb 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -1,3 +1,10 @@ +.. TODO + Sometimes application services need to create rooms (e.g. when lazy loading + from room aliases). Created rooms need to have a user that created them, so + federation works (as it relies on an entry existing in m.room.member). We + should be able to add metadata to m.room.member to state that this user is an + application service, a virtual user, etc. + Application Services HTTP API ============================= From 8aae3e58428a6d9cd15dd8238109ef7e3e50fa8e Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 19 Feb 2015 11:48:45 +0000 Subject: [PATCH 064/526] Add TODO section --- drafts/application_services.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drafts/application_services.rst b/drafts/application_services.rst index dfec19f7..9c8b6b69 100644 --- a/drafts/application_services.rst +++ b/drafts/application_services.rst @@ -1,3 +1,10 @@ +.. TODO + Sometimes application services need to create rooms (e.g. when lazy loading + from room aliases). Created rooms need to have a user that created them, so + federation works (as it relies on an entry existing in m.room.member). We should + be able to add metadata to m.room.member to state that this user is an application + service, a virtual user, etc. + Application Services ==================== From f60c0ac96b5d077c9f46029e49aa23c6b743a388 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 19 Feb 2015 13:42:01 +0000 Subject: [PATCH 065/526] Fix indentation levels on specification. --- specification/11_event_signing.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/specification/11_event_signing.rst b/specification/11_event_signing.rst index 68f9c178..baa3b6b7 100644 --- a/specification/11_event_signing.rst +++ b/specification/11_event_signing.rst @@ -1,8 +1,8 @@ Signing Events -============== +-------------- Canonical JSON --------------- +~~~~~~~~~~~~~~ Matrix events are represented using JSON objects. If we want to sign JSON events we need to encode the JSON as a binary string. Unfortunately the same @@ -38,7 +38,7 @@ using this representation. ).encode("UTF-8") Grammar -~~~~~~~ ++++++++ Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing insignificant whitespace, fractions, exponents and redundant character escapes @@ -69,14 +69,14 @@ insignificant whitespace, fractions, exponents and redundant character escapes / %x75.30.30.31 (%x30-39 / %x61-66) ; u001X Signing JSON ------------- +~~~~~~~~~~~~ We can now sign a JSON object by encoding it as a sequence of bytes, computing the signature for that sequence and then adding the signature to the original JSON object. Signing Details -~~~~~~~~~~~~~~~ ++++++++++++++++ JSON is signed by encoding the JSON object without ``signatures`` or keys grouped as ``unsigned``, using the canonical encoding described above. The JSON bytes are then signed using the @@ -133,7 +133,7 @@ and additional signatures. return json_object Checking for a Signature -~~~~~~~~~~~~~~~~~~~~~~~~ +++++++++++++++++++++++++ To check if an entity has signed a JSON object a server does the following @@ -151,7 +151,7 @@ To check if an entity has signed a JSON object a server does the following the check fails. Otherwise the check succeeds. Signing Events --------------- +~~~~~~~~~~~~~~ Signing events is a more complicated process since servers can choose to redact non-essential parts of an event. Before signing the event it is encoded as @@ -229,8 +229,8 @@ too many hashes to be listed, otherwise a server might embed illict data within the ``hash`` object. For similar reasons a server shouldn't allow hash values that are too long. -[[TODO(markjh): We might want to specify a maximum number of keys for the -``hash`` and we might want to specify the maximum output size of a hash]] - -[[TODO(markjh) We might want to allow the server to omit the output of well -known hash functions like SHA-256 when none of the keys have been redacted]] +.. TODO + [[TODO(markjh): We might want to specify a maximum number of keys for the + ``hash`` and we might want to specify the maximum output size of a hash]] + [[TODO(markjh) We might want to allow the server to omit the output of well + known hash functions like SHA-256 when none of the keys have been redacted]] From 8e75c5ed67e8acce5448a8f37858f011d167d56e Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 19 Feb 2015 14:17:31 +0000 Subject: [PATCH 066/526] Add in passive application services --- specification/40_application_service_api.rst | 444 ++++++++++++++++++- 1 file changed, 441 insertions(+), 3 deletions(-) diff --git a/specification/40_application_service_api.rst b/specification/40_application_service_api.rst index 687f477e..fdaf5e7c 100644 --- a/specification/40_application_service_api.rst +++ b/specification/40_application_service_api.rst @@ -18,10 +18,448 @@ Client-Server Services Passive Application Services ---------------------------- +"Passive" application services can only observe events from a given home server. +They cannot prevent events from being sent, nor can they modify the event being +sent. -.. TODO-spec - API that extends the client-server API to allow events to be - received with better-than-client permissions. +In order to observe events from a home server, the application service needs +to register itself and tell the home server what kinds of events it wants +to receive. + +Application Service -> Home Server API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This contains home server APIs which are used by the application service. + +Registration API +++++++++++++++++ + +This API registers the application service with its host homeserver to offer its +services. + +Inputs: + - Credentials (e.g. some kind of string token) + - Namespace[users] + - Namespace[room aliases] + - URL base to receive inbound comms +Output: + - The credentials the HS will use to query the AS with in return. (e.g. some + kind of string token) +Side effects: + - The HS will start delivering events to the URL base specified if this 200s. +API called when: + - The application service wants to register with a brand new home server. +Notes: + - An application service can state whether they should be the only ones who + can manage a specified namespace. This is referred to as an "exclusive" + namespace. An exclusive namespace prevents humans and other application + services from creating/deleting entities in that namespace. Typically, + exclusive namespaces are used when the rooms represent real rooms on + another service (e.g. IRC). Non-exclusive namespaces are used when the + application service is merely augmenting the room itself (e.g. providing + logging or searching facilities). + - Namespaces are represented by POSIX extended regular expressions in JSON. + They look like:: + + users: [ + { + "exclusive": true, + "regex": "@irc\.freenode\.net/.*" + } + ] + +:: + + POST /register + + Request format + { + url: "https://my.application.service.com/matrix/", + as_token: "some_AS_token", + namespaces: { + users: [ + { + "exclusive": true, + "regex": "@irc\.freenode\.net/.*" + } + ], + aliases: [ + { + "exclusive": true, + "regex": "#irc\.freenode\.net/.*" + } + ], + rooms: [ + { + "exclusive": true, + "regex": "!irc\.freenode\.net/.*" + } + ] + } + } + + + Returns: + 200 : Registration accepted. + 400 : Namespaces do not conform to regex + 401 : Credentials need to be supplied. + 403 : AS credentials rejected. + + + 200 OK response format + + { + hs_token: "string" + } + +Unregister API +++++++++++++++ +This API unregisters a previously registered AS from the home server. + +Inputs: + - AS token +Output: + - None. +Side effects: + - The HS will stop delivering events to the URL base specified for this AS if + this 200s. +API called when: + - The application service wants to stop receiving all events from the HS. + +:: + + POST /unregister + + Request format + { + as_token: "string" + } + + +Home Server -> Application Service API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This contains application service APIs which are used by the home server. All +application services MUST implement these APIs. + +User Query +++++++++++ + +This API is called by the HS to query the existence of a user on the Application +Service's namespace. + +Inputs: + - User ID + - HS Credentials +Output: + - Whether the user exists. +Side effects: + - User is created on the HS by the AS via CS APIs during the processing of this request. +API called when: + - HS receives an event for an unknown user ID in the AS's namespace, e.g. an + invite event to a room. +Notes: + - When the AS receives this request, if the user exists, it must create the user via + the CS API. + - It can also set arbitrary information about the user (e.g. display name, join rooms, etc) + using the CS API. + - When this setup is complete, the AS should respond to the HS request. This means the AS + blocks the HS until the user is created. + - This is deemed more flexible than alternative methods (e.g. returning a JSON blob with the + user's display name and get the HS to provision the user). +Retry notes: + - The home server cannot respond to the client's request until the response to + this API is obtained from the AS. + - Recommended that home servers try a few times then time out, returning a + 408 Request Timeout to the client. + +:: + + GET /users/$user_id?access_token=$hs_token + + Returns: + 200 : User is recognised. + 404 : User not found. + 401 : Credentials need to be supplied. + 403 : HS credentials rejected. + + + 200 OK response format + + {} + +Room Alias Query +++++++++++++++++ +This API is called by the HS to query the existence of a room alias on the +Application Service's namespace. + +Inputs: + - Room alias + - HS Credentials +Output: + - Whether the room exists. +Side effects: + - Room is created on the HS by the AS via CS APIs during the processing of + this request. +API called when: + - HS receives an event to join a room alias in the AS's namespace. +Notes: + - When the AS receives this request, if the room exists, it must create the room via + the CS API. + - It can also set arbitrary information about the room (e.g. name, topic, etc) + using the CS API. + - It can send messages as other users in order to populate scrollback. + - When this setup is complete, the AS should respond to the HS request. This means the AS + blocks the HS until the room is created and configured. + - This is deemed more flexible than alternative methods (e.g. returning an initial sync + style JSON blob and get the HS to provision the room). It also means that the AS knows + the room ID -> alias mapping. +Retry notes: + - The home server cannot respond to the client's request until the response to + this API is obtained from the AS. + - Recommended that home servers try a few times then time out, returning a + 408 Request Timeout to the client. + +:: + + GET /rooms/$room_alias?access_token=$hs_token + + Returns: + 200 : Room is recognised. + 404 : Room not found. + 401 : Credentials need to be supplied. + 403 : HS credentials rejected. + + + 200 OK response format + + {} + +Pushing ++++++++ +This API is called by the HS when the HS wants to push an event (or batch of +events) to the AS. + +Inputs: + - HS Credentials + - Event(s) to give to the AS + - HS-generated transaction ID +Output: + - None. + +Data flows: + +:: + + Typical + HS ---> AS : Home server sends events with transaction ID T. + <--- : AS sends back 200 OK. + + AS ACK Lost + HS ---> AS : Home server sends events with transaction ID T. + <-/- : AS 200 OK is lost. + HS ---> AS : Home server retries with the same transaction ID of T. + <--- : AS sends back 200 OK. If the AS had processed these events + already, it can NO-OP this request (and it knows if it is the same + events based on the transacton ID). + + +Retry notes: + - If the HS fails to pass on the events to the AS, it must retry the request. + - Since ASes by definition cannot alter the traffic being passed to it (unlike + say, a Policy Server), these requests can be done in parallel to general HS + processing; the HS doesn't need to block whilst doing this. + - Home servers should use exponential backoff as their retry algorithm. + - Home servers MUST NOT alter (e.g. add more) events they were going to + send within that transaction ID on retries, as the AS may have already + processed the events. + +Ordering notes: + - The events sent to the AS should be linearised, as they are from the event + stream. + - The home server will need to maintain a queue of transactions to send to + the AS. + +:: + + PUT /transactions/$transaction_id?access_token=$hs_token + + Request format + { + events: [ + ... + ] + } + +Client-Server v2 API Extensions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Passive application services can utilise a more powerful version of the +client-server API by identifying itself as an application service to the +home server. + +Identity assertion +++++++++++++++++++ +The client-server API infers the user ID from the ``access_token`` provided in +every request. It would be an annoying amount of book-keeping to maintain tokens +for every virtual user. It would be preferable if the application service could +use the CS API with its own ``as_token`` instead, and specify the virtual user +they wish to be acting on behalf of. For real users, this would require +additional permissions granting the AS permission to masquerade as a matrix user. + +Inputs: + - Application service token (``access_token``) + + Either: + - User ID in the AS namespace to act as. + Or: + - OAuth2 token of real user (which may end up being an access token) +Notes: + - This will apply on all aspects of the CS API, except for Account Management. + - The ``as_token`` is inserted into ``access_token`` which is usually where the + client token is. This is done on purpose to allow application services to + reuse client SDKs. + +:: + + /path?access_token=$token&user_id=$userid + + Query Parameters: + access_token: The application service token + user_id: The desired user ID to act as. + + /path?access_token=$token&user_token=$token + + Query Parameters: + access_token: The application service token + user_token: The token granted to the AS by the real user + +Timestamp massaging ++++++++++++++++++++ +The application service may want to inject events at a certain time (reflecting +the time on the network they are tracking e.g. irc, xmpp). Application services +need to be able to adjust the ``origin_server_ts`` value to do this. + +Inputs: + - Application service token (``as_token``) + - Desired timestamp +Notes: + - This will only apply when sending events. + +:: + + /path?access_token=$token&ts=$timestamp + + Query Parameters added to the send event APIs only: + access_token: The application service token + ts: The desired timestamp + +Server admin style permissions +++++++++++++++++++++++++++++++ +The home server needs to give the application service *full control* over its +namespace, both for users and for room aliases. This means that the AS should +be able to create/edit/delete any room alias in its namespace, as well as +create/delete any user in its namespace. No additional API changes need to be +made in order for control of room aliases to be granted to the AS. Creation of +users needs API changes in order to: + +- Work around captchas. +- Have a 'passwordless' user. + +This involves bypassing the registration flows entirely. This is achieved by +including the AS token on a ``/register`` request, along with a login type of +``m.login.application_service`` to set the desired user ID without a password. + +:: + + /register?access_token=$as_token + + Content: + { + type: "m.login.application_service", + user: "" + } + +Application services which attempt to create users or aliases *outside* of +their defined namespaces will receive an error code ``M_EXCLUSIVE``. Similarly, +normal users who attempt to create users or alises *inside* an application +service-defined namespace will receive the same ``M_EXCLUSIVE`` error code, +but only if the application service has defined the namespace as ``exclusive``. + +ID conventions +~~~~~~~~~~~~~~ +.. NOTE:: + - Giving HSes the freedom to namespace still feels like the Right Thing here. + - Exposing a public API provides the consistency which was the main complaint + against namespacing. + - This may have knock-on effects for the AS registration API. E.g. why don't + we let ASes specify the *URI* regex they want? + +This concerns the well-defined conventions for mapping 3P network IDs to matrix +IDs, which we expect clients to be able to do by themselves. + +User IDs +++++++++ +Matrix users may wish to directly contact a virtual user, e.g. to send an email. +The URI format is a well-structured way to represent a number of different ID +types, including: + +- MSISDNs (``tel``) +- Email addresses (``mailto``) +- IRC nicks (``irc`` - https://tools.ietf.org/html/draft-butcher-irc-url-04) +- XMPP (xep-0032) +- SIP URIs (RFC 3261) + +As a result, virtual user IDs SHOULD relate to their URI counterpart. This +mapping from URI to user ID can be expressed in a number of ways: + +- Expose a C-S API on the HS which takes URIs and responds with user IDs. +- Munge the URI with the user ID. + +Exposing an API would allow HSes to internally map user IDs however they like, +at the cost of an extra round trip (of which the response can be cached). +Munging the URI would allow clients to apply the mapping locally, but would force +user X on service Y to always map to the same munged user ID. Considering the +exposed API could just be applying this munging, there is more flexibility if +an API is exposed. + +:: + + GET /_matrix/app/v1/user?uri=$url_encoded_uri + + Returns 200 OK: + { + user_id: + } + +Room Aliases +++++++++++++ +We may want to expose some 3P network rooms so Matrix users can join them directly, +e.g. IRC rooms. We don't want to expose every 3P network room though, e.g. mailto, +tel. Rooms which are publicly accessible (e.g. IRC rooms) can be exposed as an alias by +the application service. Private rooms (e.g. sending an email to someone) should not +be exposed in this way, but should instead operate using normal invite/join semantics. +Therefore, the ID conventions discussed below are only valid for public rooms which +expose room aliases. + +Matrix users may wish to join XMPP rooms (e.g. using XEP-0045) or IRC rooms. In both +cases, these rooms can be expressed as URIs. For consistency, these "room" URIs +SHOULD be mapped in the same way as "user" URIs. + +:: + + GET /_matrix/app/v1/alias?uri=$url_encoded_uri + + Returns 200 OK: + { + alias: + } + +Event fields +++++++++++++ +We recommend that any gatewayed events should include an ``external_url`` field +in their content to provide a way for Matrix clients to link into the 'native' +client from which the event originated. For instance, this could contain the +message-ID for emails/nntp posts, or a link to a blog comment when gatewaying +blog comment traffic in & out of matrix Active Application Services ---------------------------- From 42857df26f506917aceeff9713259f7b80196c04 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 19 Feb 2015 14:25:54 +0000 Subject: [PATCH 067/526] Syntax fixes --- drafts/as-http-api.rst | 2 ++ drafts/model/protocol_examples.rst | 2 +- specification/11_event_signing.rst | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 3cc8efbb..64b0e5d2 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -5,10 +5,12 @@ should be able to add metadata to m.room.member to state that this user is an application service, a virtual user, etc. + Application Services HTTP API ============================= .. contents:: Table of Contents + .. sectnum:: Application Service -> Home Server diff --git a/drafts/model/protocol_examples.rst b/drafts/model/protocol_examples.rst index f1d31f34..d4f9a0a2 100644 --- a/drafts/model/protocol_examples.rst +++ b/drafts/model/protocol_examples.rst @@ -32,7 +32,7 @@ Content-Type: application/json } HTTP/1.1 200 OK -... + ====================================== diff --git a/specification/11_event_signing.rst b/specification/11_event_signing.rst index baa3b6b7..61493dbd 100644 --- a/specification/11_event_signing.rst +++ b/specification/11_event_signing.rst @@ -234,3 +234,4 @@ that are too long. ``hash`` and we might want to specify the maximum output size of a hash]] [[TODO(markjh) We might want to allow the server to omit the output of well known hash functions like SHA-256 when none of the keys have been redacted]] + From df448df813fdeb2b854bc71a89de8b63376f22d1 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 19 Feb 2015 16:00:21 +0000 Subject: [PATCH 068/526] Redo gendoc script. --- .gitignore | 1 + scripts/gendoc.py | 53 +++++++++++++++++++++++++++++++++++++ scripts/gendoc.sh | 23 ---------------- scripts/matrx-org-gendoc.sh | 21 +++++++++++++++ 4 files changed, 75 insertions(+), 23 deletions(-) create mode 100644 .gitignore create mode 100755 scripts/gendoc.py delete mode 100755 scripts/gendoc.sh create mode 100755 scripts/matrx-org-gendoc.sh diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..c838ab29 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +scripts/gen diff --git a/scripts/gendoc.py b/scripts/gendoc.py new file mode 100755 index 00000000..c5a7aae0 --- /dev/null +++ b/scripts/gendoc.py @@ -0,0 +1,53 @@ +#! /usr/bin/env python + +from docutils.core import publish_file +import glob +import os +import shutil +import sys + +stylesheets = { + "stylesheet_path": ["basic.css", "nature.css"] +} + +def glob_spec(out_file_name): + with open(out_file_name, "wb") as outfile: + for f in sorted(glob.glob("../specification/*.rst")): + with open(f, "rb") as infile: + outfile.write(infile.read()) + + +def rst2html(i, o): + with open(i, "r") as in_file: + with open(o, "w") as out_file: + publish_file( + source=in_file, + destination=out_file, + reader_name="standalone", + parser_name="restructuredtext", + writer_name="html", + settings_overrides=stylesheets + ) + +def prepare_env(): + try: + os.makedirs("./gen") + except OSError: + pass + try: + os.makedirs("./tmp") + except OSError: + pass + +def cleanup_env(): + shutil.rmtree("./tmp") + +def main(): + prepare_env() + glob_spec("tmp/full_spec.rst") + rst2html("tmp/full_spec.rst", "gen/specification.html") + rst2html("../howtos/client-server.rst", "gen/howtos.html") + cleanup_env() + +if __name__ == '__main__': + main() diff --git a/scripts/gendoc.sh b/scripts/gendoc.sh deleted file mode 100755 index 4a92f681..00000000 --- a/scripts/gendoc.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash - -MATRIXDOTORG=$HOME/workspace/matrix.org - -# list of places we mess with the template: -# -# docs/*/*.html -# mailman -# alpha -# swagger -# blog - -cat ../specification/*.rst > /tmp/spec.rst -rst2html-2.7.py --stylesheet=basic.css,nature.css /tmp/spec.rst > $MATRIXDOTORG/docs/spec/index.html -rst2html-2.7.py --stylesheet=basic.css,nature.css ../howtos/client-server.rst > $MATRIXDOTORG/docs/howtos/client-server.html - -perl -pi -e 's###' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html - -perl -MFile::Slurp -pi -e 'BEGIN { $nav = read_file("'$MATRIXDOTORG'/includes/nav.html") } s##
[matrix]
#' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html - -perl -pi -e 's##
#' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html - -scp -r $MATRIXDOTORG/docs matrix@ldc-prd-matrix-001:/sites/matrix diff --git a/scripts/matrx-org-gendoc.sh b/scripts/matrx-org-gendoc.sh new file mode 100755 index 00000000..18961046 --- /dev/null +++ b/scripts/matrx-org-gendoc.sh @@ -0,0 +1,21 @@ +#! /bin/bash + +if [ -z "$1" ]; then + echo "Expected /includes/nav.html file as arg." + exit 1 +fi + +NAV_BAR=$1 + +if [ ! -f $NAV_BAR ]; then + echo $NAV_BAR " does not exist" + exit 1 +fi + +python gendoc.py + +perl -pi -e 's###' gen/specification.html gen/howtos.html + +perl -MFile::Slurp -pi -e 'BEGIN { $nav = read_file("'$NAV_BAR'") } s##
[matrix]
#' gen/specification.html gen/howtos.html + +perl -pi -e 's##
#' gen/specification.html gen/howtos.html From 4b0e858529c3522dc2c622b335729d82fde3e859 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 19 Feb 2015 16:32:45 +0000 Subject: [PATCH 069/526] Add $GIT_VERSION to the spec which is replaced by the gendoc script. --- scripts/gendoc.py | 62 ++++++++++++++++++++++++++++++++++++++ specification/00_basis.rst | 3 +- 2 files changed, 64 insertions(+), 1 deletion(-) diff --git a/scripts/gendoc.py b/scripts/gendoc.py index c5a7aae0..7af17259 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -1,20 +1,81 @@ #! /usr/bin/env python from docutils.core import publish_file +import fileinput import glob import os import shutil +import subprocess import sys stylesheets = { "stylesheet_path": ["basic.css", "nature.css"] } + +def get_git_ver_string(): + null = open(os.devnull, 'w') + cwd = os.path.dirname(os.path.abspath(__file__)) + try: + git_branch = subprocess.check_output( + ['git', 'rev-parse', '--abbrev-ref', 'HEAD'], + stderr=null, + cwd=cwd, + ).strip() + except subprocess.CalledProcessError: + git_branch = "" + try: + git_tag = subprocess.check_output( + ['git', 'describe', '--exact-match'], + stderr=null, + cwd=cwd, + ).strip() + git_tag = "tag=" + git_tag + except subprocess.CalledProcessError: + git_tag = "" + try: + git_commit = subprocess.check_output( + ['git', 'rev-parse', '--short', 'HEAD'], + stderr=null, + cwd=cwd, + ).strip() + except subprocess.CalledProcessError: + git_commit = "" + try: + dirty_string = "-this_is_a_dirty_checkout" + is_dirty = subprocess.check_output( + ['git', 'describe', '--dirty=' + dirty_string, "--all"], + stderr=null, + cwd=cwd, + ).strip().endswith(dirty_string) + git_dirty = "dirty" if is_dirty else "" + except subprocess.CalledProcessError: + git_dirty = "" + + if git_branch or git_tag or git_commit or git_dirty: + git_version = ",".join( + s for s in + (git_branch, git_tag, git_commit, git_dirty,) + if s + ) + return git_version.encode("ascii") + return "Unknown rev" + + def glob_spec(out_file_name): with open(out_file_name, "wb") as outfile: for f in sorted(glob.glob("../specification/*.rst")): with open(f, "rb") as infile: outfile.write(infile.read()) + +def set_git_version(filename): + git_ver = get_git_ver_string() + # inplace search and replace, stdout is redirected to the output + # file, hence the "print" lines here. + for line in fileinput.input(filename, inplace=True): + if "$GIT_VERSION" in line: + line = line.replace("$GIT_VERSION", git_ver) + print line.rstrip("\n") def rst2html(i, o): @@ -45,6 +106,7 @@ def cleanup_env(): def main(): prepare_env() glob_spec("tmp/full_spec.rst") + set_git_version("tmp/full_spec.rst") rst2html("tmp/full_spec.rst", "gen/specification.html") rst2html("../howtos/client-server.rst", "gen/howtos.html") cleanup_env() diff --git a/specification/00_basis.rst b/specification/00_basis.rst index 77aec367..22a78a2c 100644 --- a/specification/00_basis.rst +++ b/specification/00_basis.rst @@ -1,5 +1,6 @@ Matrix Specification ==================== +$GIT_VERSION Table of Contents ================= @@ -85,7 +86,7 @@ Version ======= The Matrix spec is currently rapidly evolving, and there have been no versioned -releases as yet. Versions should be identified by git revision, or failing that +releases as yet. Versions should be identified by git revision, or failing that timestamp. Overview From 5a0e40b9846ec569a9282ff482fae3153a5df7f6 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 19 Feb 2015 16:36:53 +0000 Subject: [PATCH 070/526] Mark CS API as v1 in the spec. --- specification/20_client_server_api.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specification/20_client_server_api.rst b/specification/20_client_server_api.rst index 7e06cdd0..af071d6f 100644 --- a/specification/20_client_server_api.rst +++ b/specification/20_client_server_api.rst @@ -1,5 +1,7 @@ -Client-Server API -================= +Client-Server API v1 +==================== + +This outlines version 1 of the client-server API. Registration and Login ---------------------- From 1388612c4877d0b0b33e148edd9cd86b5130a47c Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 19 Feb 2015 16:53:54 +0000 Subject: [PATCH 071/526] Add a README, spell matrix correctly. --- scripts/README | 10 ++++++++++ scripts/{matrx-org-gendoc.sh => matrix-org-gendoc.sh} | 0 2 files changed, 10 insertions(+) create mode 100644 scripts/README rename scripts/{matrx-org-gendoc.sh => matrix-org-gendoc.sh} (100%) diff --git a/scripts/README b/scripts/README new file mode 100644 index 00000000..b9285412 --- /dev/null +++ b/scripts/README @@ -0,0 +1,10 @@ +Requirements: + - docutils (for converting RST to HTML) + +To generate the complete specification along with supporting documentation, run: + python gendoc.py + +The output of this will be inside the "gen" folder. + +Matrix.org only ("gen" folder has matrix.org tweaked pages): + ./matrix-org-gendoc.sh /includes/nav.html diff --git a/scripts/matrx-org-gendoc.sh b/scripts/matrix-org-gendoc.sh similarity index 100% rename from scripts/matrx-org-gendoc.sh rename to scripts/matrix-org-gendoc.sh From 4fd5b9ced82ec9ed57a091e30a476f32acd8bf1a Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 19 Feb 2015 17:14:25 +0000 Subject: [PATCH 072/526] Update doc style We use RST everywhere, which dictates most of the style, so the styles mentioned previously were either obsolete or inaccurate. Updated the doc to be clearer on things which RST does not specify/care about. --- meta/documentation_style.rst | 54 +++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 26 deletions(-) diff --git a/meta/documentation_style.rst b/meta/documentation_style.rst index c365d09d..5cc5ce82 100644 --- a/meta/documentation_style.rst +++ b/meta/documentation_style.rst @@ -1,43 +1,45 @@ -=================== Documentation Style =================== A brief single sentence to describe what this file contains; in this case a description of the style to write documentation in. +Format +------ -Sections -======== - -Each section should be separated from the others by two blank lines. Headings -should be underlined using a row of equals signs (===). Paragraphs should be -separated by a single blank line, and wrap to no further than 80 columns. +We use restructured text throughout all the documentation. You should NOT use +markdown (github-flavored or otherwise). This format was chosen partly because +Sphinx only supports RST. -[[TODO(username): if you want to leave some unanswered questions, notes for -further consideration, or other kinds of comment, use a TODO section. Make sure -to notate it with your name so we know who to ask about it!]] -Subsections ------------ +Sections +-------- -If required, subsections can use a row of dashes to underline their header. A -single blank line between subsections of a single section. +RST support lots of different punctuation characters for underlines on sections. +Content in the specification MUST use the same characters in order for the +complete specification to be merged correctly. These characters are: +- ``=`` : Top-level sections +- ``-`` : Second-level sections +- ``~`` : Third-level sections +- ``+`` : Fourth-level sections +- You should rethink your document layout if you require a fifth level. -Bullet Lists -============ +TODOs +----- - * Bullet lists can use asterisks with a single space either side. - - * Another blank line between list elements. +Any RST file in this repository may make it onto ``matrix.org``. We do not want +``TODO`` markers visible there. For internal comments, notes, TODOs, use standard +RST comments like so:: + .. TODO-Bob + There is something to do here. This will not be rendered by something like + rst2html.py so it is safe to put internal comments here. -Definition Lists -================ +You SHOULD put your username with the TODO so we know who to ask about it. -Terms: - Start in the first column, ending with a colon +Line widths +----------- -Definitions: - Take a two space indent, following immediately from the term without a blank - line before it, but having a blank line afterwards. +We use 80 characters for line widths. This is a guideline and can be flouted IF +AND ONLY IF it makes reading more legible. Use common sense. From 6af1627b518fc8fd24588649d9e11570a245528c Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 19 Feb 2015 17:16:59 +0000 Subject: [PATCH 073/526] Minor tweak for matrix.org example --- scripts/README | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/README b/scripts/README index b9285412..bf97843a 100644 --- a/scripts/README +++ b/scripts/README @@ -7,4 +7,4 @@ To generate the complete specification along with supporting documentation, run: The output of this will be inside the "gen" folder. Matrix.org only ("gen" folder has matrix.org tweaked pages): - ./matrix-org-gendoc.sh /includes/nav.html + ./matrix-org-gendoc.sh /path/to/matrix.org/includes/nav.html From 006932c70f23b06a6e5cc4c07656b4a4033b9896 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 19 Feb 2015 17:25:15 +0000 Subject: [PATCH 074/526] Be clearer when specifying the version of the spec. --- specification/00_basis.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specification/00_basis.rst b/specification/00_basis.rst index 22a78a2c..e97d3406 100644 --- a/specification/00_basis.rst +++ b/specification/00_basis.rst @@ -1,6 +1,8 @@ Matrix Specification ==================== -$GIT_VERSION + +.. NOTE:: + The git version of this document is ``$GIT_VERSION`` Table of Contents ================= From d552a043f810c62fed3933cfc571a90fa7b4c2e6 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 19 Feb 2015 20:22:57 +0000 Subject: [PATCH 075/526] remove LI section entirely until we decide how best to discourage it, otherwise it looks like we're endorsing it --- specification/50_appendices.rst | 6 ------ 1 file changed, 6 deletions(-) diff --git a/specification/50_appendices.rst b/specification/50_appendices.rst index 2ed2a2ce..295c8f69 100644 --- a/specification/50_appendices.rst +++ b/specification/50_appendices.rst @@ -136,9 +136,3 @@ Identity Servers .. TODO-doc Dave - 3PIDs and identity server, functions - -Lawful Interception -------------------- - -Key Escrow Servers -~~~~~~~~~~~~~~~~~~ From c4056bec1b6d9e49b9690db414b8cc486b6a73c7 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 20 Feb 2015 10:43:06 +0000 Subject: [PATCH 076/526] Shuffle around structure of matrix-doc and add a README. --- {swagger => api}/README | 0 {swagger => api}/client-server/api-docs | 0 {swagger => api}/client-server/api-docs-content | 0 {swagger => api}/client-server/api-docs-directory | 0 {swagger => api}/client-server/api-docs-events | 0 {swagger => api}/client-server/api-docs-login | 0 {swagger => api}/client-server/api-docs-presence | 0 {swagger => api}/client-server/api-docs-profile | 0 {swagger => api}/client-server/api-docs-registration | 0 {swagger => api}/client-server/api-docs-rooms | 0 {swagger => api}/files/backbone-min.js | 0 {swagger => api}/files/css | 0 {swagger => api}/files/handlebars-1.0.0.js | 0 {swagger => api}/files/highlight.7.3.pack.js | 0 {swagger => api}/files/jquery-1.8.0.min.js | 0 {swagger => api}/files/jquery.ba-bbq.min.js | 0 {swagger => api}/files/jquery.slideto.min.js | 0 {swagger => api}/files/jquery.wiggle.min.js | 0 {swagger => api}/files/reset.css | 0 {swagger => api}/files/screen.css | 0 {swagger => api}/files/shred.bundle.js | 0 {swagger => api}/files/swagger-oauth.js | 0 {swagger => api}/files/swagger-ui.js | 0 {swagger => api}/files/swagger.js | 0 {swagger => api}/files/underscore-min.js | 0 {swagger => api}/swagger.html | 0 {howtos => supporting-docs/howtos}/client-server.rst | 0 .../howtos/jsfiddles}/create_room_send_msg/demo.css | 0 .../howtos/jsfiddles}/create_room_send_msg/demo.html | 0 .../howtos/jsfiddles}/create_room_send_msg/demo.js | 0 .../howtos/jsfiddles}/event_stream/demo.css | 0 .../howtos/jsfiddles}/event_stream/demo.html | 0 .../howtos/jsfiddles}/event_stream/demo.js | 0 .../howtos/jsfiddles}/example_app/demo.css | 0 .../howtos/jsfiddles}/example_app/demo.details | 0 .../howtos/jsfiddles}/example_app/demo.html | 0 .../howtos/jsfiddles}/example_app/demo.js | 0 .../howtos/jsfiddles}/register_login/demo.css | 0 .../howtos/jsfiddles}/register_login/demo.html | 0 .../howtos/jsfiddles}/register_login/demo.js | 0 .../howtos/jsfiddles}/room_memberships/demo.css | 0 .../howtos/jsfiddles}/room_memberships/demo.html | 0 .../howtos/jsfiddles}/room_memberships/demo.js | 0 43 files changed, 0 insertions(+), 0 deletions(-) rename {swagger => api}/README (100%) rename {swagger => api}/client-server/api-docs (100%) rename {swagger => api}/client-server/api-docs-content (100%) rename {swagger => api}/client-server/api-docs-directory (100%) rename {swagger => api}/client-server/api-docs-events (100%) rename {swagger => api}/client-server/api-docs-login (100%) rename {swagger => api}/client-server/api-docs-presence (100%) rename {swagger => api}/client-server/api-docs-profile (100%) rename {swagger => api}/client-server/api-docs-registration (100%) rename {swagger => api}/client-server/api-docs-rooms (100%) rename {swagger => api}/files/backbone-min.js (100%) rename {swagger => api}/files/css (100%) rename {swagger => api}/files/handlebars-1.0.0.js (100%) rename {swagger => api}/files/highlight.7.3.pack.js (100%) rename {swagger => api}/files/jquery-1.8.0.min.js (100%) rename {swagger => api}/files/jquery.ba-bbq.min.js (100%) rename {swagger => api}/files/jquery.slideto.min.js (100%) rename {swagger => api}/files/jquery.wiggle.min.js (100%) rename {swagger => api}/files/reset.css (100%) rename {swagger => api}/files/screen.css (100%) rename {swagger => api}/files/shred.bundle.js (100%) rename {swagger => api}/files/swagger-oauth.js (100%) rename {swagger => api}/files/swagger-ui.js (100%) rename {swagger => api}/files/swagger.js (100%) rename {swagger => api}/files/underscore-min.js (100%) rename {swagger => api}/swagger.html (100%) rename {howtos => supporting-docs/howtos}/client-server.rst (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/create_room_send_msg/demo.css (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/create_room_send_msg/demo.html (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/create_room_send_msg/demo.js (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/event_stream/demo.css (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/event_stream/demo.html (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/event_stream/demo.js (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/example_app/demo.css (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/example_app/demo.details (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/example_app/demo.html (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/example_app/demo.js (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/register_login/demo.css (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/register_login/demo.html (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/register_login/demo.js (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/room_memberships/demo.css (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/room_memberships/demo.html (100%) rename {jsfiddles => supporting-docs/howtos/jsfiddles}/room_memberships/demo.js (100%) diff --git a/swagger/README b/api/README similarity index 100% rename from swagger/README rename to api/README diff --git a/swagger/client-server/api-docs b/api/client-server/api-docs similarity index 100% rename from swagger/client-server/api-docs rename to api/client-server/api-docs diff --git a/swagger/client-server/api-docs-content b/api/client-server/api-docs-content similarity index 100% rename from swagger/client-server/api-docs-content rename to api/client-server/api-docs-content diff --git a/swagger/client-server/api-docs-directory b/api/client-server/api-docs-directory similarity index 100% rename from swagger/client-server/api-docs-directory rename to api/client-server/api-docs-directory diff --git a/swagger/client-server/api-docs-events b/api/client-server/api-docs-events similarity index 100% rename from swagger/client-server/api-docs-events rename to api/client-server/api-docs-events diff --git a/swagger/client-server/api-docs-login b/api/client-server/api-docs-login similarity index 100% rename from swagger/client-server/api-docs-login rename to api/client-server/api-docs-login diff --git a/swagger/client-server/api-docs-presence b/api/client-server/api-docs-presence similarity index 100% rename from swagger/client-server/api-docs-presence rename to api/client-server/api-docs-presence diff --git a/swagger/client-server/api-docs-profile b/api/client-server/api-docs-profile similarity index 100% rename from swagger/client-server/api-docs-profile rename to api/client-server/api-docs-profile diff --git a/swagger/client-server/api-docs-registration b/api/client-server/api-docs-registration similarity index 100% rename from swagger/client-server/api-docs-registration rename to api/client-server/api-docs-registration diff --git a/swagger/client-server/api-docs-rooms b/api/client-server/api-docs-rooms similarity index 100% rename from swagger/client-server/api-docs-rooms rename to api/client-server/api-docs-rooms diff --git a/swagger/files/backbone-min.js b/api/files/backbone-min.js similarity index 100% rename from swagger/files/backbone-min.js rename to api/files/backbone-min.js diff --git a/swagger/files/css b/api/files/css similarity index 100% rename from swagger/files/css rename to api/files/css diff --git a/swagger/files/handlebars-1.0.0.js b/api/files/handlebars-1.0.0.js similarity index 100% rename from swagger/files/handlebars-1.0.0.js rename to api/files/handlebars-1.0.0.js diff --git a/swagger/files/highlight.7.3.pack.js b/api/files/highlight.7.3.pack.js similarity index 100% rename from swagger/files/highlight.7.3.pack.js rename to api/files/highlight.7.3.pack.js diff --git a/swagger/files/jquery-1.8.0.min.js b/api/files/jquery-1.8.0.min.js similarity index 100% rename from swagger/files/jquery-1.8.0.min.js rename to api/files/jquery-1.8.0.min.js diff --git a/swagger/files/jquery.ba-bbq.min.js b/api/files/jquery.ba-bbq.min.js similarity index 100% rename from swagger/files/jquery.ba-bbq.min.js rename to api/files/jquery.ba-bbq.min.js diff --git a/swagger/files/jquery.slideto.min.js b/api/files/jquery.slideto.min.js similarity index 100% rename from swagger/files/jquery.slideto.min.js rename to api/files/jquery.slideto.min.js diff --git a/swagger/files/jquery.wiggle.min.js b/api/files/jquery.wiggle.min.js similarity index 100% rename from swagger/files/jquery.wiggle.min.js rename to api/files/jquery.wiggle.min.js diff --git a/swagger/files/reset.css b/api/files/reset.css similarity index 100% rename from swagger/files/reset.css rename to api/files/reset.css diff --git a/swagger/files/screen.css b/api/files/screen.css similarity index 100% rename from swagger/files/screen.css rename to api/files/screen.css diff --git a/swagger/files/shred.bundle.js b/api/files/shred.bundle.js similarity index 100% rename from swagger/files/shred.bundle.js rename to api/files/shred.bundle.js diff --git a/swagger/files/swagger-oauth.js b/api/files/swagger-oauth.js similarity index 100% rename from swagger/files/swagger-oauth.js rename to api/files/swagger-oauth.js diff --git a/swagger/files/swagger-ui.js b/api/files/swagger-ui.js similarity index 100% rename from swagger/files/swagger-ui.js rename to api/files/swagger-ui.js diff --git a/swagger/files/swagger.js b/api/files/swagger.js similarity index 100% rename from swagger/files/swagger.js rename to api/files/swagger.js diff --git a/swagger/files/underscore-min.js b/api/files/underscore-min.js similarity index 100% rename from swagger/files/underscore-min.js rename to api/files/underscore-min.js diff --git a/swagger/swagger.html b/api/swagger.html similarity index 100% rename from swagger/swagger.html rename to api/swagger.html diff --git a/howtos/client-server.rst b/supporting-docs/howtos/client-server.rst similarity index 100% rename from howtos/client-server.rst rename to supporting-docs/howtos/client-server.rst diff --git a/jsfiddles/create_room_send_msg/demo.css b/supporting-docs/howtos/jsfiddles/create_room_send_msg/demo.css similarity index 100% rename from jsfiddles/create_room_send_msg/demo.css rename to supporting-docs/howtos/jsfiddles/create_room_send_msg/demo.css diff --git a/jsfiddles/create_room_send_msg/demo.html b/supporting-docs/howtos/jsfiddles/create_room_send_msg/demo.html similarity index 100% rename from jsfiddles/create_room_send_msg/demo.html rename to supporting-docs/howtos/jsfiddles/create_room_send_msg/demo.html diff --git a/jsfiddles/create_room_send_msg/demo.js b/supporting-docs/howtos/jsfiddles/create_room_send_msg/demo.js similarity index 100% rename from jsfiddles/create_room_send_msg/demo.js rename to supporting-docs/howtos/jsfiddles/create_room_send_msg/demo.js diff --git a/jsfiddles/event_stream/demo.css b/supporting-docs/howtos/jsfiddles/event_stream/demo.css similarity index 100% rename from jsfiddles/event_stream/demo.css rename to supporting-docs/howtos/jsfiddles/event_stream/demo.css diff --git a/jsfiddles/event_stream/demo.html b/supporting-docs/howtos/jsfiddles/event_stream/demo.html similarity index 100% rename from jsfiddles/event_stream/demo.html rename to supporting-docs/howtos/jsfiddles/event_stream/demo.html diff --git a/jsfiddles/event_stream/demo.js b/supporting-docs/howtos/jsfiddles/event_stream/demo.js similarity index 100% rename from jsfiddles/event_stream/demo.js rename to supporting-docs/howtos/jsfiddles/event_stream/demo.js diff --git a/jsfiddles/example_app/demo.css b/supporting-docs/howtos/jsfiddles/example_app/demo.css similarity index 100% rename from jsfiddles/example_app/demo.css rename to supporting-docs/howtos/jsfiddles/example_app/demo.css diff --git a/jsfiddles/example_app/demo.details b/supporting-docs/howtos/jsfiddles/example_app/demo.details similarity index 100% rename from jsfiddles/example_app/demo.details rename to supporting-docs/howtos/jsfiddles/example_app/demo.details diff --git a/jsfiddles/example_app/demo.html b/supporting-docs/howtos/jsfiddles/example_app/demo.html similarity index 100% rename from jsfiddles/example_app/demo.html rename to supporting-docs/howtos/jsfiddles/example_app/demo.html diff --git a/jsfiddles/example_app/demo.js b/supporting-docs/howtos/jsfiddles/example_app/demo.js similarity index 100% rename from jsfiddles/example_app/demo.js rename to supporting-docs/howtos/jsfiddles/example_app/demo.js diff --git a/jsfiddles/register_login/demo.css b/supporting-docs/howtos/jsfiddles/register_login/demo.css similarity index 100% rename from jsfiddles/register_login/demo.css rename to supporting-docs/howtos/jsfiddles/register_login/demo.css diff --git a/jsfiddles/register_login/demo.html b/supporting-docs/howtos/jsfiddles/register_login/demo.html similarity index 100% rename from jsfiddles/register_login/demo.html rename to supporting-docs/howtos/jsfiddles/register_login/demo.html diff --git a/jsfiddles/register_login/demo.js b/supporting-docs/howtos/jsfiddles/register_login/demo.js similarity index 100% rename from jsfiddles/register_login/demo.js rename to supporting-docs/howtos/jsfiddles/register_login/demo.js diff --git a/jsfiddles/room_memberships/demo.css b/supporting-docs/howtos/jsfiddles/room_memberships/demo.css similarity index 100% rename from jsfiddles/room_memberships/demo.css rename to supporting-docs/howtos/jsfiddles/room_memberships/demo.css diff --git a/jsfiddles/room_memberships/demo.html b/supporting-docs/howtos/jsfiddles/room_memberships/demo.html similarity index 100% rename from jsfiddles/room_memberships/demo.html rename to supporting-docs/howtos/jsfiddles/room_memberships/demo.html diff --git a/jsfiddles/room_memberships/demo.js b/supporting-docs/howtos/jsfiddles/room_memberships/demo.js similarity index 100% rename from jsfiddles/room_memberships/demo.js rename to supporting-docs/howtos/jsfiddles/room_memberships/demo.js From 31ec39db7f6105323c28949a31289373cd4abaeb Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 20 Feb 2015 10:43:54 +0000 Subject: [PATCH 077/526] Actually add the README --- README.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 README.rst diff --git a/README.rst b/README.rst new file mode 100644 index 00000000..69952933 --- /dev/null +++ b/README.rst @@ -0,0 +1,24 @@ +This repository contains the documentation for Matrix. + +Structure +========= + +- ``api`` : Contains the HTTP API specification. +- ``drafts`` : Contains documents which will make it into the specification + and/or supporting documentation at some point in the future. +- ``meta`` : Contains documents outlining the processes involved when writing + documents, e.g. documentation style, guidelines. +- ``scripts`` : Contains scripts to generate formatted versions of the + documentation, typically HTML. +- ``specification`` : Contains the specification split up into sections. +- ``supporting-docs`` : Contains additional documents which explain design + decisions, examples, use cases, etc. + +Contributing +============ + +Known issues with the specification are represented as JIRA issues at +https://matrix.org/jira/browse/SPEC + +If you want to ask more about the specification, or have suggestions for +improvements, join us on ``#matrix-dev:matrix.org``. From 505c804426739f0251ac1fc81b5847f87ed9ede8 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 20 Feb 2015 10:49:36 +0000 Subject: [PATCH 078/526] Update scripts to look in the right places. Update jsfiddles to be more obvious in the HOWTO. --- scripts/gendoc.py | 2 +- supporting-docs/howtos/client-server.rst | 25 ++++++++++++++---------- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 7af17259..82d76a15 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -108,7 +108,7 @@ def main(): glob_spec("tmp/full_spec.rst") set_git_version("tmp/full_spec.rst") rst2html("tmp/full_spec.rst", "gen/specification.html") - rst2html("../howtos/client-server.rst", "gen/howtos.html") + rst2html("../supporting-docs/howtos/client-server.rst", "gen/howtos.html") cleanup_env() if __name__ == '__main__': diff --git a/supporting-docs/howtos/client-server.rst b/supporting-docs/howtos/client-server.rst index fe1538a6..bfb606a4 100644 --- a/supporting-docs/howtos/client-server.rst +++ b/supporting-docs/howtos/client-server.rst @@ -22,9 +22,10 @@ Accounts Before you can send and receive messages, you must **register** for an account. If you already have an account, you must **login** into it. -`Try out the fiddle`__ +.. NOTE:: + `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/register_login + .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/register_login Registration ------------ @@ -86,9 +87,10 @@ Communicating In order to communicate with another user, you must **create a room** with that user and **send a message** to that room. -`Try out the fiddle`__ +.. NOTE:: + `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/create_room_send_msg + .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/create_room_send_msg Creating a room --------------- @@ -136,9 +138,10 @@ these rules may specify if you require an **invitation** from someone already in the room in order to **join the room**. In addition, you may also be able to join a room **via a room alias** if one was set up. -`Try out the fiddle`__ +.. NOTE:: + `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/room_memberships + .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/room_memberships Inviting a user to a room ------------------------- @@ -182,9 +185,10 @@ An event is some interesting piece of data that a client may be interested in. It can be a message in a room, a room invite, etc. There are many different ways of getting events, depending on what the client already knows. -`Try out the fiddle`__ +.. NOTE:: + `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/event_stream + .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/event_stream Getting all state ----------------- @@ -632,6 +636,7 @@ creating and joining rooms, sending messages, getting member lists and getting historical messages for a room. This covers most functionality of a messaging application. -`Try out the fiddle`__ +.. NOTE:: + `Try out the fiddle`__ -.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/jsfiddles/example_app + .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/example_app From a6fa4e53fb76c1a0e618dcff786af75d1a4fb662 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 20 Feb 2015 11:25:05 +0000 Subject: [PATCH 079/526] Update gendoc.py Copy howto to tmp/ so the $GIT_VERSION can be dropped in. --- scripts/gendoc.py | 8 +++++--- supporting-docs/howtos/client-server.rst | 4 +++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 82d76a15..1b008685 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -62,7 +62,7 @@ def get_git_ver_string(): return "Unknown rev" -def glob_spec(out_file_name): +def glob_spec_to(out_file_name): with open(out_file_name, "wb") as outfile: for f in sorted(glob.glob("../specification/*.rst")): with open(f, "rb") as infile: @@ -105,10 +105,12 @@ def cleanup_env(): def main(): prepare_env() - glob_spec("tmp/full_spec.rst") + glob_spec_to("tmp/full_spec.rst") + shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst") set_git_version("tmp/full_spec.rst") + set_git_version("tmp/howto.rst") rst2html("tmp/full_spec.rst", "gen/specification.html") - rst2html("../supporting-docs/howtos/client-server.rst", "gen/howtos.html") + rst2html("tmp/howto.rst", "gen/howtos.html") cleanup_env() if __name__ == '__main__': diff --git a/supporting-docs/howtos/client-server.rst b/supporting-docs/howtos/client-server.rst index bfb606a4..f2830ff2 100644 --- a/supporting-docs/howtos/client-server.rst +++ b/supporting-docs/howtos/client-server.rst @@ -4,10 +4,12 @@ would be better if /register used the same technique as /login? /register should be "user" not "user_id". - How to use the client-server API ================================ +.. NOTE:: + The git version of this document is ``$GIT_VERSION`` + This guide focuses on how the client-server APIs *provided by the reference home server* can be used. Since this is specific to a home server implementation, there may be variations in relation to registering/logging in From 5cb656fba71e719c7321dc1c51e77a1d220a5a52 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 20 Feb 2015 11:32:53 +0000 Subject: [PATCH 080/526] Minor swagger HTML tweaks --- api/swagger.html | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/api/swagger.html b/api/swagger.html index 27e398f0..be6fb6b9 100644 --- a/api/swagger.html +++ b/api/swagger.html @@ -61,11 +61,9 @@ -