From 5f97e28c9bffc7aa268f8c45dd568bae25724a20 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 30 Dec 2014 03:00:37 +0000 Subject: [PATCH 01/62] initial formal proposal for the AS API --- drafts/applications_services.rst | 170 +++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 drafts/applications_services.rst diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst new file mode 100644 index 00000000..cee29212 --- /dev/null +++ b/drafts/applications_services.rst @@ -0,0 +1,170 @@ +Application Services +==================== + +Overview +======== + +Application services provide a way of implementing custom serverside functionality +on top of Matrix without the complexity of implementing the full federation API. +By acting as a trusted service logically located behind an existing homeserver, +Application services are decoupled from: + +* Signing or validating federated traffic or conversation history +* Validating authorisation constraints on federated traffic +* Managing routing or retry schemes to the rest of the Matrix federation + +As such, developers can focus entirely on implementing application logic rather +than being concerned with the details of managing Matrix federation. + +Features available to application services include: + +* Privileged subscription to any events available to the homeserver +* Synthesising virtual users +* Synthesising virtual rooms +* Injecting message history for virtual rooms + +Features not provided by application services include: + +* Intercepting and filtering/modifying message or behaviour within a room + (this is a job for a Policy Server, as it requires a single logical focal + point for messages in order to consistently apply the custom business logic) + +Example use cases for application services include: + +* Exposing existing communication services in Matrix + + * Gateways to/from standards-based protocols (SIP, XMPP, IRC, RCS (MSRP), + SIMPLE, Lync, etc) + * Gateways to/from closed services (e.g. WhatsApp) + * Gateways could be architected as: + + * Act as a virtual client on the non-Matrix network + (e.g. connect as multiple virtual clients to an IRC or XMPP server) + * Act as a server on the non-Matrix network + (e.g. speak s2s XMPP federation, or IRC link protocol) + * Act as an application service on the non-Matrix network + (e.g. link up as IRC services, or an XMPP component) + * Exposing a non-Matrix client interface listener from the AS + (e.g. listen on port 6667 for IRC clients, or port 5222 for XMPP clients) + + * Bridging existing APIs into Matrix + + * e.g. SMS/MMS aggregator APIs + * Domain-specific APIs such as SABRE + + * Integrating more exotic content into Matrix + + * e.g. MIDI<->Matrix gateway/bridge + * 3D world <-> Matrix bridge + + * Application services: + + * VoIP Conference services + * Text-to-speech and Speech-to-text services + * Signal processing + * IVR + * Server-machine translation + * Censorship service + * Multi-User Gaming (Dungeons etc) + * Other "constrained worlds" (e.g. 3D geometry representations) + + * applying physics to a 3D world on the serverside + + * (applying gravity and friction and air resistance... collision detection) + * domain-specific merge conflict resolution of events + + * Payment style transactional usecases with transactional guarantees + +Architecture Outline +==================== + +The application service registers with its host homeserver to offer its services. + +In the registration process, the AS provides: + + * Credentials to identify itself as an approved application service for that HS + * Details of the namespaces of users and rooms the AS is acting on behalf of and + "subscribing to" + * A URL base for receiving requests from the HS (as the AS is a server, + implementers expect to receive data via inbound requests rather than + long-poll outbound requests) + +On HS handling events to unknown users: + + * If the HS receives an event for an unknown user who is in the namespace delegated to + the AS, then the HS queries the AS for the profile of that user. If the AS + confirms the existence of that user (from its perspective), then the HS + creates an account to represent the virtual user. + * The namespace of virtual user accounts could conform to a structure like + @.irc.freenode.Arathorn:matrix.org or similar. + * The AS can preprovision virtual users using the existing CS API rather than + lazy-loading them in this manner. + +On HS handling events to unknown rooms: + + * If the HS receives an invite to an unknown room which is in the namespace + delegated to the AS, then the HS queries the AS for the existence of that room. + If the AS confirms its existence (from its perspective), then the HS creates + the room. + * The initial state of the room may be populated by the AS by querying an + initialSync API (probably a subset of the CS initialSync API, to reuse the + same pattern for the equivalent function). As messages have to be signed + from the point of m.room.create, we will not be able to back-populate + arbitrary history for rooms which are lazy-created in this manner, and instead + have to chose the amount of history to be synchronised into the AS as a one-off. + * If exposing arbitrary history is required, then either the room history must be + preemptively provisioned in the HS by the AS via the CS API (TODO: meaning the + CS API needs to support massaged timestamps), or the HS must delegate conversation + storage entirely to the AS using a Storage API (not defined here) which allows + the existing conversation store to back the HS, complete with all necessary + Matrix metadata (e.g. hashes, signatures, federation DAG, etc). + +On HS handling events to existing users and rooms: + + * If the HS receives an event for a user or room that already exist (either + provisioned by the AS or by normal client interactions), then the message + is handled as normal. + * Events in the namespaces of rooms and users that the AS has subscribed to + are pushed to the AS using the same pattern as the federation API (without + any of the encryption or federation metadata). TODO: are they linearised? + +On AS relaying events from unknown-to-HS users: + + * AS injects the event to the HS using the CS API, irrespective of whether the + target user or room is known to the HS or not. If the HS doesn't recognise + the target it goes through the same lazy-load provisioning as per above. + * The reason for not using a subset of the federation API here is because it + allows AS developers to reuse existing CS SDKs and benefit from the more + meaningful error handling of the CS API. The sending user ID must be + explicitly specified, as it cannot be inferred from the access_token, which + will be the same for all AS requests. + + * TODO: or do we maintain a separate access_token mapping? It seems like + unnecessary overhead for the AS developer; easier to just use a single + privileged access_token and just track which userid is emitting events? + +On AS relaying events in unknown-to-HS rooms: + + * See above. + +On AS publishing aliases for virtual rooms: + + * AS uses the normal alias management API to preemptively create/delete public + directory entries for aliases for virtual rooms provided by the AS. + * In order to create these aliases, the underlying room ID must also exist, so + at least the m.room.create of that room must also be prepopulated. It seems + sensible to prepopulate the required initial state and history of the room to + avoid a two-phase prepopulation process. + +On unregistering the AS from the HS: + + * An AS must tell the HS when it is going offline in order to stop receiving + requests from the HS. It does this by hitting an API on the HS. + +Extensions to CS API +==================== + + * Ability to assert the identity of the virtual user for all methods. + * Ability to massage timestamps when prepopulating historical state and + messages of virtual rooms. + * Ability to delete aliases (including from the directory) as well as create them. \ No newline at end of file From f91f569f3007007ea2c07718f8cde7f562f0d868 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 30 Dec 2014 03:10:11 +0000 Subject: [PATCH 02/62] explain why the @.irc.freenode style prefixes are a good idae --- drafts/applications_services.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst index cee29212..ca012a81 100644 --- a/drafts/applications_services.rst +++ b/drafts/applications_services.rst @@ -95,8 +95,10 @@ On HS handling events to unknown users: the AS, then the HS queries the AS for the profile of that user. If the AS confirms the existence of that user (from its perspective), then the HS creates an account to represent the virtual user. - * The namespace of virtual user accounts could conform to a structure like - @.irc.freenode.Arathorn:matrix.org or similar. + * The namespace of virtual user accounts should conform to a structure like + @.irc.freenode.Arathorn:matrix.org. This lets Matrix users communicate with + foreign users who are not yet mapped into Matrix via 3PID mappings or through + an existing non-virtual Matrix user by trying to talk to them via a gateway. * The AS can preprovision virtual users using the existing CS API rather than lazy-loading them in this manner. From 01bc9eb4a1dba776d14f5048e339ecfc85805f07 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 5 Jan 2015 15:16:10 +0000 Subject: [PATCH 03/62] some forgotten AS use cases --- drafts/applications_services.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst index ca012a81..0333a89a 100644 --- a/drafts/applications_services.rst +++ b/drafts/applications_services.rst @@ -59,6 +59,8 @@ Example use cases for application services include: * Application services: + * Search engines (e.g. elasticsearch search indices) + * Notification systems (e.g. send custom pushes for various hooks) * VoIP Conference services * Text-to-speech and Speech-to-text services * Signal processing From deb24aecf40cb7e6e7fdd4fad8320820ae117c4d Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 5 Jan 2015 18:42:22 +0000 Subject: [PATCH 04/62] some thoughts on invisibility --- drafts/applications_services.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst index 0333a89a..45bb8276 100644 --- a/drafts/applications_services.rst +++ b/drafts/applications_services.rst @@ -159,12 +159,28 @@ On AS publishing aliases for virtual rooms: at least the m.room.create of that room must also be prepopulated. It seems sensible to prepopulate the required initial state and history of the room to avoid a two-phase prepopulation process. - + On unregistering the AS from the HS: * An AS must tell the HS when it is going offline in order to stop receiving requests from the HS. It does this by hitting an API on the HS. +AS Visibility: + + * If an AS needs to sniff events in a room in order to operate on them (e.g. + to act as a search engine) but not inject traffic into the room, it should + do so by subscribing to the relevant events without actually joining the room. + * If the AS needs to participate in the room as a virtual user (e.g. an IVR + service, or a bot, or a gatewayed virtual user), it should join the room + normally. + * There are rare instances where an AS may wish to participate in a room + (including inserting messages), but be hidden from the room list - e.g. a + conferencing server focus bot may wish to join many rooms as the focus and + both listen to VoIP setups and inject its own VoIP answers, without ever + being physically seen in the room. In this scenario, the user should set + its presence to 'invisible', a state that HSes should only allow AS-authed + users to set. + Extensions to CS API ==================== From 259cdd253a001f582a7029a68730155bfcfade16 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 6 Jan 2015 11:54:02 +0000 Subject: [PATCH 05/62] fix plural --- drafts/applications_services.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/applications_services.rst b/drafts/applications_services.rst index 45bb8276..07a70ce2 100644 --- a/drafts/applications_services.rst +++ b/drafts/applications_services.rst @@ -125,7 +125,7 @@ On HS handling events to unknown rooms: On HS handling events to existing users and rooms: - * If the HS receives an event for a user or room that already exist (either + * If the HS receives an event for a user or room that already exists (either provisioned by the AS or by normal client interactions), then the message is handled as normal. * Events in the namespaces of rooms and users that the AS has subscribed to From b557e71121a420270d5f79ab124596d0d627223b Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 13 Jan 2015 16:25:28 +0000 Subject: [PATCH 06/62] update AS API to merge in feedback from the pull request --- ..._services.rst => application_services.rst} | 63 ++++++++++++++++--- 1 file changed, 53 insertions(+), 10 deletions(-) rename drafts/{applications_services.rst => application_services.rst} (70%) diff --git a/drafts/applications_services.rst b/drafts/application_services.rst similarity index 70% rename from drafts/applications_services.rst rename to drafts/application_services.rst index 07a70ce2..9ec271ec 100644 --- a/drafts/applications_services.rst +++ b/drafts/application_services.rst @@ -87,6 +87,11 @@ In the registration process, the AS provides: * Credentials to identify itself as an approved application service for that HS * Details of the namespaces of users and rooms the AS is acting on behalf of and "subscribing to" + * Namespaces are defined as a list of regexps against which to match room aliases, + room IDs, and user IDs. + * There is overlap between selecting events via the csv2 Filter API and subscribing + to events here - perhaps subscription involves passing a filter token into the + registration API. * A URL base for receiving requests from the HS (as the AS is a server, implementers expect to receive data via inbound requests rather than long-poll outbound requests) @@ -101,8 +106,13 @@ On HS handling events to unknown users: @.irc.freenode.Arathorn:matrix.org. This lets Matrix users communicate with foreign users who are not yet mapped into Matrix via 3PID mappings or through an existing non-virtual Matrix user by trying to talk to them via a gateway. - * The AS can preprovision virtual users using the existing CS API rather than - lazy-loading them in this manner. + * The AS can alternatively preprovision virtual users using the existing CS API + rather than lazy-loading them in this manner. + * The AS may want to link the matrix ID of the sender through to their 3PID in + the remote ecosystem. E.g. a message sent from @matthew:matrix.org may wish + to originate from Arathorn on irc.freenode.net in the case of an IRC bridge. + It's left as an AS implementation detail as to how the user should authorise + the AS to act on its behalf. On HS handling events to unknown rooms: @@ -116,12 +126,19 @@ On HS handling events to unknown rooms: from the point of m.room.create, we will not be able to back-populate arbitrary history for rooms which are lazy-created in this manner, and instead have to chose the amount of history to be synchronised into the AS as a one-off. - * If exposing arbitrary history is required, then either the room history must be - preemptively provisioned in the HS by the AS via the CS API (TODO: meaning the - CS API needs to support massaged timestamps), or the HS must delegate conversation - storage entirely to the AS using a Storage API (not defined here) which allows - the existing conversation store to back the HS, complete with all necessary - Matrix metadata (e.g. hashes, signatures, federation DAG, etc). + * If exposing arbitrary history is required, then: + + * either: the room history must be preemptively provisioned in the HS by the AS via + the CS API (TODO: meaning the CS API needs to support massaged + timestamps), resulting in conversation history being replicated between + the HS and the source store. + * or: the HS must delegate conversation storage entirely to the + AS using a Storage API (not defined here) which allows the existing + conversation store to back the HS, complete with all necessary Matrix + metadata (e.g. hashes, signatures, federation DAG, etc). This obviously + increases the burden of implementing an AS considerably, but is the only + option if the implementer wants to avoid duplicating conversation history + between the external data source and the HS. On HS handling events to existing users and rooms: @@ -130,7 +147,20 @@ On HS handling events to existing users and rooms: is handled as normal. * Events in the namespaces of rooms and users that the AS has subscribed to are pushed to the AS using the same pattern as the federation API (without - any of the encryption or federation metadata). TODO: are they linearised? + any of the encryption or federation metadata). This serves precisely the + same purpose as the CS event stream and has the same data flow semantics + (and indeed an AS implementer could chose to use the CS event stream instead) + + * Events are linearised to avoid the AS having to handle the complexity of + linearisation, and because if linearisation is good enough for CS, it + should be good enough for AS. Should the AS require non-linearised events + from Matrix, it should implement the federation API rather than the AS API + instead. + * HS->AS event pushes are retried for reliability with sequence numbers + (or logical timestamping?) to presereve the linearisation order and ensure + a reliable event stream. + * Clustered HSes must linearise just as they do for the CS API. Clustered + ASes must loadbalance the inbound stream across the cluster as required. On AS relaying events from unknown-to-HS users: @@ -146,6 +176,12 @@ On AS relaying events from unknown-to-HS users: * TODO: or do we maintain a separate access_token mapping? It seems like unnecessary overhead for the AS developer; easier to just use a single privileged access_token and just track which userid is emitting events? + * If the AS is spoofing the identity of a real (not virtual) matrix user, + we should actually let them log themselves in via OAuth2 to give permission + to the AS to act on their behalf. + * We can't auth gatewayed virtual users from 3rd party systems who are being + relayed into Matrix, as the relaying is happening whether the user likes it + or not. Therefore we do need to be able to spoof sender ID for virtual users. On AS relaying events in unknown-to-HS rooms: @@ -180,11 +216,18 @@ AS Visibility: being physically seen in the room. In this scenario, the user should set its presence to 'invisible', a state that HSes should only allow AS-authed users to set. + +E2E Encryption + + * The AS obviously has no visibility to E2E encrypted messages, unless it is + explicitly added to an encrypted room and participates in the group chat + itself. Extensions to CS API ==================== * Ability to assert the identity of the virtual user for all methods. * Ability to massage timestamps when prepopulating historical state and - messages of virtual rooms. + messages of virtual rooms (either by overriding origin_server_ts (preferred) or + adding an as_ts which we expect clients to honour) * Ability to delete aliases (including from the directory) as well as create them. \ No newline at end of file From 540b3a41ff69cbd0c7a7ca49432ef12e274f01f4 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 14 Jan 2015 15:15:40 +0000 Subject: [PATCH 07/62] RSTify; typos --- drafts/application_services.rst | 267 ++++++++++++++++---------------- 1 file changed, 132 insertions(+), 135 deletions(-) diff --git a/drafts/application_services.rst b/drafts/application_services.rst index 9ec271ec..2c73107f 100644 --- a/drafts/application_services.rst +++ b/drafts/application_services.rst @@ -33,32 +33,29 @@ Example use cases for application services include: * Exposing existing communication services in Matrix - * Gateways to/from standards-based protocols (SIP, XMPP, IRC, RCS (MSRP), - SIMPLE, Lync, etc) - * Gateways to/from closed services (e.g. WhatsApp) - * Gateways could be architected as: - - * Act as a virtual client on the non-Matrix network - (e.g. connect as multiple virtual clients to an IRC or XMPP server) - * Act as a server on the non-Matrix network - (e.g. speak s2s XMPP federation, or IRC link protocol) - * Act as an application service on the non-Matrix network - (e.g. link up as IRC services, or an XMPP component) - * Exposing a non-Matrix client interface listener from the AS - (e.g. listen on port 6667 for IRC clients, or port 5222 for XMPP clients) - - * Bridging existing APIs into Matrix - + * Gateways to/from standards-based protocols (SIP, XMPP, IRC, RCS (MSRP), SIMPLE, Lync, etc) + * Gateways to/from closed services (e.g. WhatsApp) + * Gateways could be architected as: + + * Act as a virtual client on the non-Matrix network + (e.g. connect as multiple virtual clients to an IRC or XMPP server) + * Act as a server on the non-Matrix network + (e.g. speak s2s XMPP federation, or IRC link protocol) + * Act as an application service on the non-Matrix network + (e.g. link up as IRC services, or an XMPP component) + * Exposing a non-Matrix client interface listener from the AS + (e.g. listen on port 6667 for IRC clients, or port 5222 for XMPP clients) + + +* Bridging existing APIs into Matrix * e.g. SMS/MMS aggregator APIs * Domain-specific APIs such as SABRE - * Integrating more exotic content into Matrix - +* Integrating more exotic content into Matrix * e.g. MIDI<->Matrix gateway/bridge * 3D world <-> Matrix bridge - * Application services: - +* Application services: * Search engines (e.g. elasticsearch search indices) * Notification systems (e.g. send custom pushes for various hooks) * VoIP Conference services @@ -84,150 +81,150 @@ The application service registers with its host homeserver to offer its services In the registration process, the AS provides: - * Credentials to identify itself as an approved application service for that HS - * Details of the namespaces of users and rooms the AS is acting on behalf of and - "subscribing to" - * Namespaces are defined as a list of regexps against which to match room aliases, - room IDs, and user IDs. - * There is overlap between selecting events via the csv2 Filter API and subscribing - to events here - perhaps subscription involves passing a filter token into the - registration API. - * A URL base for receiving requests from the HS (as the AS is a server, - implementers expect to receive data via inbound requests rather than - long-poll outbound requests) +* Credentials to identify itself as an approved application service for that HS +* Details of the namespaces of users and rooms the AS is acting on behalf of and + "subscribing to" +* Namespaces are defined as a list of regexps against which to match room aliases, + room IDs, and user IDs. +* There is overlap between selecting events via the csv2 Filter API and subscribing + to events here - perhaps subscription involves passing a filter token into the + registration API. +* A URL base for receiving requests from the HS (as the AS is a server, + implementers expect to receive data via inbound requests rather than + long-poll outbound requests) On HS handling events to unknown users: - * If the HS receives an event for an unknown user who is in the namespace delegated to - the AS, then the HS queries the AS for the profile of that user. If the AS - confirms the existence of that user (from its perspective), then the HS - creates an account to represent the virtual user. - * The namespace of virtual user accounts should conform to a structure like - @.irc.freenode.Arathorn:matrix.org. This lets Matrix users communicate with - foreign users who are not yet mapped into Matrix via 3PID mappings or through - an existing non-virtual Matrix user by trying to talk to them via a gateway. - * The AS can alternatively preprovision virtual users using the existing CS API - rather than lazy-loading them in this manner. - * The AS may want to link the matrix ID of the sender through to their 3PID in - the remote ecosystem. E.g. a message sent from @matthew:matrix.org may wish - to originate from Arathorn on irc.freenode.net in the case of an IRC bridge. - It's left as an AS implementation detail as to how the user should authorise - the AS to act on its behalf. +* If the HS receives an event for an unknown user who is in the namespace delegated to + the AS, then the HS queries the AS for the profile of that user. If the AS + confirms the existence of that user (from its perspective), then the HS + creates an account to represent the virtual user. +* The namespace of virtual user accounts should conform to a structure like + ``@.irc.freenode.Arathorn:matrix.org``. This lets Matrix users communicate with + foreign users who are not yet mapped into Matrix via 3PID mappings or through + an existing non-virtual Matrix user by trying to talk to them via a gateway. +* The AS can alternatively preprovision virtual users using the existing CS API + rather than lazy-loading them in this manner. +* The AS may want to link the matrix ID of the sender through to their 3PID in + the remote ecosystem. E.g. a message sent from ``@matthew:matrix.org`` may wish + to originate from Arathorn on irc.freenode.net in the case of an IRC bridge. + It's left as an AS implementation detail as to how the user should authorise + the AS to act on its behalf. On HS handling events to unknown rooms: - * If the HS receives an invite to an unknown room which is in the namespace - delegated to the AS, then the HS queries the AS for the existence of that room. - If the AS confirms its existence (from its perspective), then the HS creates - the room. - * The initial state of the room may be populated by the AS by querying an - initialSync API (probably a subset of the CS initialSync API, to reuse the - same pattern for the equivalent function). As messages have to be signed - from the point of m.room.create, we will not be able to back-populate - arbitrary history for rooms which are lazy-created in this manner, and instead - have to chose the amount of history to be synchronised into the AS as a one-off. - * If exposing arbitrary history is required, then: +* If the HS receives an invite to an unknown room which is in the namespace + delegated to the AS, then the HS queries the AS for the existence of that room. + If the AS confirms its existence (from its perspective), then the HS creates + the room. +* The initial state of the room may be populated by the AS by querying an + initialSync API (probably a subset of the CS initialSync API, to reuse the + same pattern for the equivalent function). As messages have to be signed + from the point of ``m.room.create``, we will not be able to back-populate + arbitrary history for rooms which are lazy-created in this manner, and instead + have to chose the amount of history to be synchronised into the AS as a one-off. +* If exposing arbitrary history is required, then: - * either: the room history must be preemptively provisioned in the HS by the AS via - the CS API (TODO: meaning the CS API needs to support massaged - timestamps), resulting in conversation history being replicated between - the HS and the source store. - * or: the HS must delegate conversation storage entirely to the - AS using a Storage API (not defined here) which allows the existing - conversation store to back the HS, complete with all necessary Matrix - metadata (e.g. hashes, signatures, federation DAG, etc). This obviously - increases the burden of implementing an AS considerably, but is the only - option if the implementer wants to avoid duplicating conversation history - between the external data source and the HS. + * either: the room history must be preemptively provisioned in the HS by the AS via + the CS API (TODO: meaning the CS API needs to support massaged + timestamps), resulting in conversation history being replicated between + the HS and the source store. + * or: the HS must delegate conversation storage entirely to the + AS using a Storage API (not defined here) which allows the existing + conversation store to back the HS, complete with all necessary Matrix + metadata (e.g. hashes, signatures, federation DAG, etc). This obviously + increases the burden of implementing an AS considerably, but is the only + option if the implementer wants to avoid duplicating conversation history + between the external data source and the HS. On HS handling events to existing users and rooms: - * If the HS receives an event for a user or room that already exists (either - provisioned by the AS or by normal client interactions), then the message - is handled as normal. - * Events in the namespaces of rooms and users that the AS has subscribed to - are pushed to the AS using the same pattern as the federation API (without - any of the encryption or federation metadata). This serves precisely the - same purpose as the CS event stream and has the same data flow semantics - (and indeed an AS implementer could chose to use the CS event stream instead) - - * Events are linearised to avoid the AS having to handle the complexity of - linearisation, and because if linearisation is good enough for CS, it - should be good enough for AS. Should the AS require non-linearised events - from Matrix, it should implement the federation API rather than the AS API - instead. - * HS->AS event pushes are retried for reliability with sequence numbers - (or logical timestamping?) to presereve the linearisation order and ensure - a reliable event stream. - * Clustered HSes must linearise just as they do for the CS API. Clustered - ASes must loadbalance the inbound stream across the cluster as required. +* If the HS receives an event for a user or room that already exists (either + provisioned by the AS or by normal client interactions), then the message + is handled as normal. +* Events in the namespaces of rooms and users that the AS has subscribed to + are pushed to the AS using the same pattern as the federation API (without + any of the encryption or federation metadata). This serves precisely the + same purpose as the CS event stream and has the same data flow semantics + (and indeed an AS implementer could chose to use the CS event stream instead) + + * Events are linearised to avoid the AS having to handle the complexity of + linearisation, and because if linearisation is good enough for CS, it + should be good enough for AS. Should the AS require non-linearised events + from Matrix, it should implement the federation API rather than the AS API + instead. + * HS->AS event pushes are retried for reliability with sequence numbers + (or logical timestamping?) to preserve the linearisation order and ensure + a reliable event stream. + * Clustered HSes must linearise just as they do for the CS API. Clustered + ASes must loadbalance the inbound stream across the cluster as required. On AS relaying events from unknown-to-HS users: - * AS injects the event to the HS using the CS API, irrespective of whether the - target user or room is known to the HS or not. If the HS doesn't recognise - the target it goes through the same lazy-load provisioning as per above. - * The reason for not using a subset of the federation API here is because it - allows AS developers to reuse existing CS SDKs and benefit from the more - meaningful error handling of the CS API. The sending user ID must be - explicitly specified, as it cannot be inferred from the access_token, which - will be the same for all AS requests. - - * TODO: or do we maintain a separate access_token mapping? It seems like - unnecessary overhead for the AS developer; easier to just use a single - privileged access_token and just track which userid is emitting events? - * If the AS is spoofing the identity of a real (not virtual) matrix user, - we should actually let them log themselves in via OAuth2 to give permission - to the AS to act on their behalf. - * We can't auth gatewayed virtual users from 3rd party systems who are being - relayed into Matrix, as the relaying is happening whether the user likes it - or not. Therefore we do need to be able to spoof sender ID for virtual users. +* AS injects the event to the HS using the CS API, irrespective of whether the + target user or room is known to the HS or not. If the HS doesn't recognise + the target it goes through the same lazy-load provisioning as per above. +* The reason for not using a subset of the federation API here is because it + allows AS developers to reuse existing CS SDKs and benefit from the more + meaningful error handling of the CS API. The sending user ID must be + explicitly specified, as it cannot be inferred from the access_token, which + will be the same for all AS requests. + + * TODO: or do we maintain a separate ``access_token`` mapping? It seems like + unnecessary overhead for the AS developer; easier to just use a single + privileged ``access_token`` and just track which ``user_id`` is emitting events? + * If the AS is spoofing the identity of a real (not virtual) matrix user, + we should actually let them log themselves in via OAuth2 to give permission + to the AS to act on their behalf. + * We can't auth gatewayed virtual users from 3rd party systems who are being + relayed into Matrix, as the relaying is happening whether the user likes it + or not. Therefore we do need to be able to spoof sender ID for virtual users. On AS relaying events in unknown-to-HS rooms: - * See above. +* See above. On AS publishing aliases for virtual rooms: - * AS uses the normal alias management API to preemptively create/delete public - directory entries for aliases for virtual rooms provided by the AS. - * In order to create these aliases, the underlying room ID must also exist, so - at least the m.room.create of that room must also be prepopulated. It seems - sensible to prepopulate the required initial state and history of the room to - avoid a two-phase prepopulation process. +* AS uses the normal alias management API to preemptively create/delete public + directory entries for aliases for virtual rooms provided by the AS. +* In order to create these aliases, the underlying room ID must also exist, so + at least the ``m.room.create`` of that room must also be prepopulated. It seems + sensible to prepopulate the required initial state and history of the room to + avoid a two-phase prepopulation process. On unregistering the AS from the HS: - * An AS must tell the HS when it is going offline in order to stop receiving - requests from the HS. It does this by hitting an API on the HS. +* An AS must tell the HS when it is going offline in order to stop receiving + requests from the HS. It does this by hitting an API on the HS. AS Visibility: - * If an AS needs to sniff events in a room in order to operate on them (e.g. - to act as a search engine) but not inject traffic into the room, it should - do so by subscribing to the relevant events without actually joining the room. - * If the AS needs to participate in the room as a virtual user (e.g. an IVR - service, or a bot, or a gatewayed virtual user), it should join the room - normally. - * There are rare instances where an AS may wish to participate in a room - (including inserting messages), but be hidden from the room list - e.g. a - conferencing server focus bot may wish to join many rooms as the focus and - both listen to VoIP setups and inject its own VoIP answers, without ever - being physically seen in the room. In this scenario, the user should set - its presence to 'invisible', a state that HSes should only allow AS-authed - users to set. +* If an AS needs to sniff events in a room in order to operate on them (e.g. + to act as a search engine) but not inject traffic into the room, it should + do so by subscribing to the relevant events without actually joining the room. +* If the AS needs to participate in the room as a virtual user (e.g. an IVR + service, or a bot, or a gatewayed virtual user), it should join the room + normally. +* There are rare instances where an AS may wish to participate in a room + (including inserting messages), but be hidden from the room list - e.g. a + conferencing server focus bot may wish to join many rooms as the focus and + both listen to VoIP setups and inject its own VoIP answers, without ever + being physically seen in the room. In this scenario, the user should set + its presence to 'invisible', a state that HSes should only allow AS-authed + users to set. E2E Encryption - * The AS obviously has no visibility to E2E encrypted messages, unless it is - explicitly added to an encrypted room and participates in the group chat - itself. +* The AS obviously has no visibility to E2E encrypted messages, unless it is + explicitly added to an encrypted room and participates in the group chat + itself. Extensions to CS API ==================== - * Ability to assert the identity of the virtual user for all methods. - * Ability to massage timestamps when prepopulating historical state and - messages of virtual rooms (either by overriding origin_server_ts (preferred) or - adding an as_ts which we expect clients to honour) - * Ability to delete aliases (including from the directory) as well as create them. \ No newline at end of file +* Ability to assert the identity of the virtual user for all methods. +* Ability to massage timestamps when prepopulating historical state and + messages of virtual rooms (either by overriding ``origin_server_ts`` (preferred) or + adding an ``as_ts`` which we expect clients to honour) +* Ability to delete aliases (including from the directory) as well as create them. From 68c774689aa67231421e6d669a6d240bb94de3ac Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Jan 2015 15:34:07 +0000 Subject: [PATCH 08/62] Add stub AS HTTP API doc --- drafts/as-http-api.rst | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 drafts/as-http-api.rst diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst new file mode 100644 index 00000000..e69de29b From ad7d675d3d269de2be9cd7d1b4f7dd1b7772d3cd Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 14 Jan 2015 16:40:50 +0000 Subject: [PATCH 09/62] Draft Registration and User Query APIs --- drafts/as-http-api.rst | 140 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index e69de29b..41afc213 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -0,0 +1,140 @@ +Application Services HTTP API +============================= + +.. contents:: Table of Contents +.. sectnum:: + +Application Service -> Home Server +---------------------------------- +This contains home server APIs which are used by the application service. + +Registration API ``[Draft]`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This API registers the application service with its host homeserver to offer its services. + +Inputs: + - Credentials (e.g. some kind of string token) + - Namespace[users] + - Namespace[room aliases] + - URL base to receive inbound comms +Output: + - The credentials the HS will use to query the AS with in return. (e.g. some kind of string token) + - The complete namespace prefix for each namespace requested. +Side effects: + - The HS will start delivering events to the URL base specified if this 200s. +API called when: + - The application service wants to register with a brand new home server. +Notes: + - Namespaces are represented in JSON, with a well-defined conversion to IDs. This prevents + parsing errors and allows the HS to enforce their own namespacing semantics. They look like:: + users: { + irc: ["freenode", "rizon"] + } + which represents 2 namespaces: ``@.irc.freenode.*`` and ``@.irc.rizon.*``. The leaf nodes + must be an array. Intermediate nodes must be JSON objects with the key as the desired string + segment. A more complicated example:: + rooms: { + irc: { + freenode: ["matrix"], + rizon: ["matrixorg"] + } + } + which represents 2 namespaces: ``#.irc.freenode.matrix.*`` and ``#.irc.rizon.matrixorg.*``. + - By specifying namespaces like this, you allow home servers to namespace application services + sensibly, rather than every IRC AS trying to nab all ``@.irc.*`` users. In order for home + servers to do this, they need to tell the application service the actual namespaces allocated + for them. This is returned in the JSON response, with the same structure as the original request, + but with the strings now representing the namespace prefix allocated, e.g:: + users: { + irc: [".applicationservice_146.irc.freenode", ".applicationservice_146.irc.rizon"] + } + The sigil prefix ``@`` is omitted since it is clear from the ``users`` key that these namespace + prefixes are for users. + - This makes the request/response JSON structure deliciously symmetric. +:: + + POST /register + + Request format + { + url: "https://my.application.service.com/matrix/", + as_token: "some_AS_token", + namespaces: { + users: { + irc: ["freenode", "rizon"] + }, + rooms: { + irc: { + freenode: ["matrix"], + rizon: ["matrixorg"] + } + } + } + } + + + Returns: + 200 : Registration accepted. + 400 : Namespaces do not conform to regex + 401 : Credentials need to be supplied. + 403 : AS credentials rejected. + + + 200 OK response format + + { + hs_token: "foobar", + namespaces: { + users: { + irc: [".applicationservice_146.irc.freenode", ".applicationservice_146.irc.rizon"] + }, + rooms: { + irc: { + freenode: [".irc.freenode.matrix"], + rizon: [".applicationservice_146.this.can.be.any.prefix.you.like"] + } + } + } + } + + +Home Server -> Application Service +---------------------------------- +This contains application service APIs which are used by the home server. + +User Query ``[Draft]`` +~~~~~~~~~~~~~~~~~~~~~~ +This API is called by the HS to query the existence of a user on the Application Service's namespace. + +Inputs: + - User ID + - HS Credentials +Output: + - Profile info +Side effects: + - User is created on the HS if this response 200s. +API called when: + - HS receives an event for an unknown user ID in the AS's namespace. +Notes: + - The created user will have their profile info set based on the output. + +:: + + GET /users/$user_id?access_token=$hs_token + + Returns: + 200 : User is recognised. + 404 : User not found. + 401 : Credentials need to be supplied. + 403 : HS credentials rejected. + + + 200 OK response format + + { + profile: { + display_name: "Foo" + avatar_url: "mxc://foo/bar" + } + } + From d240e1f856584c598bfa7ef4c14f19b8b9f72535 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 14 Jan 2015 17:01:33 +0000 Subject: [PATCH 10/62] Add TODO APIs --- drafts/as-http-api.rst | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 41afc213..4915ab6e 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -96,6 +96,10 @@ Notes: } } } + +Unregister API ``[TODO]`` +~~~~~~~~~~~~~~~~~~~~~~~~~ + Home Server -> Application Service @@ -137,4 +141,35 @@ Notes: avatar_url: "mxc://foo/bar" } } + +Room Query ``[TODO]`` +~~~~~~~~~~~~~~~~~~~~~ +This API is called by the HS to query the existence of a room on the Application Service's namespace. + +Pushing ``[TODO]`` +~~~~~~~~~~~~~~~~~~ +This API is called by the HS when the HS wants to push an event (or batch of events) to the AS. + + - Retry semantics + - Ordering + + +Client -> Application Service +----------------------------- +This contains application service APIs which are used by the client. + +Linking ``[TODO]`` +~~~~~~~~~~~~~~~~~~ +Clients may want to link their matrix user ID to their 3PID (e.g. IRC nick). This +API allows the AS to do this, so messages sent from the AS are sent as the client. + +- Probably OAuth2 + +Client-Server v2 API Extensions +------------------------------- + + - Identity assertion (rather than access token inference) + - timestamp massaging (for inserting messages between other messages) + - alias mastery over the ASes namespace + - user ID mastery over the ASes namespace From cdeb0faf853c40dd587507ca534708c98d5aadb5 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Wed, 14 Jan 2015 17:17:18 +0000 Subject: [PATCH 11/62] Remove block quotes of shame --- drafts/as-http-api.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 4915ab6e..826a774d 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -150,8 +150,8 @@ Pushing ``[TODO]`` ~~~~~~~~~~~~~~~~~~ This API is called by the HS when the HS wants to push an event (or batch of events) to the AS. - - Retry semantics - - Ordering +- Retry semantics +- Ordering @@ -169,7 +169,7 @@ API allows the AS to do this, so messages sent from the AS are sent as the clien Client-Server v2 API Extensions ------------------------------- - - Identity assertion (rather than access token inference) - - timestamp massaging (for inserting messages between other messages) - - alias mastery over the ASes namespace - - user ID mastery over the ASes namespace +- Identity assertion (rather than access token inference) +- timestamp massaging (for inserting messages between other messages) +- alias mastery over the ASes namespace +- user ID mastery over the ASes namespace From b19a799a0e1f222d777106e7d924c63bd298c60e Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 15 Jan 2015 01:56:47 +0000 Subject: [PATCH 12/62] notes on how to build pstn gateway ASes --- drafts/pstn_gatewaying.txt | 48 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 drafts/pstn_gatewaying.txt diff --git a/drafts/pstn_gatewaying.txt b/drafts/pstn_gatewaying.txt new file mode 100644 index 00000000..2cb8c649 --- /dev/null +++ b/drafts/pstn_gatewaying.txt @@ -0,0 +1,48 @@ +Gatewaying to the PSTN via Matrix Application Services +====================================================== + +Matrix Application Services (AS) provides a way for PSTN users to interact +with Matrix via an AS acting as a gateway. Each PSTN user is represented as a +virtual user on a specific homeserver maintained by the AS. Typically the AS +is provisioned on a well-known AS-supplier HS (e.g. matrix.openmarket.com) or +is a service provisioned on the user's local HS. + +In either scenario, the AS maintains virtual users of form +@.tel.e164:homeserver. These are lazily created (as per the AS spec) when +matrix users try to contact a user id of form @.tel.*:homeserver, or when the +AS needs to inject traffic into the HS on behalf of the PSTN user. The reason +for these being a visible virtual user rather than an invisible user or an +invisible sniffing AS is because they do represent real physical 3rd party +endpoints in the PSTN, and need to be able to send return messages. + +Communication with an actual PSTN user happens in a normal Matrix room, which +for 1:1 matrix<->pstn contact will typically store all conversation history +with that user. On first contact, the matrix user invites the virtual user +into the room (or vice versa). In the event of switching to another AS-enabled +HS, the matrix user would kick the old AS and invite the new one. In the event +of needing loadbalancing between two SMS gateways (for instance), the user +would set visibility flags (TODO: specify per-message ACLs, or use crypto to +only sign messages so they're visible to certain other users?) to adjust which +virtual AS users could see which messages in the room. + +For group chat, one or more AS virtual users may be invited to a group chat, +where-upon they will relay all the traffic in that group chat through to their +PSTN counterpart (and vice versa). This behaviour requires no additional +functionality beyond that required to support 1:1 chat. + +When contacting a user, Matrix clients should check whether a given E.164 +number is already mapped to a real Matrix user by querying the identity +servers (or subscribing to identity updates for a given E.164 number. TODO: ID +server subscriptions). If the E.164 number has a validated mapping in the ID +server to a Matrix ID, then this target ID should be used instead of +contacting the virtual user. + +It's likely that PSTN gateway ASes will need to charge the end-user for use of +the gateway. The AS must therefore track credit per matrix ID it interacts +with, and stop gatewaying as desired once credit is exhausted. The task of +extracting credit from the end-user and adding it to the AS is not covered by +the Matrix specification. + +For SMS routing, options are: + * Terminate traffic only (from a shared shortcode originator) + * Two-way traffic via a VMN. To save allocating huge numbers of VMNs to Matrix users, the VMN can be allocated from a pool such that each {caller,callee} tuple is unique (but the caller number will only work from that specific callee). \ No newline at end of file From d8cd4a45d4a04cc9de02f7c36bb860bec42720db Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 09:50:38 +0000 Subject: [PATCH 13/62] Modify Register API As per https://github.com/matrix-org/matrix-doc/pull/5#issuecomment-70059200 --- drafts/as-http-api.rst | 65 ++++++++++++------------------------------ 1 file changed, 19 insertions(+), 46 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 826a774d..d7c749f9 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -10,6 +10,9 @@ This contains home server APIs which are used by the application service. Registration API ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. NOTE:: + - Do we really have to use regex for this? Can't we do this a nicer way? + This API registers the application service with its host homeserver to offer its services. Inputs: @@ -19,38 +22,18 @@ Inputs: - URL base to receive inbound comms Output: - The credentials the HS will use to query the AS with in return. (e.g. some kind of string token) - - The complete namespace prefix for each namespace requested. Side effects: - The HS will start delivering events to the URL base specified if this 200s. API called when: - The application service wants to register with a brand new home server. Notes: - - Namespaces are represented in JSON, with a well-defined conversion to IDs. This prevents - parsing errors and allows the HS to enforce their own namespacing semantics. They look like:: - users: { - irc: ["freenode", "rizon"] - } - which represents 2 namespaces: ``@.irc.freenode.*`` and ``@.irc.rizon.*``. The leaf nodes - must be an array. Intermediate nodes must be JSON objects with the key as the desired string - segment. A more complicated example:: - rooms: { - irc: { - freenode: ["matrix"], - rizon: ["matrixorg"] - } - } - which represents 2 namespaces: ``#.irc.freenode.matrix.*`` and ``#.irc.rizon.matrixorg.*``. - - By specifying namespaces like this, you allow home servers to namespace application services - sensibly, rather than every IRC AS trying to nab all ``@.irc.*`` users. In order for home - servers to do this, they need to tell the application service the actual namespaces allocated - for them. This is returned in the JSON response, with the same structure as the original request, - but with the strings now representing the namespace prefix allocated, e.g:: - users: { - irc: [".applicationservice_146.irc.freenode", ".applicationservice_146.irc.rizon"] - } + - Namespaces are represented by POSIX extended regular expressions in JSON. + They look like:: + users: [ + "irc\.freenode\.net/.*", + ] The sigil prefix ``@`` is omitted since it is clear from the ``users`` key that these namespace prefixes are for users. - - This makes the request/response JSON structure deliciously symmetric. :: POST /register @@ -60,15 +43,12 @@ Notes: url: "https://my.application.service.com/matrix/", as_token: "some_AS_token", namespaces: { - users: { - irc: ["freenode", "rizon"] - }, - rooms: { - irc: { - freenode: ["matrix"], - rizon: ["matrixorg"] - } - } + users: [ + "irc\.freenode\.net/.*" + ], + rooms: [ + "irc\.freenode\.net/.*" + ] } } @@ -83,18 +63,7 @@ Notes: 200 OK response format { - hs_token: "foobar", - namespaces: { - users: { - irc: [".applicationservice_146.irc.freenode", ".applicationservice_146.irc.rizon"] - }, - rooms: { - irc: { - freenode: [".irc.freenode.matrix"], - rizon: [".applicationservice_146.this.can.be.any.prefix.you.like"] - } - } - } + hs_token: "foobar" } Unregister API ``[TODO]`` @@ -108,6 +77,10 @@ This contains application service APIs which are used by the home server. User Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~ +.. NOTE:: + - This API may be called a lot by the HS (e.g. incoming events for unknown user IDs, profile + requests, etc. Is this okay? + This API is called by the HS to query the existence of a user on the Application Service's namespace. Inputs: From 9401d9a62e8a792f646d5f89a55d1d90d75e8f07 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 13:48:50 +0000 Subject: [PATCH 14/62] Flesh out the required CS extensions --- drafts/as-http-api.rst | 72 +++++++++++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 12 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index d7c749f9..a66628c3 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -46,7 +46,7 @@ Notes: users: [ "irc\.freenode\.net/.*" ], - rooms: [ + aliases: [ "irc\.freenode\.net/.*" ] } @@ -63,12 +63,30 @@ Notes: 200 OK response format { - hs_token: "foobar" + hs_token: "string" } -Unregister API ``[TODO]`` +Unregister API ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~ +This API unregisters a previously registered AS from the home server. +Inputs: + - AS token +Output: + - None. +Side effects: + - The HS will stop delivering events to the URL base specified for this AS if this 200s. +API called when: + - The application service wants to stop receiving all events from the HS. + +:: + + POST /unregister + + Request format + { + as_token: "string" + } Home Server -> Application Service @@ -77,9 +95,6 @@ This contains application service APIs which are used by the home server. User Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~ -.. NOTE:: - - This API may be called a lot by the HS (e.g. incoming events for unknown user IDs, profile - requests, etc. Is this okay? This API is called by the HS to query the existence of a user on the Application Service's namespace. @@ -110,8 +125,8 @@ Notes: { profile: { - display_name: "Foo" - avatar_url: "mxc://foo/bar" + display_name: "string" + avatar_url: "string(uri)" } } @@ -142,7 +157,40 @@ API allows the AS to do this, so messages sent from the AS are sent as the clien Client-Server v2 API Extensions ------------------------------- -- Identity assertion (rather than access token inference) -- timestamp massaging (for inserting messages between other messages) -- alias mastery over the ASes namespace -- user ID mastery over the ASes namespace +Identity assertion +~~~~~~~~~~~~~~~~~~ +The client-server API infers the user ID from the ``access_token`` provided in every +request. It would be an annoying amount of book-keeping to maintain tokens for every +virtual user. It would be preferable if the application service could use the CS API +with its own ``as_token`` instead, and specify the virtual user they wish to be +acting on behalf of. For real users, this would require additional permissions (see +"C-AS Linking"). + +Inputs: + - Application service token (``as_token``) + Either: + - User ID in the AS namespace to act as. + Or: + - OAuth2 token of real user (which may end up being an access token) +Notes: + - This will apply on all aspects of the CS API, except for Account Management. + + +Timestamp massaging +~~~~~~~~~~~~~~~~~~~ +The application service may want to inject events at a certain time (reflecting +the time on the network they are tracking e.g. irc, xmpp). Application services +need to be able to adjust the ``origin_server_ts`` value to do this. + +Inputs: + - Application service token (``as_token``) + - Desired timestamp +Notes: + - This will only apply when sending events. + +Server admin style permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The home server needs to give the application service *full control* over its +namespace, both for users and for room aliases. This means that the AS should +be able to create/edit/delete any room alias in its namespace, as well as +create/delete any user in its namespace. From d4ea43d2b9126f3004cf948c470010528cd7a655 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 13:56:03 +0000 Subject: [PATCH 15/62] Outline what the CS extensions look like in HTTP --- drafts/as-http-api.rst | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index a66628c3..666d996c 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -175,6 +175,19 @@ Inputs: Notes: - This will apply on all aspects of the CS API, except for Account Management. +:: + + /path?as_token=$token&user_id=$userid + + Query Parameters: + as_token: The application service token + user_id: The desired user ID to act as. + + /path?as_token=$token&user_token=$token + + Query Parameters: + as_token: The application service token + user_token: The token granted to the AS by the real user Timestamp massaging ~~~~~~~~~~~~~~~~~~~ @@ -187,10 +200,19 @@ Inputs: - Desired timestamp Notes: - This will only apply when sending events. + +:: + + /path?as_token=$token&ts=$timestamp + + Query Parameters added to the send event APIs only: + as_token: The application service token + ts: The desired timestamp Server admin style permissions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The home server needs to give the application service *full control* over its namespace, both for users and for room aliases. This means that the AS should be able to create/edit/delete any room alias in its namespace, as well as -create/delete any user in its namespace. +create/delete any user in its namespace. This does not require any additional +public APIs. From 7d56113b4fa1ff3f5d13fa76a3acd005b9ae66da Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 14:27:20 +0000 Subject: [PATCH 16/62] Clarification and tweaks Room ID namespace clarifications and s/as_token/access_token/g for client-server extensions. --- drafts/as-http-api.rst | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 666d996c..5a1ca364 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -33,7 +33,7 @@ Notes: "irc\.freenode\.net/.*", ] The sigil prefix ``@`` is omitted since it is clear from the ``users`` key that these namespace - prefixes are for users. + prefixes are for users. Likewise, ``#`` for ``aliases`` and ``!`` for ``rooms``. :: POST /register @@ -48,6 +48,9 @@ Notes: ], aliases: [ "irc\.freenode\.net/.*" + ], + rooms: [ + "irc\.freenode\.net/.*" ] } } @@ -167,23 +170,26 @@ acting on behalf of. For real users, this would require additional permissions ( "C-AS Linking"). Inputs: - - Application service token (``as_token``) + - Application service token (``access_token``) Either: - User ID in the AS namespace to act as. Or: - OAuth2 token of real user (which may end up being an access token) Notes: - This will apply on all aspects of the CS API, except for Account Management. + - The ``as_token`` is inserted into ``access_token`` which is usually where the client + token is. This is done on purpose to allow application services to reuse client + SDKs. :: - /path?as_token=$token&user_id=$userid + /path?access_token=$token&user_id=$userid Query Parameters: as_token: The application service token user_id: The desired user ID to act as. - /path?as_token=$token&user_token=$token + /path?access_token=$token&user_token=$token Query Parameters: as_token: The application service token From e0e9beebc0923f0fa07c4719f0cf00bfc5e78a11 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 16:48:39 +0000 Subject: [PATCH 17/62] Add draft Linking API Also re-add sigils to regex. --- drafts/as-http-api.rst | 63 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 52 insertions(+), 11 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 5a1ca364..8995ffd8 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -30,10 +30,8 @@ Notes: - Namespaces are represented by POSIX extended regular expressions in JSON. They look like:: users: [ - "irc\.freenode\.net/.*", + "@irc\.freenode\.net/.*", ] - The sigil prefix ``@`` is omitted since it is clear from the ``users`` key that these namespace - prefixes are for users. Likewise, ``#`` for ``aliases`` and ``!`` for ``rooms``. :: POST /register @@ -44,13 +42,13 @@ Notes: as_token: "some_AS_token", namespaces: { users: [ - "irc\.freenode\.net/.*" + "@irc\.freenode\.net/.*" ], aliases: [ - "irc\.freenode\.net/.*" + "#irc\.freenode\.net/.*" ], rooms: [ - "irc\.freenode\.net/.*" + "!irc\.freenode\.net/.*" ] } } @@ -146,16 +144,59 @@ This API is called by the HS when the HS wants to push an event (or batch of eve -Client -> Application Service ------------------------------ -This contains application service APIs which are used by the client. +Client -> Server -> Application Service +--------------------------------------- +This contains home server APIs which are used by the client, to communicate with +the application service. -Linking ``[TODO]`` +Linking ``[Draft]`` ~~~~~~~~~~~~~~~~~~ +.. NOTE:: + - How does the application service know that the matrix user really is the virtual + user they claim to be? If we force an IS lookup, then this would resolve on its + own as anyone who wants to talk to the virtual user will do a lookup before trying + the application service... + Clients may want to link their matrix user ID to their 3PID (e.g. IRC nick). This API allows the AS to do this, so messages sent from the AS are sent as the client. -- Probably OAuth2 +This is not achieved using OAuth2 or similar because the trust relationships are +different. The application service already has a huge amount of trust from the +home server, unlike a random third-party web app. As a result, the data flow is +different because the third-party (the application service) is trusted by the +authorisation entity (the home server). Furthermore, it is desirable to not have +the clients communicate directly with the application service in order to +decrease the complexity of AS design. + +To grant permission for an application service to masquerade as a user: + +Inputs: + - Credentials of user (e.g. ``access_token``) + - The user ID within an application service's namespace to claim. + - Restrictions if any (e.g. only for rooms X,Y,Z. Only for 10 hours. etc) +Output: + - None. +Side effects: + - The home server will generate an ``access_token`` and push it to the + application service along with both user IDs if this response 200s. +Notes: + - Repeated calls to this API should invalidate any existing token for this + user ID / application service combo and provision a new token which is then + pushed. + - The generated access token MUST honour the restrictions laid out by the + client. + +To revoke permission for an application service to masquerade as a user: + +Inputs: + - Credentials of user (e.g. ``access_token``) + - The user ID within an application service's namespace to revoke. If blank, + revokes all user IDs linked to this matrix user ID. +Output: + - None. +Side effects: + - The home server invalidate all access tokens for this user ID / AS combo + and push this invalidation to the application service if this response 200s. Client-Server v2 API Extensions ------------------------------- From 0313397e42591c89f47b65ee35eb40258481f750 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 16:51:20 +0000 Subject: [PATCH 18/62] Clarification on IDs Specifically using the term "virtual user ID" in more places to mean the namespace of AS user IDs. --- drafts/as-http-api.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 8995ffd8..2fb9aed2 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -157,8 +157,9 @@ Linking ``[Draft]`` own as anyone who wants to talk to the virtual user will do a lookup before trying the application service... -Clients may want to link their matrix user ID to their 3PID (e.g. IRC nick). This -API allows the AS to do this, so messages sent from the AS are sent as the client. +Clients may want to link their matrix user ID to their virtual user ID. This +API allows the AS to do this, so messages sent from the AS are sent as the client's +user ID, instead of the virtual user ID. This is not achieved using OAuth2 or similar because the trust relationships are different. The application service already has a huge amount of trust from the @@ -191,7 +192,7 @@ To revoke permission for an application service to masquerade as a user: Inputs: - Credentials of user (e.g. ``access_token``) - The user ID within an application service's namespace to revoke. If blank, - revokes all user IDs linked to this matrix user ID. + revokes all virtual user IDs linked to this matrix user ID. Output: - None. Side effects: From 1b3e27a05a8666e02e031972a7bf0336195d5599 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 15 Jan 2015 17:26:36 +0000 Subject: [PATCH 19/62] Add Linking HTTP API Also add note on masquerading problem. --- drafts/as-http-api.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 2fb9aed2..727a5ff9 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -156,6 +156,8 @@ Linking ``[Draft]`` user they claim to be? If we force an IS lookup, then this would resolve on its own as anyone who wants to talk to the virtual user will do a lookup before trying the application service... + - In other words, what is preventing ``@bob:matrix.org`` masquerading as + ``@.irc.freenode.alice:matrix.org``? Clients may want to link their matrix user ID to their virtual user ID. This API allows the AS to do this, so messages sent from the AS are sent as the client's @@ -186,6 +188,21 @@ Notes: pushed. - The generated access token MUST honour the restrictions laid out by the client. + +:: + + PUT /appservices/$virtual_user_id?access_token=$token + + Request format + { + restrictions: { + expires_in: 3600, + rooms: [ + "!fl3rwfehw:matrix.org", + "!fwet2yugs:matrix.org" + ] + } + } To revoke permission for an application service to masquerade as a user: @@ -198,6 +215,11 @@ Output: Side effects: - The home server invalidate all access tokens for this user ID / AS combo and push this invalidation to the application service if this response 200s. + +:: + + DELETE /appservices/$virtual_user_id?access_token=$token + Client-Server v2 API Extensions ------------------------------- From f33b722e4b006f8ffaee89f38648df0c2b9b2dde Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 16 Jan 2015 08:28:55 +0000 Subject: [PATCH 20/62] More clarifications s/as_token/access_token/ for CS extensions. Clarify when room queries are made. --- drafts/as-http-api.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 727a5ff9..59d5a3e4 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -131,9 +131,11 @@ Notes: } } -Room Query ``[TODO]`` -~~~~~~~~~~~~~~~~~~~~~ -This API is called by the HS to query the existence of a room on the Application Service's namespace. +Room Alias Query ``[TODO]`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This API is called by the HS to query the existence of a room alias on the Application +Service's namespace. + Pushing ``[TODO]`` ~~~~~~~~~~~~~~~~~~ @@ -250,13 +252,13 @@ Notes: /path?access_token=$token&user_id=$userid Query Parameters: - as_token: The application service token + access_token: The application service token user_id: The desired user ID to act as. /path?access_token=$token&user_token=$token Query Parameters: - as_token: The application service token + access_token: The application service token user_token: The token granted to the AS by the real user Timestamp massaging @@ -273,10 +275,10 @@ Notes: :: - /path?as_token=$token&ts=$timestamp + /path?access_token=$token&ts=$timestamp Query Parameters added to the send event APIs only: - as_token: The application service token + access_token: The application service token ts: The desired timestamp Server admin style permissions From 1d45dc08ca07cecb7f9c618591753c4f6540fe06 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 16 Jan 2015 10:56:42 +0000 Subject: [PATCH 21/62] Add room query API. Remove linking API. Also add section on ID conventions which need thinking about. --- drafts/as-http-api.rst | 171 +++++++++++++++++++---------------------- 1 file changed, 81 insertions(+), 90 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 59d5a3e4..fd0b302e 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -13,7 +13,8 @@ Registration API ``[Draft]`` .. NOTE:: - Do we really have to use regex for this? Can't we do this a nicer way? -This API registers the application service with its host homeserver to offer its services. +This API registers the application service with its host homeserver to offer its +services. Inputs: - Credentials (e.g. some kind of string token) @@ -21,7 +22,8 @@ Inputs: - Namespace[room aliases] - URL base to receive inbound comms Output: - - The credentials the HS will use to query the AS with in return. (e.g. some kind of string token) + - The credentials the HS will use to query the AS with in return. (e.g. some + kind of string token) Side effects: - The HS will start delivering events to the URL base specified if this 200s. API called when: @@ -76,7 +78,8 @@ Inputs: Output: - None. Side effects: - - The HS will stop delivering events to the URL base specified for this AS if this 200s. + - The HS will stop delivering events to the URL base specified for this AS if + this 200s. API called when: - The application service wants to stop receiving all events from the HS. @@ -97,7 +100,8 @@ This contains application service APIs which are used by the home server. User Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~ -This API is called by the HS to query the existence of a user on the Application Service's namespace. +This API is called by the HS to query the existence of a user on the Application +Service's namespace. Inputs: - User ID @@ -131,96 +135,71 @@ Notes: } } -Room Alias Query ``[TODO]`` +Room Alias Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This API is called by the HS to query the existence of a room alias on the Application -Service's namespace. - - -Pushing ``[TODO]`` -~~~~~~~~~~~~~~~~~~ -This API is called by the HS when the HS wants to push an event (or batch of events) to the AS. - -- Retry semantics -- Ordering - - - -Client -> Server -> Application Service ---------------------------------------- -This contains home server APIs which are used by the client, to communicate with -the application service. - -Linking ``[Draft]`` -~~~~~~~~~~~~~~~~~~ -.. NOTE:: - - How does the application service know that the matrix user really is the virtual - user they claim to be? If we force an IS lookup, then this would resolve on its - own as anyone who wants to talk to the virtual user will do a lookup before trying - the application service... - - In other words, what is preventing ``@bob:matrix.org`` masquerading as - ``@.irc.freenode.alice:matrix.org``? - -Clients may want to link their matrix user ID to their virtual user ID. This -API allows the AS to do this, so messages sent from the AS are sent as the client's -user ID, instead of the virtual user ID. - -This is not achieved using OAuth2 or similar because the trust relationships are -different. The application service already has a huge amount of trust from the -home server, unlike a random third-party web app. As a result, the data flow is -different because the third-party (the application service) is trusted by the -authorisation entity (the home server). Furthermore, it is desirable to not have -the clients communicate directly with the application service in order to -decrease the complexity of AS design. - -To grant permission for an application service to masquerade as a user: +This API is called by the HS to query the existence of a room alias on the +Application Service's namespace. Inputs: - - Credentials of user (e.g. ``access_token``) - - The user ID within an application service's namespace to claim. - - Restrictions if any (e.g. only for rooms X,Y,Z. Only for 10 hours. etc) + - Room alias + - HS Credentials Output: - - None. + - The current state events for the room if any. + - The message events for the room if any. Side effects: - - The home server will generate an ``access_token`` and push it to the - application service along with both user IDs if this response 200s. + - A new room will be created with the alias input if this response 200s. +API called when: + - HS receives an event to join a room alias in the AS's namespace. Notes: - - Repeated calls to this API should invalidate any existing token for this - user ID / application service combo and provision a new token which is then - pushed. - - The generated access token MUST honour the restrictions laid out by the - client. - + - This can be thought of as an ``initialSync`` but for a 3P networked room, + which is lazily loaded when a matrix user tries to join the room. + :: - PUT /appservices/$virtual_user_id?access_token=$token + GET /rooms/$room_alias?access_token=$hs_token - Request format - { - restrictions: { - expires_in: 3600, - rooms: [ - "!fl3rwfehw:matrix.org", - "!fwet2yugs:matrix.org" + Returns: + 200 : Room is recognised. + 404 : Room not found. + 401 : Credentials need to be supplied. + 403 : HS credentials rejected. + + + 200 OK response format + + { + events: [ + { + content: { + ... + }, + user_id: "string", + origin_server_ts: integer, // massaged timestamps + type: "string" + }, + ... + ], + state: [ + { + content: { + ... + }, + user_id: "string(virtual user id)", + origin_server_ts: integer, + state_key: "string", + type: "string" // e.g. m.room.name + }, + ... ] } - } -To revoke permission for an application service to masquerade as a user: - -Inputs: - - Credentials of user (e.g. ``access_token``) - - The user ID within an application service's namespace to revoke. If blank, - revokes all virtual user IDs linked to this matrix user ID. -Output: - - None. -Side effects: - - The home server invalidate all access tokens for this user ID / AS combo - and push this invalidation to the application service if this response 200s. - -:: +Pushing ``[TODO]`` +~~~~~~~~~~~~~~~~~~ +This API is called by the HS when the HS wants to push an event (or batch of +events) to the AS. - DELETE /appservices/$virtual_user_id?access_token=$token +- Retry semantics +- Ordering Client-Server v2 API Extensions @@ -228,12 +207,12 @@ Client-Server v2 API Extensions Identity assertion ~~~~~~~~~~~~~~~~~~ -The client-server API infers the user ID from the ``access_token`` provided in every -request. It would be an annoying amount of book-keeping to maintain tokens for every -virtual user. It would be preferable if the application service could use the CS API -with its own ``as_token`` instead, and specify the virtual user they wish to be -acting on behalf of. For real users, this would require additional permissions (see -"C-AS Linking"). +The client-server API infers the user ID from the ``access_token`` provided in +every request. It would be an annoying amount of book-keeping to maintain tokens +for every virtual user. It would be preferable if the application service could +use the CS API with its own ``as_token`` instead, and specify the virtual user +they wish to be acting on behalf of. For real users, this would require +additional permissions (see "C-AS Linking"). Inputs: - Application service token (``access_token``) @@ -243,9 +222,9 @@ Inputs: - OAuth2 token of real user (which may end up being an access token) Notes: - This will apply on all aspects of the CS API, except for Account Management. - - The ``as_token`` is inserted into ``access_token`` which is usually where the client - token is. This is done on purpose to allow application services to reuse client - SDKs. + - The ``as_token`` is inserted into ``access_token`` which is usually where the + client token is. This is done on purpose to allow application services to + reuse client SDKs. :: @@ -288,3 +267,15 @@ namespace, both for users and for room aliases. This means that the AS should be able to create/edit/delete any room alias in its namespace, as well as create/delete any user in its namespace. This does not require any additional public APIs. + + +ID conventions +-------------- +This concerns the well-defined conventions for mapping 3P network IDs to matrix +IDs, which we expect clients to be able to do by themselves. + +- What do user IDs look like? Butchered URIs? Can all 3P network IDs be + reasonably expressed as URIs? (e.g. tel, email, irc, xmpp, ...) +- What do room aliases look like? Some cases are clear (e.g. IRC) but others + are a lot more fiddly (e.g. email? You don't want to share a room with + everyone who has ever sent an email to ``bob@gmail.com``)... \ No newline at end of file From bd6fc4b5bfbf1b7e7c9867f1c7f1b48f4b3f4821 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 16 Jan 2015 14:29:53 +0000 Subject: [PATCH 22/62] Add draft Pushing API. Add Examples section. Work through an IRC example; raise issues in the notes section. Add retry notes on HS>AS API calls. --- drafts/as-http-api.rst | 227 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 216 insertions(+), 11 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index fd0b302e..dd2dbf80 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -31,9 +31,11 @@ API called when: Notes: - Namespaces are represented by POSIX extended regular expressions in JSON. They look like:: + users: [ "@irc\.freenode\.net/.*", ] + :: POST /register @@ -70,7 +72,7 @@ Notes: } Unregister API ``[Draft]`` -~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~ This API unregisters a previously registered AS from the home server. Inputs: @@ -111,10 +113,16 @@ Output: Side effects: - User is created on the HS if this response 200s. API called when: - - HS receives an event for an unknown user ID in the AS's namespace. + - HS receives an event for an unknown user ID in the AS's namespace, e.g. an + invite event to a room. Notes: - The created user will have their profile info set based on the output. - +Retry notes: + - The home server cannot respond to the client's request until the response to + this API is obtained from the AS. + - Recommended that home servers try a few times then time out, returning a + 408 Request Timeout to the client. + :: GET /users/$user_id?access_token=$hs_token @@ -136,7 +144,7 @@ Notes: } Room Alias Query ``[Draft]`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This API is called by the HS to query the existence of a room alias on the Application Service's namespace. @@ -153,6 +161,11 @@ API called when: Notes: - This can be thought of as an ``initialSync`` but for a 3P networked room, which is lazily loaded when a matrix user tries to join the room. +Retry notes: + - The home server cannot respond to the client's request until the response to + this API is obtained from the AS. + - Recommended that home servers try a few times then time out, returning a + 408 Request Timeout to the client. :: @@ -193,14 +206,61 @@ Notes: ] } -Pushing ``[TODO]`` -~~~~~~~~~~~~~~~~~~ +Pushing ``[Draft]`` +~~~~~~~~~~~~~~~~~~~ This API is called by the HS when the HS wants to push an event (or batch of events) to the AS. -- Retry semantics -- Ordering +Inputs: + - HS Credentials + - Event(s) to give to the AS + - HS-generated transaction ID +Output: + - None. + +Data flows: + +:: + + Typical + HS ---> AS : Home server sends events with transaction ID T. + <--- : AS sends back 200 OK. + + AS ACK Lost + HS ---> AS : Home server sends events with transaction ID T. + <-/- : AS 200 OK is lost. + HS ---> AS : Home server retries with the same transaction ID of T. + <--- : AS sends back 200 OK. If the AS had processed these events + already, it can NO-OP this request (and it knows if it is the same + events based on the transacton ID). + + +Retry notes: + - If the HS fails to pass on the events to the AS, it must retry the request. + - Since ASes by definition cannot alter the traffic being passed to it (unlike + say, a Policy Server), these requests can be done in parallel to general HS + processing; the HS doesn't need to block whilst doing this. + - Home servers should use exponential backoff as their retry algorithm. + - Home servers MUST NOT alter (e.g. add more) events they were going to + send within that transaction ID on retries, as the AS may have already + processed the events. + +Ordering notes: + - The events sent to the AS should be linearised, as they are from the event + stream. + - The home server will need to maintain a queue of transactions to send to + the AS. +:: + + PUT /transactions/$transaction_id?access_token=$hs_token + + Request format + { + events: [ + ... + ] + } Client-Server v2 API Extensions ------------------------------- @@ -216,6 +276,7 @@ additional permissions (see "C-AS Linking"). Inputs: - Application service token (``access_token``) + Either: - User ID in the AS namespace to act as. Or: @@ -269,8 +330,8 @@ create/delete any user in its namespace. This does not require any additional public APIs. -ID conventions --------------- +ID conventions ``[TODO]`` +------------------------- This concerns the well-defined conventions for mapping 3P network IDs to matrix IDs, which we expect clients to be able to do by themselves. @@ -278,4 +339,148 @@ IDs, which we expect clients to be able to do by themselves. reasonably expressed as URIs? (e.g. tel, email, irc, xmpp, ...) - What do room aliases look like? Some cases are clear (e.g. IRC) but others are a lot more fiddly (e.g. email? You don't want to share a room with - everyone who has ever sent an email to ``bob@gmail.com``)... \ No newline at end of file + everyone who has ever sent an email to ``bob@gmail.com``)... + +Examples +-------- +.. NOTE:: + - User/Alias namespaces are subject to change depending on ID conventions. + - Should home servers by default generate fixed room IDs which match the room + alias? Otherwise, you need to tell the AS that room alias X matches room ID + Y so when the home server pushes events with room ID Y the AS knows which + room that is. + +IRC +~~~ +Pre-conditions: + - Server admin stores the AS token "T_a" on the home server. + - Home server has a token "T_h". + - Home server has the domain "hsdomain.com" + +1. Application service registration +:: + + AS -> HS: Registers itself with the home server + POST /register + { + url: "https://someapp.com/matrix", + as_token: "T_a", + namespaces: { + users: [ + "@irc\.freenode\.net/.*" + ], + aliases: [ + "#irc\.freenode\.net/.*" + ], + rooms: [ + "!irc\.freenode\.net/.*" + ] + } + } + + Returns 200 OK: + { + hs_token: "T_h" + } + +2. IRC user "Bob" says "hello?" on "#matrix" at timestamp 1421416883133: +:: + + - AS stores message as potential scrollback. + - Nothing happens as no Matrix users are in the room. + +3. Matrix user "@alice:hsdomain.com" wants to join "#matrix": +:: + + User -> HS: Request to join "#irc.freenode.net/#matrix:hsdomain.com" + + HS -> AS: Room Query "#irc.freenode.net/#matrix:hsdomain.com" + GET /rooms/%23irc.freenode.net%2F%23matrix%3Ahsdomain.com?access_token=T_h + Returns 200 OK: + { + events: [ + { + content: { + body: "hello?", + msgtype: "m.text" + } + origin_server_ts: 1421416883133, + user_id: "@irc.freenode.net/Bob:hsdomain.com" + type: "m.room.message" + } + ], + state: [ + { + content: { + name: "#matrix" + } + origin_server_ts: 1421416883133, // default this to the first msg? + user_id: "@irc.freenode.net/Bob:hsdomain.com", // see above + state_key: "", + type: "m.room.name" + } + ] + } + + - HS provisions new room with *FIXED* room ID (see notes section) + "!irc.freenode.net/#matrix:hsdomain.com" with normal state events + (e.g. m.room.create). join_rules can be overridden by the AS if supplied in + "state". + - HS injects messages into room. Finds unknown user ID + "@irc.freenode.net/Bob:hsdomain.com" in AS namespace, so queries AS. + + HS -> AS: User Query "@irc.freenode.net/Bob:hsdomain.com" + GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h + Returns 200 OK: + { + profile: { + display_name: "Bob" + } + } + + - HS provisions new user with display name "Bob". + - HS sends room information back to client. + +4. @alice:hsdomain.com says "hi!" in this room: +:: + + User -> HS: Send message "hi!" in room !irc.freenode.net/#matrix:hsdomain.com + + - HS sends message. + - HS sees the room ID is in the AS namespace and pushes it to the AS. + + HS -> AS: Push event + PUT /transactions/1?access_token=T_h + { + events: [ + { + content: { + body: "hi!", + msgtype: "m.text" + }, + origin_server_ts: , + user_id: "@alice:hsdomain.com", + room_id: "!irc.freenode.net/#matrix:hsdomain.com", + type: "m.room.message" + } + ] + } + + - AS passes this through to IRC. + + +5. IRC user "Bob" says "what's up?" on "#matrix" at timestamp 1421418084816: +:: + + IRC -> AS: "what's up?" + AS -> HS: Send message via CS API extension + PUT /rooms/%21irc.freenode.net%2F%23matrix%3Ahsdomain.com/send/m.room.message + ?access_token=T_a + &user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com + &ts=1421418084816 + { + body: "what's up?" + msgtype: "m.text" + } + + - HS modifies the user_id and origin_server_ts on the event and sends it. \ No newline at end of file From 828d6992e563acf78d4a595f446ba1b518844ea4 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 16 Jan 2015 14:35:20 +0000 Subject: [PATCH 23/62] Small thoughts about the IRC example. --- drafts/as-http-api.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index dd2dbf80..1bd247f8 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -348,7 +348,10 @@ Examples - Should home servers by default generate fixed room IDs which match the room alias? Otherwise, you need to tell the AS that room alias X matches room ID Y so when the home server pushes events with room ID Y the AS knows which - room that is. + room that is. Alternatively, do we expect ASes to query the HS via the room + alias APIs just like clients do? This doesn't get us the reverse mapping + though which is what we really want when events are pushed to the AS. Maybe + a special room_id_to_alias event poke is pushed to the AS? IRC ~~~ From 603361bfd68674dae57624122eb4adb5ee6bd689 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Mon, 19 Jan 2015 10:20:39 +0000 Subject: [PATCH 24/62] Update user and room query APIs Expect the AS to create these entities, rather than the HS. --- drafts/as-http-api.rst | 126 ++++++++++++++--------------------------- 1 file changed, 41 insertions(+), 85 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 1bd247f8..931c8a07 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -109,14 +109,21 @@ Inputs: - User ID - HS Credentials Output: - - Profile info + - Whether the user exists. Side effects: - - User is created on the HS if this response 200s. + - User is created on the HS by the AS via CS APIs during the processing of this request. API called when: - HS receives an event for an unknown user ID in the AS's namespace, e.g. an invite event to a room. Notes: - - The created user will have their profile info set based on the output. + - When the AS receives this request, if the user exists, it must create the user via + the CS API. + - It can also set arbitrary information about the user (e.g. display name, join rooms, etc) + using the CS API. + - When this setup is complete, the AS should respond to the HS request. This means the AS + blocks the HS until the user is created. + - This is deemed more flexible than alternative methods (e.g. returning a JSON blob with the + user's display name and get the HS to provision the user). Retry notes: - The home server cannot respond to the client's request until the response to this API is obtained from the AS. @@ -136,12 +143,7 @@ Retry notes: 200 OK response format - { - profile: { - display_name: "string" - avatar_url: "string(uri)" - } - } + {} Room Alias Query ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -155,12 +157,20 @@ Output: - The current state events for the room if any. - The message events for the room if any. Side effects: - - A new room will be created with the alias input if this response 200s. + - Room is created on the HS by the AS via CS APIs during the processing of + this request. API called when: - HS receives an event to join a room alias in the AS's namespace. Notes: - - This can be thought of as an ``initialSync`` but for a 3P networked room, - which is lazily loaded when a matrix user tries to join the room. + - When the AS receives this request, if the room exists, it must create the room via + the CS API. + - It can also set arbitrary information about the room (e.g. name, topic, etc) + using the CS API. + - When this setup is complete, the AS should respond to the HS request. This means the AS + blocks the HS until the room is created and configured. + - This is deemed more flexible than alternative methods (e.g. returning an initial sync + style JSON blob and get the HS to provision the room). It also means that the AS knows + the room ID -> alias mapping. Retry notes: - The home server cannot respond to the client's request until the response to this API is obtained from the AS. @@ -180,31 +190,7 @@ Retry notes: 200 OK response format - { - events: [ - { - content: { - ... - }, - user_id: "string", - origin_server_ts: integer, // massaged timestamps - type: "string" - }, - ... - ], - state: [ - { - content: { - ... - }, - user_id: "string(virtual user id)", - origin_server_ts: integer, - state_key: "string", - type: "string" // e.g. m.room.name - }, - ... - ] - } + {} Pushing ``[Draft]`` ~~~~~~~~~~~~~~~~~~~ @@ -345,13 +331,6 @@ Examples -------- .. NOTE:: - User/Alias namespaces are subject to change depending on ID conventions. - - Should home servers by default generate fixed room IDs which match the room - alias? Otherwise, you need to tell the AS that room alias X matches room ID - Y so when the home server pushes events with room ID Y the AS knows which - room that is. Alternatively, do we expect ASes to query the HS via the room - alias APIs just like clients do? This doesn't get us the reverse mapping - though which is what we really want when events are pushed to the AS. Maybe - a special room_id_to_alias event poke is pushed to the AS? IRC ~~~ @@ -399,49 +378,26 @@ Pre-conditions: HS -> AS: Room Query "#irc.freenode.net/#matrix:hsdomain.com" GET /rooms/%23irc.freenode.net%2F%23matrix%3Ahsdomain.com?access_token=T_h - Returns 200 OK: - { - events: [ - { - content: { - body: "hello?", - msgtype: "m.text" - } - origin_server_ts: 1421416883133, - user_id: "@irc.freenode.net/Bob:hsdomain.com" - type: "m.room.message" - } - ], - state: [ + [Starts blocking] + AS -> HS: Creates room. Gets room ID "!aasaasasa:matrix.org". + AS -> HS: Sets room name to "#matrix". + AS -> HS: Sends message as ""@irc.freenode.net/Bob:hsdomain.com" + PUT /rooms/%21irc.freenode.net%2F%23matrix%3Ahsdomain.com/send/m.room.message + ?access_token=T_a + &user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com + &ts=1421416883133 { - content: { - name: "#matrix" - } - origin_server_ts: 1421416883133, // default this to the first msg? - user_id: "@irc.freenode.net/Bob:hsdomain.com", // see above - state_key: "", - type: "m.room.name" + body: "hello?" + msgtype: "m.text" } - ] - } - - - HS provisions new room with *FIXED* room ID (see notes section) - "!irc.freenode.net/#matrix:hsdomain.com" with normal state events - (e.g. m.room.create). join_rules can be overridden by the AS if supplied in - "state". - - HS injects messages into room. Finds unknown user ID - "@irc.freenode.net/Bob:hsdomain.com" in AS namespace, so queries AS. - - HS -> AS: User Query "@irc.freenode.net/Bob:hsdomain.com" - GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h - Returns 200 OK: - { - profile: { - display_name: "Bob" - } - } + HS -> AS: User Query "@irc.freenode.net/Bob:hsdomain.com" + GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h + [Starts blocking] + AS -> HS: Creates user using CS API extension. + AS -> HS: Set user display name to "Bob". + [Finishes blocking] + [Finished blocking] - - HS provisions new user with display name "Bob". - HS sends room information back to client. 4. @alice:hsdomain.com says "hi!" in this room: @@ -486,4 +442,4 @@ Pre-conditions: msgtype: "m.text" } - - HS modifies the user_id and origin_server_ts on the event and sends it. \ No newline at end of file + - HS modifies the user_id and origin_server_ts on the event and sends it. From 117b0901927ac814a0bc1e8f17532ed74ce19f73 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Mon, 19 Jan 2015 10:33:08 +0000 Subject: [PATCH 25/62] Add registration API extension. For AS provisioning of users with its namespace. --- drafts/as-http-api.rst | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 931c8a07..11a404b2 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -166,6 +166,7 @@ Notes: the CS API. - It can also set arbitrary information about the room (e.g. name, topic, etc) using the CS API. + - It can send messages as other users in order to populate scrollback. - When this setup is complete, the AS should respond to the HS request. This means the AS blocks the HS until the room is created and configured. - This is deemed more flexible than alternative methods (e.g. returning an initial sync @@ -258,7 +259,7 @@ every request. It would be an annoying amount of book-keeping to maintain tokens for every virtual user. It would be preferable if the application service could use the CS API with its own ``as_token`` instead, and specify the virtual user they wish to be acting on behalf of. For real users, this would require -additional permissions (see "C-AS Linking"). +additional permissions granting the AS permission to masquerade as a matrix user. Inputs: - Application service token (``access_token``) @@ -312,9 +313,26 @@ Server admin style permissions The home server needs to give the application service *full control* over its namespace, both for users and for room aliases. This means that the AS should be able to create/edit/delete any room alias in its namespace, as well as -create/delete any user in its namespace. This does not require any additional -public APIs. +create/delete any user in its namespace. No additional API changes need to be +made in order for control of room aliases to be granted to the AS. Creation of +users needs API changes in order to: +- Work around captchas. +- Have a 'passwordless' user. + +This involves bypassing the registration flows entirely. This is achieved by +including the AS token on a ``/register`` request, along with a login type of +``m.login.application_service`` to set the desired user ID without a password. + +:: + + /register?access_token=$as_token + + Content: + { + type: "m.login.application_service", + user: "" + } ID conventions ``[TODO]`` ------------------------- @@ -394,6 +412,11 @@ Pre-conditions: GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h [Starts blocking] AS -> HS: Creates user using CS API extension. + POST /register?access_token=T_a + { + type: "m.login.application_service", + user: "irc.freenode.net/Bob" + } AS -> HS: Set user display name to "Bob". [Finishes blocking] [Finished blocking] From ae0fb938adf7c333e47720394c0d9f73f9330402 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Mon, 19 Jan 2015 10:45:51 +0000 Subject: [PATCH 26/62] Update IRC example to reflect new room ID --- drafts/as-http-api.rst | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 11a404b2..c8aeca35 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -154,8 +154,7 @@ Inputs: - Room alias - HS Credentials Output: - - The current state events for the room if any. - - The message events for the room if any. + - Whether the room exists. Side effects: - Room is created on the HS by the AS via CS APIs during the processing of this request. @@ -397,10 +396,10 @@ Pre-conditions: HS -> AS: Room Query "#irc.freenode.net/#matrix:hsdomain.com" GET /rooms/%23irc.freenode.net%2F%23matrix%3Ahsdomain.com?access_token=T_h [Starts blocking] - AS -> HS: Creates room. Gets room ID "!aasaasasa:matrix.org". + AS -> HS: Creates room. Gets room ID "!aasaasasa:hsdomain.com". AS -> HS: Sets room name to "#matrix". AS -> HS: Sends message as ""@irc.freenode.net/Bob:hsdomain.com" - PUT /rooms/%21irc.freenode.net%2F%23matrix%3Ahsdomain.com/send/m.room.message + PUT /rooms/%21aasaasasa%3Ahsdomain.com/send/m.room.message ?access_token=T_a &user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com &ts=1421416883133 @@ -426,7 +425,7 @@ Pre-conditions: 4. @alice:hsdomain.com says "hi!" in this room: :: - User -> HS: Send message "hi!" in room !irc.freenode.net/#matrix:hsdomain.com + User -> HS: Send message "hi!" in room !aasaasasa:hsdomain.com - HS sends message. - HS sees the room ID is in the AS namespace and pushes it to the AS. @@ -442,7 +441,7 @@ Pre-conditions: }, origin_server_ts: , user_id: "@alice:hsdomain.com", - room_id: "!irc.freenode.net/#matrix:hsdomain.com", + room_id: "!aasaasasa:hsdomain.com", type: "m.room.message" } ] @@ -456,7 +455,7 @@ Pre-conditions: IRC -> AS: "what's up?" AS -> HS: Send message via CS API extension - PUT /rooms/%21irc.freenode.net%2F%23matrix%3Ahsdomain.com/send/m.room.message + PUT /rooms/%21aasaasasa%3Ahsdomain.com/send/m.room.message ?access_token=T_a &user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com &ts=1421418084816 From 7a2b6abee5ecc713452283034a196d70e5b2bc45 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Tue, 20 Jan 2015 14:22:48 +0000 Subject: [PATCH 27/62] Flesh out ID conventions Going down the API route. --- drafts/as-http-api.rst | 70 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 64 insertions(+), 6 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index c8aeca35..cb64a53c 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -333,16 +333,74 @@ including the AS token on a ``/register`` request, along with a login type of user: "" } -ID conventions ``[TODO]`` +ID conventions ``[Draft]`` ------------------------- +.. NOTE:: + - Giving HSes the freedom to namespace still feels like the Right Thing here. + - Exposing a public API provides the consistency which was the main complaint + against namespacing. + - This may have knock-on effects for the AS registration API. E.g. why don't + we let ASes specify the *URI* regex they want? + This concerns the well-defined conventions for mapping 3P network IDs to matrix IDs, which we expect clients to be able to do by themselves. -- What do user IDs look like? Butchered URIs? Can all 3P network IDs be - reasonably expressed as URIs? (e.g. tel, email, irc, xmpp, ...) -- What do room aliases look like? Some cases are clear (e.g. IRC) but others - are a lot more fiddly (e.g. email? You don't want to share a room with - everyone who has ever sent an email to ``bob@gmail.com``)... +User IDs +~~~~~~~~ +Matrix users may wish to directly contact a virtual user, e.g. to send an email. +The URI format is a well-structured way to represent a number of different ID +types, including: + +- MSISDNs (``tel``) +- Email addresses (``mailto``) +- IRC nicks (``irc`` - https://tools.ietf.org/html/draft-butcher-irc-url-04) +- XMPP (xep-0032) +- SIP URIs (RFC 3261) + +As a result, virtual user IDs SHOULD relate to their URI counterpart. This +mapping from URI to user ID can be expressed in a number of ways: + +- Expose a C-S API on the HS which takes URIs and responds with user IDs. +- Munge the URI with the user ID. + +Exposing an API would allow HSes to internally map user IDs however they like, +at the cost of an extra round trip (of which the response can be cached). +Munging the URI would allow clients to apply the mapping locally, but would force +user X on service Y to always map to the same munged user ID. Considering the +exposed API could just be applying this munging, there is more flexibility if +an API is exposed. + +:: + + GET /_matrix/app/v1/user?uri=$url_encoded_uri + + Returns 200 OK: + { + user_id: + } + +Room Aliases +~~~~~~~~~~~~ +We may want to expose some 3P network rooms so Matrix users can join them directly, +e.g. IRC rooms. We don't want to expose every 3P network room though, e.g. mailto, +tel. Rooms which are publicly accessible (e.g. IRC rooms) can be exposed as an alias by +the application service. Private rooms (e.g. sending an email to someone) should not +be exposed in this way, but should instead operate using normal invite/join semantics. +Therefore, the ID conventions discussed below are only valid for public rooms which +expose room aliases. + +Matrix users may wish to join XMPP rooms (e.g. using XEP-0045) or IRC rooms. In both +cases, these rooms can be expressed as URIs. For consistency, these "room" URIs +SHOULD be mapped in the same way as "user" URIs. + +:: + + GET /_matrix/app/v1/alias?uri=$url_encoded_uri + + Returns 200 OK: + { + alias: + } Examples -------- From 23d3e5081272a8ca53b092e79d1e716641ad8f30 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Tue, 20 Jan 2015 17:48:43 +0000 Subject: [PATCH 28/62] Add use case justification for regex --- drafts/application_services.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drafts/application_services.rst b/drafts/application_services.rst index 2c73107f..c654c010 100644 --- a/drafts/application_services.rst +++ b/drafts/application_services.rst @@ -85,7 +85,8 @@ In the registration process, the AS provides: * Details of the namespaces of users and rooms the AS is acting on behalf of and "subscribing to" * Namespaces are defined as a list of regexps against which to match room aliases, - room IDs, and user IDs. + room IDs, and user IDs. Regexps give the flexibility to say, sub-domain MSISDN + ranges per AS, whereas a blunt prefix string does not. * There is overlap between selecting events via the csv2 Filter API and subscribing to events here - perhaps subscription involves passing a filter token into the registration API. From 90dfee1c275f350773f583d3a87f054e65b4688f Mon Sep 17 00:00:00 2001 From: Kegsay Date: Tue, 20 Jan 2015 17:49:09 +0000 Subject: [PATCH 29/62] Remove TODO note on regex --- drafts/as-http-api.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index cb64a53c..1c1f342f 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -10,8 +10,6 @@ This contains home server APIs which are used by the application service. Registration API ``[Draft]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. NOTE:: - - Do we really have to use regex for this? Can't we do this a nicer way? This API registers the application service with its host homeserver to offer its services. From 0a07d688184cd0658cd03c25c747706ef4098434 Mon Sep 17 00:00:00 2001 From: David Baker Date: Wed, 28 Jan 2015 18:20:28 +0000 Subject: [PATCH 30/62] Add overview on push. --- drafts/push_overview.rst | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 drafts/push_overview.rst diff --git a/drafts/push_overview.rst b/drafts/push_overview.rst new file mode 100644 index 00000000..6cee68c5 --- /dev/null +++ b/drafts/push_overview.rst @@ -0,0 +1,31 @@ +Push Notifications +================== + +Matrix supports push notifications as a first class citizen. Home Servers send +notifications of user events to user-configured HTTP endpoints. User may also +configure a number of rules that determine what events generate notifications. +These are all stored and managed by the users home server such that settings can +be reused between client apps as appropriate. + +Nomenclature +------------ + +Pusher + A 'pusher' is an activity in the Home Server that manages the sending + of HTTP notifications for a single device of a single user. + +Push Rules + A push rule is a single rule, configured by a matrix user, that gives + instructions to the Home Server about whether an event should be notified + about and how given a set of conditions. Matrix clients allow the user to + configure these. They create and view them via the Client to Server REST API. + +Push Gateway + A push gateway is a server that receives HTTP event notifications from Home + Servers and passes them on to a different protocol such as APNS for iOS + devices or GCM for Android devices. Matrix.org provides a reference push + gateway, 'sygnal'. + +For information on the client-server API for setting pushers and push rules, see +the Client Server API section. For more information on the format of HTTP +notifications, see the HTTP Notification Protocol section. From c27fb8a2620ed952bde6c0032ed25daacab913e3 Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 13:49:18 +0000 Subject: [PATCH 31/62] Add WIP docs for the client-server bit of push. --- drafts/push_csapi.rst | 52 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 drafts/push_csapi.rst diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst new file mode 100644 index 00000000..bc960fe4 --- /dev/null +++ b/drafts/push_csapi.rst @@ -0,0 +1,52 @@ +Push Notifications +================== + +Pushers +------- +To receive any notification pokes at all, it is necessary to configure a +'pusher' on the Home Server that you wish to receive notifications from. There +is a single API endpoint for this:: + +Fetching a user account displayname:: + + POST $PREFIX/pushers/set + +This takes a JSON object with the following keys: + +pushkey + This is a unique identifier for this pusher. The value you should use for this + is the routing or destination address information for the notification, for + example, the APNS token for APNS or the Registration ID for GCM. If your + notification client has no such concept, use any unique identifier. +kind + The kind of pusher to configure. 'http' makes a pusher that sends HTTP pokes. + null deletes the pusher. +instance_handle + This is an identifier for the device which owns the pusher. It may be up to 32 + characters long. It must be unique among all the pushers for a given user + (therefore the device ID may not be used). It is advised that when an app's + data is copied or restored to a different device, this ID remain the same (ie. + be shared by multiple pushers for multiple devices). Client apps should be + aware that this situation can occur and be able to rectify it (eg. by + offerring to reset the instance_hanlde, optionally duplicating all push rules + to new instance handle). +app_id + appId is a reverse-DNS style identifier for the application. It is recommended + that this end with the platform, such that different platform versions get + different app identifiers. Max length, 64 chars. +app_display_name + A string that will allow the user to identify what application owns this + pusher. +device_display_name + A string that will allow the user to identify what device owns this pusher. +lang + The preferred language for receiving notifications (eg, 'en' or 'en-US') +data + A dictionary of information for the pusher implementation itself. For HTTP + pushers, this must contain a 'url' key which is a string of the URL that + should be used to send notifications. + +If the pusher was created successfully, an empty JSON dictionary is returned. + + + From f68fa8d6f887cc58e591cc06c708548b6c2636a7 Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 15:14:45 +0000 Subject: [PATCH 32/62] Push rules c/s spec. --- drafts/push_csapi.rst | 105 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index bc960fe4..89212576 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -49,4 +49,109 @@ data If the pusher was created successfully, an empty JSON dictionary is returned. +Push Rules +---------- +Home Servers have an interface to configure what events trigger notifications. +This behaviour is configured through 'Push Rules'. Push Rules come in a variety +of different kinds and each kind of rule has an associated priority. The +different kinds of rule, in descending order of priority, are: + +Override Rules + The highest priority rules are user-configured overrides. +Content Rules + These configure behaviour for (unencrypted) messages that match certain + patterns. +Room Rules + These change the behaviour of all messages to a given room. The rule_id of a + room rule is always the room that it affects. +Sender + These rules configure notification behaviour for messages from a specific, + named Matrix user ID. The rule_id of Sender rules is always the Matrix user + ID of the user whose messages theyt apply to. +Underride + These are identical to override rules, but have a lower priority than content, + room and sender rules. +Default + These are rules provided by the home server and cannot be changed by the user. + They are the lowest priority rule and establish baseline behaviour. + +In addition, each kind of rule except default may be either global or +device-specific. Device specific rules only affect delivery of notifications via +pushers with a matching instance_handle. All device-specific rules are higher +priority than all global rules. Thusly, the full list of rule kinds, in +descending priority order, is as follows: + + * Device-specific Override + * Device-specific Content + * Device-specific Room + * Device-specific Sender + * Device-specific Underride + * Global Override + * Global Content + * Global Room + * Global Sender + * Global Underride + * Global Default + +For some kinds of rule, rules of the same kind also have an ordering with +respect to one another. The kinds that do not are room and sender rules where +the rules are mutually exclusive by definition and therefore an ordering would +be redundant. Actions for the highest priority rule and only that rule apply +(for example, a set_sound action in a lower priority rule will not apply if a +higher priority rule matches, even if that rule does not specify a sound). + +Push Rules: Actions: +-------------------- +All rules have an associated list of 'actions'. An action affects if and how a +notification is delievered for a matching event. This standard defines the +following actions, although if Home servers wish to support more, they are free +to do so: + +notify + This causes each matching event to generate a notification. +dont_notify + Prevents this event from generating a notification +coalesce + This enables notifications for matching events but activates Home Server + specific behaviour to intelligently coalesce multiple events into a single + notification. Not all Home Servers may support this. Those that do not should + treat it as the 'notify' action. +set_sound + Sets the value 'sound' key that is sent in the notification poke. This has an + associated string which is the value to set the 'sound' key to. + +Actions that have no parameter are represented as a string. Those with a +parameter are represented as a dictionary with a single key/value pair where the +key is the name of the action and the value is the parameter, eg. { "set_sound": +"ping.wav" } + +Push Rules: Conditions: +----------------------- +Override, Underride and Default rules have a list of 'conditions'. All +conditions must hold true for an event in order for a rule to be applied to an +event. Matrix specifies the following conditions, although if Home Servers wish +to support others, they are free to do so: + +event_match + This is a glob pattern match on a field of the event. Parameters: + * 'key': The dot-separated field of the event to match, eg. content.body + * 'pattern': The glob-style pattern to match against. Patterns with no + special glob characters should be treated as having asterisks + prepended and appended when testing the condition. +device + Matches the instance_handle of the device that the notification would be + delivered to. Parameters: + * 'instance_handle': The instance_handle of the device. +contains_display_name + This matches unencrypted messages where content.body contains the owner's + display name in that room. This is a separate rule because display names may + change and as such it would be hard to maintain a rule that matched the user's + display name. This condition has no parameters. +room_member_count + This matches the current number of members in the room. + * 'is': A decimal integer optionally prefixed by one of, '==', '<', '>', + '>=' or '<='. A prefix of '<' matches rooms where the member count is + strictly less than the given number and so forth. If no prefix is present, + this matches rooms where the member count is exactly equal to the given + number (ie. the same as '=='). From b7a6e8f7b52c366cbcb6ed8577e0d12b6e99cf0d Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 16:25:53 +0000 Subject: [PATCH 33/62] Finish non-API doc parts of push rules. --- drafts/push_csapi.rst | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 89212576..bd204389 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -60,7 +60,9 @@ Override Rules The highest priority rules are user-configured overrides. Content Rules These configure behaviour for (unencrypted) messages that match certain - patterns. + patterns. Content rules take one parameter, 'pattern', that gives the pattern + to match against. This is treated in the same way as pattern for event_match + conditions, below. Room Rules These change the behaviour of all messages to a given room. The rule_id of a room rule is always the room that it affects. @@ -100,6 +102,8 @@ be redundant. Actions for the highest priority rule and only that rule apply (for example, a set_sound action in a lower priority rule will not apply if a higher priority rule matches, even if that rule does not specify a sound). +Rules also have an identifier, rule_id, which is a string. + Push Rules: Actions: -------------------- All rules have an associated list of 'actions'. An action affects if and how a @@ -155,3 +159,8 @@ room_member_count this matches rooms where the member count is exactly equal to the given number (ie. the same as '=='). +Room, Sender, User and Content rules do not have conditions in the same way, +but instead have predefined conditions, the behaviour of which can be configured +using parameters named as described above. In the cases of room and sender +rules, the rule_id of the rule determines its behaviour. + From 0c79798f708334c19097f873d24e37e46d75b6da Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 17:10:09 +0000 Subject: [PATCH 34/62] More WIP API docs and remove spurious line. --- drafts/push_csapi.rst | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index bd204389..1c99a859 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -7,8 +7,6 @@ To receive any notification pokes at all, it is necessary to configure a 'pusher' on the Home Server that you wish to receive notifications from. There is a single API endpoint for this:: -Fetching a user account displayname:: - POST $PREFIX/pushers/set This takes a JSON object with the following keys: @@ -164,3 +162,20 @@ but instead have predefined conditions, the behaviour of which can be configured using parameters named as described above. In the cases of room and sender rules, the rule_id of the rule determines its behaviour. +Push Rules: API +--------------- +Rules live under a hierarchy in the REST API that resembles:: + + $PREFIX/pushrules/// + +To add or change a rule, a client performs a PUT request to the appropriate URL. +When adding rules of a type that has an ordering, the client can add parameters +that define the priority of the rule: + +before + Use 'before' with a rule_id as its value to make the new rule the next-less + important rule with respect to the given rule. +after + This makes the new rule the next-most important rule relative to the given + rule. + From 88178e376e6b352dedfe6b98b05c034e3158c94a Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 17:26:58 +0000 Subject: [PATCH 35/62] Oops, before is *more* important. --- drafts/push_csapi.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 1c99a859..115b9216 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -173,9 +173,8 @@ When adding rules of a type that has an ordering, the client can add parameters that define the priority of the rule: before - Use 'before' with a rule_id as its value to make the new rule the next-less + Use 'before' with a rule_id as its value to make the new rule the next-more important rule with respect to the given rule. after - This makes the new rule the next-most important rule relative to the given + This makes the new rule the next-less important rule relative to the given rule. - From c40b97aaf5089140ab29a53529dc9f67b63e820e Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 18:16:36 +0000 Subject: [PATCH 36/62] Rule adding API with examples --- drafts/push_csapi.rst | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 115b9216..3c6edac6 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -168,6 +168,17 @@ Rules live under a hierarchy in the REST API that resembles:: $PREFIX/pushrules/// +The component parts are as follows: + +scope + Either 'global' or 'device/' to specify global rules or + device rules for the given instance_handle. +kind + The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content' or + 'default'. +rule_id + The identifier for the rule. + To add or change a rule, a client performs a PUT request to the appropriate URL. When adding rules of a type that has an ordering, the client can add parameters that define the priority of the rule: @@ -178,3 +189,33 @@ before after This makes the new rule the next-less important rule relative to the given rule. + +All requests to the push rules API also require an access_token as a query +paraemter. + +The content of the PUT request is a JSON object with a list of actions under the +'actions' key and either conditions (under the 'conditions' key) or the +appropriate parameters for the rule (under the appropriate key name). + +Examples: + +To create a rule that suppresses notifications for the room '#spam:matrix.org':: + + curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456" + +To suppress notifications for the user '@spambot:matrix.org':: + + curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/room/%40spambot%3Amatrix.org?access_token=123456" + +To always notify for messages that contain the work 'cake' and set a specific sound (with a rule_id of 'SSByZWFsbHkgbGlrZSBjYWtl'):: + + curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" + +To add a rule suppressing notifications for messages starting with 'cake' but ending with 'lie', superseeding the previous rule:: + + curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake*lie", "actions" : ["notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" + +To add a custom sound for notifications messages containing the word 'beer' in any rooms with 10 members or fewer (with greater importance than the room, sender and content rules):: + + curl -X PUT -H "Content-Type: application/json" -d '{ "conditions": [{"kind": "event_match", "key": "content.body", "pattern": "beer" }, {"kind": "room_member_count", "is": "<=10"}], "actions" : ["notify", {"set_sound":"beeroclock.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456 + From 1a5678786fb34d35110d44ca5a5aefd53509f34a Mon Sep 17 00:00:00 2001 From: David Baker Date: Mon, 2 Feb 2015 18:36:04 +0000 Subject: [PATCH 37/62] Doc deleting & getting rules. --- drafts/push_csapi.rst | 98 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 3c6edac6..0a3294d4 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -219,3 +219,101 @@ To add a custom sound for notifications messages containing the word 'beer' in a curl -X PUT -H "Content-Type: application/json" -d '{ "conditions": [{"kind": "event_match", "key": "content.body", "pattern": "beer" }, {"kind": "room_member_count", "is": "<=10"}], "actions" : ["notify", {"set_sound":"beeroclock.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456 + +To delete rules, a client would just make a DELETE request to the same URL:: + + curl -X DELETE "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456" + + +Retrieving the current ruleset can be done either by fetching individual rules +using the scheme as specified above. This returns the rule in the same format as +would be given in the PUT API with the addition of a rule_id:: + + curl "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456" + +Returns:: + + { + "actions": [ + "dont_notify" + ], + "rule_id": "#spam:matrix.org" + } + +Clients can also fetch broader sets of rules by removing path components. +Requesting the root level returns a structure as follows:: + + { + "device": { + "exampledevice": { + "content": [], + "default": [], + "override": [], + "room": [ + { + "actions": [ + "dont_notify" + ], + "rule_id": "#spam:matrix.org" + } + ], + "sender": [], + "underride": [] + } + }, + "global": { + "content": [], + "default": [ + { + "actions": [ + "notify", + { + "set_sound": "default" + } + ], + "conditions": [ + { + "key": "content.body", + "kind": "event_match", + "pattern": "*@test:steve*" + } + ] + }, + { + "actions": [ + "notify", + { + "set_sound": "default" + } + ], + "conditions": [ + { + "kind": "contains_display_name" + } + ] + }, + { + "actions": [ + "notify", + { + "set_sound": "default" + } + ], + "conditions": [ + { + "is": "2", + "kind": "room_member_count" + } + ] + } + ], + "override": [], + "room": [], + "sender": [], + "underride": [] + } + } + +Adding patch components to the request drills down into this structure to filter +to only the requested set of rules. + From 067e4bb7771282a71567bc60fc28b99dc75ad5e9 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 3 Feb 2015 14:21:56 +0000 Subject: [PATCH 38/62] Add docs for http notification poke protocol. --- drafts/push_overview.rst | 3 +- drafts/push_pgwapi.rst | 128 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+), 1 deletion(-) create mode 100644 drafts/push_pgwapi.rst diff --git a/drafts/push_overview.rst b/drafts/push_overview.rst index 6cee68c5..216ebdcd 100644 --- a/drafts/push_overview.rst +++ b/drafts/push_overview.rst @@ -24,7 +24,8 @@ Push Gateway A push gateway is a server that receives HTTP event notifications from Home Servers and passes them on to a different protocol such as APNS for iOS devices or GCM for Android devices. Matrix.org provides a reference push - gateway, 'sygnal'. + gateway, 'sygnal'. A client app tells a Home Server what push gateway + to send notifications to when it sets up a pusher. For information on the client-server API for setting pushers and push rules, see the Client Server API section. For more information on the format of HTTP diff --git a/drafts/push_pgwapi.rst b/drafts/push_pgwapi.rst new file mode 100644 index 00000000..bf0a4c37 --- /dev/null +++ b/drafts/push_pgwapi.rst @@ -0,0 +1,128 @@ +Push Notifications: HTTP Notification Protocol +============================================== +This describes the format used by "http" pushers to send notifications of +events. + +Notifications are sent as HTTP POST requests to the URL configured when the +pusher is created. The body of the POST request is a JSON dictionary. The format +is as follows:: + + { + "notification": { + "id": "$3957tyerfgewrf384", + "type": "m.room.message", + "sender": "@exampleuser:matrix.org", + "sender_display_name": "Major Tom", + "room_name": "Mission Control", + "room_alias": "#exampleroom:matrix.org", + "prio": "high", + "content": { + "msgtype": "m.text", + "body": "I'm floating in a most peculiar way." + } + }, + "counts": { + "unread" : 2, + "missed_calls": 1 + } + "devices": [ + { + "app_id": "org.matrix.matrixConsole.ios", + "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/", + "pushkey_ts": 12345678, + "data" : { + }, + "tweaks": { + "sound": "bing.wav" + } + } + ] + } + } + +The contents of this dictionary are defined as follows: + +id + An identifier for this notification that may be used to detect duplicate + notification requests. This is not necessarily the ID of the event that + triggered the notification. +type + The type of the event as in the event's 'type' field. +sender + The sender of the event as in the corresponding event field. +sender_display_name + The current display name of the sender in the room in which the event + occurred. +room_name + The name of the room in which the event occurred. +room_alias + An alias to display for the room in which the event occurred. +prio + The priority of the notification. Acceptable values are 'high' or 'low. If + omitted, 'high' is assumed. This may be used by push gateways to deliver less + time-sensitive notifications in a way that will preserve battery power on + mobile devices. +content + The 'content' field from the event, if present. If the event had no content + field, this field is omitted. +counts + This is a dictionary of the current number of unacknowledged communications + for the recipient user. Counts whose value is zero are omitted. +unread + The number of unread messages a user has accross all of the rooms they are a + member of. +missed_calls + The number of unacknowledged missed calls a user has accross all rooms of + which they are a member. +device + This is an array of devices that the notification should be sent to. +app_id + The app_id given when the pusher was created. +pushkey + The pushkey given when the pusher was created. +pushkey_ts + The unix timestamp (in seconds) when the pushkey was last updated. +data + A dictionary of additional pusher-specific data. For 'http' pushers, this is + the data dictionary passed in at pusher creation minus the 'url' key. +tweaks + A dictionary of customisations made to the way this notification is to be + presented. These are added by push rules. +sound + Sets the sound file that should be played. 'default' means that a default + sound should be played. + +The recipient of an HTTP notification should respond with an HTTP 2xx response +when the notification has been processed. If the endpoint returns an HTTP error +code, the Home Server should retry for a reasonable amount of time with a +reasonable backoff scheme. + +The endpoint should return a JSON dictionary as follows:: + + { + "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ] + } + +Whose keys are: + +rejected + A list of all pushkeys given in the notification request that are not valid. + These could have been rejected by an upstream gateway because they have + expired or have never been valid. Home Servers must cease sending notification + requests for these pushkeys and remove the associated pushers. It may not + necessarily be the notification in the request that failed: it could be that + a previous notification to the same pushkey failed. + +Push: Recommendations for APNS +------------------------------ +For sending APNS notifications, the exact format is flexible and up to the +client app and its push gateway to agree on (since APNS requires that the sender +have a private key owned by the app developer, each app must have its own push +gateway). However, Matrix strongly recommends: + + * That the APNS token be base64 encoded and used as the pushkey. + * That a different app_id be used for apps on the production and sandbox + APS environments. + * That PANS push gateways do not attempt to wait for errors from the APNS + gateway before returning and instead to store failures and return + 'rejected' responses next time that pushkey is used. From 8920d58cc43a6767c042a635ac6109713ce788ac Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 3 Feb 2015 14:35:01 +0000 Subject: [PATCH 39/62] Strongly recommend standard URI scheme for notification pokes. --- drafts/push_pgwapi.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drafts/push_pgwapi.rst b/drafts/push_pgwapi.rst index bf0a4c37..8dd054f6 100644 --- a/drafts/push_pgwapi.rst +++ b/drafts/push_pgwapi.rst @@ -4,7 +4,11 @@ This describes the format used by "http" pushers to send notifications of events. Notifications are sent as HTTP POST requests to the URL configured when the -pusher is created. The body of the POST request is a JSON dictionary. The format +pusher is created, but Matrix strongly recommends that the path should be:: + + /_matrix/push/v1/notify + +The body of the POST request is a JSON dictionary. The format is as follows:: { From 91d12dc9553995fd5dc8907cfaed7b842755c6b5 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 3 Feb 2015 15:38:12 +0000 Subject: [PATCH 40/62] Replace set_sound with more generic set_tweaks. --- drafts/push_csapi.rst | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 0a3294d4..b979331f 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -97,8 +97,8 @@ For some kinds of rule, rules of the same kind also have an ordering with respect to one another. The kinds that do not are room and sender rules where the rules are mutually exclusive by definition and therefore an ordering would be redundant. Actions for the highest priority rule and only that rule apply -(for example, a set_sound action in a lower priority rule will not apply if a -higher priority rule matches, even if that rule does not specify a sound). +(for example, a set_tweak action in a lower priority rule will not apply if a +higher priority rule matches, even if that rule does not specify any tweaks). Rules also have an identifier, rule_id, which is a string. @@ -118,14 +118,28 @@ coalesce specific behaviour to intelligently coalesce multiple events into a single notification. Not all Home Servers may support this. Those that do not should treat it as the 'notify' action. -set_sound - Sets the value 'sound' key that is sent in the notification poke. This has an - associated string which is the value to set the 'sound' key to. - -Actions that have no parameter are represented as a string. Those with a -parameter are represented as a dictionary with a single key/value pair where the -key is the name of the action and the value is the parameter, eg. { "set_sound": -"ping.wav" } +set_tweak + Sets an entry in the 'tweaks' dictionary key that is sent in the notification + poke. This takes the form of a dictionary with a 'set_tweak' key whose value + is the name of the tweak to set. It must also have a 'value' key which is + the value to which it should be set. + +Actions that have no parameters are represented as a string. Otherwise, they are +represented as a dictionary with a key equal to their name and other keys as +their parameters, eg. { "set_tweak": "sound", "value": "default" } + +Push Rule Actions: Tweaks +------------------------- +The 'set_tweak' key action is used to add an entry to the 'tweaks' dictionary +that is sent in the notification poke. The following tweaks are e defined: + +sound + A sound to be played when this notification arrives. 'default' means to + play a default sound. + +Tweaks are passed transparently through the Home Server so client applications +and push gateways may agree on additional tweaks, for example, how to flash the +notification light on a mobile device. Push Rules: Conditions: ----------------------- From f24e951b024c206aa9fa4caee77576d6a1ed12ac Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 3 Feb 2015 16:21:09 +0000 Subject: [PATCH 41/62] s/instance_handle/profile_tag/ --- drafts/push_csapi.rst | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index b979331f..8e1fd99c 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -19,15 +19,13 @@ pushkey kind The kind of pusher to configure. 'http' makes a pusher that sends HTTP pokes. null deletes the pusher. -instance_handle - This is an identifier for the device which owns the pusher. It may be up to 32 - characters long. It must be unique among all the pushers for a given user - (therefore the device ID may not be used). It is advised that when an app's - data is copied or restored to a different device, this ID remain the same (ie. - be shared by multiple pushers for multiple devices). Client apps should be - aware that this situation can occur and be able to rectify it (eg. by - offerring to reset the instance_hanlde, optionally duplicating all push rules - to new instance handle). +profile_tag + This is a string that determines what set of device rules will be matched when + evaluating push rules for this pusher. It is an arbitrary string. Multiple + devices maybe use the same profile_tag. It is advised that when an app's + data is copied or restored to a different device, this value remain the same. + Client apps should offer ways to change the profile_tag, optionally copying + rules from the old profile tag. app_id appId is a reverse-DNS style identifier for the application. It is recommended that this end with the platform, such that different platform versions get @@ -77,7 +75,7 @@ Default In addition, each kind of rule except default may be either global or device-specific. Device specific rules only affect delivery of notifications via -pushers with a matching instance_handle. All device-specific rules are higher +pushers with a matching profile_tag. All device-specific rules are higher priority than all global rules. Thusly, the full list of rule kinds, in descending priority order, is as follows: @@ -154,10 +152,10 @@ event_match * 'pattern': The glob-style pattern to match against. Patterns with no special glob characters should be treated as having asterisks prepended and appended when testing the condition. -device - Matches the instance_handle of the device that the notification would be +profile_tag + Matches the profile_tag of the device that the notification would be delivered to. Parameters: - * 'instance_handle': The instance_handle of the device. + * 'profile_tag': The profile_tag to match with. contains_display_name This matches unencrypted messages where content.body contains the owner's display name in that room. This is a separate rule because display names may @@ -185,8 +183,8 @@ Rules live under a hierarchy in the REST API that resembles:: The component parts are as follows: scope - Either 'global' or 'device/' to specify global rules or - device rules for the given instance_handle. + Either 'global' or 'device/' to specify global rules or + device rules for the given profile_tag. kind The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content' or 'default'. From dbbf9626b4e2d1c0549bae0daee8891553179fa2 Mon Sep 17 00:00:00 2001 From: David Baker Date: Wed, 4 Feb 2015 11:34:32 +0000 Subject: [PATCH 42/62] Fix various incorreect examples in the doc. --- drafts/push_csapi.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 8e1fd99c..9016d0e2 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -61,7 +61,7 @@ Content Rules conditions, below. Room Rules These change the behaviour of all messages to a given room. The rule_id of a - room rule is always the room that it affects. + room rule is always the ID of the room that it affects. Sender These rules configure notification behaviour for messages from a specific, named Matrix user ID. The rule_id of Sender rules is always the Matrix user @@ -211,21 +211,21 @@ appropriate parameters for the rule (under the appropriate key name). Examples: -To create a rule that suppresses notifications for the room '#spam:matrix.org':: +To create a rule that suppresses notifications for the room with ID '!dj234r78wl45Gh4D:matrix.org':: - curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456" + curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" To suppress notifications for the user '@spambot:matrix.org':: - curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/room/%40spambot%3Amatrix.org?access_token=123456" + curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" To always notify for messages that contain the work 'cake' and set a specific sound (with a rule_id of 'SSByZWFsbHkgbGlrZSBjYWtl'):: - curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" + curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" To add a rule suppressing notifications for messages starting with 'cake' but ending with 'lie', superseeding the previous rule:: - curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake*lie", "actions" : ["notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/sender/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" + curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake*lie", "actions" : ["notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" To add a custom sound for notifications messages containing the word 'beer' in any rooms with 10 members or fewer (with greater importance than the room, sender and content rules):: From 8184855d5ad4dfe03957732b1b3d202c5cce8fdb Mon Sep 17 00:00:00 2001 From: David Baker Date: Wed, 4 Feb 2015 16:30:57 +0000 Subject: [PATCH 43/62] add note about scope of rule_ids. --- drafts/push_csapi.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 9016d0e2..e9abf326 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -98,7 +98,10 @@ be redundant. Actions for the highest priority rule and only that rule apply (for example, a set_tweak action in a lower priority rule will not apply if a higher priority rule matches, even if that rule does not specify any tweaks). -Rules also have an identifier, rule_id, which is a string. +Rules also have an identifier, rule_id, which is a string. The rule_id may +alphanumeric characters only. The rule_id is unique within the kind of rule and +scope: rule_ids need not be unique between rules of the same kind on different +devices. Push Rules: Actions: -------------------- From e6c9e77d350dbc5eceaf6c6a85f3b4dbc19e9a22 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 5 Feb 2015 10:49:48 +0000 Subject: [PATCH 44/62] Change server default rules. --- drafts/push_csapi.rst | 69 ++++++++----------------------------------- 1 file changed, 12 insertions(+), 57 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index e9abf326..ae9591bf 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -69,15 +69,12 @@ Sender Underride These are identical to override rules, but have a lower priority than content, room and sender rules. -Default - These are rules provided by the home server and cannot be changed by the user. - They are the lowest priority rule and establish baseline behaviour. -In addition, each kind of rule except default may be either global or -device-specific. Device specific rules only affect delivery of notifications via -pushers with a matching profile_tag. All device-specific rules are higher -priority than all global rules. Thusly, the full list of rule kinds, in -descending priority order, is as follows: +In addition, each kind of rule may be either global or device-specific. Device +specific rules only affect delivery of notifications via pushers with a matching +profile_tag. All device-specific rules are higher priority than all global +rules. Thusly, the full list of rule kinds, in descending priority order, is as +follows: * Device-specific Override * Device-specific Content @@ -89,7 +86,6 @@ descending priority order, is as follows: * Global Room * Global Sender * Global Underride - * Global Default For some kinds of rule, rules of the same kind also have an ordering with respect to one another. The kinds that do not are room and sender rules where @@ -101,7 +97,12 @@ higher priority rule matches, even if that rule does not specify any tweaks). Rules also have an identifier, rule_id, which is a string. The rule_id may alphanumeric characters only. The rule_id is unique within the kind of rule and scope: rule_ids need not be unique between rules of the same kind on different -devices. +devices. + +A home server may also have server default rules of each kind and in each scope. +Server default rules are lower priority than user-defined rules in eacgh scope. +Server defined rules do not have a rule_id. A rule has a rule_id if and only if +it is a user-defined rule. Push Rules: Actions: -------------------- @@ -189,8 +190,7 @@ scope Either 'global' or 'device/' to specify global rules or device rules for the given profile_tag. kind - The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content' or - 'default'. + The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content'. rule_id The identifier for the rule. @@ -262,7 +262,6 @@ Requesting the root level returns a structure as follows:: "device": { "exampledevice": { "content": [], - "default": [], "override": [], "room": [ { @@ -278,50 +277,6 @@ Requesting the root level returns a structure as follows:: }, "global": { "content": [], - "default": [ - { - "actions": [ - "notify", - { - "set_sound": "default" - } - ], - "conditions": [ - { - "key": "content.body", - "kind": "event_match", - "pattern": "*@test:steve*" - } - ] - }, - { - "actions": [ - "notify", - { - "set_sound": "default" - } - ], - "conditions": [ - { - "kind": "contains_display_name" - } - ] - }, - { - "actions": [ - "notify", - { - "set_sound": "default" - } - ], - "conditions": [ - { - "is": "2", - "kind": "room_member_count" - } - ] - } - ], "override": [], "room": [], "sender": [], From 94af47ffd4eb051282779df1ab6f54f906ae0489 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 5 Feb 2015 13:03:10 +0000 Subject: [PATCH 45/62] Server default riules will have a rule_id when you need it to work out what the rule does. --- drafts/push_csapi.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index ae9591bf..39645ae2 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -101,8 +101,8 @@ devices. A home server may also have server default rules of each kind and in each scope. Server default rules are lower priority than user-defined rules in eacgh scope. -Server defined rules do not have a rule_id. A rule has a rule_id if and only if -it is a user-defined rule. +Server defined rules do not have a rule_id except when it is necessary to derive +the function of the rule (ie. in room and sender rules). Push Rules: Actions: -------------------- From 8360be247b45236410423563298957323265bac7 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 5 Feb 2015 13:15:34 +0000 Subject: [PATCH 46/62] Leave out what characters are allowed in a rule ID as we allow most thigs in matrix usernames currently. --- drafts/push_csapi.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 39645ae2..5f8515f1 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -94,10 +94,9 @@ be redundant. Actions for the highest priority rule and only that rule apply (for example, a set_tweak action in a lower priority rule will not apply if a higher priority rule matches, even if that rule does not specify any tweaks). -Rules also have an identifier, rule_id, which is a string. The rule_id may -alphanumeric characters only. The rule_id is unique within the kind of rule and -scope: rule_ids need not be unique between rules of the same kind on different -devices. +Rules also have an identifier, rule_id, which is a string. The rule_id is +unique within the kind of rule and scope: rule_ids need not be unique between +rules of the same kind on different devices. A home server may also have server default rules of each kind and in each scope. Server default rules are lower priority than user-defined rules in eacgh scope. From 7c0d1c86d8f38f19a6c114961e13f6a83472907b Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 5 Feb 2015 14:56:07 +0000 Subject: [PATCH 47/62] How to tell server default rules from other rules. --- drafts/push_csapi.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 5f8515f1..1cc0131e 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -101,7 +101,8 @@ rules of the same kind on different devices. A home server may also have server default rules of each kind and in each scope. Server default rules are lower priority than user-defined rules in eacgh scope. Server defined rules do not have a rule_id except when it is necessary to derive -the function of the rule (ie. in room and sender rules). +the function of the rule (ie. in room and sender rules). Server default rules +have an attribute, "default" set to true. Push Rules: Actions: -------------------- From e7c522b6068670173a519639b5fa96f831631323 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 5 Feb 2015 16:06:47 +0000 Subject: [PATCH 48/62] Add information on exclusive namespaces --- drafts/application_services.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/drafts/application_services.rst b/drafts/application_services.rst index c654c010..dfec19f7 100644 --- a/drafts/application_services.rst +++ b/drafts/application_services.rst @@ -86,7 +86,13 @@ In the registration process, the AS provides: "subscribing to" * Namespaces are defined as a list of regexps against which to match room aliases, room IDs, and user IDs. Regexps give the flexibility to say, sub-domain MSISDN - ranges per AS, whereas a blunt prefix string does not. + ranges per AS, whereas a blunt prefix string does not. These namespaces are further + configured by setting whether they are ``exclusive`` or not. An exclusive namespace + prevents entities other than the aforementioned AS from creating/editing/deleting + entries within that namespace. This does not affect the visibility/readability of + entries within that namespace (e.g. it doesn't prevent users joining exclusive + aliases, or ASes from listening to exclusive aliases, but does prevent both users + and ASes from creating/editing/deleting aliases within that namespace). * There is overlap between selecting events via the csv2 Filter API and subscribing to events here - perhaps subscription involves passing a filter token into the registration API. From 36395c26111ef7eb09b3cad2d6d2ce4a4bbc40cb Mon Sep 17 00:00:00 2001 From: Kegsay Date: Fri, 6 Feb 2015 11:01:20 +0000 Subject: [PATCH 49/62] Add info on M_EXCLUSIVE --- drafts/as-http-api.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 1c1f342f..0d040868 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -330,6 +330,11 @@ including the AS token on a ``/register`` request, along with a login type of type: "m.login.application_service", user: "" } + +Application services which attempt to create users or aliases *outside* of +their defined namespaces will receive an error code ``M_EXCLUSIVE``. Similarly, +normal users who attempt to create users or alises *inside* an application +service-defined namespace will receive the same ``M_EXCLUSIVE`` error code. ID conventions ``[Draft]`` ------------------------- From 6c5d07a0d14d5ad8a9084bf6b8aa1096cea648d7 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 8 Feb 2015 02:32:06 +0000 Subject: [PATCH 50/62] typoe --- drafts/push_pgwapi.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/push_pgwapi.rst b/drafts/push_pgwapi.rst index 8dd054f6..bb95acaf 100644 --- a/drafts/push_pgwapi.rst +++ b/drafts/push_pgwapi.rst @@ -127,6 +127,6 @@ gateway). However, Matrix strongly recommends: * That the APNS token be base64 encoded and used as the pushkey. * That a different app_id be used for apps on the production and sandbox APS environments. - * That PANS push gateways do not attempt to wait for errors from the APNS + * That APNS push gateways do not attempt to wait for errors from the APNS gateway before returning and instead to store failures and return 'rejected' responses next time that pushkey is used. From 2c8b875cf133b14604f6cb45b55f5eb76511e6ed Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 8 Feb 2015 02:33:53 +0000 Subject: [PATCH 51/62] typoe --- drafts/push_csapi.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 1cc0131e..c23ff0b1 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -133,7 +133,7 @@ their parameters, eg. { "set_tweak": "sound", "value": "default" } Push Rule Actions: Tweaks ------------------------- The 'set_tweak' key action is used to add an entry to the 'tweaks' dictionary -that is sent in the notification poke. The following tweaks are e defined: +that is sent in the notification poke. The following tweaks are defined: sound A sound to be played when this notification arrives. 'default' means to From 7a59d401f64fd6d1fd3c1c97b4c822b2bb2a0442 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 9 Feb 2015 09:17:55 +0000 Subject: [PATCH 52/62] Fix syntax errors --- drafts/as-http-api.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 0d040868..0dfb6ca7 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -337,7 +337,7 @@ normal users who attempt to create users or alises *inside* an application service-defined namespace will receive the same ``M_EXCLUSIVE`` error code. ID conventions ``[Draft]`` -------------------------- +-------------------------- .. NOTE:: - Giving HSes the freedom to namespace still feels like the Right Thing here. - Exposing a public API provides the consistency which was the main complaint @@ -418,6 +418,7 @@ Pre-conditions: - Home server has the domain "hsdomain.com" 1. Application service registration + :: AS -> HS: Registers itself with the home server @@ -444,12 +445,14 @@ Pre-conditions: } 2. IRC user "Bob" says "hello?" on "#matrix" at timestamp 1421416883133: + :: - AS stores message as potential scrollback. - Nothing happens as no Matrix users are in the room. 3. Matrix user "@alice:hsdomain.com" wants to join "#matrix": + :: User -> HS: Request to join "#irc.freenode.net/#matrix:hsdomain.com" @@ -484,6 +487,7 @@ Pre-conditions: - HS sends room information back to client. 4. @alice:hsdomain.com says "hi!" in this room: + :: User -> HS: Send message "hi!" in room !aasaasasa:hsdomain.com @@ -512,6 +516,7 @@ Pre-conditions: 5. IRC user "Bob" says "what's up?" on "#matrix" at timestamp 1421418084816: + :: IRC -> AS: "what's up?" From 56f4467cede56b3014318f914705e8b7f372bdbd Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 9 Feb 2015 09:19:45 +0000 Subject: [PATCH 53/62] Fix syntax error --- drafts/push_csapi.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index c23ff0b1..9c89170d 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -159,6 +159,7 @@ event_match profile_tag Matches the profile_tag of the device that the notification would be delivered to. Parameters: + * 'profile_tag': The profile_tag to match with. contains_display_name This matches unencrypted messages where content.body contains the owner's From 246531813bc7e3893596d20ba27aff5f27bd1516 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 12 Feb 2015 20:23:36 +0000 Subject: [PATCH 54/62] typo --- drafts/push_csapi.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 9c89170d..01bbe00a 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -99,7 +99,7 @@ unique within the kind of rule and scope: rule_ids need not be unique between rules of the same kind on different devices. A home server may also have server default rules of each kind and in each scope. -Server default rules are lower priority than user-defined rules in eacgh scope. +Server default rules are lower priority than user-defined rules in each scope. Server defined rules do not have a rule_id except when it is necessary to derive the function of the rule (ie. in room and sender rules). Server default rules have an attribute, "default" set to true. From 7c65e4748e201ec3ea19c04c5ec4b00a98fa290c Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 12 Feb 2015 20:24:14 +0000 Subject: [PATCH 55/62] typo --- drafts/media_repository.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/media_repository.rst b/drafts/media_repository.rst index 14a7c967..25ef9dd6 100644 --- a/drafts/media_repository.rst +++ b/drafts/media_repository.rst @@ -73,4 +73,4 @@ Clients may try to access a large number of remote files through a homeserver. Homeservers should restrict the number and size of remote files that it caches. Clients or remote homeservers may try to upload malicious files targeting -vunerabilities in either the homeserver thumbnailing or the client decoders. +vulnerabilities in either the homeserver thumbnailing or the client decoders. From b02f2aa3641064211d705b643caeed06e8afd302 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 12 Feb 2015 20:31:44 +0000 Subject: [PATCH 56/62] never upscale images --- drafts/media_repository.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drafts/media_repository.rst b/drafts/media_repository.rst index 25ef9dd6..879e1b85 100644 --- a/drafts/media_repository.rst +++ b/drafts/media_repository.rst @@ -52,7 +52,8 @@ within a given rectangle. Homeservers may generate thumbnails for content uploaded to remote homeservers themselves or may rely on the remote homeserver to thumbnail the content. Homeservers may return thumbnails of a different size to that -requested. However homeservers should provide extact matches where reasonable. +requested. However homeservers should provide exact matches where reasonable. +Homeservers must never upscale images. Security -------- From 7a0f7b4b43c5eb34e9fb7e62a270edd2a8e08c90 Mon Sep 17 00:00:00 2001 From: David Baker Date: Fri, 13 Feb 2015 13:40:44 +0000 Subject: [PATCH 57/62] Add that rules with no conditions always match. --- drafts/push_csapi.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drafts/push_csapi.rst b/drafts/push_csapi.rst index 01bbe00a..9cef9004 100644 --- a/drafts/push_csapi.rst +++ b/drafts/push_csapi.rst @@ -147,8 +147,9 @@ Push Rules: Conditions: ----------------------- Override, Underride and Default rules have a list of 'conditions'. All conditions must hold true for an event in order for a rule to be applied to an -event. Matrix specifies the following conditions, although if Home Servers wish -to support others, they are free to do so: +event. A rule with no conditions always matches. Matrix specifies the following +conditions, although if Home Servers wish to support others, they are free to +do so: event_match This is a glob pattern match on a field of the event. Parameters: From bcba42651a2782998bc731f89f9761577d8a5820 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Mon, 16 Feb 2015 13:10:25 +0000 Subject: [PATCH 58/62] Add support for exclusive namespaces As per https://github.com/matrix-org/matrix-doc/pull/5#issuecomment-73071259 --- drafts/as-http-api.rst | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 0dfb6ca7..d7b076c3 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -27,11 +27,22 @@ Side effects: API called when: - The application service wants to register with a brand new home server. Notes: + - An application service can state whether they should be the only ones who + can manage a specified namespace. This is referred to as an "exclusive" + namespace. An exclusive namespace prevents humans and other application + services from creating/deleting entities in that namespace. Typically, + exclusive namespaces are used when the rooms represent real rooms on + another service (e.g. IRC). Non-exclusive namespaces are used when the + application service is merely augmenting the room itself (e.g. providing + logging or searching facilities). - Namespaces are represented by POSIX extended regular expressions in JSON. They look like:: users: [ - "@irc\.freenode\.net/.*", + { + "exclusive": true, + "regex": "@irc\.freenode\.net/.*" + } ] :: @@ -44,13 +55,22 @@ Notes: as_token: "some_AS_token", namespaces: { users: [ - "@irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "@irc\.freenode\.net/.*" + } ], aliases: [ - "#irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "#irc\.freenode\.net/.*" + } ], rooms: [ - "!irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "!irc\.freenode\.net/.*" + } ] } } @@ -428,13 +448,16 @@ Pre-conditions: as_token: "T_a", namespaces: { users: [ - "@irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "@irc\.freenode\.net/.*" + } ], aliases: [ - "#irc\.freenode\.net/.*" - ], - rooms: [ - "!irc\.freenode\.net/.*" + { + "exclusive": true, + "regex": "#irc\.freenode\.net/.*" + } ] } } From f3a498d0bb49a68fcffaadb65afd6ccc2a06e6a3 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 16 Feb 2015 22:09:10 +0000 Subject: [PATCH 59/62] document the concept of a standard `external_url` field for gatewayed events --- drafts/as-http-api.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index d7b076c3..488bfe30 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -425,6 +425,15 @@ SHOULD be mapped in the same way as "user" URIs. alias: } +Event fields +~~~~~~~~~~~~ +We recommend that any gatewayed events should include an `external_url` field in +their content to provide a way for Matrix clients to link into the 'native' +client from which the event originated. For instance, this could contain the +message-ID for emails/nntp posts, or a link to a blog comment when gatewaying +blog comment traffic in & out of matrix + + Examples -------- .. NOTE:: From f0cc0e082c39d3054ba3d381c8022be25ff41879 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 19 Feb 2015 10:13:21 +0000 Subject: [PATCH 60/62] Add room_id in push poke to spec. --- drafts/push_pgwapi.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drafts/push_pgwapi.rst b/drafts/push_pgwapi.rst index bb95acaf..92035581 100644 --- a/drafts/push_pgwapi.rst +++ b/drafts/push_pgwapi.rst @@ -14,6 +14,7 @@ is as follows:: { "notification": { "id": "$3957tyerfgewrf384", + "room_id": "!slw48wfj34rtnrf:example.com", "type": "m.room.message", "sender": "@exampleuser:matrix.org", "sender_display_name": "Major Tom", @@ -50,6 +51,8 @@ id An identifier for this notification that may be used to detect duplicate notification requests. This is not necessarily the ID of the event that triggered the notification. +room_id + The ID of the room in which this event occurred. type The type of the event as in the event's 'type' field. sender From 3ce1e559545b1ecd0e52e63b89be609142f19dbc Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 19 Feb 2015 11:44:10 +0000 Subject: [PATCH 61/62] Move TODO comments into the doc in prep for merging --- drafts/as-http-api.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drafts/as-http-api.rst b/drafts/as-http-api.rst index 488bfe30..3cc8efbb 100644 --- a/drafts/as-http-api.rst +++ b/drafts/as-http-api.rst @@ -1,3 +1,10 @@ +.. TODO + Sometimes application services need to create rooms (e.g. when lazy loading + from room aliases). Created rooms need to have a user that created them, so + federation works (as it relies on an entry existing in m.room.member). We + should be able to add metadata to m.room.member to state that this user is an + application service, a virtual user, etc. + Application Services HTTP API ============================= From 8aae3e58428a6d9cd15dd8238109ef7e3e50fa8e Mon Sep 17 00:00:00 2001 From: Kegsay Date: Thu, 19 Feb 2015 11:48:45 +0000 Subject: [PATCH 62/62] Add TODO section --- drafts/application_services.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drafts/application_services.rst b/drafts/application_services.rst index dfec19f7..9c8b6b69 100644 --- a/drafts/application_services.rst +++ b/drafts/application_services.rst @@ -1,3 +1,10 @@ +.. TODO + Sometimes application services need to create rooms (e.g. when lazy loading + from room aliases). Created rooms need to have a user that created them, so + federation works (as it relies on an entry existing in m.room.member). We should + be able to add metadata to m.room.member to state that this user is an application + service, a virtual user, etc. + Application Services ====================