You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
447 lines
23 KiB
ReStructuredText
447 lines
23 KiB
ReStructuredText
.. Copyright 2016 OpenMarket Ltd
|
|
.. Copyright 2018 New Vector Ltd
|
|
..
|
|
.. Licensed under the Apache License, Version 2.0 (the "License");
|
|
.. you may not use this file except in compliance with the License.
|
|
.. You may obtain a copy of the License at
|
|
..
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
..
|
|
.. Unless required by applicable law or agreed to in writing, software
|
|
.. distributed under the License is distributed on an "AS IS" BASIS,
|
|
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
.. See the License for the specific language governing permissions and
|
|
.. limitations under the License.
|
|
|
|
Application Service API
|
|
=======================
|
|
|
|
{{unstable_warning_block_APPSERVICE_RELEASE_LABEL}}
|
|
|
|
The Matrix client-server API and server-server APIs provide the means to
|
|
implement a consistent self-contained federated messaging fabric. However, they
|
|
provide limited means of implementing custom server-side behaviour in Matrix
|
|
(e.g. gateways, filters, extensible hooks etc). The Application Service API (AS API)
|
|
defines a standard API to allow such extensible functionality to be implemented
|
|
irrespective of the underlying homeserver implementation.
|
|
|
|
.. TODO-spec
|
|
Add in Client-Server services? Overview of bots? Seems weird to be in the spec
|
|
given it is VERY implementation specific.
|
|
|
|
.. contents:: Table of Contents
|
|
.. sectnum::
|
|
|
|
Changelog
|
|
---------
|
|
|
|
|
|
.. topic:: Version: unstable
|
|
{{application_service_changelog}}
|
|
|
|
This version of the specification is generated from
|
|
`matrix-doc <https://github.com/matrix-org/matrix-doc>`_ as of Git commit
|
|
`{{git_version}} <https://github.com/matrix-org/matrix-doc/tree/{{git_rev}}>`_.
|
|
|
|
For the full historical changelog, see
|
|
https://github.com/matrix-org/matrix-doc/blob/master/changelogs/application_service.rst
|
|
|
|
Other versions of this specification
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The following other versions are also available, in reverse chronological order:
|
|
|
|
- `HEAD <https://matrix.org/docs/spec/application_service/unstable.html>`_: Includes all changes since the latest versioned release.
|
|
- `r0.1.0 <https://matrix.org/docs/spec/application_service/r0.1.0.html>`_
|
|
|
|
|
|
Application Services
|
|
--------------------
|
|
Application services are passive and can only observe events from homeserver.
|
|
They can inject events into rooms they are participating in.
|
|
They cannot prevent events from being sent, nor can they modify the content of
|
|
the event being sent. In order to observe events from a homeserver, the
|
|
homeserver needs to be configured to pass certain types of traffic to the
|
|
application service. This is achieved by manually configuring the homeserver
|
|
with information about the application service.
|
|
|
|
Registration
|
|
~~~~~~~~~~~~
|
|
|
|
.. NOTE::
|
|
Previously, application services could register with a homeserver via HTTP
|
|
APIs. This was removed as it was seen as a security risk. A compromised
|
|
application service could re-register for a global ``*`` regex and sniff
|
|
*all* traffic on the homeserver. To protect against this, application
|
|
services now have to register via configuration files which are linked to
|
|
the homeserver configuration file. The addition of configuration files
|
|
allows homeserver admins to sanity check the registration for suspicious
|
|
regex strings.
|
|
|
|
.. TODO
|
|
Removing the API entirely is probably a mistake - having a standard cross-HS
|
|
way of doing this stops ASes being coupled to particular HS implementations.
|
|
A better solution would be to somehow mandate that the API done to avoid
|
|
abuse.
|
|
|
|
Application services register "namespaces" of user IDs, room aliases and room IDs.
|
|
These namespaces are represented as regular expressions. An application service
|
|
is said to be "interested" in a given event if one of the IDs in the event match
|
|
the regular expression provided by the application service, such as the room having
|
|
an alias or ID in the relevant namespaces. Similarly, the application service is
|
|
said to be interested in a given event if one of the application service's namespaced
|
|
users is the target of the event, or is a joined member of the room where the event
|
|
occurred.
|
|
|
|
An application service can also state whether they should be the only ones who
|
|
can manage a specified namespace. This is referred to as an "exclusive"
|
|
namespace. An exclusive namespace prevents humans and other application
|
|
services from creating/deleting entities in that namespace. Typically,
|
|
exclusive namespaces are used when the rooms represent real rooms on
|
|
another service (e.g. IRC). Non-exclusive namespaces are used when the
|
|
application service is merely augmenting the room itself (e.g. providing
|
|
logging or searching facilities). Namespaces are represented by POSIX extended
|
|
regular expressions and look like:
|
|
|
|
.. code-block:: yaml
|
|
|
|
users:
|
|
- exclusive: true
|
|
regex: "@_irc_bridge_.*"
|
|
|
|
Application services may define the following namespaces (with none being explicitly required):
|
|
|
|
+------------------+-----------------------------------------------------------+
|
|
| Name | Description |
|
|
+==================+===========================================================+
|
|
| users | Events which are sent from certain users. |
|
|
+------------------+-----------------------------------------------------------+
|
|
| aliases | Events which are sent in rooms with certain room aliases. |
|
|
+------------------+-----------------------------------------------------------+
|
|
| rooms | Events which are sent in rooms with certain room IDs. |
|
|
+------------------+-----------------------------------------------------------+
|
|
|
|
Each individual namespace MUST declare the following fields:
|
|
|
|
+------------------+-----------------------------------------------------------------------------------------------------------------------------------+
|
|
| Name | Description |
|
|
+==================+===================================================================================================================================+
|
|
| exclusive | **Required** A true or false value stating whether this application service has exclusive access to events within this namespace. |
|
|
+------------------+-----------------------------------------------------------------------------------------------------------------------------------+
|
|
| regex | **Required** A regular expression defining which values this namespace includes. |
|
|
+------------------+-----------------------------------------------------------------------------------------------------------------------------------+
|
|
|
|
Exclusive user and alias namespaces should begin with an underscore after the
|
|
sigil to avoid collisions with other users on the homeserver. Application
|
|
services should additionally attempt to identify the service they represent
|
|
in the reserved namespace. For example, ``@_irc_.*`` would be a good namespace
|
|
to register for an application service which deals with IRC.
|
|
|
|
The registration is represented by a series of key-value pairs, which this
|
|
specification will present as YAML. See below for the possible options along
|
|
with their explanation:
|
|
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
| Name | Description |
|
|
+==================+====================================================================================================================================================+
|
|
| id | **Required.** A unique, user-defined ID of the application service which will never change. |
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
| url | **Required.** The URL for the application service. May include a path after the domain name. Optionally set to ``null`` if no traffic is required. |
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
| as_token | **Required.** A unique token for application services to use to authenticate requests to Homeservers. |
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
| hs_token | **Required.** A unique token for Homeservers to use to authenticate requests to application services. |
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
| sender_localpart | **Required.** The localpart of the user associated with the application service. |
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
| namespaces | **Required.** A list of ``users``, ``aliases`` and ``rooms`` namespaces that the application service controls. |
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
| rate_limited | Whether requests from masqueraded users are rate-limited. The sender is excluded. |
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
| protocols | The external protocols which the application service provides (e.g. IRC). |
|
|
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|
|
|
|
An example registration file for an IRC-bridging application service is below:
|
|
|
|
.. code-block:: yaml
|
|
|
|
id: "IRC Bridge"
|
|
url: "http://127.0.0.1:1234"
|
|
as_token: "30c05ae90a248a4188e620216fa72e349803310ec83e2a77b34fe90be6081f46"
|
|
hs_token: "312df522183efd404ec1cd22d2ffa4bbc76a8c1ccf541dd692eef281356bb74e"
|
|
sender_localpart: "_irc_bot" # Will result in @_irc_bot:example.org
|
|
namespaces:
|
|
users:
|
|
- exclusive: true
|
|
regex: "@_irc_bridge_.*"
|
|
aliases:
|
|
- exclusive: false
|
|
regex: "#_irc_bridge_.*"
|
|
rooms: []
|
|
|
|
.. WARNING::
|
|
If the homeserver in question has multiple application services, each
|
|
``as_token`` and ``id`` MUST be unique per application service as these are
|
|
used to identify the application service. The homeserver MUST enforce this.
|
|
|
|
Homeserver -> Application Service API
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Authorization
|
|
+++++++++++++
|
|
|
|
Homeservers MUST include a query parameter named ``access_token`` containing the
|
|
``hs_token`` from the application service's registration when making requests to
|
|
the application service. Application services MUST verify the provided ``access_token``
|
|
matches their known ``hs_token``, failing the request with a ``M_FORBIDDEN`` error
|
|
if it does not match.
|
|
|
|
Legacy routes
|
|
+++++++++++++
|
|
|
|
Previous drafts of the application service specification had a mix of endpoints
|
|
that have been used in the wild for a significant amount of time. The application
|
|
service specification now defines a version on all endpoints to be more compatible
|
|
with the rest of the Matrix specification and the future.
|
|
|
|
Homeservers should attempt to use the specified endpoints first when communicating
|
|
with application services. However, if the application service receives an http status
|
|
code that does not indicate success (ie: 404, 500, 501, etc) then the homeserver
|
|
should fall back to the older endpoints for the application service.
|
|
|
|
The older endpoints have the exact same request body and response format, they
|
|
just belong at a different path. The equivalent path for each is as follows:
|
|
|
|
* ``/_matrix/app/v1/transactions/{txnId}`` becomes ``/transactions/{txnId}``
|
|
* ``/_matrix/app/v1/users/{userId}`` becomes ``/users/{userId}``
|
|
* ``/_matrix/app/v1/rooms/{roomAlias}`` becomes ``/rooms/{roomAlias}``
|
|
* ``/_matrix/app/v1/thirdparty/protocol/{protocol}`` becomes ``/_matrix/app/unstable/thirdparty/protocol/{protocol}``
|
|
* ``/_matrix/app/v1/thirdparty/user/{user}`` becomes ``/_matrix/app/unstable/thirdparty/user/{user}``
|
|
* ``/_matrix/app/v1/thirdparty/location/{location}`` becomes ``/_matrix/app/unstable/thirdparty/location/{location}``
|
|
* ``/_matrix/app/v1/thirdparty/user`` becomes ``/_matrix/app/unstable/thirdparty/user``
|
|
* ``/_matrix/app/v1/thirdparty/location`` becomes ``/_matrix/app/unstable/thirdparty/location``
|
|
|
|
Homeservers should periodically try again for the newer endpoints because the
|
|
application service may have been updated.
|
|
|
|
Pushing events
|
|
++++++++++++++
|
|
|
|
The application service API provides a transaction API for sending a list of
|
|
events. Each list of events includes a transaction ID, which works as follows:
|
|
|
|
::
|
|
|
|
Typical
|
|
HS ---> AS : Homeserver sends events with transaction ID T.
|
|
<--- : Application Service sends back 200 OK.
|
|
|
|
AS ACK Lost
|
|
HS ---> AS : Homeserver sends events with transaction ID T.
|
|
<-/- : AS 200 OK is lost.
|
|
HS ---> AS : Homeserver retries with the same transaction ID of T.
|
|
<--- : Application Service sends back 200 OK. If the AS had processed these
|
|
events already, it can NO-OP this request (and it knows if it is the
|
|
same events based on the transaction ID).
|
|
|
|
The events sent to the application service should be linearised, as if they were
|
|
from the event stream. The homeserver MUST maintain a queue of transactions to
|
|
send to the application service. If the application service cannot be reached, the
|
|
homeserver SHOULD backoff exponentially until the application service is reachable again.
|
|
As application services cannot *modify* the events in any way, these requests can
|
|
be made without blocking other aspects of the homeserver. Homeservers MUST NOT
|
|
alter (e.g. add more) events they were going to send within that transaction ID
|
|
on retries, as the application service may have already processed the events.
|
|
|
|
{{transactions_as_http_api}}
|
|
|
|
Querying
|
|
++++++++
|
|
|
|
The application service API includes two querying APIs: for room aliases and for
|
|
user IDs. The application service SHOULD create the queried entity if it desires.
|
|
During this process, the application service is blocking the homeserver until the
|
|
entity is created and configured. If the homeserver does not receive a response
|
|
to this request, the homeserver should retry several times before timing out. This
|
|
should result in an HTTP status 408 "Request Timeout" on the client which initiated
|
|
this request (e.g. to join a room alias).
|
|
|
|
.. admonition:: Rationale
|
|
|
|
Blocking the homeserver and expecting the application service to create the entity
|
|
using the client-server API is simpler and more flexible than alternative methods
|
|
such as returning an initial sync style JSON blob and get the HS to provision
|
|
the room/user. This also meant that there didn't need to be a "backchannel" to inform
|
|
the application service about information about the entity such as room ID to
|
|
room alias mappings.
|
|
|
|
{{query_user_as_http_api}}
|
|
|
|
{{query_room_as_http_api}}
|
|
|
|
|
|
Third party networks
|
|
++++++++++++++++++++
|
|
|
|
Application services may declare which protocols they support via their registration
|
|
configuration for the homeserver. These networks are generally for third party services
|
|
such as IRC that the application service is managing. Application services may populate
|
|
a Matrix room directory for their registered protocols, as defined in the Client-Server
|
|
API Extensions.
|
|
|
|
Each protocol may have several "locations" (also known as "third party locations" or "3PLs").
|
|
A location within a protocol is a place in the third party network, such as an IRC channel.
|
|
Users of the third party network may also be represented by the application service.
|
|
|
|
Locations and users can be searched by fields defined by the application service, such
|
|
as by display name or other attribute. When clients request the homeserver to search
|
|
in a particular "network" (protocol), the search fields will be passed along to the
|
|
application service for filtering.
|
|
|
|
{{protocols_as_http_api}}
|
|
|
|
|
|
.. _create the user: `sect:asapi-permissions`_
|
|
|
|
Client-Server API Extensions
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Application services can use a more powerful version of the
|
|
client-server API by identifying itself as an application service to the
|
|
homeserver.
|
|
|
|
Endpoints defined in this section MUST be supported by homeservers in the
|
|
client-server API as accessible only by application services.
|
|
|
|
Identity assertion
|
|
++++++++++++++++++
|
|
The client-server API infers the user ID from the ``access_token`` provided in
|
|
every request. To avoid the application service from having to keep track of each
|
|
user's access token, the application service should identify itself to the Client-Server
|
|
API by providing its ``as_token`` for the ``access_token`` alongside the user the
|
|
application service would like to masquerade as.
|
|
|
|
Inputs:
|
|
- Application service token (``as_token``)
|
|
- User ID in the AS namespace to act as.
|
|
|
|
Notes:
|
|
- This applies to all aspects of the Client-Server API, except for Account Management.
|
|
- The ``as_token`` is inserted into ``access_token`` which is usually where the
|
|
client token is, such as via the query string or ``Authorization`` header. This
|
|
is done on purpose to allow application services to reuse client SDKs.
|
|
- The ``access_token`` should be supplied through the ``Authorization`` header where
|
|
possible to prevent the token appearing in HTTP request logs by accident.
|
|
|
|
The application service may specify the virtual user to act as through use of a
|
|
``user_id`` query string parameter on the request. The user specified in the query
|
|
string must be covered by one of the application service's ``user`` namespaces. If
|
|
the parameter is missing, the homeserver is to assume the application service intends
|
|
to act as the user implied by the ``sender_localpart`` property of the registration.
|
|
|
|
An example request would be::
|
|
|
|
GET /_matrix/client/%CLIENT_MAJOR_VERSION%/account/whoami?user_id=@_irc_user:example.org
|
|
Authorization: Bearer YourApplicationServiceTokenHere
|
|
|
|
.. TODO-TravisR: Temporarily take out timestamp massaging while we're releasing r0.
|
|
See https://github.com/matrix-org/matrix-doc/issues/1585
|
|
.. Timestamp massaging
|
|
+++++++++++++++++++
|
|
The application service may want to inject events at a certain time (reflecting
|
|
the time on the network they are tracking e.g. irc, xmpp). Application services
|
|
need to be able to adjust the ``origin_server_ts`` value to do this.
|
|
|
|
Inputs:
|
|
- Application service token (``as_token``)
|
|
- Desired timestamp (in milliseconds since the unix epoch)
|
|
|
|
Notes:
|
|
- This will only apply when sending events.
|
|
|
|
::
|
|
|
|
PUT /_matrix/client/r0/rooms/!somewhere:example.org/send/m.room.message/txnId?ts=1534535223283
|
|
Authorization: Bearer YourApplicationServiceTokenHere
|
|
|
|
Content: The event to send, as per the Client-Server API.
|
|
|
|
Timestamp massaging
|
|
+++++++++++++++++++
|
|
|
|
Previous drafts of the Application Service API permitted application services
|
|
to alter the timestamp of their sent events by providing a ``ts`` query parameter
|
|
when sending an event. This API has been excluded from the first release due to
|
|
design concerns, however some servers may still support the feature. Please visit
|
|
`issue #1585 <https://github.com/matrix-org/matrix-doc/issues/1585>`_ for more
|
|
information.
|
|
|
|
Server admin style permissions
|
|
++++++++++++++++++++++++++++++
|
|
|
|
.. _sect:asapi-permissions:
|
|
|
|
The homeserver needs to give the application service *full control* over its
|
|
namespace, both for users and for room aliases. This means that the AS should
|
|
be able to create/edit/delete any room alias in its namespace, as well as
|
|
create/delete any user in its namespace. No additional API changes need to be
|
|
made in order for control of room aliases to be granted to the AS. Creation of
|
|
users needs API changes in order to:
|
|
|
|
- Work around captchas.
|
|
- Have a 'passwordless' user.
|
|
|
|
This involves bypassing the registration flows entirely. This is achieved by
|
|
including the ``as_token`` on a ``/register`` request, along with a login type of
|
|
``m.login.application_service`` to set the desired user ID without a password.
|
|
|
|
::
|
|
|
|
POST /_matrix/client/%CLIENT_MAJOR_VERSION%/register
|
|
Authorization: Bearer YourApplicationServiceTokenHere
|
|
|
|
Content:
|
|
{
|
|
type: "m.login.application_service",
|
|
username: "_irc_example"
|
|
}
|
|
|
|
Application services which attempt to create users or aliases *outside* of
|
|
their defined namespaces will receive an error code ``M_EXCLUSIVE``. Similarly,
|
|
normal users who attempt to create users or aliases *inside* an application
|
|
service-defined namespace will receive the same ``M_EXCLUSIVE`` error code,
|
|
but only if the application service has defined the namespace as ``exclusive``.
|
|
|
|
Using ``/sync`` and ``/events``
|
|
+++++++++++++++++++++++++++++++
|
|
|
|
Application services wishing to use ``/sync`` or ``/events`` from the Client-Server
|
|
API MUST do so with a virtual user (provide a ``user_id`` via the query string). It
|
|
is expected that the application service use the transactions pushed to it to
|
|
handle events rather than syncing with the user implied by ``sender_localpart``.
|
|
|
|
Application service room directories
|
|
++++++++++++++++++++++++++++++++++++
|
|
|
|
Application services can maintain their own room directories for their defined
|
|
third party protocols. These room directories may be accessed by clients through
|
|
additional parameters on the ``/publicRooms`` client-server endpoint.
|
|
|
|
{{appservice_room_directory_cs_http_api}}
|
|
|
|
Referencing messages from a third party network
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Application services should include an ``external_url`` in the ``content`` of
|
|
events it emits to indicate where the message came from. This typically applies
|
|
to application services that bridge other networks into Matrix, such as IRC,
|
|
where an HTTP URL may be available to reference.
|
|
|
|
Clients should provide users with a way to access the ``external_url`` if it
|
|
is present. Clients should additionally ensure the URL has a scheme of ``https``
|
|
or ``http`` before making use of it.
|
|
|
|
The presence of an ``external_url`` on an event does not necessarily mean the
|
|
event was sent from an application service. Clients should be wary of the URL
|
|
contained within, as it may not be a legitimate reference to the event's source.
|