From f4c5c209f346821c769a17b612cdda6d416db681 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 31 Aug 2018 12:36:08 -0600 Subject: [PATCH 1/4] Generalize language for pagination Previously the section was very strict in what pagination was, however this isn't the reality for the matrix specification. Several endpoints have their own pagination naming conventions and do not follow those mandated by this section. This commit generalizes the language to cover those endpoints while also describing how pagination works. In particular, it describes the rough API shape to expect and how to deal with the responses. This commit also removes the `M_BAD_PAGINATION` error as it is not used in the real world. Homeservers are instead encouraged to use the standard `M_INVALID_PARAM` or similar error code. Fixes https://github.com/matrix-org/matrix-doc/issues/610 Fixes https://github.com/matrix-org/matrix-doc/issues/1523 --- specification/client_server_api.rst | 80 ++++++++++------------------- 1 file changed, 28 insertions(+), 52 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 2f17e4c36..a329aa0cb 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -138,9 +138,6 @@ Some requests have unique error codes: :``M_INVALID_ROOM_STATE``: Sent when the intial state given to the ``createRoom`` API is invalid. -:``M_BAD_PAGINATION``: - Encountered when specifying bad pagination query parameters. - :``M_THREEPID_IN_USE``: Sent when a threepid given to an API cannot be used because the same threepid is already in use. @@ -1018,39 +1015,21 @@ Pagination is the process of dividing a dataset into multiple discrete pages. Matrix makes use of pagination to allow clients to view extremely large datasets. These datasets are not limited to events in a room (for example clients may want to paginate a list of rooms in addition to events within those rooms). Regardless -of *what* is being paginated, there is a common underlying API which is used to -to give clients a consistent way of selecting subsets of a potentially changing -dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters -which describe where to read from the stream. ``from`` and ``to`` are opaque -textual 'stream tokens' which describe the current position in the dataset. -The ``dir`` parameter is an enum representing the direction of events to return: -either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and -``end`` stream token values which can then be passed to subsequent requests to -continue pagination. Not all endpoints will make use of all the parameters -outlined here: see the specific endpoint in question for more information. - -Pagination Request Query Parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Query parameters: - from: - $streamtoken - The opaque token to start streaming from. - to: - $streamtoken - The opaque token to end streaming at. Typically, - clients will not know the item of data to end at, so this will usually be - omitted. - limit: - integer - An integer representing the maximum number of items to - return. - dir: - f|b - The direction to return events in. Typically this is ``b`` to paginate - backwards in time. - -'START' and 'END' are placeholder values used in these examples to describe the -start and end of the dataset respectively. - -Unless specified, the default pagination parameters are ``from=START``, -``to=END``, without a limit set. +of what is being paginated, there is a common approach which is used to give +clients an easy way of selecting subsets of a potentially changing dataset. Each +endpoint that uses pagination may use different parameters, however the theme +amoung them is that they take a ``from`` and ``to`` token, and occasionally +a ``limit`` and ``dir`` to describe the direction to look in. Together, these +parameters describe the position in a data set, where ``from`` and ``to`` are +known as "stream tokens" matching the regular expression ``[a-zA-Z0-9.=_-]+``. +If supported, the ``dir`` defines the direction of events to return: either +forwards (``f``) or backwards (``b``). The response contains a ``start`` or +``prev_batch`` token which references the result set immediately prior to the +returned set. The response might additionally have an ``end`` or ``next_batch`` +token to indicate the results after the returned set. + +In the following examples, 'START' and 'END' are placeholders to signify the +start and end of the data sets respectively. For example, if an endpoint had events E1 -> E15. The client wants the last 5 events and doesn't know any previous events:: @@ -1067,8 +1046,8 @@ events and doesn't know any previous events:: Another example: a public room list has rooms R1 -> R17. The client is showing 5 -rooms at a time on screen, and is on page 2. They want to -now show page 3 (rooms R11 -> 15):: +rooms at a time on screen, and is on page 2. They want to now show page 3 (rooms +R11 -> 15):: S E | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | stream token @@ -1086,20 +1065,17 @@ token from the initial request was '9' which corresponded to R10. When the 2nd request was made, R10 did not appear again, even though from=9 was specified. If you know the token, you already have the data. -Pagination Response -~~~~~~~~~~~~~~~~~~~ - -Responses to pagination requests MUST follow the format:: - - { - "chunk": [ ... , Responses , ... ], - "start" : $streamtoken, - "end" : $streamtoken - } - -Where $streamtoken is an opaque token which can be used in another query to -get the next set of results. The "start" and "end" keys can only be omitted if -the complete dataset is provided in "chunk". +Responses for pagination-capable endpoints SHOULD have a ``chunk`` array alongside +the applicable stream tokens to represent the result set. + +In general, when the end of a result set is reached the applicable stream token +will be excluded from the response. For example, if a user was backwards-paginating +events in a room they'd eventually reach the first event in the room. In this scenario, +the ``prev_batch`` token would be excluded from the response. Some paginated +endpoints are open-ended in one direction, such as endpoints which expose an event +stream for an active room. In this case, it is not possible for the client to reach +the true "end" of the data set and therefore should always be presented with a token +to keep moving forwards. .. _`filter`: From e49ed5d1ecce075a46fb137a7c8219eb96167bc5 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 31 Aug 2018 12:37:24 -0600 Subject: [PATCH 2/4] Changelog --- changelogs/client_server/newsfragments/1642.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1642.clarification diff --git a/changelogs/client_server/newsfragments/1642.clarification b/changelogs/client_server/newsfragments/1642.clarification new file mode 100644 index 000000000..0ccb0f3c6 --- /dev/null +++ b/changelogs/client_server/newsfragments/1642.clarification @@ -0,0 +1 @@ +Clarify and generalise the language used for describing pagination. From f299fe023ac1784e37b609b640e60f5a602dca24 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 31 Aug 2018 12:51:31 -0600 Subject: [PATCH 3/4] English --- specification/client_server_api.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index a329aa0cb..cf5bc9ea1 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1017,16 +1017,16 @@ These datasets are not limited to events in a room (for example clients may want to paginate a list of rooms in addition to events within those rooms). Regardless of what is being paginated, there is a common approach which is used to give clients an easy way of selecting subsets of a potentially changing dataset. Each -endpoint that uses pagination may use different parameters, however the theme -amoung them is that they take a ``from`` and ``to`` token, and occasionally -a ``limit`` and ``dir`` to describe the direction to look in. Together, these -parameters describe the position in a data set, where ``from`` and ``to`` are -known as "stream tokens" matching the regular expression ``[a-zA-Z0-9.=_-]+``. -If supported, the ``dir`` defines the direction of events to return: either -forwards (``f``) or backwards (``b``). The response contains a ``start`` or -``prev_batch`` token which references the result set immediately prior to the -returned set. The response might additionally have an ``end`` or ``next_batch`` -token to indicate the results after the returned set. +endpoint that uses pagination may use different parameters. However the theme +among them is that they take a ``from`` and ``to`` token, and occasionally +a ``limit`` and ``dir``. Together, these parameters describe the position in a +data set, where ``from`` and ``to`` are known as "stream tokens" matching the +regular expression ``[a-zA-Z0-9.=_-]+``. If supported, the ``dir`` defines the +direction of events to return: either forwards (``f``) or backwards (``b``). +The response contains a ``start`` or ``prev_batch`` token which references the +result set immediately prior to the returned set. The response might additionally +have an ``end`` or ``next_batch`` token to indicate the results after the returned +set. In the following examples, 'START' and 'END' are placeholders to signify the start and end of the data sets respectively. From 835f5de387f5e09861d9065d7283815d9d099152 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 31 Aug 2018 14:22:07 -0600 Subject: [PATCH 4/4] Generalize the token name even more --- specification/client_server_api.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index cf5bc9ea1..549474af8 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1023,10 +1023,10 @@ a ``limit`` and ``dir``. Together, these parameters describe the position in a data set, where ``from`` and ``to`` are known as "stream tokens" matching the regular expression ``[a-zA-Z0-9.=_-]+``. If supported, the ``dir`` defines the direction of events to return: either forwards (``f``) or backwards (``b``). -The response contains a ``start`` or ``prev_batch`` token which references the -result set immediately prior to the returned set. The response might additionally -have an ``end`` or ``next_batch`` token to indicate the results after the returned -set. +The response may contain tokens that can be used for retrieving results before +or after the returned set. These tokens may be called `start` or `prev_batch` +for retrieving the previous result set, or `end`, `next_batch` or `next_token` +for retrieving the next result set. In the following examples, 'START' and 'END' are placeholders to signify the start and end of the data sets respectively.