From 5e3ea11050296877c53c774e588c858e1c8dd7fc Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 27 Aug 2021 17:30:30 +0100 Subject: [PATCH] Replace "Pagination" section in C-S API with text in appendices (#3366) The Pagination section in the C-S API was, basically, full of rubbish. I think that anything of any value it contained was repeated either directly on the API definitions or in the text specific to syncing at https://spec.matrix.org/unstable/client-server-api/#syncing. The conventions I've added to the Appendices are based on the discussions in #1898. They are there because I don't want to have to go through it all again next time we add a paginated API. Fixes: #1898 Fixes: #2268 --- .../{ => newsfragments}/3353.clarification | 0 .../newsfragments/3366.clarification | 1 + content/appendices.md | 30 ++++++- content/client-server-api/_index.md | 82 ------------------- 4 files changed, 29 insertions(+), 84 deletions(-) rename changelogs/client_server/{ => newsfragments}/3353.clarification (100%) create mode 100644 changelogs/client_server/newsfragments/3366.clarification diff --git a/changelogs/client_server/3353.clarification b/changelogs/client_server/newsfragments/3353.clarification similarity index 100% rename from changelogs/client_server/3353.clarification rename to changelogs/client_server/newsfragments/3353.clarification diff --git a/changelogs/client_server/newsfragments/3366.clarification b/changelogs/client_server/newsfragments/3366.clarification new file mode 100644 index 00000000..459c5075 --- /dev/null +++ b/changelogs/client_server/newsfragments/3366.clarification @@ -0,0 +1 @@ +Remove the inaccurate 'Pagination' section. diff --git a/content/appendices.md b/content/appendices.md index 25b88945..31c5fbb8 100644 --- a/content/appendices.md +++ b/content/appendices.md @@ -1064,8 +1064,8 @@ The event signing algorithm should emit the following signed event: ## Conventions for Matrix APIs This section is intended primarily to guide API designers when adding to Matrix, -setting guidelines to follow for how those APIs should work. This is important to -maintain consistency with the Matrix protocol, and thus improve developer +setting guidelines to follow for how those APIs should work. This is important to +maintain consistency with the Matrix protocol, and thus improve developer experience. ### HTTP endpoint and JSON property naming @@ -1079,3 +1079,29 @@ The key names in JSON objects passed over the API also follow this convention. There are a few historical exceptions to this rule, such as `/createRoom`. These inconsistencies may be addressed in future versions of this specification. {{% /boxes/note %}} + +### Pagination + +REST API endpoints which can return multiple "pages" of results should adopt the +following conventions. + + * If more results are available, the endpoint should return a property named + `next_batch`. The value should be a string token which can be passed into + a subsequent call to the endpoint to retrieve the next page of results. + + If no more results are available, this is indicated by *omitting* the + `next_batch` property from the results. + + * The endpoint should accept a query-parameter named `from` which the client + is expected to set to the value of a previous `next_batch`. + + * Some endpoints might support pagination in two directions (example: + `/messages`, which can be used to move forward or backwards in the timeline + from a known point). In this case, the endpoint should return a `prev_batch` + property which can be passed into `from` to receive the previous page of + results. + + Avoid having a separate "direction" parameter, which is generally redundant: + the tokens returned by `next_batch` and `prev_batch` should contain enough + information for subsequent calls to the API to know which page of results + they should return. diff --git a/content/client-server-api/_index.md b/content/client-server-api/_index.md index b81cb608..1f50ae7a 100644 --- a/content/client-server-api/_index.md +++ b/content/client-server-api/_index.md @@ -1214,88 +1214,6 @@ using an `unstable` version. When this capability is not listed, clients should use `"1"` as the default and only stable `available` room version. -## Pagination - -{{% boxes/note %}} -The paths referred to in this section are not actual endpoints. They -only serve as examples to explain how pagination functions. -{{% /boxes/note %}} - -Pagination is the process of dividing a dataset into multiple discrete -pages. Matrix makes use of pagination to allow clients to view extremely -large datasets. These datasets are not limited to events in a room (for -example clients may want to paginate a list of rooms in addition to -events within those rooms). Regardless of what is being paginated, there -is a common approach which is used to give clients an easy way of -selecting subsets of a potentially changing dataset. Each endpoint that -uses pagination may use different parameters. However the theme among -them is that they take a `from` and `to` token, and occasionally a -`limit` and `dir`. Together, these parameters describe the position in a -data set, where `from` and `to` are known as "stream tokens" matching -the regular expression `[a-zA-Z0-9.=_-]+`. If supported, the `dir` -defines the direction of events to return: either forwards (`f`) or -backwards (`b`). The response may contain tokens that can be used for -retrieving results before or after the returned set. These tokens may be -called start or prev\_batch for retrieving the previous result -set, or end, next\_batch or next\_token for retrieving the next result set. - -In the following examples, 'START' and 'END' are placeholders to signify -the start and end of the data sets respectively. - -For example, if an endpoint had events E1 -> E15. The client wants -the last 5 events and doesn't know any previous events: - -``` - S E - |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| - | | | - | _____| <--backwards-- | - |__________________ | | ________| - | | | | - GET /somepath?to=START&limit=5&dir=b&from=END - Returns: - E15,E14,E13,E12,E11 -``` - -Another example: a public room list has rooms R1 -> R17. The client -is showing 5 rooms at a time on screen, and is on page 2. They want to -now show page 3 (rooms R11 -> 15): - -``` - S E - | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | stream token - |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room - |____________| |________________| - | | - Currently | - viewing | - | - GET /roomslist?from=9&to=END&limit=5 - Returns: R11,R12,R13,R14,R15 -``` - -Note that tokens are treated in an *exclusive*, not inclusive, manner. -The end token from the initial request was '9' which corresponded to -R10. When the 2nd request was made, R10 did not appear again, even -though from=9 was specified. If you know the token, you already have the -data. - -Responses for pagination-capable endpoints SHOULD have a `chunk` array -alongside the applicable stream tokens to represent the result set. - -In general, when the end of a result set is reached the applicable -stream token will be excluded from the response. For example, if a user -was backwards-paginating events in a room they'd eventually reach the -first event in the room. In this scenario, the `prev_batch` token would -be excluded from the response. Some paginated endpoints are open-ended -in one direction, such as endpoints which expose an event stream for an -active room. In this case, it is not possible for the client to reach -the true "end" of the data set and therefore should always be presented -with a token to keep moving forwards. - ## Filtering Filters can be created on the server and can be passed as a parameter to