Replace "Pagination" section in C-S API with text in appendices (#3366)

The Pagination section in the C-S API was, basically, full of rubbish. I think that anything of any value it contained was repeated either directly on the API definitions or in the text specific to syncing at https://spec.matrix.org/unstable/client-server-api/#syncing.

The conventions I've added to the Appendices are based on the discussions in #1898. They are there because I don't want to have to go through it all again next time we add a paginated API.

Fixes: #1898
Fixes: #2268

changelogs/client_server/newsfragments/3366.clarification

@@ -0,0 +1 @@
+Remove the inaccurate 'Pagination' section.

content/appendices.md

@@ -1079,3 +1079,29 @@ The key names in JSON objects passed over the API also follow this convention.
 There are a few historical exceptions to this rule, such as `/createRoom`.
 These inconsistencies may be addressed in future versions of this specification.
 {{% /boxes/note %}}
+
+### Pagination
+
+REST API endpoints which can return multiple "pages" of results should adopt the
+following conventions.
+
+ * If more results are available, the endpoint should return a property named
+   `next_batch`. The value should be a string token which can be passed into
+   a subsequent call to the endpoint to retrieve the next page of results.
+
+   If no more results are available, this is indicated by *omitting* the
+   `next_batch` property from the results.
+
+ * The endpoint should accept a query-parameter named `from` which the client
+   is expected to set to the value of a previous `next_batch`.
+
+ * Some endpoints might support pagination in two directions (example:
+   `/messages`, which can be used to move forward or backwards in the timeline
+   from a known point). In this case, the endpoint should return a `prev_batch`
+   property which can be passed into `from` to receive the previous page of
+   results.
+
+   Avoid having a separate "direction" parameter, which is generally redundant:
+   the tokens returned by `next_batch` and `prev_batch` should contain enough
+   information for subsequent calls to the API to know which page of results
+   they should return.

content/client-server-api/_index.md

@@ -1214,88 +1214,6 @@ using an `unstable` version.
 When this capability is not listed, clients should use `"1"` as the
 default and only stable `available` room version.
 
-## Pagination
-
-{{% boxes/note %}}
-The paths referred to in this section are not actual endpoints. They
-only serve as examples to explain how pagination functions.
-{{% /boxes/note %}}
-
-Pagination is the process of dividing a dataset into multiple discrete
-pages. Matrix makes use of pagination to allow clients to view extremely
-large datasets. These datasets are not limited to events in a room (for
-example clients may want to paginate a list of rooms in addition to
-events within those rooms). Regardless of what is being paginated, there
-is a common approach which is used to give clients an easy way of
-selecting subsets of a potentially changing dataset. Each endpoint that
-uses pagination may use different parameters. However the theme among
-them is that they take a `from` and `to` token, and occasionally a
-`limit` and `dir`. Together, these parameters describe the position in a
-data set, where `from` and `to` are known as "stream tokens" matching
-the regular expression `[a-zA-Z0-9.=_-]+`. If supported, the `dir`
-defines the direction of events to return: either forwards (`f`) or
-backwards (`b`). The response may contain tokens that can be used for
-retrieving results before or after the returned set. These tokens may be
-called <span class="title-ref">start</span> or <span
-class="title-ref">prev\_batch</span> for retrieving the previous result
-set, or <span class="title-ref">end</span>, <span
-class="title-ref">next\_batch</span> or <span
-class="title-ref">next\_token</span> for retrieving the next result set.
-
-In the following examples, 'START' and 'END' are placeholders to signify
-the start and end of the data sets respectively.
-
-For example, if an endpoint had events E1 -&gt; E15. The client wants
-the last 5 events and doesn't know any previous events:
-
-```
-    S                                                    E
-    |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-|
-    |                               |                    |
-    |                          _____|  <--backwards--    |
-    |__________________       |         |        ________|
-                       |      |         |        |
-     GET /somepath?to=START&limit=5&dir=b&from=END
-     Returns:
-       E15,E14,E13,E12,E11
-```
-
-Another example: a public room list has rooms R1 -&gt; R17. The client
-is showing 5 rooms at a time on screen, and is on page 2. They want to
-now show page 3 (rooms R11 -&gt; 15):
-
-```
-    S                                                           E
-    |  0  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16 | stream token
-    |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room
-                      |____________| |________________|
-                            |                |
-                        Currently            |
-                        viewing              |
-                                             |
-                             GET /roomslist?from=9&to=END&limit=5
-                             Returns: R11,R12,R13,R14,R15
-```
-
-Note that tokens are treated in an *exclusive*, not inclusive, manner.
-The end token from the initial request was '9' which corresponded to
-R10. When the 2nd request was made, R10 did not appear again, even
-though from=9 was specified. If you know the token, you already have the
-data.
-
-Responses for pagination-capable endpoints SHOULD have a `chunk` array
-alongside the applicable stream tokens to represent the result set.
-
-In general, when the end of a result set is reached the applicable
-stream token will be excluded from the response. For example, if a user
-was backwards-paginating events in a room they'd eventually reach the
-first event in the room. In this scenario, the `prev_batch` token would
-be excluded from the response. Some paginated endpoints are open-ended
-in one direction, such as endpoints which expose an event stream for an
-active room. In this case, it is not possible for the client to reach
-the true "end" of the data set and therefore should always be presented
-with a token to keep moving forwards.
-
 ## Filtering
 
 Filters can be created on the server and can be passed as a parameter to