From e5928754a4c80846e1fb6cbb1ddfc445a60cca13 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 14 Apr 2023 18:05:52 -0500 Subject: [PATCH 1/3] Add initial proposal for forwards fill --- proposals/0000-forwards-fill.md | 148 ++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 proposals/0000-forwards-fill.md diff --git a/proposals/0000-forwards-fill.md b/proposals/0000-forwards-fill.md new file mode 100644 index 00000000..6ea9cee1 --- /dev/null +++ b/proposals/0000-forwards-fill.md @@ -0,0 +1,148 @@ +# MSCXXXX: Forwards fill (`/backfill` forwards) + +Currently, `/messages` only backfills events when you paginate backwards with `?dir=b`. +The limiting factor here is `/backfill` which only looks backwards to find preceding +events. + +The spec just says this: + +> Since users in that room are able to request the history by the `/messages` client API +> endpoint, it’s possible that they might step backwards far enough into history before +> the homeserver itself was a member of that room. +> +> To cover this case, the federation API provides a server-to-server analog of the +> `/messages` client API, allowing one homeserver to fetch history from another. This is +> the `/backfill` API. +> +> *-- https://spec.matrix.org/v1.5/server-server-api/#backfilling-and-retrieving-missing-events* + +This proposal is after adding a way to `/backfill` forwards, also known as *forwards fill*. + +This also opens the door to adding backfill support to `/context` which returns events +around a given event in both directions. Please note, that this MSC doesn't dictate when +a server should `/backfill` or not. We can address this in a future MSC with spec or +guidelines which is tracked by https://github.com/matrix-org/matrix-spec/issues/1281 + +Navigating the DAG forwards is harder because by the nature of `prev_events`, which +connect the DAG together, gives us no indication of what comes next. We could give a +list of events that we have as forward extremeties and ask other homeservers for any +successor events (events that reference the given events in their `prev_events`). + +## Proposal + +Add a new server API endpoint `GET /_matrix/federation/v1/forwards_fill/{roomId}` which +is basically a mirror image of `/backfill`. + +Retrieves a sliding-window history of subsequent PDUs that occurred in the given room. +Starting from the PDU ID(s) given in the `from_event_id` query parameter, the PDUs that +succeed them are retrieved, up to the total number given by the `limit`. + +| | | +--- | --- +Rate-limited | No +Requires authentication | Yes + +#### Request + +##### Request parameters + +Name| Type | Description +--- | --- | --- +`roomId` | `string` | **Required:** The room ID to backfill. + +##### Query parameters + +Name| Type | Description +--- | --- | --- +`limit` | `integer` | **Required:** The maximum number of PDUs to retrieve, including the given events. +`from_event_id` | `[string]` | **Required:** The event IDs to find successors from. TODO: Should we spec a hard limit to the number of events you can specify? + + +#### Responses + +Status | Description +--- | --- +`200` | A transaction containing the PDUs that succeeded the given event(s), up to the given limit. + +##### 200 response transaction + +Name| Type | Description +--- | --- | --- +`origin` | `string` | **Required:** The server_name of the homeserver sending this transaction. +`origin_server_ts` | `integer` | **Required:** POSIX timestamp in milliseconds on originating homeserver when this transaction started. +`pdus` | `[PDU]` | **Required:** List of persistent updates to rooms. Note that events have a different format depending on the room version - check the room version specification for precise event formats. + + +## Potential issues + +Forwards fill has the same normal problems that `/backfill` has where the server will +probably only exhaust a single chain of events up to the `limit` from the many given +events to navigate from. This makes it harder to fill in all the gaps in one pass. But +servers can keep hitting the endpoint with the new extremities until they've filled in +everything. Is there a smarter heuristic guideline that servers should follow? Or +something we should spec for how much depth/breadth a server should return given a list +of many events to navigate from? + +For example with Synapse, when it does a `/messages?dir=b` and there are backward +extremities, it just does a single `/backfill` with the closest 5 extremities and just +moves on with the `/messages` response regardless of how much timeline is missing. +Perhaps this is better solved by allowing the server to indicate where the gaps are so +the client can choose if it's important to them to fill them in, see +*[MSC3871](https://github.com/matrix-org/matrix-spec-proposals/pull/3871): Gappy +timelines*. + + +## Alternatives + +We could add a direction query parameter to `/backfill` but the semantics for are +slightly different as backfill normally includes the events you requested with `v` as +well preceding events versus with going forward, we just want to find the successors. + + +## Security considerations + +We're only exposing messages according to the existing message history setting in the +room (`m.room.history_visibility`). No extra data is exposed, just a new way to navigate +and fetch it all. + + +## Auxiliary references + +Other links: + + - https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r802851952 + - https://github.com/matrix-org/matrix-spec/issues/601#issuecomment-1082198910 + - https://github.com/matrix-org/matrix-public-archive/issues/72 + + +## Unstable prefix + +While this feature is in development, the endpoint can be used with the unstable prefix +at `GET /_matrix/federation/org.matrix.mscXXXX/forwards_fill/{roomId}` instead of it's +stable `/v1` location as described above. + +### While the MSC is unstable + +During this period, to detect server support, other servers should check for the +presence of the `org.matrix.mscXXXX` flag in `unstable_features` on `/versions`. Servers +are also required to use the unstable prefixes (see [unstable prefix](#unstable-prefix)) +during this time. + +### Once the MSC is merged but not in a spec version + +Once this MSC is merged, but is not yet part of the spec, servers should rely on the +presence of the `org.matrix.mscXXXX.stable` flag in `unstable_features` to determine +server support. If the flag is present, servers are required to use stable prefixes (see +[unstable prefix](#unstable-prefix)). + +### Once the MSC is in a spec version + +Once this MSC becomes a part of a spec version, servers should rely on the presence of +the spec version, that supports the MSC, in `versions` on `/versions`, to determine +support. Servers are encouraged to keep the `org.matrix.mscXXXX.stable` flag around for +a reasonable amount of time to help smooth over the transition for other servers. +"Reasonable" is intentionally left as an implementation detail, however the MSC process +currently recommends *at most* 2 months from the date of spec release. + + + From 1cdfd0bccab98d3d9ecb727727604fc6c4c31ebc Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 14 Apr 2023 18:08:02 -0500 Subject: [PATCH 2/3] Use MSC number --- .../{0000-forwards-fill.md => 4000-forwards-fill.md} | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) rename proposals/{0000-forwards-fill.md => 4000-forwards-fill.md} (95%) diff --git a/proposals/0000-forwards-fill.md b/proposals/4000-forwards-fill.md similarity index 95% rename from proposals/0000-forwards-fill.md rename to proposals/4000-forwards-fill.md index 6ea9cee1..1d48dbf2 100644 --- a/proposals/0000-forwards-fill.md +++ b/proposals/4000-forwards-fill.md @@ -1,4 +1,4 @@ -# MSCXXXX: Forwards fill (`/backfill` forwards) +# MSC4000: Forwards fill (`/backfill` forwards) Currently, `/messages` only backfills events when you paginate backwards with `?dir=b`. The limiting factor here is `/backfill` which only looks backwards to find preceding @@ -118,20 +118,20 @@ Other links: ## Unstable prefix While this feature is in development, the endpoint can be used with the unstable prefix -at `GET /_matrix/federation/org.matrix.mscXXXX/forwards_fill/{roomId}` instead of it's +at `GET /_matrix/federation/org.matrix.msc4000/forwards_fill/{roomId}` instead of it's stable `/v1` location as described above. ### While the MSC is unstable During this period, to detect server support, other servers should check for the -presence of the `org.matrix.mscXXXX` flag in `unstable_features` on `/versions`. Servers +presence of the `org.matrix.msc4000` flag in `unstable_features` on `/versions`. Servers are also required to use the unstable prefixes (see [unstable prefix](#unstable-prefix)) during this time. ### Once the MSC is merged but not in a spec version Once this MSC is merged, but is not yet part of the spec, servers should rely on the -presence of the `org.matrix.mscXXXX.stable` flag in `unstable_features` to determine +presence of the `org.matrix.msc4000.stable` flag in `unstable_features` to determine server support. If the flag is present, servers are required to use stable prefixes (see [unstable prefix](#unstable-prefix)). @@ -139,7 +139,7 @@ server support. If the flag is present, servers are required to use stable prefi Once this MSC becomes a part of a spec version, servers should rely on the presence of the spec version, that supports the MSC, in `versions` on `/versions`, to determine -support. Servers are encouraged to keep the `org.matrix.mscXXXX.stable` flag around for +support. Servers are encouraged to keep the `org.matrix.msc4000.stable` flag around for a reasonable amount of time to help smooth over the transition for other servers. "Reasonable" is intentionally left as an implementation detail, however the MSC process currently recommends *at most* 2 months from the date of spec release. From c3bba8447ea7cd17654a9bd89a7eecf93e9164f9 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Mon, 17 Apr 2023 16:11:08 -0500 Subject: [PATCH 3/3] Determine support by optimistically trying and looking at error response See https://github.com/matrix-org/matrix-spec-proposals/pull/4000#discussion_r1167982535 --- proposals/4000-forwards-fill.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/proposals/4000-forwards-fill.md b/proposals/4000-forwards-fill.md index 1d48dbf2..42d73223 100644 --- a/proposals/4000-forwards-fill.md +++ b/proposals/4000-forwards-fill.md @@ -123,26 +123,28 @@ stable `/v1` location as described above. ### While the MSC is unstable -During this period, to detect server support, other servers should check for the -presence of the `org.matrix.msc4000` flag in `unstable_features` on `/versions`. Servers -are also required to use the unstable prefixes (see [unstable prefix](#unstable-prefix)) -during this time. +During this period, servers should optimistically try the unstable endpoint and look for +the `404` `M_UNRECOGNIZED` error code to determine support (see [MSC3743: *Standardized +error response for unknown +endpoints*](https://github.com/matrix-org/matrix-spec-proposals/pull/3743)). ### Once the MSC is merged but not in a spec version -Once this MSC is merged, but is not yet part of the spec, servers should rely on the -presence of the `org.matrix.msc4000.stable` flag in `unstable_features` to determine -server support. If the flag is present, servers are required to use stable prefixes (see -[unstable prefix](#unstable-prefix)). +Once this MSC is merged, but is not yet part of the spec, servers should optimistically +use the stable endpoint and fallback to the unstable endpoint to have maximum +compatibility if desired. ### Once the MSC is in a spec version -Once this MSC becomes a part of a spec version, servers should rely on the presence of -the spec version, that supports the MSC, in `versions` on `/versions`, to determine -support. Servers are encouraged to keep the `org.matrix.msc4000.stable` flag around for -a reasonable amount of time to help smooth over the transition for other servers. -"Reasonable" is intentionally left as an implementation detail, however the MSC process -currently recommends *at most* 2 months from the date of spec release. +Once this MSC becomes a part of a spec version, servers should optimistically use the +stable endpoint. If a given server doesn't support the endpoint, another server in the +room can be tried until all are exhausted. + +Servers can keep falling back to the unstable endpoint and are encouraged (not +obligated) to serve the unstable fallback endpoint for a reasonable amount of time to +help smooth over the transition for other servers. "Reasonable" is intentionally left as +an implementation detail, however the MSC process currently recommends *at most* 2 +months from the date of spec release.