From 5b6f858802332e8fcee368e9db6cdfac7533c995 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Wed, 28 Oct 2015 20:28:49 +0000 Subject: [PATCH 01/13] Some initial notes by way of the remote join handshake; with several TODOs and unanswered questions --- specification/server_server_api.rst | 149 +++++++++++++++++++++++++++- 1 file changed, 147 insertions(+), 2 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 26e040ca..f0239128 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -533,6 +533,150 @@ part of the path specifies the kind of query being made, and its query arguments have a meaning specific to that kind of query. The response is a JSON-encoded object whose meaning also depends on the kind of query. + +To join a room:: + + GET .../make_join// + Response: JSON encoding of a join proto-event + + PUT .../send_join// + Response: JSON encoding of the state of the room at the time of the event + +Performs the room join handshake. For more information, see "Joining Rooms" +below. + +Joining Rooms +------------- + +When a new user wishes to join room that the user's homeserver already knows +about, the homeserver can immediately determine if this is allowable by +inspecting the state of the room, and if it is acceptable, it can generate, +sign, and emit a new ``m.room.member`` state event adding the user into that +room. When the homeserver does not yet know about the room it cannot do this +directly. Instead, it must take a longer multi-stage handshaking process by +which it first selects a remote homeserver which is already participating in +that room, and uses it to assist in the joining procss. This is the remote join +handshake. + +This handshake involves the homeserver of the new member wishing to join +(referred to here as the "joining" server), the directory server hosting the +room alias the user is requesting to join with, and a homeserver where existing +room members are already present (referred to as the "resident" server). + +In summary, the remote join handshake consists of the joining server querying +the directory server for information about the room alias; receiving a room ID +and a list of join candidates. The joining server then requests information +about the current state of the room from one of the residents. It uses this +information to construct a ``m.room.member`` event which it finally sends to +a resident server. + +Conceptually these are three different roles of homeserver. In practice the +directory server is likely to be resident in the room, and so may be selected +by the joining server to be the assisting resident. Likewise, it is likely that +the joining server picks the same candidate resident for both phases of event +construction, though in principle any valid candidate may be used at each time. +Thus, any join handshake can potentially involve anywhere from two to four +homeservers, though most in practice will use just two. + +.. TODO-doc + - Consider drawing a request/response diagram here + +The first part of the handshake involves using the directory server to request +the room ID and join candidates. This is covered in more detail on the +directory server documentation, below. + +Once the joining server has the room ID and the join candidates, it then needs +to obtain enough of the current state of the room to fill in the required +fields of the ``m.room.member`` event. It obtains this by selecting a resident +from the candidate list, and requesting the ``make_join`` endpoint using a +``GET`` request, specifying the room ID and the user ID of the new member who +is attempting to join. + +The resident server replies to this request with a JSON-encoded object having a +single key called ``event``; within this is an object whose fields contain some +of the information that the joining server will need. Despite its name, this +object is not a full object; notably it does not need to be hashed or signed by +the assisting resident. The required fields are: + +==================== ======== ============ + Key Type Description +==================== ======== ============ +``type`` String The value ``m.room.member`` +``auth_events`` String An event-reference list containing the + authorization events that would allow this member + to join +``content`` Object The event content +``depth`` Integer (this field must be present but is ignored; it + may be 0) +``event_id`` String A new event ID specified by the assisting + resident +``origin`` String The name of the assisting resident homeserver +``origin_server_ts`` Integer A timestamp added by the resident homeserver +``prev_events`` List (TODO(paul): ? - I notice these can be blank) +``prev_state`` List (TODO(paul): ? - I notice these can be blank) +``room_id`` String The room ID of the room +``sender`` String The user ID of the joining member +``state_key`` String The user ID of the joining member +==================== ======== ============ + +The ``content`` field itself must be an object, containing: + +============== ====== ============ + Key Type Description +============== ====== ============ +``membership`` String The value ``join`` +============== ====== ============ + +The joining server now has sufficient information to construct the real join +event from these protoevent fields. It copies the values of most of them, +adding (or replacing) the following fields: + +==================== ======= ============ + Key Type Description +==================== ======= ============ +``event_id`` String A new event ID specified by the joining homeserver +``origin`` String The name of the joining homeserver +``origin_server_ts`` Integer A timestamp added by the joining homeserver +==================== ======= ============ + +.. TODO-spec + - Why does the protoevent have an event_id, only for the real event to ignore + it and specify a different one? We should definitely pick one or the other. + +This will be a true event, so the joining server should apply the event-signing +algorithm to it, resulting in the addition of the ``hashes`` and ``signatures`` +fields. + +To complete the join handshake, the joining server must now submit this new +event to an assisting resident, by using the ``send_join`` endpoint. This is +invoked using the room ID and the event ID of the new member event. + +The assisting resident then accepts this event into the room's event graph, and +responds to the joining server with the full set of state for the newly-joined +room. This is returned as a two-element list, whose first element is the +integer 200, and whose second element contains the following keys: + +.. TODO-spec + - This is likely an implementation bug; see SYN-490. This should probably + actually just return the object directly + +============== ===== ============ + Key Type Description +============== ===== ============ +``auth_chain`` List A list of events giving the authorization chain for this + join event +``state`` List A complete list of the prevailing state events at the + instant just before accepting the new ``m.room.member`` + event +============== ===== ============ + +.. TODO-spec + - (paul) I don't really understand why the full auth_chain events are given + here. What purpose does it serve expanding them out in full, when surely + they'll appear in the state anyway? + - (paul) the state seems to be entirely ignored by synapse, so I'm not really + sure what ought to be there. + Backfilling ----------- .. NOTE:: @@ -763,6 +907,7 @@ Querying directory information:: servers: list of strings giving the join candidates The list of join candidates is a list of server names that are likely to hold -the given room; these are servers that the requesting server may wish to try -joining with. This list may or may not include the server answering the query. +the given room; these are servers that the requesting server may wish to use as +assisting resident servers as part of the remote join handshake. This list may +or may not include the server answering the query. From 40fa339cf7790837f550cf1ac7fa580a8442ad80 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Mon, 2 Nov 2015 15:00:18 +0000 Subject: [PATCH 02/13] Draw a pretty (well at least I think it's pretty) ASCII diagram of the remote join handshake --- specification/server_server_api.rst | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index f0239128..66633376 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -578,8 +578,23 @@ construction, though in principle any valid candidate may be used at each time. Thus, any join handshake can potentially involve anywhere from two to four homeservers, though most in practice will use just two. -.. TODO-doc - - Consider drawing a request/response diagram here +:: + + Client Joining Directory Resident + Server Server Server + + join request --> + | + directory request -------> + <---------- directory response + | + make_join request -----------------------> + <------------------------------- make_join response + | + send_join request -----------------------> + <------------------------------- send_join response + | + <---------- join response The first part of the handshake involves using the directory server to request the room ID and join candidates. This is covered in more detail on the From f6c55979e0cdf6881ca4211745b9fc5cd6f6fbb7 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Mon, 2 Nov 2015 15:17:18 +0000 Subject: [PATCH 03/13] Remove TODO comment about SYN-490 as it's unlikely to matter for v1; we'll fix it in v2 --- specification/server_server_api.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 66633376..d26cc26e 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -671,10 +671,6 @@ responds to the joining server with the full set of state for the newly-joined room. This is returned as a two-element list, whose first element is the integer 200, and whose second element contains the following keys: -.. TODO-spec - - This is likely an implementation bug; see SYN-490. This should probably - actually just return the object directly - ============== ===== ============ Key Type Description ============== ===== ============ From d7c69fae438410e1b91030e60466f48f28e27360 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Fri, 13 Nov 2015 16:33:34 +0000 Subject: [PATCH 04/13] Fix typo 'process' --- specification/server_server_api.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index d26cc26e..1c3e761d 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -555,8 +555,8 @@ sign, and emit a new ``m.room.member`` state event adding the user into that room. When the homeserver does not yet know about the room it cannot do this directly. Instead, it must take a longer multi-stage handshaking process by which it first selects a remote homeserver which is already participating in -that room, and uses it to assist in the joining procss. This is the remote join -handshake. +that room, and uses it to assist in the joining process. This is the remote +join handshake. This handshake involves the homeserver of the new member wishing to join (referred to here as the "joining" server), the directory server hosting the From aac45295ee9dc615cfc7290cc928c011ca0d4e79 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Fri, 13 Nov 2015 16:34:22 +0000 Subject: [PATCH 05/13] Remark that the directory server step could be skipped in an invite case --- specification/server_server_api.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 1c3e761d..e158ae23 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -596,9 +596,12 @@ homeservers, though most in practice will use just two. | <---------- join response -The first part of the handshake involves using the directory server to request -the room ID and join candidates. This is covered in more detail on the -directory server documentation, below. +The first part of the handshake usually involves using the directory server to +request the room ID and join candidates. This is covered in more detail on the +directory server documentation, below. In the case of a new user joining a +room as a result of a received invite, the joining user's homeserver could +optimise this step away by picking the origin server of that invite message as +the join candidate. Once the joining server has the room ID and the join candidates, it then needs to obtain enough of the current state of the room to fill in the required From db5a90edcd7fcc33385d2145b2ad1ed93313182f Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Fri, 13 Nov 2015 16:36:26 +0000 Subject: [PATCH 06/13] Avoid using the words 'current state' when talking about the result of the /make_join request --- specification/server_server_api.rst | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index e158ae23..6dfbd86f 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -566,9 +566,8 @@ room members are already present (referred to as the "resident" server). In summary, the remote join handshake consists of the joining server querying the directory server for information about the room alias; receiving a room ID and a list of join candidates. The joining server then requests information -about the current state of the room from one of the residents. It uses this -information to construct a ``m.room.member`` event which it finally sends to -a resident server. +about the room from one of the residents. It uses this information to construct +a ``m.room.member`` event which it finally sends to a resident server. Conceptually these are three different roles of homeserver. In practice the directory server is likely to be resident in the room, and so may be selected @@ -604,11 +603,11 @@ optimise this step away by picking the origin server of that invite message as the join candidate. Once the joining server has the room ID and the join candidates, it then needs -to obtain enough of the current state of the room to fill in the required -fields of the ``m.room.member`` event. It obtains this by selecting a resident -from the candidate list, and requesting the ``make_join`` endpoint using a -``GET`` request, specifying the room ID and the user ID of the new member who -is attempting to join. +to obtain enough information about the room to fill in the required fields of +the ``m.room.member`` event. It obtains this by selecting a resident from the +candidate list, and requesting the ``make_join`` endpoint using a ``GET`` +request, specifying the room ID and the user ID of the new member who is +attempting to join. The resident server replies to this request with a JSON-encoded object having a single key called ``event``; within this is an object whose fields contain some From 885dd1e86cc8d6ee818ab32ab246eeec9a1f5ede Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Fri, 13 Nov 2015 16:39:31 +0000 Subject: [PATCH 07/13] Explain the 'prev_events' join protoevent key --- specification/server_server_api.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 6dfbd86f..ebf55eb4 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -629,7 +629,8 @@ the assisting resident. The required fields are: resident ``origin`` String The name of the assisting resident homeserver ``origin_server_ts`` Integer A timestamp added by the resident homeserver -``prev_events`` List (TODO(paul): ? - I notice these can be blank) +``prev_events`` List An event-reference list containing the immediate + predecessor events ``prev_state`` List (TODO(paul): ? - I notice these can be blank) ``room_id`` String The room ID of the room ``sender`` String The user ID of the joining member From 988d77347651ce2504ab1e52196166358dffe7f7 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Fri, 13 Nov 2015 17:49:22 +0000 Subject: [PATCH 08/13] /make_join protoevent no longer needs the pointless 'prev_state' key (SYN-517) --- specification/server_server_api.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index ebf55eb4..9cc10141 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -631,7 +631,6 @@ the assisting resident. The required fields are: ``origin_server_ts`` Integer A timestamp added by the resident homeserver ``prev_events`` List An event-reference list containing the immediate predecessor events -``prev_state`` List (TODO(paul): ? - I notice these can be blank) ``room_id`` String The room ID of the room ``sender`` String The user ID of the joining member ``state_key`` String The user ID of the joining member @@ -688,8 +687,6 @@ integer 200, and whose second element contains the following keys: - (paul) I don't really understand why the full auth_chain events are given here. What purpose does it serve expanding them out in full, when surely they'll appear in the state anyway? - - (paul) the state seems to be entirely ignored by synapse, so I'm not really - sure what ought to be there. Backfilling ----------- From 36af793f05992d0bef5ef325c303a753f16f24e0 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Mon, 16 Nov 2015 15:30:22 +0000 Subject: [PATCH 09/13] s/full object/full event/ --- specification/server_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 9cc10141..127a3572 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -612,7 +612,7 @@ attempting to join. The resident server replies to this request with a JSON-encoded object having a single key called ``event``; within this is an object whose fields contain some of the information that the joining server will need. Despite its name, this -object is not a full object; notably it does not need to be hashed or signed by +object is not a full event; notably it does not need to be hashed or signed by the assisting resident. The required fields are: ==================== ======== ============ From 923f05e5541f354c2003acf77dbe4e2c3d6f90ec Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Mon, 16 Nov 2015 15:34:11 +0000 Subject: [PATCH 10/13] More consistency around 'resident homeserver' --- specification/server_server_api.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 127a3572..4c315edc 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -613,7 +613,7 @@ The resident server replies to this request with a JSON-encoded object having a single key called ``event``; within this is an object whose fields contain some of the information that the joining server will need. Despite its name, this object is not a full event; notably it does not need to be hashed or signed by -the assisting resident. The required fields are: +the resident homeserver. The required fields are: ==================== ======== ============ Key Type Description @@ -625,9 +625,9 @@ the assisting resident. The required fields are: ``content`` Object The event content ``depth`` Integer (this field must be present but is ignored; it may be 0) -``event_id`` String A new event ID specified by the assisting - resident -``origin`` String The name of the assisting resident homeserver +``event_id`` String A new event ID specified by the resident + homeserver +``origin`` String The name of the resident homeserver ``origin_server_ts`` Integer A timestamp added by the resident homeserver ``prev_events`` List An event-reference list containing the immediate predecessor events @@ -665,12 +665,12 @@ algorithm to it, resulting in the addition of the ``hashes`` and ``signatures`` fields. To complete the join handshake, the joining server must now submit this new -event to an assisting resident, by using the ``send_join`` endpoint. This is +event to an resident homeserver, by using the ``send_join`` endpoint. This is invoked using the room ID and the event ID of the new member event. -The assisting resident then accepts this event into the room's event graph, and -responds to the joining server with the full set of state for the newly-joined -room. This is returned as a two-element list, whose first element is the +The resident homeserver then accepts this event into the room's event graph, +and responds to the joining server with the full set of state for the newly- +joined room. This is returned as a two-element list, whose first element is the integer 200, and whose second element contains the following keys: ============== ===== ============ @@ -919,6 +919,6 @@ Querying directory information:: The list of join candidates is a list of server names that are likely to hold the given room; these are servers that the requesting server may wish to use as -assisting resident servers as part of the remote join handshake. This list may -or may not include the server answering the query. +resident servers as part of the remote join handshake. This list may or may not +include the server answering the query. From 122c082fcf0ef14cb93e09d1b8d2b7469866c786 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Mon, 16 Nov 2015 16:29:43 +0000 Subject: [PATCH 11/13] Comment about origin servers of invites having subsequently left the room --- specification/server_server_api.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 4c315edc..20026890 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -600,7 +600,9 @@ request the room ID and join candidates. This is covered in more detail on the directory server documentation, below. In the case of a new user joining a room as a result of a received invite, the joining user's homeserver could optimise this step away by picking the origin server of that invite message as -the join candidate. +the join candidate. However, the joining server should be aware that the origin +server of the invite might since have left the room, so should be prepared to +fall back on the regular join flow if this optimisation fails. Once the joining server has the room ID and the join candidates, it then needs to obtain enough information about the room to fill in the required fields of From 6cbfba70113e302985c4f93af181141e7b0c84f1 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Mon, 16 Nov 2015 16:33:26 +0000 Subject: [PATCH 12/13] 'auth_events' is a List, not a String --- specification/server_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 20026890..6c66b3d9 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -621,7 +621,7 @@ the resident homeserver. The required fields are: Key Type Description ==================== ======== ============ ``type`` String The value ``m.room.member`` -``auth_events`` String An event-reference list containing the +``auth_events`` List An event-reference list containing the authorization events that would allow this member to join ``content`` Object The event content From 233e8486bc41e4ad87fc8e06a57bbcd4b8579974 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Mon, 16 Nov 2015 16:50:58 +0000 Subject: [PATCH 13/13] Wording fix - objects contain keys, not list elements directly --- specification/server_server_api.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 6c66b3d9..cc9426e2 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -673,7 +673,8 @@ invoked using the room ID and the event ID of the new member event. The resident homeserver then accepts this event into the room's event graph, and responds to the joining server with the full set of state for the newly- joined room. This is returned as a two-element list, whose first element is the -integer 200, and whose second element contains the following keys: +integer 200, and whose second element is an object which contains the +following keys: ============== ===== ============ Key Type Description