From 902c7d3ea69c56b9c1576756123d98721135af12 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 19 Aug 2015 15:00:22 +0100
Subject: [PATCH 01/85] Add draft macaroon caveat specification

---
 drafts/macaroons_caveats.rst | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 drafts/macaroons_caveats.rst

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
new file mode 100644
index 00000000..b6920d6c
--- /dev/null
+++ b/drafts/macaroons_caveats.rst
@@ -0,0 +1,30 @@
+Macaroon Caveats
+================
+
+Macaroons (http://theory.stanford.edu/~ataly/Papers/macaroons.pdf) are issued by Matrix servers as authorization tokens. Macaroons may be restricted, by adding caveats to them.
+
+Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
+
+Some caveats are specified in this specification, and must be understood by all servers. The use of non-standard caveats is allowed.
+
+All caveats must take the form:
+
+`key` `operator` `value`
+where `key` is a non-empty string drawn from the character set [A-Za-z0-9_]
+`operator` is a non-empty string which does not contain whitespace
+`value` is a non-empty string
+And these are joined by single space characters.
+
+Specified caveats:
+
++-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
+| Caveat name | Description                                      | Legal Values                                                                               |
++-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
+| gen         | Generation of the macaroon caveat spec.          | 1                                                                                          |
+| user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                   |
+| type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                 |
+|                                                                   refresh - only used to authorize a token refresh                                          |
+| time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                |
+|                                                                  Operator < means the macaroon is valid before the timestamp, as interpreted by the server. |
+|                                                                  Operator > means the macaroon is valid after the timestamp, as interpreted by the server.  |
++-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+

From 0440983c4ab74f30a38f27d70c51f47ea1a8ff30 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 28 Aug 2015 10:31:45 +0100
Subject: [PATCH 02/85] Spec exchanging refresh tokens for new access tokens

---
 specification/10_client_server_api.rst | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index 83c8b7a8..6b8ffbcc 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1087,10 +1087,33 @@ On success, this returns a JSON object with keys:
 user_id
   The fully-qualified Matrix ID that has been registered.
 access_token
-  An access token for the new account.
+  An access token for the account. This token may expire at some point, and if
+  so, it MAY come with a refersh_token, described below.
+refresh_token (optional)
+  A refresh token may be exchanged for a new access_token as described in
+  `Refreshing access tokens`.
 home_server
   The hostname of the Home Server on which the account has been registered.
 
+Refreshing access tokens
+~~~~~~~~~~~~~~~~~~~~~~~~
+Exchanging a refresh token for an access token is done using the request::
+
+  POST $PREFIX/tokenrefresh
+
+The body of the POST request is a JSON object containing:
+
+refresh_token
+  The refresh token.
+
+On success, this invalidates the refresh token, so that it cannot be used again,
+and returns a JSON object with keys:
+
+access_token
+  An access token for the account, as is returned from login.
+refresh_token (optional)
+  A refresh token, as is returned from login.
+
 Changing Password
 ~~~~~~~~~~~~~~~~~
 This section refers to API Version 2. These API calls currently use the prefix

From 9515640d5e0ef51e075a0635ebd8261888e04501 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 28 Aug 2015 10:41:49 +0100
Subject: [PATCH 03/85] Add note about no specific 'token expired' error

---
 specification/10_client_server_api.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index 6b8ffbcc..5dd215d4 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1114,6 +1114,11 @@ access_token
 refresh_token (optional)
   A refresh token, as is returned from login.
 
+There is no specific error message to indicate that a request has failed because
+an access token has expired; instead, if a client has reason to believe its
+access token is valid, and it receives an auth error, they should attempt to
+refresh for a new token on failure, and re-try the request with the new token.
+
 Changing Password
 ~~~~~~~~~~~~~~~~~
 This section refers to API Version 2. These API calls currently use the prefix

From b2477614658d6b4267ffcb1f35cfc9adb7f388ad Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 10:33:25 +0100
Subject: [PATCH 04/85] .gitignore speculator

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index e2250131..74b1c7d2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
 scripts/gen
 scripts/continuserv/continuserv
+scripts/speculator/speculator
 templating/out
 *.pyc
 supporting-docs/_site

From 0341c30824ed89b64cb09b5eea74f011eed81495 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 10:39:22 +0100
Subject: [PATCH 05/85] Minor typo and formatting fixes

---
 drafts/macaroons_caveats.rst           | 2 +-
 specification/10_client_server_api.rst | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index b6920d6c..2a45fe12 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -1,7 +1,7 @@
 Macaroon Caveats
 ================
 
-Macaroons (http://theory.stanford.edu/~ataly/Papers/macaroons.pdf) are issued by Matrix servers as authorization tokens. Macaroons may be restricted, by adding caveats to them.
+Macaroons (http://theory.stanford.edu/~ataly/Papers/macaroons.pdf) are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
 
 Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
 
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index 6a589f8e..440f98e4 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1035,7 +1035,7 @@ user_id
   The fully-qualified Matrix ID that has been registered.
 access_token
   An access token for the account. This token may expire at some point, and if
-  so, it MAY come with a refersh_token, described below.
+  so, it MAY come with a refresh_token, described below.
 refresh_token (optional)
   A refresh token may be exchanged for a new access_token as described in
   `Refreshing access tokens`.
@@ -1064,7 +1064,7 @@ refresh_token (optional)
 There is no specific error message to indicate that a request has failed because
 an access token has expired; instead, if a client has reason to believe its
 access token is valid, and it receives an auth error, they should attempt to
-refresh for a new token on failure, and re-try the request with the new token.
+refresh for a new token on failure, and retry the request with the new token.
 
 Changing Password
 ~~~~~~~~~~~~~~~~~

From b8e2ac5c1b9408a4dc3d72b4bc2935dd278cfeda Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 13:49:27 +0100
Subject: [PATCH 06/85] Swaggerify /login

---
 api/client-server/v1/registration.yaml  | 79 +++++++++++++++++++++++++
 specification/10_client_server_api.rst  | 27 +--------
 templating/matrix_templates/sections.py |  6 ++
 3 files changed, 86 insertions(+), 26 deletions(-)
 create mode 100644 api/client-server/v1/registration.yaml

diff --git a/api/client-server/v1/registration.yaml b/api/client-server/v1/registration.yaml
new file mode 100644
index 00000000..2df695be
--- /dev/null
+++ b/api/client-server/v1/registration.yaml
@@ -0,0 +1,79 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Registration and Login API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/login":
+    post:
+      summary: Authenticates the user.
+      description: |-
+        Authenticates the user by password, and issues an access token they can
+        use to authorize themself in subsequent requests.
+      security:
+        - accessToken: []
+      parameters:
+        - in: body
+          name: body
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "username": "cheeky_monkey",
+                "password": "ilovebananas"
+              }
+            properties:
+              username:
+                type: string
+                description: The fully qualified user ID or just local part of the user ID, to log in.
+              password:
+                type: string
+                description: The user's password.
+            required: ["username", "password"]
+      responses:
+        200:
+          description: The user has been authenticated.
+          examples:
+            application/json: |-
+              {
+                "user_id": "@cheeky_monkey:matrix.org",
+                "access_token": "abc123",
+                "home_server": "matrix.org"
+              }
+          schema:
+            type: object
+            properties:
+              user_id:
+                type: string
+                description: The fully-qualified Matrix ID that has been registered.
+              access_token:
+                type: string
+                description: An access token for the account. This access token can then be used to authorize other requests.
+              home_server:
+                type: string
+                description: The hostname of the Home Server on which the account has been registered.
+        403:
+          description: |-
+            The login attempt failed. For example, the password may have been incorrect.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN"}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index f0d6be42..b55118be 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1011,32 +1011,7 @@ was registered whilst the client was performing authentication.
 
 Old V1 API docs: |register|_
 
-Login
-~~~~~
-This section refers to API Version 1.
-
-API docs: |login|_
-
-Obtaining an access token for an existing user account is done using the
-request::
-
-  POST $PREFIX/login
-
-The body of the POST request is a JSON object containing:
-
-username
-  The full qualified or local part of the Matrix ID to log in with.
-password
-  The password for the account.
-
-On success, this returns a JSON object with keys:
-
-user_id
-  The fully-qualified Matrix ID that has been registered.
-access_token
-  An access token for the new account.
-home_server
-  The hostname of the Home Server on which the account has been registered.
+{{registration_http_api}}
 
 Changing Password
 ~~~~~~~~~~~~~~~~~
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 7e21434f..f28fe22d 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -96,6 +96,12 @@ class MatrixSections(Sections):
             title_kind="~"
         )
 
+    def render_registration_http_api(self):
+        return self._render_http_api_group(
+            "registration",
+            title_kind="~"
+        )
+
     def render_room_events(self):
         def filterFn(eventType):
             return (

From 59a4e843612cd8b5cf954b1f1cac6bf9fdb732e8 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 17:10:46 +0100
Subject: [PATCH 07/85] s/registration/login/

---
 api/client-server/v1/{registration.yaml => login.yaml} | 0
 specification/10_client_server_api.rst                 | 2 +-
 templating/matrix_templates/sections.py                | 4 ++--
 3 files changed, 3 insertions(+), 3 deletions(-)
 rename api/client-server/v1/{registration.yaml => login.yaml} (100%)

diff --git a/api/client-server/v1/registration.yaml b/api/client-server/v1/login.yaml
similarity index 100%
rename from api/client-server/v1/registration.yaml
rename to api/client-server/v1/login.yaml
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index b55118be..fe11d199 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1011,7 +1011,7 @@ was registered whilst the client was performing authentication.
 
 Old V1 API docs: |register|_
 
-{{registration_http_api}}
+{{login_http_api}}
 
 Changing Password
 ~~~~~~~~~~~~~~~~~
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index f28fe22d..729157bb 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -96,9 +96,9 @@ class MatrixSections(Sections):
             title_kind="~"
         )
 
-    def render_registration_http_api(self):
+    def render_login_http_api(self):
         return self._render_http_api_group(
-            "registration",
+            "login",
             title_kind="~"
         )
 

From 3b44fb92f972104bc781999303631d7ba5d649f2 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 17:13:54 +0100
Subject: [PATCH 08/85] Clarify == case

---
 drafts/macaroons_caveats.rst | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index 2a45fe12..791d217a 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -17,14 +17,16 @@ And these are joined by single space characters.
 
 Specified caveats:
 
-+-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
-| Caveat name | Description                                      | Legal Values                                                                               |
-+-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
-| gen         | Generation of the macaroon caveat spec.          | 1                                                                                          |
-| user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                   |
-| type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                 |
-|                                                                   refresh - only used to authorize a token refresh                                          |
-| time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                |
-|                                                                  Operator < means the macaroon is valid before the timestamp, as interpreted by the server. |
-|                                                                  Operator > means the macaroon is valid after the timestamp, as interpreted by the server.  |
-+-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
+| Caveat name | Description                                      | Legal Values                                                                                   |
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
+| gen         | Generation of the macaroon caveat spec.          | 1                                                                                              |
+| user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                       |
+| type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                     |
+|                                                                   refresh - only used to authorize a token refresh                                              |
+| time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                    |
+|                                                                  Operator < means the macaroon is valid before the timestamp, as interpreted by the server.     |
+|                                                                  Operator > means the macaroon is valid after the timestamp, as interpreted by the server.      |
+|                                                                  Operator == means the macaroon is valid at exactly the timestamp, as interpreted by the server.|
+|                                                                  Note that exact equality of time is largely meaningless.                                       |
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+

From c3eab1cc3c81cc8a589ed4343dbbbeb2a0027a94 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 15 Sep 2015 11:41:40 +0100
Subject: [PATCH 09/85] Set the current working directory in gendoc.py to the
 script directory as that is where it expects to be run

---
 scripts/gendoc.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index e3c3797e..10c3fa90 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -8,6 +8,8 @@ import shutil
 import subprocess
 import sys
 
+os.chdir(os.path.dirname(__file__))
+
 stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css"]
 }

From 7eb8b4fde275834496580cf642de6dd5eca8c377 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 15 Sep 2015 15:52:36 +0100
Subject: [PATCH 10/85] Add new-style docs for the APIs for getting events for
 a room

---
 api/client-server/v1/rooms.yaml         | 433 ++++++++++++++++++++++++
 specification/10_client_server_api.rst  |  44 +--
 templating/matrix_templates/sections.py |   6 +
 templating/matrix_templates/units.py    |   4 +-
 4 files changed, 442 insertions(+), 45 deletions(-)
 create mode 100644 api/client-server/v1/rooms.yaml

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
new file mode 100644
index 00000000..9a73bb05
--- /dev/null
+++ b/api/client-server/v1/rooms.yaml
@@ -0,0 +1,433 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Rooms API"
+  version: "1.0.0"
+host: example.com:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/state/{eventType}/{stateKey}":
+    get:
+      summary: Get the state identified by the type and key.
+      description: |-
+        Looks up the contents of a state event in a room. If the user is
+        joined to the room then the state is taken from the current
+        state of the room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to look up the state in.
+          required: true
+          x-example: "!room:example.com"
+        - in: path
+          type: string
+          name: eventType
+          description: The type of state to look up.
+          required: true
+          x-example: "m.room.name"
+        - in: path
+          type: string
+          name: stateKey
+          description: |-
+            The key of the state to look up. Defaults to the empty string.
+          required: false
+          x-example: ""
+      responses:
+        200:
+          description: The content of the state event.
+          examples:
+            application/json: |-
+              {"name": "Example room name"}
+          schema:
+            type: object
+        404:
+          description: The room has no state with the given type or key.
+        403:
+          description: You are not joined to the room.
+
+  "/rooms/{roomId}/state":
+    get:
+      summary: Get all state events in the current state of a room.
+      description: |-
+        Get the state events for the current state of a room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to look up the state for.
+          required: true
+          x-example: "!room:example.com"
+      responses:
+        200:
+          description: The current state of the room
+          examples:
+            application/json: |-
+              [
+                {
+                    "age": 7148266897,
+                    "content": {
+                        "join_rule": "public"
+                    },
+                    "event_id": "$14259997323TLwtb:example.com",
+                    "origin_server_ts": 1425999732392,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.join_rules",
+                    "user_id": "@alice:example.com"
+                },
+                {
+                    "age": 6547561012,
+                    "content": {
+                        "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto",
+                        "displayname": null,
+                        "membership": "join"
+                    },
+                    "event_id": "$1426600438280zExKY:example.com",
+                    "membership": "join",
+                    "origin_server_ts": 1426600438277,
+                    "room_id": "!room:example.com",
+                    "state_key": "@alice:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@alice:example.com"
+                },
+                {
+                    "age": 7148267200,
+                    "content": {
+                        "creator": "@alice:example.com"
+                    },
+                    "event_id": "$14259997320KhbwJ:example.com",
+                    "origin_server_ts": 1425999732089,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.create",
+                    "user_id": "@alice:example.com"
+                },
+                {
+                    "age": 1622568720,
+                    "content": {
+                        "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto",
+                        "displayname": "Bob",
+                        "membership": "join"
+                    },
+                    "event_id": "$1431525430134MxlLX:example.com",
+                    "origin_server_ts": 1431525430569,
+                    "replaces_state": "$142652023736BSXcM:example.com",
+                    "room_id": "!room:example.com",
+                    "state_key": "@bob:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@bob:example.com"
+                },
+                {
+                    "age": 7148267004,
+                    "content": {
+                        "ban": 50,
+                        "events": {
+                            "m.room.name": 100,
+                            "m.room.power_levels": 100
+                         },
+                         "events_default": 0,
+                         "kick": 50,
+                         "redact": 50,
+                         "state_default": 50,
+                         "users": {
+                             "@alice:example.com": 100
+                         },
+                         "users_default": 0
+                    },
+                    "event_id": "$14259997322mqfaq:example.com",
+                    "origin_server_ts": 1425999732285,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.power_levels",
+                    "user_id": "@alice:example.com"
+                }
+              ]
+            schema:
+              type: array
+              title: RoomState
+              description: |-
+                If the user is a member of the room this will be the
+                current state of the room as a list of events.
+              items:
+                title: StateEvent
+                type: object
+                allOf:
+                  - "$ref": "definitions/state_event.yaml"
+        403:
+          description: You are not joined to the room.
+
+  "/rooms/{roomId}/initialSync":
+    get:
+      summary: Snapshot the current state of a room and its most recent messages.
+      description: |-
+        Get a copy of the current state and the most recent messages in a room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to get the data.
+          required: true
+          x-example: "!room:example.com"
+      responses:
+        200:
+          description: The current state of the room
+          examples:
+            application/json: |-
+              {
+                "membership": "join",
+                "messages": {
+                  "chunk": [
+                    {
+                      "age": 343513403,
+                      "content": {
+                        "body": "foo",
+                        "msgtype": "m.text"
+                      },
+                      "event_id": "$14328044851tzTJS:example.com",
+                      "origin_server_ts": 1432804485886,
+                      "room_id": "!room:example.com",
+                      "type": "m.room.message",
+                      "user_id": "@alice:example.com"
+                    },
+                    {
+                      "age": 343511809,
+                      "content": {
+                        "body": "bar",
+                        "msgtype": "m.text"
+                      },
+                      "event_id": "$14328044872spjFg:example.com",
+                      "origin_server_ts": 1432804487480,
+                      "room_id": "!room:example.com",
+                      "type": "m.room.message",
+                      "user_id": "@bob:example.com"
+                    }
+                  ],
+                  "end": "s3456_9_0",
+                  "start": "t44-3453_9_0"
+                },
+                "room_id": "!room:example.com",
+                "state": [
+                  {
+                    "age": 7148266897,
+                    "content": {
+                      "join_rule": "public"
+                    },
+                    "event_id": "$14259997323TLwtb:example.com",
+                    "origin_server_ts": 1425999732392,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.join_rules",
+                    "user_id": "@alice:example.com"
+                  },
+                  {
+                    "age": 6547561012,
+                    "content": {
+                      "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto",
+                      "displayname": null,
+                      "membership": "join"
+                    },
+                    "event_id": "$1426600438280zExKY:example.com",
+                    "membership": "join",
+                    "origin_server_ts": 1426600438277,
+                    "room_id": "!room:example.com",
+                    "state_key": "@alice:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@alice:example.com"
+                  },
+                  {
+                    "age": 7148267200,
+                    "content": {
+                        "creator": "@alice:example.com"
+                    },
+                    "event_id": "$14259997320KhbwJ:example.com",
+                    "origin_server_ts": 1425999732089,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.create",
+                    "user_id": "@alice:example.com"
+                  },
+                  {
+                    "age": 1622568720,
+                    "content": {
+                        "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto",
+                        "displayname": "Bob",
+                        "membership": "join"
+                    },
+                    "event_id": "$1431525430134MxlLX:example.com",
+                    "origin_server_ts": 1431525430569,
+                    "replaces_state": "$142652023736BSXcM:example.com",
+                    "room_id": "!room:example.com",
+                    "state_key": "@bob:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@bob:example.com"
+                  },
+                  {
+                    "age": 7148267004,
+                    "content": {
+                      "ban": 50,
+                      "events": {
+                        "m.room.name": 100,
+                        "m.room.power_levels": 100
+                      },
+                      "events_default": 0,
+                      "kick": 50,
+                      "redact": 50,
+                      "state_default": 50,
+                      "users": {
+                        "@alice:example.com": 100
+                      },
+                      "users_default": 0
+                    },
+                    "event_id": "$14259997322mqfaq:example.com",
+                    "origin_server_ts": 1425999732285,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.power_levels",
+                    "user_id": "@alice:example.com"
+                  }
+                ],
+                "visibility": "private"
+              }
+          schema:
+            title: RoomInfo
+            type: object
+            properties:
+              room_id:
+                type: string
+                description: "The ID of this room."
+              membership:
+                type: string
+                description: "The user's membership state in this room."
+                enum: ["invite", "join", "leave", "ban"]
+              messages:
+                type: object
+                title: PaginationChunk
+                description: "The pagination chunk for this room."
+                properties:
+                  start:
+                    type: string
+                    description: |-
+                      A token which correlates to the first value in ``chunk``.
+                      Used for pagination.
+                  end:
+                    type: string
+                    description: |-
+                      A token which correlates to the last value in ``chunk``.
+                      Used for pagination.
+                  chunk:
+                    type: array
+                    description: |-
+                      If the user is a member of the room this will be a
+                      list of the most recent messages for this room. If
+                      the user has left the room this will be the
+                      messages that preceeded them leaving. This array
+                      will consist of at most ``limit`` elements.
+                    items:
+                      type: object
+                      title: RoomEvent
+                      allOf:
+                        - "$ref": "definitions/room_event.yaml"
+                required: ["start", "end", "chunk"]
+              state:
+                type: array
+                description: |-
+                  If the user is a member of the room this will be the
+                  current state of the room as a list of events. If the
+                  user has left the room this will be the state of the
+                  room when they left it.
+                items:
+                  title: StateEvent
+                  type: object
+                  allOf:
+                    - "$ref": "definitions/state_event.yaml"
+              visibility:
+                type: string
+                enum: ["private", "public"]
+                description: |-
+                  Whether this room is visible to the ``/publicRooms`` API
+                  or not."
+            required: ["room_id", "membership"]
+
+  "/rooms/{roomId}/members":
+    get:
+      summary: Get the m.room.member events for the room.
+      description:
+        Get the list of members of the room.
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to get the member events for.
+          required: true,
+          x-example: "!room:example.com"
+      responses:
+        200:
+          description: The members of the room.
+          examples:
+            application/json: |-
+              {
+                "chunk": [
+                  {
+                    "age": 6547561012,
+                    "content": {
+                      "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto",
+                      "displayname": null,
+                      "membership": "join"
+                    },
+                    "event_id": "$1426600438280zExKY:example.com",
+                    "membership": "join",
+                    "origin_server_ts": 1426600438277,
+                    "room_id": "!room:example.com",
+                    "state_key": "@alice:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@alice:example.com"
+                  },
+                  {
+                    "age": 1622568720,
+                    "content": {
+                        "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto",
+                        "displayname": "Bob",
+                        "membership": "join"
+                    },
+                    "event_id": "$1431525430134MxlLX:example.com",
+                    "origin_server_ts": 1431525430569,
+                    "replaces_state": "$142652023736BSXcM:example.com",
+                    "room_id": "!room:example.com",
+                    "state_key": "@bob:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@bob:example.com"
+                  },
+                ]
+              }
+          schema:
+            object:
+            properities:
+              chunk:
+                type: array
+                items:
+                  title: MemberEvent
+                  type: object
+                  allOf:
+                    - "$ref": "../../event-schemas/schema/v1/m.room.member"
+              visibility:
+                type: string
+
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index fe11d199..2c85a4c2 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -631,49 +631,7 @@ Getting events for a room
 
 There are several APIs provided to ``GET`` events for a room:
 
-``/rooms/<room id>/state/<event type>/<state key>``
-  Description:
-    Get the state event identified.
-  Response format:
-    A JSON object representing the state event **content**.
-  Example:
-    ``/rooms/!room:domain.com/state/m.room.name`` returns ``{ "name": "Room name" }``
-
-|/rooms/<room_id>/state|_
-  Description:
-    Get all state events for a room.
-  Response format:
-    ``[ { state event }, { state event }, ... ]``
-  Example:
-    TODO-doc
-
-|/rooms/<room_id>/members|_
-  Description:
-    Get all ``m.room.member`` state events.
-  Response format:
-    ``{ "start": "<token>", "end": "<token>", "chunk": [ { m.room.member event }, ... ] }``
-  Example:
-    TODO-doc
-
-|/rooms/<room_id>/messages|_
-  Description:
-    Get all events from the room's timeline. This API supports
-    pagination using ``from`` and ``to`` query parameters, coupled with the
-    ``start`` and ``end`` tokens from an |initialSync|_ API.
-    
-  Response format:
-    ``{ "start": "<token>", "end": "<token>" }``
-  Example:
-    TODO-doc
-
-|/rooms/<room_id>/initialSync|_
-  Description:
-    Get all relevant events for a room. This includes state events, paginated
-    non-state events and presence events.
-  Response format:
-    `` { TODO-doc } ``
-  Example:
-    TODO-doc
+{{rooms_http_api}}
 
 Redactions
 ~~~~~~~~~~
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 729157bb..2a591c11 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -102,6 +102,12 @@ class MatrixSections(Sections):
             title_kind="~"
         )
 
+    def render_rooms_http_api(self):
+        return self._render_http_api_group(
+            "rooms",
+            title_kind="+"
+        )
+
     def render_room_events(self):
         def filterFn(eventType):
             return (
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 0c072df3..98f0f3a9 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -149,8 +149,8 @@ class MatrixUnits(Units):
                     # object with some keys; we'll add entries f.e one)
                     if "schema" not in param:
                         raise Exception(
-                            "API endpoint group=%s path=%s method=%s param=%s"+
-                            " has no valid parameter value." % (
+                            ("API endpoint group=%s path=%s method=%s param=%s"+
+                            " has no valid parameter value.") % (
                                 group_name, path, method, param
                             )
                         )

From 2e9d3d283a351dae22bf2d88e8dfed821442d6de Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 15 Sep 2015 16:23:19 +0100
Subject: [PATCH 11/85] Swagger refresh tokens

---
 api/client-server/v1/login.yaml | 71 ++++++++++++++++++++++++++++++++-
 1 file changed, 70 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index 2df695be..0852db6b 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -63,7 +63,19 @@ paths:
                 description: The fully-qualified Matrix ID that has been registered.
               access_token:
                 type: string
-                description: An access token for the account. This access token can then be used to authorize other requests.
+                description: |-
+                  An access token for the account.
+                  This access token can then be used to authorize other requests.
+                  The access token may expire at some point, and if so, it SHOULD come with a refresh_token.
+                  There is no specific error message to indicate that a request has failed because
+                  an access token has expired; instead, if a client has reason to believe its
+                  access token is valid, and it receives an auth error, they should attempt to
+                  refresh for a new token on failure, and retry the request with the new token.
+              refresh_token:
+                type: string
+                # TODO: Work out how to linkify /tokenrefresh
+                description: |-
+                  (optional) A ``refresh_token`` may be exchanged for a new ``access_token`` using the /tokenrefresh API endpoint.
               home_server:
                 type: string
                 description: The hostname of the Home Server on which the account has been registered.
@@ -77,3 +89,60 @@ paths:
           description: This request was rate-limited.
           schema:
             "$ref": "definitions/error.yaml"
+  "/tokenrefresh":
+    post:
+      summary: Exchanges a refresh token for an access token.
+      description: |-
+        Exchanges a refresh token for a new access token.
+        This is intended to be used if the access token has expired.
+      security:
+        - accessToken: []
+      parameters:
+        - in: body
+          name: body
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "refresh_token": "a1b2c3"
+              }
+            properties:
+              refresh_token:
+                type: string
+                description: The refresh token which was issued by the server.
+            required: ["refresh_token"]
+      responses:
+        200:
+          description: |-
+            The refresh token was accepted, and a new access token has been issued.
+            The passed refresh token is no longer valid, and cannot be used.
+            A new refresh token may have been returned.
+          examples:
+            application/json: |-
+              {
+                "access_token": "bearwithme123",
+                "refresh_token": "exchangewithme987"
+              }
+          schema:
+            type: object
+            properties:
+              access_token:
+                type: string
+                description: |-
+                  An access token for the account.
+                  This access token can then be used to authorize other requests.
+                  The access token may expire at some point, and if so, it SHOULD come with a refresh_token.
+              refresh_token:
+                type: string
+                description: (optional) A ``refresh_token`` may be exchanged for a new ``access_token`` using the TODO Linkify /tokenrefresh API endpoint.
+        403:
+          description: |-
+            The exchange attempt failed. For example, the refresh token may have already been used.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN"}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"

From 21411309e0569ffa8a0b95bff06e1e989821de2d Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 15 Sep 2015 16:27:10 +0100
Subject: [PATCH 12/85] Fix chdir path

---
 scripts/gendoc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 10c3fa90..ed172726 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -8,7 +8,7 @@ import shutil
 import subprocess
 import sys
 
-os.chdir(os.path.dirname(__file__))
+os.chdir(os.path.dirname(os.path.abspath(__file__)))
 
 stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css"]

From 6147562d14a0efb7fe14b64c1cf8637a0c86bf8f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 10:09:37 +0100
Subject: [PATCH 13/85] Split the "core" definition files into separate files.
 To make it easier to use the schema files with tools that don't support deep
 links.

---
 event-schemas/schema/v1/core                  | 88 -------------------
 event-schemas/schema/v1/core/event.json       | 24 +++++
 .../v1/core/msgtype_infos/image_info.json     | 23 +++++
 event-schemas/schema/v1/core/room_event.json  | 16 ++++
 event-schemas/schema/v1/core/state_event.json | 20 +++++
 event-schemas/schema/v1/m.call.answer         |  2 +-
 event-schemas/schema/v1/m.call.candidates     |  2 +-
 event-schemas/schema/v1/m.call.hangup         |  2 +-
 event-schemas/schema/v1/m.call.invite         |  2 +-
 event-schemas/schema/v1/m.room.aliases        |  2 +-
 .../schema/v1/m.room.canonical_alias          |  2 +-
 event-schemas/schema/v1/m.room.create         |  2 +-
 .../schema/v1/m.room.history_visibility       |  2 +-
 event-schemas/schema/v1/m.room.join_rules     |  2 +-
 event-schemas/schema/v1/m.room.member         |  2 +-
 event-schemas/schema/v1/m.room.message        |  2 +-
 .../schema/v1/m.room.message#m.audio          |  2 +-
 .../schema/v1/m.room.message#m.emote          |  2 +-
 event-schemas/schema/v1/m.room.message#m.file |  4 +-
 .../schema/v1/m.room.message#m.image          |  4 +-
 .../schema/v1/m.room.message#m.location       |  4 +-
 .../schema/v1/m.room.message#m.notice         |  2 +-
 event-schemas/schema/v1/m.room.message#m.text |  2 +-
 .../schema/v1/m.room.message#m.video          |  4 +-
 .../schema/v1/m.room.message.feedback         |  2 +-
 event-schemas/schema/v1/m.room.name           |  2 +-
 event-schemas/schema/v1/m.room.power_levels   |  2 +-
 event-schemas/schema/v1/m.room.redaction      |  2 +-
 event-schemas/schema/v1/m.room.topic          |  2 +-
 29 files changed, 111 insertions(+), 116 deletions(-)
 delete mode 100644 event-schemas/schema/v1/core
 create mode 100644 event-schemas/schema/v1/core/event.json
 create mode 100644 event-schemas/schema/v1/core/msgtype_infos/image_info.json
 create mode 100644 event-schemas/schema/v1/core/room_event.json
 create mode 100644 event-schemas/schema/v1/core/state_event.json

diff --git a/event-schemas/schema/v1/core b/event-schemas/schema/v1/core
deleted file mode 100644
index ed374a69..00000000
--- a/event-schemas/schema/v1/core
+++ /dev/null
@@ -1,88 +0,0 @@
-{
-  "$schema": "http://json-schema.org/draft-04/schema#",
-  "definitions": {
-    "event": {
-      "title": "Event",
-      "description": "The basic set of fields all events must have.",
-      "type": "object",
-      "properties": {
-        "event_id": {
-          "type": "string",
-          "description": "The globally unique event identifier."
-        },
-        "user_id": {
-          "type": "string",
-          "description": "Contains the fully-qualified ID of the user who *sent* this event."
-        },
-        "content": {
-          "type": "object",
-          "description": "The fields in this object will vary depending on the type of event. When interacting with the REST API, this is the HTTP body."
-        },
-        "type": {
-          "type": "string",
-          "description": "The type of event. This SHOULD be namespaced similar to Java package naming conventions e.g. 'com.example.subdomain.event.type'"
-        }
-      },
-      "required": ["event_id", "user_id", "content", "type"]
-    },
-    "room_event": {
-        "type": "object",
-        "title": "Room Event",
-        "description": "In addition to the Event fields, Room Events MUST have the following additional field.",
-        "allOf":[{
-            "$ref": "#/definitions/event"
-        }],
-        "properties": {
-            "room_id": {
-                "type": "string",
-                "description": "The ID of the room associated with this event."
-            }
-        },
-        "required": ["room_id"]
-    },
-    "state_event": {
-        "type": "object",
-        "title": "State Event",
-        "description": "In addition to the Room Event fields, State Events have the following additional fields.",
-        "allOf":[{
-            "$ref": "#/definitions/room_event"
-        }],
-        "properties": {
-            "state_key": {
-                "type": "string",
-                "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
-            },
-            "prev_content": {
-                "type": "object",
-                "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
-            }
-        },
-        "required": ["state_key"]
-    },
-    "msgtype_infos": {
-      "image_info": {
-        "type": "object",
-        "title": "ImageInfo",
-        "description": "Metadata about an image.",
-        "properties": {
-            "size": {
-                "type": "integer",
-                "description": "Size of the image in bytes."
-            },
-            "w": {
-                "type": "integer",
-                "description": "The width of the image in pixels."
-            },
-            "h": {
-                "type": "integer",
-                "description": "The height of the image in pixels."
-            },
-            "mimetype": {
-                "type": "string",
-                "description": "The mimetype of the image, e.g. ``image/jpeg``."
-            }
-        }
-      }
-    }
-  }
-}
diff --git a/event-schemas/schema/v1/core/event.json b/event-schemas/schema/v1/core/event.json
new file mode 100644
index 00000000..b4824079
--- /dev/null
+++ b/event-schemas/schema/v1/core/event.json
@@ -0,0 +1,24 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "type": "object",
+  "title": "Event",
+  "description": "The basic set of fields all events must have.",
+  "properties": {
+    "event_id": {
+      "type": "string",
+      "description": "The globally unique event identifier."
+    },
+    "user_id": {
+      "type": "string",
+      "description": "Contains the fully-qualified ID of the user who *sent* this event."
+    },
+    "content": {
+      "type": "object",
+      "description": "The fields in this object will vary depending on the type of event. When interacting with the REST API, this is the HTTP body."
+    },
+    "type": {
+      "type": "string",
+      "description": "The type of event. This SHOULD be namespaced similar to Java package naming conventions e.g. 'com.example.subdomain.event.type'"
+    }
+  }
+}
diff --git a/event-schemas/schema/v1/core/msgtype_infos/image_info.json b/event-schemas/schema/v1/core/msgtype_infos/image_info.json
new file mode 100644
index 00000000..ee75745e
--- /dev/null
+++ b/event-schemas/schema/v1/core/msgtype_infos/image_info.json
@@ -0,0 +1,23 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "title": "ImageInfo",
+  "description": "Metadata about an image.",
+  "properties": {
+    "size": {
+      "type": "integer",
+      "description": "Size of the image in bytes."
+    },
+    "w": {
+      "type": "integer",
+      "description": "The width of the image in pixels."
+    },
+    "h": {
+      "type": "integer",
+      "description": "The height of the image in pixels."
+    },
+    "mimetype": {
+      "type": "string",
+      "description": "The mimetype of the image, e.g. ``image/jpeg``."
+    }
+  }
+}
diff --git a/event-schemas/schema/v1/core/room_event.json b/event-schemas/schema/v1/core/room_event.json
new file mode 100644
index 00000000..717cdaae
--- /dev/null
+++ b/event-schemas/schema/v1/core/room_event.json
@@ -0,0 +1,16 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "type": "object",
+  "title": "Room Event",
+  "description": "In addition to the Event fields, Room Events MUST have the following additional field.",
+  "allOf":[{
+    "$ref": "core/event.json"
+  }],
+  "properties": {
+    "room_id": {
+      "type": "string",
+      "description": "The ID of the room associated with this event."
+    }
+  },
+  "required": ["room_id"]
+}
diff --git a/event-schemas/schema/v1/core/state_event.json b/event-schemas/schema/v1/core/state_event.json
new file mode 100644
index 00000000..a237f4d1
--- /dev/null
+++ b/event-schemas/schema/v1/core/state_event.json
@@ -0,0 +1,20 @@
+{
+    "$schema": "http://json-schema.org/draft-04/schema#",
+        "type": "object",
+        "title": "State Event",
+        "description": "In addition to the Room Event fields, State Events have the following additional fields.",
+        "allOf":[{
+            "$ref": "core/room_event.json"
+        }],
+        "properties": {
+            "state_key": {
+                "type": "string",
+                "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
+            },
+            "prev_content": {
+                "type": "object",
+                "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
+            }
+        },
+        "required": ["state_key"]
+}
diff --git a/event-schemas/schema/v1/m.call.answer b/event-schemas/schema/v1/m.call.answer
index 598554f0..abdf4a9b 100644
--- a/event-schemas/schema/v1/m.call.answer
+++ b/event-schemas/schema/v1/m.call.answer
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by the callee when they wish to answer the call.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.candidates b/event-schemas/schema/v1/m.call.candidates
index 6475cf51..052ead0b 100644
--- a/event-schemas/schema/v1/m.call.candidates
+++ b/event-schemas/schema/v1/m.call.candidates
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by callers after sending an invite and by the callee after answering. Its purpose is to give the other party additional ICE candidates to try using to communicate.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.hangup b/event-schemas/schema/v1/m.call.hangup
index 3704ec75..383952d8 100644
--- a/event-schemas/schema/v1/m.call.hangup
+++ b/event-schemas/schema/v1/m.call.hangup
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.invite b/event-schemas/schema/v1/m.call.invite
index 2be01466..a852ff43 100644
--- a/event-schemas/schema/v1/m.call.invite
+++ b/event-schemas/schema/v1/m.call.invite
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by the caller when they wish to establish a call.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.aliases b/event-schemas/schema/v1/m.room.aliases
index 6585f13d..79fd259b 100644
--- a/event-schemas/schema/v1/m.room.aliases
+++ b/event-schemas/schema/v1/m.room.aliases
@@ -4,7 +4,7 @@
     "title": "Informs the room about what room aliases it has been given.",
     "description": "This event is sent by a homeserver directly to inform of changes to the list of aliases it knows about for that room. The ``state_key`` for this event is set to the homeserver which owns the room alias. The entire set of known aliases for the room is the union of all the ``m.room.aliases`` events, one for each homeserver. Clients **should** check the validity of any room alias given in this list before presenting it to the user as trusted fact. The lists given by this event should be considered simply as advice on which aliases might exist, for which the client can perform the lookup to confirm whether it receives the correct room ID.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
index 9f044e58..17caa321 100644
--- a/event-schemas/schema/v1/m.room.canonical_alias
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -4,7 +4,7 @@
     "title": "Informs the room as to which alias is the canonical one.",
     "description": "This event is used to inform the room about which alias should be considered the canonical one. This could be for display purposes or as suggestion to users which alias to use to advertise the room.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.create b/event-schemas/schema/v1/m.room.create
index 8f74e69b..ecc2ed63 100644
--- a/event-schemas/schema/v1/m.room.create
+++ b/event-schemas/schema/v1/m.room.create
@@ -4,7 +4,7 @@
     "title": "The first event in the room.",
     "description": "This is the first event in a room and cannot be changed. It acts as the root of all other events.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
index 07bd39ff..48b0a5c4 100644
--- a/event-schemas/schema/v1/m.room.history_visibility
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -4,7 +4,7 @@
     "title": "Controls visibility of history.",
     "description": "This event controls whether a member of a room can see the events that happened in a room from before they joined.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.join_rules b/event-schemas/schema/v1/m.room.join_rules
index 2ba20b89..567247eb 100644
--- a/event-schemas/schema/v1/m.room.join_rules
+++ b/event-schemas/schema/v1/m.room.join_rules
@@ -4,7 +4,7 @@
     "title": "Describes how users are allowed to join the room.",
     "description": "A room may be ``public`` meaning anyone can join the room without any prior action. Alternatively, it can be ``invite`` meaning that a user who wishes to join the room must first receive an invite to the room from someone already inside of the room. Currently, ``knock`` and ``private`` are reserved keywords which are not implemented.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index ab616176..67e10d51 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -4,7 +4,7 @@
     "title": "The current membership state of a user in the room.",
     "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message
index a849f07f..61b6256f 100644
--- a/event-schemas/schema/v1/m.room.message
+++ b/event-schemas/schema/v1/m.room.message
@@ -4,7 +4,7 @@
     "title": "Message",
     "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.audio b/event-schemas/schema/v1/m.room.message#m.audio
index 9910cb0c..b236e5b0 100644
--- a/event-schemas/schema/v1/m.room.message#m.audio
+++ b/event-schemas/schema/v1/m.room.message#m.audio
@@ -4,7 +4,7 @@
     "title": "AudioMessage",
     "description": "This message represents a single audio clip.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.emote b/event-schemas/schema/v1/m.room.message#m.emote
index 886abafc..9f17180c 100644
--- a/event-schemas/schema/v1/m.room.message#m.emote
+++ b/event-schemas/schema/v1/m.room.message#m.emote
@@ -4,7 +4,7 @@
     "title": "EmoteMessage",
     "description": "This message is similar to ``m.text`` except that the sender is 'performing' the action contained in the ``body`` key, similar to ``/me`` in IRC. This message should be prefixed by the name of the sender. This message could also be represented in a different colour to distinguish it from regular ``m.text`` messages.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.file b/event-schemas/schema/v1/m.room.message#m.file
index cf6d6552..d8c63b33 100644
--- a/event-schemas/schema/v1/m.room.message#m.file
+++ b/event-schemas/schema/v1/m.room.message#m.file
@@ -4,7 +4,7 @@
     "title": "FileMessage",
     "description": "This message represents a generic file.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
@@ -50,7 +50,7 @@
                     "title": "ImageInfo",
                     "description": "Metadata about the image referred to in ``thumbnail_url``.",
                     "allOf": [{
-                        "$ref": "core#/definitions/msgtype_infos/image_info"
+                        "$ref": "core/msgtype_infos/image_info.json"
                     }]
                 }
             },
diff --git a/event-schemas/schema/v1/m.room.message#m.image b/event-schemas/schema/v1/m.room.message#m.image
index 8c010b0d..ca22654f 100644
--- a/event-schemas/schema/v1/m.room.message#m.image
+++ b/event-schemas/schema/v1/m.room.message#m.image
@@ -4,7 +4,7 @@
     "title": "ImageMessage",
     "description": "This message represents a single image and an optional thumbnail.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
@@ -31,7 +31,7 @@
                     "title": "ImageInfo",
                     "description": "Metadata about the image referred to in ``thumbnail_url``.",
                     "allOf": [{
-                        "$ref": "core#/definitions/msgtype_infos/image_info"
+                        "$ref": "core/msgtype_infos/image_info.json"
                     }]
                 },
                 "info": {
diff --git a/event-schemas/schema/v1/m.room.message#m.location b/event-schemas/schema/v1/m.room.message#m.location
index cd15f31e..919c3325 100644
--- a/event-schemas/schema/v1/m.room.message#m.location
+++ b/event-schemas/schema/v1/m.room.message#m.location
@@ -4,7 +4,7 @@
     "title": "LocationMessage",
     "description": "This message represents a real-world location.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
@@ -30,7 +30,7 @@
                     "type": "object",
                     "title": "ImageInfo",
                     "allOf": [{
-                        "$ref": "core#/definitions/msgtype_infos/image_info"
+                        "$ref": "core/msgtype_infos/image_info.json"
                     }]
                 }
             },
diff --git a/event-schemas/schema/v1/m.room.message#m.notice b/event-schemas/schema/v1/m.room.message#m.notice
index 0f8044f5..e6ce6f89 100644
--- a/event-schemas/schema/v1/m.room.message#m.notice
+++ b/event-schemas/schema/v1/m.room.message#m.notice
@@ -4,7 +4,7 @@
     "title": "NoticeMessage",
     "description": "A m.notice message should be considered similar to a plain m.text message except that clients should visually distinguish it in some way. It is intended to be used by automated clients, such as bots, bridges, and other entities, rather than humans. Additionally, such automated agents which watch a room for messages and respond to them ought to ignore m.notice messages. This helps to prevent infinite-loop situations where two automated clients continuously exchange messages, as each responds to the other.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.text b/event-schemas/schema/v1/m.room.message#m.text
index 054d4a96..ceadbcd0 100644
--- a/event-schemas/schema/v1/m.room.message#m.text
+++ b/event-schemas/schema/v1/m.room.message#m.text
@@ -4,7 +4,7 @@
     "title": "TextMessage",
     "description": "This message is the most basic message and is used to represent text.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.video b/event-schemas/schema/v1/m.room.message#m.video
index 22266506..667832ff 100644
--- a/event-schemas/schema/v1/m.room.message#m.video
+++ b/event-schemas/schema/v1/m.room.message#m.video
@@ -4,7 +4,7 @@
     "title": "VideoMessage",
     "description": "This message represents a single video clip.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
@@ -55,7 +55,7 @@
                             "type": "object",
                             "title": "ImageInfo",
                             "allOf": [{
-                                "$ref": "core#/definitions/msgtype_infos/image_info"
+                                "$ref": "core/msgtype_infos/image_info.json"
                             }]
                         }
                     }
diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback
index 4cfd44d1..b662e9e6 100644
--- a/event-schemas/schema/v1/m.room.message.feedback
+++ b/event-schemas/schema/v1/m.room.message.feedback
@@ -4,7 +4,7 @@
     "title": "MessageFeedback",
     "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.name b/event-schemas/schema/v1/m.room.name
index 5077774c..5565147c 100644
--- a/event-schemas/schema/v1/m.room.name
+++ b/event-schemas/schema/v1/m.room.name
@@ -4,7 +4,7 @@
     "description": "A room has an opaque room ID which is not human-friendly to read. A room alias is human-friendly, but not all rooms have room aliases. The room name is a human-friendly string designed to be displayed to the end-user. The room name is not unique, as multiple rooms can have the same room name set. The room name can also be set when creating a room using ``/createRoom`` with the ``name`` key.",
     "type": "object",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.power_levels b/event-schemas/schema/v1/m.room.power_levels
index 9e0cb81a..95850fcb 100644
--- a/event-schemas/schema/v1/m.room.power_levels
+++ b/event-schemas/schema/v1/m.room.power_levels
@@ -4,7 +4,7 @@
     "title": "Defines the power levels (privileges) of users in the room.",
     "description": "This event specifies the minimum level a user must have in order to perform a certain action. It also specifies the levels of each user in the room. If a ``user_id`` is in the ``users`` list, then that ``user_id`` has the associated power level. Otherwise they have the default level ``users_default``. If ``users_default`` is not supplied, it is assumed to be 0. The level required to send a certain event is governed by ``events``, ``state_default`` and ``events_default``. If an event type is specified in ``events``, then the user must have at least the level specified in order to send that event. If the event type is not supplied, it defaults to ``events_default`` for Message Events and ``state_default`` for State Events.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.redaction b/event-schemas/schema/v1/m.room.redaction
index 2896c0da..b095b17d 100644
--- a/event-schemas/schema/v1/m.room.redaction
+++ b/event-schemas/schema/v1/m.room.redaction
@@ -4,7 +4,7 @@
     "title": "Redaction",
     "description": "Events can be redacted by either room or server admins. Redacting an event means that all keys not required by the protocol are stripped off, allowing admins to remove offensive or illegal content that may have been attached to any event. This cannot be undone, allowing server owners to physically delete the offending data.  There is also a concept of a moderator hiding a message event, which can be undone, but cannot be applied to state events. The event that has been redacted is specified in the ``redacts`` event level key.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.topic b/event-schemas/schema/v1/m.room.topic
index 266df8fa..32319c38 100644
--- a/event-schemas/schema/v1/m.room.topic
+++ b/event-schemas/schema/v1/m.room.topic
@@ -4,7 +4,7 @@
     "title": "Topic",
     "description": "A topic is a short message detailing what is currently being discussed in the room.  It can also be used as a way to display extra information about the room, which may not be suitable for the room name. The room topic can also be set when creating a room using ``/createRoom`` with the ``topic`` key.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {

From 2cf8da6b20dabe3d97ac9c97a6d356b4aa2c4d5d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 10:28:57 +0100
Subject: [PATCH 14/85] Update the gendoc script to load the core event schema
 from separate files.

---
 templating/matrix_templates/units.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 0c072df3..6bb9b7d7 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -283,17 +283,25 @@ class MatrixUnits(Units):
     def load_common_event_fields(self):
         path = "../event-schemas/schema/v1/core"
         event_types = {}
-        with open(path, "r") as f:
-            core_json = json.loads(f.read())
-            for event_type in core_json["definitions"]:
+
+        for (root, dirs, files) in os.walk(path):
+            for filename in files:
+                if not filename.endswith(".json"):
+                    continue
+
+                event_type = filename[:-5]  # strip the ".json"
+                with open(os.path.join(root, filename)) as f:
+                    event_info = json.loads(f.read())
+
                 if "event" not in event_type:
                     continue  # filter ImageInfo and co
-                event_info = core_json["definitions"][event_type]
+
                 table = {
                     "title": event_info["title"],
                     "desc": event_info["description"],
                     "rows": []
                 }
+
                 for prop in sorted(event_info["properties"]):
                     row = {
                         "key": prop,
@@ -301,6 +309,7 @@ class MatrixUnits(Units):
                         "desc": event_info["properties"][prop].get("description","")
                     }
                     table["rows"].append(row)
+
                 event_types[event_type] = table
         return event_types
 

From c49338006aa1fe8da43185ea77afa7510673c99d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 13:09:35 +0100
Subject: [PATCH 15/85] replace definitions/*event.yaml with symlinks to
 /event-schemas/

---
 api/client-server/v1/core                     |   1 +
 api/client-server/v1/definitions/event.yaml   |   7 ----
 .../v1/definitions/room_event.yaml            |   9 -----
 .../v1/definitions/state_event.yaml           |  11 ------
 api/client-server/v1/events                   |   1 +
 api/client-server/v1/presence.yaml            |   2 +-
 api/client-server/v1/sync.yaml                |  10 ++---
 .../schema/v1/core/.room_event.json.swp       | Bin 0 -> 12288 bytes
 event-schemas/schema/v1/core/event.json       |   9 -----
 event-schemas/schema/v1/core/room_event.json  |  11 +++++-
 event-schemas/schema/v1/core/state_event.json |  35 +++++++++---------
 11 files changed, 34 insertions(+), 62 deletions(-)
 create mode 120000 api/client-server/v1/core
 delete mode 100644 api/client-server/v1/definitions/event.yaml
 delete mode 100644 api/client-server/v1/definitions/room_event.yaml
 delete mode 100644 api/client-server/v1/definitions/state_event.yaml
 create mode 120000 api/client-server/v1/events
 create mode 100644 event-schemas/schema/v1/core/.room_event.json.swp

diff --git a/api/client-server/v1/core b/api/client-server/v1/core
new file mode 120000
index 00000000..9fc0cd87
--- /dev/null
+++ b/api/client-server/v1/core
@@ -0,0 +1 @@
+events/core
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/event.yaml b/api/client-server/v1/definitions/event.yaml
deleted file mode 100644
index cfb1b924..00000000
--- a/api/client-server/v1/definitions/event.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-type: object
-description: A Matrix Event
-properties:
-  event_id:
-    type: string
-    description: An event ID.
-required: ["event_id"]
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/room_event.yaml b/api/client-server/v1/definitions/room_event.yaml
deleted file mode 100644
index 5a9e3a09..00000000
--- a/api/client-server/v1/definitions/room_event.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-type: object
-description: A Matrix Room Event
-properties:
-  event_id:
-    type: string
-    description: An event ID.
-  room_id:
-    type: string
-required: ["event_id", "room_id"]
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/state_event.yaml b/api/client-server/v1/definitions/state_event.yaml
deleted file mode 100644
index f8e6f4d4..00000000
--- a/api/client-server/v1/definitions/state_event.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-type: object
-description: A Matrix State Event
-properties:
-  event_id:
-    type: string
-    description: An event ID.
-  room_id:
-    type: string
-  state_key:
-    type: string
-required: ["event_id", "room_id", "state_key"]
\ No newline at end of file
diff --git a/api/client-server/v1/events b/api/client-server/v1/events
new file mode 120000
index 00000000..7a0d0326
--- /dev/null
+++ b/api/client-server/v1/events
@@ -0,0 +1 @@
+../../../event-schemas/schema/v1
\ No newline at end of file
diff --git a/api/client-server/v1/presence.yaml b/api/client-server/v1/presence.yaml
index 6f8311e4..472a2d7f 100644
--- a/api/client-server/v1/presence.yaml
+++ b/api/client-server/v1/presence.yaml
@@ -205,4 +205,4 @@ paths:
               type: object
               title: PresenceEvent
               allOf:
-                - "$ref": "definitions/event.yaml"
+                - "$ref": "events/core/event.json"
diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 777e27db..b9525491 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -82,7 +82,7 @@ paths:
                   type: object
                   title: RoomEvent
                   allOf:
-                   - "$ref": "definitions/room_event.yaml"
+                   - "$ref": "events/core/room_event.json"
         400:
           description: "Bad pagination ``from`` parameter."
   "/initialSync":
@@ -253,7 +253,7 @@ paths:
                   type: object
                   title: Event
                   allOf:
-                   - "$ref": "definitions/event.yaml"
+                   - "$ref": "events/core/event.json"
               rooms:
                 type: array
                 items:
@@ -294,7 +294,7 @@ paths:
                             type: object
                             title: RoomEvent
                             allOf:
-                              - "$ref": "definitions/room_event.yaml"
+                              - "$ref": "events/core/room_event.json"
                       required: ["start", "end", "chunk"]
                     state:
                       type: array
@@ -307,7 +307,7 @@ paths:
                         title: StateEvent
                         type: object
                         allOf:
-                          - "$ref": "definitions/state_event.yaml"
+                          - "$ref": "events/core/state_event.json"
                     visibility:
                       type: string
                       enum: ["private", "public"]
@@ -350,6 +350,6 @@ paths:
               }
           schema:
             allOf:
-              - "$ref": "definitions/event.yaml"
+              - "$ref": "events/core/event.json"
         404:
           description: The event was not found or you do not have permission to read this event.
diff --git a/event-schemas/schema/v1/core/.room_event.json.swp b/event-schemas/schema/v1/core/.room_event.json.swp
new file mode 100644
index 0000000000000000000000000000000000000000..fce0fd5f873a95140af3b68ffd4843fa3cb522c4
GIT binary patch
literal 12288
zcmeI2&2G~`5XZN0E}$a8g~PA~AyrM>@S#fOz=01bhpG}ZNT{l+jlGUHI9_LWolsQ~
z2i^b>7oH&!2e|PPyatcZ17g;85~+t!j-XlTpS3;XotfX9tW0<7WqTL4oA(*6D~#>E
zKkeL}-nlq!F*Xq9bwBZj<tr=Pa);U)SQ7?4<&vTs41_c4O+%aB>@w0}AV)HD4I3tM
zAZ$?W!Kf94+Q`6YJvb^ontiLYbA==)B0vP@1XkI-yR9vMyUuUI)wRytLKj4U2oM1x
zKm>>Y5g-CYfC&8m1YEwt-eQ>->Qb-Q`^uTUzNL)_5CI}U1c(3;AOb{y2oM1xKm>>Y
z5%>oQh={QptBid_`TzfC-~YcZGxihp0~MoQp&p_hpw>}qsMAY~eML=CpHc5n&rpw0
zP1H5icRcqS>I>=<%E$SDqD2IV01+SpM1Tko0U|&IhyW2F0*p-<05@`2C?g}j0|&eu
zeO3`aM9X9of94x{M$(2x<xc4go%v&(IiWHOE|CxyX?olk79v%#k`c6@fR0N`VWoj_
zqG7}0%=HF7Qx>Y}YF-AwlihJH{gD=DwX+_t0(1Z75hk+(C(AS0#{_z*?h4ESicAd)
zSw#R9VN9QqY5pNtnbEnUEz0x6Kv=6oB^(w$#<BCisBs3os*&eL=h8SOZC&-`u!eEV
z$k-z)C)5|i>AotCb?>3^z+*bwrFI4)ij;2}I9+z+sULm^gpx*f6Q275%10J<J1_Pj
q5hGdp#X3#(7#lJ_L8R4d&9X(@DVLUgTbxlF>VE2OUxp5!ov`26(Di%(

literal 0
HcmV?d00001

diff --git a/event-schemas/schema/v1/core/event.json b/event-schemas/schema/v1/core/event.json
index b4824079..e73aec80 100644
--- a/event-schemas/schema/v1/core/event.json
+++ b/event-schemas/schema/v1/core/event.json
@@ -1,17 +1,8 @@
 {
-  "$schema": "http://json-schema.org/draft-04/schema#",
   "type": "object",
   "title": "Event",
   "description": "The basic set of fields all events must have.",
   "properties": {
-    "event_id": {
-      "type": "string",
-      "description": "The globally unique event identifier."
-    },
-    "user_id": {
-      "type": "string",
-      "description": "Contains the fully-qualified ID of the user who *sent* this event."
-    },
     "content": {
       "type": "object",
       "description": "The fields in this object will vary depending on the type of event. When interacting with the REST API, this is the HTTP body."
diff --git a/event-schemas/schema/v1/core/room_event.json b/event-schemas/schema/v1/core/room_event.json
index 717cdaae..a64ba2ea 100644
--- a/event-schemas/schema/v1/core/room_event.json
+++ b/event-schemas/schema/v1/core/room_event.json
@@ -1,5 +1,4 @@
 {
-  "$schema": "http://json-schema.org/draft-04/schema#",
   "type": "object",
   "title": "Room Event",
   "description": "In addition to the Event fields, Room Events MUST have the following additional field.",
@@ -10,7 +9,15 @@
     "room_id": {
       "type": "string",
       "description": "The ID of the room associated with this event."
-    }
+    },
+    "event_id": {
+      "type": "string",
+      "description": "The globally unique event identifier."
+    },
+    "user_id": {
+      "type": "string",
+      "description": "Contains the fully-qualified ID of the user who *sent* this event."
+    },
   },
   "required": ["room_id"]
 }
diff --git a/event-schemas/schema/v1/core/state_event.json b/event-schemas/schema/v1/core/state_event.json
index a237f4d1..60de9413 100644
--- a/event-schemas/schema/v1/core/state_event.json
+++ b/event-schemas/schema/v1/core/state_event.json
@@ -1,20 +1,19 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
-        "type": "object",
-        "title": "State Event",
-        "description": "In addition to the Room Event fields, State Events have the following additional fields.",
-        "allOf":[{
-            "$ref": "core/room_event.json"
-        }],
-        "properties": {
-            "state_key": {
-                "type": "string",
-                "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
-            },
-            "prev_content": {
-                "type": "object",
-                "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
-            }
-        },
-        "required": ["state_key"]
+  "type": "object",
+  "title": "State Event",
+  "description": "In addition to the Room Event fields, State Events have the following additional fields.",
+  "allOf":[{
+    "$ref": "core/room_event.json"
+  }],
+  "properties": {
+    "state_key": {
+      "type": "string",
+      "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
+    },
+    "prev_content": {
+      "type": "object",
+      "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
+    }
+  },
+  "required": ["state_key"]
 }

From 380f186273b4b7e045c76b0078b091063347aef6 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 14:42:43 +0100
Subject: [PATCH 16/85] Log which file a json parse error occurred in

---
 templating/matrix_templates/units.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 6bb9b7d7..4b8127ae 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -290,8 +290,14 @@ class MatrixUnits(Units):
                     continue
 
                 event_type = filename[:-5]  # strip the ".json"
-                with open(os.path.join(root, filename)) as f:
-                    event_info = json.loads(f.read())
+                filepath = os.path.join(root, filename)
+                with open(filepath) as f:
+                    try:
+                        event_info = json.load(f)
+                    except Exception as e:
+                        raise ValueError(
+                            "Error reading file %r" % (filepath,), e
+                        )
 
                 if "event" not in event_type:
                     continue  # filter ImageInfo and co

From d0b018c46ecee5ee01d11a3c244d5e4fd3e30bef Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 14:43:09 +0100
Subject: [PATCH 17/85] Fix syntax error in JSON

---
 .../schema/v1/core/.room_event.json.swp       | Bin 12288 -> 12288 bytes
 event-schemas/schema/v1/core/room_event.json  |   2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/core/.room_event.json.swp b/event-schemas/schema/v1/core/.room_event.json.swp
index fce0fd5f873a95140af3b68ffd4843fa3cb522c4..8140e9ceb080ae0bf455a04a57e8d6966854a472 100644
GIT binary patch
delta 73
zcmZojXh@JsG6?hZRWR2xVE_UF28JgGe}$^8RcAgcyHV<uJgY1(1H;tGoC?vr+kmW%
XKs+5Jvbj(}g?I84UA4`-bzPYN>!=h&

delta 77
zcmZojXh@JsG6?hZRWR2xVE_UF1_u90zd}{lsxz-O+$i-*o>hjIfnmyIPK9XRtw7cW
bAf5&i*<7fg!aI4IuIgkLJ&DbiboH135Umu8

diff --git a/event-schemas/schema/v1/core/room_event.json b/event-schemas/schema/v1/core/room_event.json
index a64ba2ea..7ac66b30 100644
--- a/event-schemas/schema/v1/core/room_event.json
+++ b/event-schemas/schema/v1/core/room_event.json
@@ -17,7 +17,7 @@
     "user_id": {
       "type": "string",
       "description": "Contains the fully-qualified ID of the user who *sent* this event."
-    },
+    }
   },
   "required": ["room_id"]
 }

From fd3401fc7a78b3ccbfd9825f01272225d7858407 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 14:44:17 +0100
Subject: [PATCH 18/85] Remove accidentally committed swp file

---
 .../schema/v1/core/.room_event.json.swp         | Bin 12288 -> 0 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)
 delete mode 100644 event-schemas/schema/v1/core/.room_event.json.swp

diff --git a/event-schemas/schema/v1/core/.room_event.json.swp b/event-schemas/schema/v1/core/.room_event.json.swp
deleted file mode 100644
index 8140e9ceb080ae0bf455a04a57e8d6966854a472..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 12288
zcmeI2&1w`u5XU=uj!FbQ1<H;ngzQe@2SPk}@B{V`1kDN(Lg=08+39t5dbhi0*ANyD
zK7xn`pCJc9JbCc}JopZte1sgt>YmwMh!EpZsfPdb%v4QR)vr#=R%iWLYYXl)Z!ugK
z82kR^sD175`q{6`jP-?i(aXHy<W-dJWQW=sSrbQH<+8F9^@TI)WkZ|p^fu9PBuBDv
z4I5{&FKkrp(P$%zwULq0y1y4ZnmwzFQ-vfaB0vOY1QyxN8yoBXafM%ji%ad9g>Hxd
z5g-CYfCvx)B0vO)01^2A3An)mdxd45sY|_F?+eHF`jIvwKm>>Y5g-CYfCvx)B0vO)
z01+SpMBpDJAQHxw7a4nt^8f!&fB*kH&)5&tH&lvxj=GDwjaor1p^nZm_7Qc6dXIXI
zdW5=%YN9ToKI6HcP#;k5P(IEZ6fGh^1c(3;AOb{y2oM1xKm>>Y5nyaGVF297VX2Hv
z_$KV|F#D_$zKfR08eVV<S9DCIjg1;Or3>`s_jTcfDlE86LR#keeq&gQT%}4T(0Twm
z4VKbM1LI7?s>PY>ReYx`RMXXbda~OeNPncoS?!|BtGL{Mxre!I!a;DX^Z7lKZA_q>
z>yE%2pe)p|lvM;!3C8ppndTpYg^W%eZC;)S2Etk$E8(#4F^--8MU7M5RgHXL^gtS?
zq^+x-?A9=@8ku@TVM2XAobIdQSodQz?s!bcyVNQ`B#H7(1E)hr9{S;TKqz@)*WigS
yAiT1$)qc7SnHb67m+Cy%V{FLm1d&&tHA9QIQ!WpEo1ak|>VE1@PsR@49k5>;A@-yI


From 9b18fd252ebab7244c3ebdd005e013d54b3a263a Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 15:46:37 +0100
Subject: [PATCH 19/85] Log port being listened on

---
 scripts/continuserv/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index e7757c06..dc7705a8 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -56,7 +56,7 @@ func main() {
 	go doPopulate(ch, dir)
 
 	go watchFS(ch, w)
-
+    fmt.Printf("Listening on port %d\n", *port)
 	http.HandleFunc("/", serve)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 

From 9c3a333a6dc1cad4decd8e38c2b64feea6506131 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 15:49:01 +0100
Subject: [PATCH 20/85] Apparently tabs are A Thing

---
 scripts/continuserv/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index dc7705a8..658ae0fb 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -56,7 +56,7 @@ func main() {
 	go doPopulate(ch, dir)
 
 	go watchFS(ch, w)
-    fmt.Printf("Listening on port %d\n", *port)
+	fmt.Printf("Listening on port %d\n", *port)
 	http.HandleFunc("/", serve)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 

From 654ed9b99e7dfae5cd227e192c5a5ace111dacf0 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 16:00:06 +0100
Subject: [PATCH 21/85] Remove  keys from the m.room.* schema files so that
 they may be used in the node swagger validator

---
 event-schemas/schema/v1/m.call.answer             | 1 -
 event-schemas/schema/v1/m.call.candidates         | 1 -
 event-schemas/schema/v1/m.call.hangup             | 1 -
 event-schemas/schema/v1/m.call.invite             | 1 -
 event-schemas/schema/v1/m.presence                | 1 -
 event-schemas/schema/v1/m.receipt                 | 1 -
 event-schemas/schema/v1/m.room.aliases            | 1 -
 event-schemas/schema/v1/m.room.canonical_alias    | 1 -
 event-schemas/schema/v1/m.room.create             | 1 -
 event-schemas/schema/v1/m.room.history_visibility | 1 -
 event-schemas/schema/v1/m.room.join_rules         | 1 -
 event-schemas/schema/v1/m.room.member             | 1 -
 event-schemas/schema/v1/m.room.message            | 1 -
 event-schemas/schema/v1/m.room.message#m.audio    | 1 -
 event-schemas/schema/v1/m.room.message#m.emote    | 1 -
 event-schemas/schema/v1/m.room.message#m.file     | 1 -
 event-schemas/schema/v1/m.room.message#m.image    | 1 -
 event-schemas/schema/v1/m.room.message#m.location | 1 -
 event-schemas/schema/v1/m.room.message#m.notice   | 1 -
 event-schemas/schema/v1/m.room.message#m.text     | 1 -
 event-schemas/schema/v1/m.room.message#m.video    | 1 -
 event-schemas/schema/v1/m.room.message.feedback   | 1 -
 event-schemas/schema/v1/m.room.name               | 1 -
 event-schemas/schema/v1/m.room.power_levels       | 1 -
 event-schemas/schema/v1/m.room.redaction          | 1 -
 event-schemas/schema/v1/m.room.topic              | 1 -
 26 files changed, 26 deletions(-)

diff --git a/event-schemas/schema/v1/m.call.answer b/event-schemas/schema/v1/m.call.answer
index abdf4a9b..3b2c48d4 100644
--- a/event-schemas/schema/v1/m.call.answer
+++ b/event-schemas/schema/v1/m.call.answer
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "description": "This event is sent by the callee when they wish to answer the call.",
     "allOf": [{
diff --git a/event-schemas/schema/v1/m.call.candidates b/event-schemas/schema/v1/m.call.candidates
index 052ead0b..f3c86296 100644
--- a/event-schemas/schema/v1/m.call.candidates
+++ b/event-schemas/schema/v1/m.call.candidates
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "description": "This event is sent by callers after sending an invite and by the callee after answering. Its purpose is to give the other party additional ICE candidates to try using to communicate.",
     "allOf": [{
diff --git a/event-schemas/schema/v1/m.call.hangup b/event-schemas/schema/v1/m.call.hangup
index 383952d8..0f253ef9 100644
--- a/event-schemas/schema/v1/m.call.hangup
+++ b/event-schemas/schema/v1/m.call.hangup
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call.",
     "allOf": [{
diff --git a/event-schemas/schema/v1/m.call.invite b/event-schemas/schema/v1/m.call.invite
index a852ff43..393e2455 100644
--- a/event-schemas/schema/v1/m.call.invite
+++ b/event-schemas/schema/v1/m.call.invite
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "description": "This event is sent by the caller when they wish to establish a call.",
     "allOf": [{
diff --git a/event-schemas/schema/v1/m.presence b/event-schemas/schema/v1/m.presence
index bb69ce40..79852ac6 100644
--- a/event-schemas/schema/v1/m.presence
+++ b/event-schemas/schema/v1/m.presence
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Presence Event",
     "description": "Informs the client of a user's presence state change.",
diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt
index 4bd9f17b..0f365eed 100644
--- a/event-schemas/schema/v1/m.receipt
+++ b/event-schemas/schema/v1/m.receipt
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Receipt Event",
     "description": "Informs the client of new receipts.",
diff --git a/event-schemas/schema/v1/m.room.aliases b/event-schemas/schema/v1/m.room.aliases
index 79fd259b..38fd7d18 100644
--- a/event-schemas/schema/v1/m.room.aliases
+++ b/event-schemas/schema/v1/m.room.aliases
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Informs the room about what room aliases it has been given.",
     "description": "This event is sent by a homeserver directly to inform of changes to the list of aliases it knows about for that room. The ``state_key`` for this event is set to the homeserver which owns the room alias. The entire set of known aliases for the room is the union of all the ``m.room.aliases`` events, one for each homeserver. Clients **should** check the validity of any room alias given in this list before presenting it to the user as trusted fact. The lists given by this event should be considered simply as advice on which aliases might exist, for which the client can perform the lookup to confirm whether it receives the correct room ID.",
diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
index 17caa321..f79ce234 100644
--- a/event-schemas/schema/v1/m.room.canonical_alias
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Informs the room as to which alias is the canonical one.",
     "description": "This event is used to inform the room about which alias should be considered the canonical one. This could be for display purposes or as suggestion to users which alias to use to advertise the room.",
diff --git a/event-schemas/schema/v1/m.room.create b/event-schemas/schema/v1/m.room.create
index ecc2ed63..9561f19b 100644
--- a/event-schemas/schema/v1/m.room.create
+++ b/event-schemas/schema/v1/m.room.create
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "The first event in the room.",
     "description": "This is the first event in a room and cannot be changed. It acts as the root of all other events.",
diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
index 48b0a5c4..22b39623 100644
--- a/event-schemas/schema/v1/m.room.history_visibility
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Controls visibility of history.",
     "description": "This event controls whether a member of a room can see the events that happened in a room from before they joined.",
diff --git a/event-schemas/schema/v1/m.room.join_rules b/event-schemas/schema/v1/m.room.join_rules
index 567247eb..095ccfa7 100644
--- a/event-schemas/schema/v1/m.room.join_rules
+++ b/event-schemas/schema/v1/m.room.join_rules
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Describes how users are allowed to join the room.",
     "description": "A room may be ``public`` meaning anyone can join the room without any prior action. Alternatively, it can be ``invite`` meaning that a user who wishes to join the room must first receive an invite to the room from someone already inside of the room. Currently, ``knock`` and ``private`` are reserved keywords which are not implemented.",
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 67e10d51..5acc5f7c 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "The current membership state of a user in the room.",
     "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail.",
diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message
index 61b6256f..c8c5931e 100644
--- a/event-schemas/schema/v1/m.room.message
+++ b/event-schemas/schema/v1/m.room.message
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Message",
     "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.",
diff --git a/event-schemas/schema/v1/m.room.message#m.audio b/event-schemas/schema/v1/m.room.message#m.audio
index b236e5b0..f943fc7f 100644
--- a/event-schemas/schema/v1/m.room.message#m.audio
+++ b/event-schemas/schema/v1/m.room.message#m.audio
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "AudioMessage",
     "description": "This message represents a single audio clip.",
diff --git a/event-schemas/schema/v1/m.room.message#m.emote b/event-schemas/schema/v1/m.room.message#m.emote
index 9f17180c..3379d635 100644
--- a/event-schemas/schema/v1/m.room.message#m.emote
+++ b/event-schemas/schema/v1/m.room.message#m.emote
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "EmoteMessage",
     "description": "This message is similar to ``m.text`` except that the sender is 'performing' the action contained in the ``body`` key, similar to ``/me`` in IRC. This message should be prefixed by the name of the sender. This message could also be represented in a different colour to distinguish it from regular ``m.text`` messages.",
diff --git a/event-schemas/schema/v1/m.room.message#m.file b/event-schemas/schema/v1/m.room.message#m.file
index d8c63b33..b1cdff22 100644
--- a/event-schemas/schema/v1/m.room.message#m.file
+++ b/event-schemas/schema/v1/m.room.message#m.file
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "FileMessage",
     "description": "This message represents a generic file.",
diff --git a/event-schemas/schema/v1/m.room.message#m.image b/event-schemas/schema/v1/m.room.message#m.image
index ca22654f..5d4f811b 100644
--- a/event-schemas/schema/v1/m.room.message#m.image
+++ b/event-schemas/schema/v1/m.room.message#m.image
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "ImageMessage",
     "description": "This message represents a single image and an optional thumbnail.",
diff --git a/event-schemas/schema/v1/m.room.message#m.location b/event-schemas/schema/v1/m.room.message#m.location
index 919c3325..f02bea78 100644
--- a/event-schemas/schema/v1/m.room.message#m.location
+++ b/event-schemas/schema/v1/m.room.message#m.location
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "LocationMessage",
     "description": "This message represents a real-world location.",
diff --git a/event-schemas/schema/v1/m.room.message#m.notice b/event-schemas/schema/v1/m.room.message#m.notice
index e6ce6f89..972eb45e 100644
--- a/event-schemas/schema/v1/m.room.message#m.notice
+++ b/event-schemas/schema/v1/m.room.message#m.notice
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "NoticeMessage",
     "description": "A m.notice message should be considered similar to a plain m.text message except that clients should visually distinguish it in some way. It is intended to be used by automated clients, such as bots, bridges, and other entities, rather than humans. Additionally, such automated agents which watch a room for messages and respond to them ought to ignore m.notice messages. This helps to prevent infinite-loop situations where two automated clients continuously exchange messages, as each responds to the other.",
diff --git a/event-schemas/schema/v1/m.room.message#m.text b/event-schemas/schema/v1/m.room.message#m.text
index ceadbcd0..00aa1818 100644
--- a/event-schemas/schema/v1/m.room.message#m.text
+++ b/event-schemas/schema/v1/m.room.message#m.text
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "TextMessage",
     "description": "This message is the most basic message and is used to represent text.",
diff --git a/event-schemas/schema/v1/m.room.message#m.video b/event-schemas/schema/v1/m.room.message#m.video
index 667832ff..b91fd48a 100644
--- a/event-schemas/schema/v1/m.room.message#m.video
+++ b/event-schemas/schema/v1/m.room.message#m.video
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "VideoMessage",
     "description": "This message represents a single video clip.",
diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback
index b662e9e6..3a069654 100644
--- a/event-schemas/schema/v1/m.room.message.feedback
+++ b/event-schemas/schema/v1/m.room.message.feedback
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "MessageFeedback",
     "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.",
diff --git a/event-schemas/schema/v1/m.room.name b/event-schemas/schema/v1/m.room.name
index 5565147c..2a72feeb 100644
--- a/event-schemas/schema/v1/m.room.name
+++ b/event-schemas/schema/v1/m.room.name
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "title": "RoomName",
     "description": "A room has an opaque room ID which is not human-friendly to read. A room alias is human-friendly, but not all rooms have room aliases. The room name is a human-friendly string designed to be displayed to the end-user. The room name is not unique, as multiple rooms can have the same room name set. The room name can also be set when creating a room using ``/createRoom`` with the ``name`` key.",
     "type": "object",
diff --git a/event-schemas/schema/v1/m.room.power_levels b/event-schemas/schema/v1/m.room.power_levels
index 95850fcb..b5cb1049 100644
--- a/event-schemas/schema/v1/m.room.power_levels
+++ b/event-schemas/schema/v1/m.room.power_levels
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Defines the power levels (privileges) of users in the room.",
     "description": "This event specifies the minimum level a user must have in order to perform a certain action. It also specifies the levels of each user in the room. If a ``user_id`` is in the ``users`` list, then that ``user_id`` has the associated power level. Otherwise they have the default level ``users_default``. If ``users_default`` is not supplied, it is assumed to be 0. The level required to send a certain event is governed by ``events``, ``state_default`` and ``events_default``. If an event type is specified in ``events``, then the user must have at least the level specified in order to send that event. If the event type is not supplied, it defaults to ``events_default`` for Message Events and ``state_default`` for State Events.",
diff --git a/event-schemas/schema/v1/m.room.redaction b/event-schemas/schema/v1/m.room.redaction
index b095b17d..970679c7 100644
--- a/event-schemas/schema/v1/m.room.redaction
+++ b/event-schemas/schema/v1/m.room.redaction
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Redaction",
     "description": "Events can be redacted by either room or server admins. Redacting an event means that all keys not required by the protocol are stripped off, allowing admins to remove offensive or illegal content that may have been attached to any event. This cannot be undone, allowing server owners to physically delete the offending data.  There is also a concept of a moderator hiding a message event, which can be undone, but cannot be applied to state events. The event that has been redacted is specified in the ``redacts`` event level key.",
diff --git a/event-schemas/schema/v1/m.room.topic b/event-schemas/schema/v1/m.room.topic
index 32319c38..a9956228 100644
--- a/event-schemas/schema/v1/m.room.topic
+++ b/event-schemas/schema/v1/m.room.topic
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Topic",
     "description": "A topic is a short message detailing what is currently being discussed in the room.  It can also be used as a way to display extra information about the room, which may not be suitable for the room name. The room topic can also be set when creating a room using ``/createRoom`` with the ``topic`` key.",

From 5b59c67510be4d17a81dd437af0d7d1235da24c1 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 16:34:20 +0100
Subject: [PATCH 22/85] Minor formatting fixes. Fix state event templating.

---
 specification/00_basis.rst             |  5 ++--
 specification/10_client_server_api.rst |  7 ++---
 specification/20_events.rst            | 39 +++++++++++++-------------
 templating/matrix_templates/units.py   |  5 ++--
 4 files changed, 26 insertions(+), 30 deletions(-)

diff --git a/specification/00_basis.rst b/specification/00_basis.rst
index 10709f1d..f6d50802 100644
--- a/specification/00_basis.rst
+++ b/specification/00_basis.rst
@@ -171,13 +171,14 @@ All data exchanged over Matrix is expressed as an "event". Typically each client
 action (e.g. sending a message) correlates with exactly one event. Each event
 has a ``type`` which is used to differentiate different kinds of data. ``type``
 values MUST be uniquely globally namespaced following Java's `package naming
-conventions
-<http://docs.oracle.com/javase/specs/jls/se5.0/html/packages.html#7.7>`, e.g.
+conventions`_, e.g.
 ``com.example.myapp.event``. The special top-level namespace ``m.`` is reserved
 for events defined in the Matrix specification - for instance ``m.room.message``
 is the event type for instant messages. Events are usually sent in the context
 of a "Room".
 
+.. _package naming conventions: https://en.wikipedia.org/wiki/Java_package#Package_naming_conventions
+
 Event Graphs
 ~~~~~~~~~~~~
 
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index fe11d199..f0416c02 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -932,11 +932,8 @@ directly by sending the following request to
     "membership": "leave"
   }
 
-See the `Room events`_ section for more information on ``m.room.member``.
-
-Once a user has left a room, that room will no longer appear on the
-|initialSync|_ API.
-
+See the `Room events`_ section for more information on ``m.room.member``. Once a
+user has left a room, that room will no longer appear on the |initialSync|_ API.
 If all members in a room leave, that room becomes eligible for deletion.
 
 Banning users in a room
diff --git a/specification/20_events.rst b/specification/20_events.rst
index b0d5d716..ffc23ec4 100644
--- a/specification/20_events.rst
+++ b/specification/20_events.rst
@@ -92,11 +92,10 @@ Voice over IP
 -------------
 Matrix can also be used to set up VoIP calls. This is part of the core
 specification, although is at a relatively early stage. Voice (and video) over
-Matrix is built on the WebRTC 1.0 standard.
-
-Call events are sent to a room, like any other event. This means that clients
-must only send call events to rooms with exactly two participants as currently
-the WebRTC standard is based around two-party communication.
+Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like
+any other event. This means that clients must only send call events to rooms
+with exactly two participants as currently the WebRTC standard is based around
+two-party communication.
 
 {{voip_events}}
 
@@ -106,25 +105,26 @@ A call is set up with messages exchanged as follows:
 
 ::
 
-   Caller                   Callee
+   Caller                    Callee
+ [Place Call]
  m.call.invite ----------->
  m.call.candidate -------->
- [more candidates events]
-                         User answers call
-                  <------ m.call.answer
-               [...]
-                  <------ m.call.hangup
+ [..candidates..] -------->
+                           [Answers call]
+          <--------------- m.call.answer
+    [Call is active and ongoing]
+          <--------------- m.call.hangup
 
 Or a rejected call:
 
 ::
 
-   Caller                   Callee
- m.call.invite ----------->
- m.call.candidate -------->
- [more candidates events]
-                        User rejects call
-                 <------- m.call.hangup
+   Caller                      Callee
+ m.call.invite ------------>
+ m.call.candidate --------->
+ [..candidates..] --------->
+                            [Rejects call]
+            <-------------- m.call.hangup
 
 Calls are negotiated according to the WebRTC specification.
 
@@ -137,9 +137,8 @@ better experience for the users if their calls are connected if it is clear
 that their intention is to set up a call with one another.
 
 In Matrix, calls are to rooms rather than users (even if those rooms may only
-contain one other user) so we consider calls which are to the same room.
-
-The rules for dealing with such a situation are as follows:
+contain one other user) so we consider calls which are to the same room. The
+rules for dealing with such a situation are as follows:
 
  - If an invite to a room is received whilst the client is preparing to send an
    invite to the same room, the client should cancel its outgoing call and
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 4b8127ae..3a8825ae 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -361,8 +361,8 @@ class MatrixUnits(Units):
 
                 # add typeof
                 base_defs = {
-                    "core#/definitions/room_event": "Message Event",
-                    "core#/definitions/state_event": "State Event"
+                    "core/room_event.json": "Message Event",
+                    "core/state_event.json": "State Event"
                 }
                 if type(json_schema.get("allOf")) == list:
                     schema["typeof"] = base_defs.get(
@@ -399,7 +399,6 @@ class MatrixUnits(Units):
                         "`m.room.message msgtypes`_."
                     )
 
-
                 # Assign state key info if it has some
                 if schema["typeof"] == "State Event":
                     skey_desc = Units.prop(

From af32ec194a0fdad546df6f6e35bbd13d8b476c23 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 16:38:58 +0100
Subject: [PATCH 23/85] Move VoIP events to 22_voip_events

20_events is a huge as it is, this makes it easier to find what section
you actually want when editing.
---
 specification/20_events.rst      | 66 --------------------------------
 specification/22_voip_events.rst | 65 +++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 66 deletions(-)
 create mode 100644 specification/22_voip_events.rst

diff --git a/specification/20_events.rst b/specification/20_events.rst
index ffc23ec4..ce36b040 100644
--- a/specification/20_events.rst
+++ b/specification/20_events.rst
@@ -88,69 +88,3 @@ users, they should include the displayname and avatar URL fields in these
 events so that clients already have these details to hand, and do not have to
 perform extra roundtrips to query it.
 
-Voice over IP
--------------
-Matrix can also be used to set up VoIP calls. This is part of the core
-specification, although is at a relatively early stage. Voice (and video) over
-Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like
-any other event. This means that clients must only send call events to rooms
-with exactly two participants as currently the WebRTC standard is based around
-two-party communication.
-
-{{voip_events}}
-
-Message Exchange
-~~~~~~~~~~~~~~~~
-A call is set up with messages exchanged as follows:
-
-::
-
-   Caller                    Callee
- [Place Call]
- m.call.invite ----------->
- m.call.candidate -------->
- [..candidates..] -------->
-                           [Answers call]
-          <--------------- m.call.answer
-    [Call is active and ongoing]
-          <--------------- m.call.hangup
-
-Or a rejected call:
-
-::
-
-   Caller                      Callee
- m.call.invite ------------>
- m.call.candidate --------->
- [..candidates..] --------->
-                            [Rejects call]
-            <-------------- m.call.hangup
-
-Calls are negotiated according to the WebRTC specification.
-
-
-Glare
-~~~~~
-This specification aims to address the problem of two users calling each other
-at roughly the same time and their invites crossing on the wire. It is a far
-better experience for the users if their calls are connected if it is clear
-that their intention is to set up a call with one another.
-
-In Matrix, calls are to rooms rather than users (even if those rooms may only
-contain one other user) so we consider calls which are to the same room. The
-rules for dealing with such a situation are as follows:
-
- - If an invite to a room is received whilst the client is preparing to send an
-   invite to the same room, the client should cancel its outgoing call and
-   instead automatically accept the incoming call on behalf of the user.
- - If an invite to a room is received after the client has sent an invite to
-   the same room and is waiting for a response, the client should perform a
-   lexicographical comparison of the call IDs of the two calls and use the
-   lesser of the two calls, aborting the greater. If the incoming call is the
-   lesser, the client should accept this call on behalf of the user.
-
-The call setup should appear seamless to the user as if they had simply placed
-a call and the other party had accepted. Thusly, any media stream that had been
-setup for use on a call should be transferred and used for the call that
-replaces it.
-
diff --git a/specification/22_voip_events.rst b/specification/22_voip_events.rst
new file mode 100644
index 00000000..d75db7e7
--- /dev/null
+++ b/specification/22_voip_events.rst
@@ -0,0 +1,65 @@
+Voice over IP
+-------------
+Matrix can also be used to set up VoIP calls. This is part of the core
+specification, although is at a relatively early stage. Voice (and video) over
+Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like
+any other event. This means that clients must only send call events to rooms
+with exactly two participants as currently the WebRTC standard is based around
+two-party communication.
+
+{{voip_events}}
+
+Message Exchange
+~~~~~~~~~~~~~~~~
+A call is set up with messages exchanged as follows:
+
+::
+
+   Caller                    Callee
+ [Place Call]
+ m.call.invite ----------->
+ m.call.candidate -------->
+ [..candidates..] -------->
+                           [Answers call]
+          <--------------- m.call.answer
+    [Call is active and ongoing]
+          <--------------- m.call.hangup
+
+Or a rejected call:
+
+::
+
+   Caller                      Callee
+ m.call.invite ------------>
+ m.call.candidate --------->
+ [..candidates..] --------->
+                            [Rejects call]
+            <-------------- m.call.hangup
+
+Calls are negotiated according to the WebRTC specification.
+
+
+Glare
+~~~~~
+This specification aims to address the problem of two users calling each other
+at roughly the same time and their invites crossing on the wire. It is a far
+better experience for the users if their calls are connected if it is clear
+that their intention is to set up a call with one another.
+
+In Matrix, calls are to rooms rather than users (even if those rooms may only
+contain one other user) so we consider calls which are to the same room. The
+rules for dealing with such a situation are as follows:
+
+ - If an invite to a room is received whilst the client is preparing to send an
+   invite to the same room, the client should cancel its outgoing call and
+   instead automatically accept the incoming call on behalf of the user.
+ - If an invite to a room is received after the client has sent an invite to
+   the same room and is waiting for a response, the client should perform a
+   lexicographical comparison of the call IDs of the two calls and use the
+   lesser of the two calls, aborting the greater. If the incoming call is the
+   lesser, the client should accept this call on behalf of the user.
+
+The call setup should appear seamless to the user as if they had simply placed
+a call and the other party had accepted. Thusly, any media stream that had been
+setup for use on a call should be transferred and used for the call that
+replaces it.

From 46d29e9eea31d32e4865554e0c36badcd4006091 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 16:48:07 +0100
Subject: [PATCH 24/85] fix the room api swagger to be valid swagger

---
 api/client-server/v1/rooms.yaml | 41 +++++++++++++++------------------
 1 file changed, 19 insertions(+), 22 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 9a73bb05..c038dba5 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -43,9 +43,8 @@ paths:
         - in: path
           type: string
           name: stateKey
-          description: |-
-            The key of the state to look up. Defaults to the empty string.
-          required: false
+          description: The key of the state to look up. Defaults to the empty string.
+          required: true
           x-example: ""
       responses:
         200:
@@ -159,17 +158,17 @@ paths:
                     "user_id": "@alice:example.com"
                 }
               ]
-            schema:
-              type: array
-              title: RoomState
-              description: |-
-                If the user is a member of the room this will be the
-                current state of the room as a list of events.
-              items:
-                title: StateEvent
-                type: object
-                allOf:
-                  - "$ref": "definitions/state_event.yaml"
+          schema:
+            type: array
+            title: RoomState
+            description: |-
+              If the user is a member of the room this will be the
+              current state of the room as a list of events.
+            items:
+              title: StateEvent
+              type: object
+              allOf:
+                - "$ref": "events/core/state_event.json"
         403:
           description: You are not joined to the room.
 
@@ -345,7 +344,7 @@ paths:
                       type: object
                       title: RoomEvent
                       allOf:
-                        - "$ref": "definitions/room_event.yaml"
+                        - "$ref": "events/core/room_event.json"
                 required: ["start", "end", "chunk"]
               state:
                 type: array
@@ -358,7 +357,7 @@ paths:
                   title: StateEvent
                   type: object
                   allOf:
-                    - "$ref": "definitions/state_event.yaml"
+                    - "$ref": "events/core/state_event.json"
               visibility:
                 type: string
                 enum: ["private", "public"]
@@ -377,7 +376,7 @@ paths:
           type: string
           name: roomId
           description: The room to get the member events for.
-          required: true,
+          required: true
           x-example: "!room:example.com"
       responses:
         200:
@@ -419,15 +418,13 @@ paths:
                 ]
               }
           schema:
-            object:
-            properities:
+            type: object
+            properties:
               chunk:
                 type: array
                 items:
                   title: MemberEvent
                   type: object
                   allOf:
-                    - "$ref": "../../event-schemas/schema/v1/m.room.member"
-              visibility:
-                type: string
+                    - "$ref": "events/m.room.member"
 

From faa95e172f0ec7f1f40416b1df094380a72f2c98 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 17:49:57 +0100
Subject: [PATCH 25/85] More formatting fixes; typos; etc

---
 specification/22_voip_events.rst             |  1 +
 specification/25_application_service_api.rst | 11 ++--
 specification/30_server_server_api.rst       | 55 ++++++++++----------
 specification/31_event_signing.rst           |  2 +-
 specification/40_content_repository.rst      |  8 +--
 specification/43_push_cs_api.rst             | 20 +++----
 6 files changed, 50 insertions(+), 47 deletions(-)

diff --git a/specification/22_voip_events.rst b/specification/22_voip_events.rst
index d75db7e7..a5468237 100644
--- a/specification/22_voip_events.rst
+++ b/specification/22_voip_events.rst
@@ -63,3 +63,4 @@ The call setup should appear seamless to the user as if they had simply placed
 a call and the other party had accepted. Thusly, any media stream that had been
 setup for use on a call should be transferred and used for the call that
 replaces it.
+
diff --git a/specification/25_application_service_api.rst b/specification/25_application_service_api.rst
index 8236b2de..9b58c861 100644
--- a/specification/25_application_service_api.rst
+++ b/specification/25_application_service_api.rst
@@ -66,13 +66,13 @@ An example HS configuration required to pass traffic to the AS is:
   application service is merely augmenting the room itself (e.g. providing
   logging or searching facilities).
 - Namespaces are represented by POSIX extended regular expressions, 
-  e.g.:
+  e.g:
 
 .. code-block:: yaml
 
    users:
      - exclusive: true
-       regex: @irc.freenode.net/.*
+       regex: @irc.freenode.net_.*
 
 
 Home Server -> Application Service API
@@ -326,7 +326,7 @@ but only if the application service has defined the namespace as ``exclusive``.
 
 ID conventions
 ~~~~~~~~~~~~~~
-.. NOTE::
+.. TODO-spec
   - Giving HSes the freedom to namespace still feels like the Right Thing here.
   - Exposing a public API provides the consistency which was the main complaint
     against namespacing.
@@ -345,7 +345,7 @@ types, including:
 - MSISDNs (``tel``)
 - Email addresses (``mailto``)
 - IRC nicks (``irc`` - https://tools.ietf.org/html/draft-butcher-irc-url-04)
-- XMPP (xep-0032)
+- XMPP (XEP-0032)
 - SIP URIs (RFC 3261)
 
 As a result, virtual user IDs SHOULD relate to their URI counterpart. This
@@ -403,6 +403,8 @@ blog comment traffic in & out of matrix
 
 Active Application Services
 ----------------------------
+.. NOTE::
+  This section is a work in progress.
 
 .. TODO-spec
   API that provides hooks into the server so that you can intercept and
@@ -419,3 +421,4 @@ Policy Servers
 
 Enforcing policies
 ------------------
+
diff --git a/specification/30_server_server_api.rst b/specification/30_server_server_api.rst
index 1c7bf3ed..f5cadabf 100644
--- a/specification/30_server_server_api.rst
+++ b/specification/30_server_server_api.rst
@@ -2,10 +2,9 @@ Federation API
 ==============
 
 Matrix home servers use the Federation APIs (also known as server-server APIs)
-to communicate with each other.
-Home servers use these APIs to push messages to each other in real-time, to
-request historic messages from each other, and to query profile and presence
-information about users on each other's servers.
+to communicate with each other. Home servers use these APIs to push messages to
+each other in real-time, to request historic messages from each other, and to
+query profile and presence information about users on each other's servers.
 
 The APIs are implemented using HTTPS GETs and PUTs between each of the
 servers. These HTTPS requests are strongly authenticated using public key
@@ -21,7 +20,7 @@ Persisted Data Units (PDUs):
     context.
 
     Like email, it is the responsibility of the originating server of a PDU
-    to deliver that event to its recepient servers. However PDUs are signed
+    to deliver that event to its recipient servers. However PDUs are signed
     using the originating server's public key so that it is possible to
     deliver them through third-party servers.
 
@@ -84,18 +83,19 @@ directly or by querying an intermediate notary server using a
 response with their own key. A server may query multiple notary servers to
 ensure that they all report the same public keys.
 
-This approach is borrowed from the Perspectives Project
-(http://perspectives-project.org/), but modified to include the NACL keys and to
-use JSON instead of XML. It has the advantage of avoiding a single trust-root
-since each server is free to pick which notary servers they trust and can
-corroborate the keys returned by a given notary server by querying other
-servers.
+This approach is borrowed from the `Perspectives Project`_, but modified to
+include the NACL keys and to use JSON instead of XML. It has the advantage of
+avoiding a single trust-root since each server is free to pick which notary
+servers they trust and can corroborate the keys returned by a given notary
+server by querying other servers.
+
+.. _Perspectives Project: http://perspectives-project.org/
 
 Publishing Keys
 _______________
 
 Home servers publish the allowed TLS fingerprints and signing keys in a JSON
-object at ``/_matrix/key/v2/server/${key_id}``. The response contains a list of
+object at ``/_matrix/key/v2/server/{key_id}``. The response contains a list of
 ``verify_keys`` that are valid for signing federation requests made by the
 server and for signing events. It contains a list of ``old_verify_keys``
 which are only valid for signing events. Finally the response contains a list
@@ -510,7 +510,7 @@ To backfill events on a given context::
 
 Retrieves a sliding-window history of previous PDUs that occurred on the given
 context. Starting from the PDU ID(s) given in the "v" argument, the PDUs that
-preceeded it are retrieved, up to a total number given by the "limit" argument.
+preceded it are retrieved, up to a total number given by the "limit" argument.
 These are then returned in a new Transaction containing all of the PDUs.
 
 
@@ -554,9 +554,7 @@ Every HTTP request made by a homeserver is authenticated using public key
 digital signatures. The request method, target and body are signed by wrapping
 them in a JSON object and signing it using the JSON signing algorithm. The
 resulting signatures are added as an Authorization header with an auth scheme
-of X-Matrix.
-
-Note that the target field should include the full path starting with
+of X-Matrix. Note that the target field should include the full path starting with
 ``/_matrix/...``, including the ``?`` and any query parameters if present, but
 should not include the leading ``https:``, nor the destination server's
 hostname.
@@ -656,12 +654,12 @@ State Conflict Resolution
   - How does this work with deleting current state
   - How do we reject invalid federation traffic?
 
-[[TODO(paul): At this point we should probably have a long description of how
-State management works, with descriptions of clobbering rules, power levels, etc
-etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
-so on. This part needs refining. And writing in its own document as the details
-relate to the server/system as a whole, not specifically to server-server
-federation.]]
+  [[TODO(paul): At this point we should probably have a long description of how
+  State management works, with descriptions of clobbering rules, power levels, etc
+  etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
+  so on. This part needs refining. And writing in its own document as the details
+  relate to the server/system as a whole, not specifically to server-server
+  federation.]]
 
 Presence
 --------
@@ -677,8 +675,8 @@ Performing a presence update and poll subscription request::
       Each should be an object with the following keys:
         user_id: string containing a User ID
         presence: "offline"|"unavailable"|"online"|"free_for_chat"
-        status_msg: (optional) string of freeform text
-        last_active_ago: miliseconds since the last activity by the user
+        status_msg: (optional) string of free-form text
+        last_active_ago: milliseconds since the last activity by the user
 
     poll: (optional): list of strings giving User IDs
 
@@ -696,7 +694,7 @@ removed until explicitly requested by a later ``unpoll``.
 On receipt of a message containing a non-empty ``poll`` list, the receiving
 server should immediately send the sending server a presence update EDU of its
 own, containing in a ``push`` list the current state of every user that was in
-the orginal EDU's ``poll`` list.
+the original EDU's ``poll`` list.
 
 Sending a presence invite::
 
@@ -721,7 +719,7 @@ Rejecting a presence invite::
   Content keys - as for m.presence_invite
 
 .. TODO-doc
-  - Explain the timing-based roundtrip reduction mechanism for presence
+  - Explain the timing-based round-trip reduction mechanism for presence
     messages
   - Explain the zero-byte presence inference logic
   See also: docs/client-server/model/presence
@@ -742,8 +740,8 @@ Querying profile information::
     field: (optional) string giving a field name
 
   Returns: JSON object containing the following keys:
-    displayname: string of freeform text
-    avatar_url: string containing an http-scheme URL
+    displayname: string of free-form text
+    avatar_url: string containing an HTTP-scheme URL
 
 If the query contains the optional ``field`` key, it should give the name of a
 result field. If such is present, then the result should contain only a field
@@ -769,3 +767,4 @@ Querying directory information::
 The list of join candidates is a list of server names that are likely to hold
 the given room; these are servers that the requesting server may wish to try
 joining with. This list may or may not include the server answering the query.
+
diff --git a/specification/31_event_signing.rst b/specification/31_event_signing.rst
index 9351cd57..f3a22d1d 100644
--- a/specification/31_event_signing.rst
+++ b/specification/31_event_signing.rst
@@ -30,7 +30,7 @@ using this representation.
          value,
          # Encode code-points outside of ASCII as UTF-8 rather than \u escapes
          ensure_ascii=False,
-         # Remove unecessary white space.
+         # Remove unnecessary white space.
          separators=(',',':'),
          # Sort the keys of dictionaries.
          sort_keys=True,
diff --git a/specification/40_content_repository.rst b/specification/40_content_repository.rst
index df04605a..2c45ced7 100644
--- a/specification/40_content_repository.rst
+++ b/specification/40_content_repository.rst
@@ -39,10 +39,10 @@ thumbnailing method::
 
        <thumbnail>
 
-The thumbnail methods are "crop" and "scale". "scale" trys to return an
+The thumbnail methods are "crop" and "scale". "scale" tries to return an
 image where either the width or the height is smaller than the requested
 size. The client should then scale and letterbox the image if it needs to
-fit within a given rectangle. "crop" trys to return an image where the
+fit within a given rectangle. "crop" tries to return an image where the
 width and height are close to the requested size and the aspect matches
 the requested size. The client should scale the image if it needs to fit
 within a given rectangle.
@@ -53,8 +53,8 @@ the content. Homeservers may return thumbnails of a different size to that
 requested. However homeservers should provide exact matches where reasonable.
 Homeservers must never upscale images.
 
-Security
---------
+Security considerations
+-----------------------
 
 Clients may try to upload very large files. Homeservers should not store files
 that are too large and should not serve them to clients.
diff --git a/specification/43_push_cs_api.rst b/specification/43_push_cs_api.rst
index 022f50c3..b78b44fd 100644
--- a/specification/43_push_cs_api.rst
+++ b/specification/43_push_cs_api.rst
@@ -70,7 +70,7 @@ Room Rules
 Sender
   These rules configure notification behaviour for messages from a specific,
   named Matrix user ID. The rule_id of Sender rules is always the Matrix user
-  ID of the user whose messages theyt apply to.
+  ID of the user whose messages they'd apply to.
 Underride
   These are identical to override rules, but have a lower priority than content,
   room and sender rules.
@@ -112,7 +112,7 @@ In addition, all rules may be enabled or disabled. Disabled rules never match.
 
 If no rules match an event, the Home Server should not notify for the message
 (that is to say, the default action is "dont-notify"). Events that the user sent
-themself are never alerted for.
+themselves are never alerted for.
 
 Predefined Rules
 ----------------
@@ -128,7 +128,7 @@ with these IDs, their semantics should match those given below:
 
     {
         "rule_id": ".m.rule.contains_user_name"
-        "pattern": "[the lcoal part of the user's Matrix ID]",
+        "pattern": "[the local part of the user's Matrix ID]",
         "actions": [
             "notify",
             {
@@ -220,7 +220,7 @@ with these IDs, their semantics should match those given below:
 Push Rules: Actions:
 --------------------
 All rules have an associated list of 'actions'. An action affects if and how a
-notification is delievered for a matching event. This standard defines the
+notification is delivered for a matching event. This standard defines the
 following actions, although if Home servers wish to support more, they are free
 to do so:
 
@@ -241,11 +241,11 @@ set_tweak
 
 Actions that have no parameters are represented as a string. Otherwise, they are
 represented as a dictionary with a key equal to their name and other keys as
-their parameters, eg. { "set_tweak": "sound", "value": "default" }
+their parameters, e.g. ``{ "set_tweak": "sound", "value": "default" }``
 
 Push Rules: Actions: Tweaks
 ---------------------------
-The 'set_tweak' key action is used to add an entry to the 'tweaks' dictionary
+The ``set_tweak`` key action is used to add an entry to the 'tweaks' dictionary
 that is sent in the notification poke. The following tweaks are defined:
 
 sound
@@ -275,7 +275,7 @@ do so:
 
 event_match
   This is a glob pattern match on a field of the event. Parameters:
-   * 'key': The dot-separated field of the event to match, eg. content.body
+   * 'key': The dot-separated field of the event to match, e.g. content.body
    * 'pattern': The glob-style pattern to match against. Patterns with no
                 special glob characters should be treated as having asterisks
                 prepended and appended when testing the condition.
@@ -295,7 +295,7 @@ room_member_count
      '>=' or '<='. A prefix of '<' matches rooms where the member count is
      strictly less than the given number and so forth. If no prefix is present,
      this matches rooms where the member count is exactly equal to the given
-     number (ie. the same as '==').
+     number (i.e. the same as '==').
 
 Room, Sender, User and Content rules do not have conditions in the same way,
 but instead have predefined conditions, the behaviour of which can be configured
@@ -314,7 +314,7 @@ scope
   Either 'global' or 'device/<profile_tag>' to specify global rules or
   device rules for the given profile_tag.
 kind
-  The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content'.
+  The kind of rule, i.e. 'override', 'underride', 'sender', 'room', 'content'.
 rule_id
   The identifier for the rule.
 
@@ -330,7 +330,7 @@ after
   rule.
 
 All requests to the push rules API also require an access_token as a query
-paraemter.
+parameter.
 
 The content of the PUT request is a JSON object with a list of actions under the
 'actions' key and either conditions (under the 'conditions' key) or the

From 620d3dcb266c432e1bc471ad63b6bed57354e46f Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 17:51:44 +0100
Subject: [PATCH 26/85] Test commit

---
 specification/44_push_push_gw_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/44_push_push_gw_api.rst b/specification/44_push_push_gw_api.rst
index b182503b..d7f16144 100644
--- a/specification/44_push_push_gw_api.rst
+++ b/specification/44_push_push_gw_api.rst
@@ -1,7 +1,7 @@
 HTTP Notification Protocol
 --------------------------
 
-This describes the format used by "http" pushers to send notifications of
+This describes the format used by "HTTP" pushers to send notifications of
 events.
 
 Notifications are sent as HTTP POST requests to the URL configured when the

From 3f9d183c2aaaf5f6d2f6c9bc39a57317145dca1b Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 09:25:40 +0100
Subject: [PATCH 27/85] More typo/layout fixes

Split out identity servers to a separate file
---
 specification/44_push_push_gw_api.rst     |  8 ++++----
 specification/45_typing_notifications.rst | 11 +++++++----
 specification/46_receipts.rst             | 18 +++++++-----------
 specification/49_other_non_core_apis.rst  |  4 ++--
 specification/50_appendices.rst           |  8 --------
 specification/60_identity_servers.rst     |  8 ++++++++
 6 files changed, 28 insertions(+), 29 deletions(-)
 create mode 100644 specification/60_identity_servers.rst

diff --git a/specification/44_push_push_gw_api.rst b/specification/44_push_push_gw_api.rst
index d7f16144..c970e93e 100644
--- a/specification/44_push_push_gw_api.rst
+++ b/specification/44_push_push_gw_api.rst
@@ -77,10 +77,10 @@ counts
   This is a dictionary of the current number of unacknowledged communications
   for the recipient user. Counts whose value is zero are omitted.
 unread
-  The number of unread messages a user has accross all of the rooms they are a
+  The number of unread messages a user has across all of the rooms they are a
   member of.
 missed_calls
-  The number of unacknowledged missed calls a user has accross all rooms of
+  The number of unacknowledged missed calls a user has across all rooms of
   which they are a member.
 device
   This is an array of devices that the notification should be sent to.
@@ -104,13 +104,13 @@ And additional key is defined but only present on member events:
 
 user_is_target
   This is true if the user receiving the notification is the subject of a member
-  event (ie. the state_key of the member event is equal to the user's Matrix
+  event (i.e. the state_key of the member event is equal to the user's Matrix
   ID).
 
 The recipient of an HTTP notification should respond with an HTTP 2xx response
 when the notification has been processed. If the endpoint returns an HTTP error
 code, the Home Server should retry for a reasonable amount of time with a
-reasonable backoff scheme.
+reasonable back-off scheme.
 
 The endpoint should return a JSON dictionary as follows::
 
diff --git a/specification/45_typing_notifications.rst b/specification/45_typing_notifications.rst
index 7df9a238..89f06b48 100644
--- a/specification/45_typing_notifications.rst
+++ b/specification/45_typing_notifications.rst
@@ -5,9 +5,10 @@ Client APIs
 -----------
 
 To set "I am typing for the next N msec"::
+
   PUT .../rooms/<room_id>/typing/<user_id>
   Content:  { "typing": true, "timeout": N }
-  # timeout is in msec; I suggest no more than 20 or 30 seconds
+  # timeout is in milliseconds; suggested no more than 20 or 30 seconds
 
 This should be re-sent by the client to continue informing the server the user
 is still typing; I suggest a safety margin of 5 seconds before the expected
@@ -15,6 +16,7 @@ timeout runs out. Just keep declaring a new timeout, it will replace the old
 one.
 
 To set "I am no longer typing"::
+
   PUT ../rooms/<room_id>/typing/<user_id>
   Content: { "typing": false }
 
@@ -46,13 +48,14 @@ Servers will emit EDUs in the following form::
     "content": {
       "room_id": "!room-id-here:matrix.org",
       "user_id": "@user-id-here:matrix.org",
-      "typing": true/false,
+      "typing": true/false
     }
   }
 
 Server EDUs don't (currently) contain timing information; it is up to
 originating HSes to ensure they eventually send "stop" notifications.
 
-((This will eventually need addressing, as part of the wider typing/presence
-timer addition work))
+.. TODO
+  ((This will eventually need addressing, as part of the wider typing/presence
+  timer addition work))
 
diff --git a/specification/46_receipts.rst b/specification/46_receipts.rst
index 6428d6b5..a0a90c16 100644
--- a/specification/46_receipts.rst
+++ b/specification/46_receipts.rst
@@ -2,9 +2,8 @@ Receipts
 ========
 
 Receipts are used to publish which events in a room the user or their devices
-have interacted with. For example, which events the user has read.
-
-For efficiency this is done as "up to" markers, i.e. marking a particular event
+have interacted with. For example, which events the user has read. For
+efficiency this is done as "up to" markers, i.e. marking a particular event
 as, say, ``read`` indicates the user has read all events *up to* that event.
 
 Client-Server API
@@ -43,14 +42,11 @@ For example::
     }
 
 For efficiency, receipts are batched into one event per room. In the initialSync
-and v2 sync APIs the receipts are listed in a seperate top level ``receipts``
-key.
-
-Each ``user_id``, ``receipt_type`` pair must be associated with only a single
-``event_id``.
-
-New receipts that come down the event streams are deltas. Deltas update
-existing mappings, clobbering based on ``user_id``, ``receipt_type`` pairs.
+and v2 sync APIs the receipts are listed in a separate top level ``receipts``
+key. Each ``user_id``, ``receipt_type`` pair must be associated with only a
+single ``event_id``. New receipts that come down the event streams are deltas.
+Deltas update existing mappings, clobbering based on ``user_id``,
+``receipt_type`` pairs.
 
 
 A client can update the markers for its user by issuing a request::
diff --git a/specification/49_other_non_core_apis.rst b/specification/49_other_non_core_apis.rst
index 8855ce9e..d6315a96 100644
--- a/specification/49_other_non_core_apis.rst
+++ b/specification/49_other_non_core_apis.rst
@@ -3,10 +3,10 @@ Address book repository
 
 .. NOTE::
   This section is a work in progress.
-  Do we even need it?  Clients can use out-of-band addressbook servers for now;
-  this should definitely not be core.
 
 .. TODO-spec
+  Do we even need it?  Clients can use out-of-band addressbook servers for now;
+  this should definitely not be core.
   - format: POST(?) wodges of json, some possible processing, then return wodges of json on GET.
   - processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of
     contacts), etc.
diff --git a/specification/50_appendices.rst b/specification/50_appendices.rst
index 295c8f69..de1ac290 100644
--- a/specification/50_appendices.rst
+++ b/specification/50_appendices.rst
@@ -128,11 +128,3 @@ Threat: Disclosure to Servers Within Chatroom
 An attacker could take control of a server within a chatroom to expose message
 contents or metadata for messages in that room.
 
-
-Identity Servers
-================
-.. NOTE::
-  This section is a work in progress.
-
-.. TODO-doc Dave
-  - 3PIDs and identity server, functions
diff --git a/specification/60_identity_servers.rst b/specification/60_identity_servers.rst
new file mode 100644
index 00000000..6ec013bd
--- /dev/null
+++ b/specification/60_identity_servers.rst
@@ -0,0 +1,8 @@
+Identity Servers
+================
+.. NOTE::
+  This section is a work in progress.
+
+.. TODO-doc Dave
+  - 3PIDs and identity server, functions
+

From 8e7b33ac9989c985ceac1cf7fccddd84c1346ec8 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 09:40:53 +0100
Subject: [PATCH 28/85] Fix more typos/spelling errors

---
 specification/00_basis.rst             | 13 ++++++-----
 specification/10_client_server_api.rst | 32 +++++++++++++-------------
 2 files changed, 23 insertions(+), 22 deletions(-)

diff --git a/specification/00_basis.rst b/specification/00_basis.rst
index f6d50802..384d583b 100644
--- a/specification/00_basis.rst
+++ b/specification/00_basis.rst
@@ -79,7 +79,7 @@ The functionality that Matrix provides includes:
 - Extensible user management (inviting, joining, leaving, kicking, banning)
   mediated by a power-level based user privilege system.
 - Extensible room state management (room naming, aliasing, topics, bans)
-- Extensible user profile management (avatars, displaynames, etc)
+- Extensible user profile management (avatars, display names, etc)
 - Managing user accounts (registration, login, logout)
 - Use of 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
   Facebook accounts to authenticate, identify and discover users on Matrix.
@@ -91,7 +91,7 @@ The functionality that Matrix provides includes:
 The end goal of Matrix is to be a ubiquitous messaging layer for synchronising
 arbitrary data between sets of people, devices and services - be that for
 instant messages, VoIP call setups, or any other objects that need to be
-reliably and persistently pushed from A to B in an interoperable and federated
+reliably and persistently pushed from A to B in an inter-operable and federated
 manner.
 
 Overview
@@ -185,7 +185,7 @@ Event Graphs
 Events exchanged in the context of a room are stored in a directed acyclic graph
 (DAG) called an ``event graph``. The partial ordering of this graph gives the
 chronological ordering of events within the room. Each event in the graph has a
-list of zero or more ``parent`` events, which refer to any preceeding events
+list of zero or more ``parent`` events, which refer to any preceding events
 which have no chronological successor from the perspective of the homeserver
 which created the event.
 
@@ -368,7 +368,8 @@ room). An example of a non-proactive client activity would be a client setting
 key called ``last_active_ago``, which gives the relative number of milliseconds
 since the message is generated/emitted that the user was last seen active.
 
-N.B. in v1 API, status/online/idle state are muxed into a single 'presence' field on the m.presence event.
+N.B. in v1 API, status/online/idle state are muxed into a single 'presence'
+field on the ``m.presence`` event.
 
 Presence Lists
 ~~~~~~~~~~~~~~
@@ -386,7 +387,7 @@ Profiles
 
 Users may publish arbitrary key/value data associated with their account - such
 as a human readable ``display name``, a profile photo URL, contact information
-(email address, phone nubers, website URLs etc).
+(email address, phone numbers, website URLs etc).
 
 In Client-Server API v2, profile data is typed using namespaced keys for
 interoperability, much like events - e.g. ``m.profile.display_name``.
@@ -443,7 +444,7 @@ The ``error`` string will be a human-readable error message, usually a sentence
 explaining what went wrong. The ``errcode`` string will be a unique string
 which can be used to handle an error message e.g. ``M_FORBIDDEN``. These error
 codes should have their namespace first in ALL CAPS, followed by a single _ to
-ease seperating the namespace from the error code.. For example, if there was a
+ease separating the namespace from the error code.. For example, if there was a
 custom namespace ``com.mydomain.here``, and a
 ``FORBIDDEN`` code, the error code should look like
 ``COM.MYDOMAIN.HERE_FORBIDDEN``. There may be additional keys depending on the
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index f0416c02..ef77a7dc 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -8,11 +8,11 @@ The client-server API provides a simple lightweight API to let clients send
 messages, control rooms and synchronise conversation history. It is designed to
 support both lightweight clients which store no state and lazy-load data from
 the server as required - as well as heavyweight clients which maintain a full
-local peristent copy of server state.
+local persistent copy of server state.
 
 This mostly describes v1 of the Client-Server API as featured in the original September
 2014 launch of Matrix, apart from user-interactive authentication where it is
-encouraged to move to V2, therefore this is the version documented here.
+encouraged to move to v2, therefore this is the version documented here.
 Version 2 is currently in development (as of Jan-March 2015) as an incremental
 but backwards-incompatible refinement of Version 1 and will be released
 shortly.
@@ -154,7 +154,7 @@ Matrix client, for example, an email confirmation may be completed when the user
 clicks on the link in the email. In this case, the client retries the request
 with an auth dict containing only the session key. The response to this will be
 the same as if the client were attempting to complete an auth state normally,
-ie. the request will either complete or request auth, with the presence or
+i.e. the request will either complete or request auth, with the presence or
 absence of that login stage type in the 'completed' array indicating whether
 that stage is complete.
 
@@ -204,7 +204,7 @@ Password-based
 :Type:
   ``m.login.password``
 :Description:
-  The client submits a username and secret password, both sent in plaintext.
+  The client submits a username and secret password, both sent in plain-text.
 
 To respond to this type, reply with an auth dict as follows::
 
@@ -247,10 +247,10 @@ service which the home server accepts when logging in, this indirection can be
 skipped and the "uri" key can be the ``Authorization Request URI``.
 
 The client then visits the ``Authorization Request URI``, which then shows the
-OAuth2 Allow/Deny prompt. Hitting 'Allow' returns the [XXX: redirects to the?]``redirect URI`` with the
-auth code. Home servers can choose any path for the ``redirect URI``. Once the
-OAuth flow has completed, the client retries the request with the session only,
-as above.
+OAuth2 Allow/Deny prompt. Hitting 'Allow' redirects to the ``redirect URI`` with
+the auth code. Home servers can choose any path for the ``redirect URI``. Once
+the OAuth flow has completed, the client retries the request with the session
+only, as above.
 
 Email-based (identity server)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -308,7 +308,7 @@ Where ``stage type`` is the type name of the stage it is attempting and
 ``session id`` is the ID of the session given by the home server.
 
 This MUST return an HTML page which can perform this authentication stage. This
-page must attempt to call the Javascript function ``window.onAuthDone`` when
+page must attempt to call the JavaScript function ``window.onAuthDone`` when
 the authentication has been completed.
 
 Pagination
@@ -373,7 +373,7 @@ now show page 3 (rooms R11 -> 15)::
                              Returns: R11,R12,R13,R14,R15
                          
 Note that tokens are treated in an *exclusive*, not inclusive, manner. The end 
-token from the intial request was '9' which corresponded to R10. When the 2nd
+token from the initial request was '9' which corresponded to R10. When the 2nd
 request was made, R10 did not appear again, even though from=9 was specified. If
 you know the token, you already have the data.
 
@@ -425,9 +425,9 @@ You can visualise the range of events being returned as::
                               |                             |
                         start: '1-2-3'                end: 'a-b-c'
                              
-Now, to receive future events in realtime on the eventstream, you simply GET
+Now, to receive future events in real-time on the eventstream, you simply GET
 $PREFIX/events with a ``from`` parameter of 'a-b-c': in other words passing in the
-``end`` token returned by initialsync. The request blocks until new events are
+``end`` token returned by initial sync. The request blocks until new events are
 available or until your specified timeout elapses, and then returns a
 new paginatable chunk of events alongside new start and end parameters::
 
@@ -467,7 +467,7 @@ event stream. When the request returns, an ``end`` token is included in the
 response. This token can be used in the next request to continue where the
 last request left off.
 
-All events must be deduplicated based on their event ID.
+All events must be de-duplicated based on their event ID.
 
 .. TODO
   is deduplication actually a hard requirement in CS v2?
@@ -493,7 +493,7 @@ Room events are split into two categories:
 
 :Message events:
   These are events which describe transient "once-off" activity in a room:
-  typically communication such as sending an instant messaage or setting up a
+  typically communication such as sending an instant message or setting up a
   VoIP call. These used to be called 'non-state' events.
 
 This specification outlines several events, all with the event type prefix
@@ -971,7 +971,7 @@ Registering for a user account is done using the request::
   POST $V2PREFIX/register
 
 This API endpoint uses the User-Interactive Authentication API.
-This API endoint does not require an access token.
+This API endpoint does not require an access token.
 
 The body of the POST request is a JSON object containing:
 
@@ -1059,7 +1059,7 @@ The third party identifier credentials object comprises:
 
 id_server
   The colon-separated hostname and port of the Identity Server used to
-  authenticate the third party identifer. If the port is the default, it and the
+  authenticate the third party identifier. If the port is the default, it and the
   colon should be omitted.
 sid
   The session ID given by the Identity Server

From ad26b7f8cb89b2a32e009bdd1d061ea6b7887b9d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 10:03:58 +0100
Subject: [PATCH 29/85] Fix a subtle rendering bug when catting spec sections

Throw in gendoc.py if a spec section doesn't end with \n\n

There needs to be TWO new lines at the end of each spec section else the
title of the next section merges into the last paragraph of the earlier
section. This happens without rst2html producing a warning, and results
in the section heading of a file disappearing(!)
---
 scripts/gendoc.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index ed172726..e871055e 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -18,7 +18,14 @@ def glob_spec_to(out_file_name):
     with open(out_file_name, "wb") as outfile:
         for f in sorted(glob.glob("../specification/*.rst")):
             with open(f, "rb") as infile:
-                outfile.write(infile.read())
+                section = infile.read()
+                # we need TWO new lines else the next file's title gets merged
+                # the last paragraph *WITHOUT RST PRODUCING A WARNING*
+                if not section[-2:] == '\n\n':
+                    raise Exception(
+                        "The file " + f + " does not end with 2 new lines."
+                    )
+                outfile.write(section)
 
 
 def rst2html(i, o):

From 08defafd98727b0011dddd6c0e8e271f61d147b2 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 18 Sep 2015 10:26:13 +0100
Subject: [PATCH 30/85] Print errors to stderr

This means that continuserv will actually display them
---
 scripts/gendoc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index ed172726..0e9626af 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -49,7 +49,7 @@ def run_through_template(input):
             )
     except subprocess.CalledProcessError as e:
         with open(tmpfile, 'r') as f:
-            print f.read()
+            sys.stderr.write(f.read() + "\n")
         raise
 
 def prepare_env():

From ef473b4161d029b35ef193309c9b4ad23f43e4f1 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 11:05:48 +0100
Subject: [PATCH 31/85] Start enforcing title styles

This depends on the number of the file.
---
 scripts/gendoc.py | 55 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 47 insertions(+), 8 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index e871055e..6bf7c1a6 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -4,6 +4,7 @@ from docutils.core import publish_file
 import fileinput
 import glob
 import os
+import re
 import shutil
 import subprocess
 import sys
@@ -14,17 +15,55 @@ stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css"]
 }
 
-def glob_spec_to(out_file_name):
+title_style_matchers = {
+    "=": re.compile("^=+$"),
+    "-": re.compile("^-+$")
+}
+TOP_LEVEL = "="
+SECOND_LEVEL = "-"
+
+
+def check_valid_section(filename, section):
+    # we need TWO new lines else the next file's title gets merged
+    # the last paragraph *WITHOUT RST PRODUCING A WARNING*
+    if not section[-2:] == '\n\n':
+        raise Exception(
+            "The file " + filename + " does not end with 2 new lines."
+        )
+
+    # Enforce some rules to reduce the risk of having mismatched title
+    # styles.
+    title_line = section.split("\n")[1]
+    if title_line != (len(title_line) * title_line[0]):
+        raise Exception(
+            "The file " + filename + " doesn't have a title style line on line 2"
+        )
+
+    # anything marked as x0_ is the start of a new top-level section
+    if re.match("^[0-9]+0_", filename):
+        if not title_style_matchers[TOP_LEVEL].match(title_line):
+            raise Exception(
+                "The file " + filename + " is a top-level section because it matches " +
+                "the filename format x0_something.rst but has the wrong title " +
+                "style: expected '" + TOP_LEVEL + "' but got '" +
+                title_line[0] + "'"
+            )
+    # anything marked as xx_ is the start of a sub-section
+    elif re.match("^[0-9]+_", filename):
+        if not title_style_matchers[SECOND_LEVEL].match(title_line):
+            raise Exception(
+                "The file " + filename + " is a 2nd-level section because it matches " +
+                "the filename format xx_something.rst but has the wrong title " +
+                "style: expected '" + SECOND_LEVEL + "' but got '" +
+                title_line[0] + "'"
+            )
+
+def cat_spec_sections_to(out_file_name):
     with open(out_file_name, "wb") as outfile:
         for f in sorted(glob.glob("../specification/*.rst")):
             with open(f, "rb") as infile:
                 section = infile.read()
-                # we need TWO new lines else the next file's title gets merged
-                # the last paragraph *WITHOUT RST PRODUCING A WARNING*
-                if not section[-2:] == '\n\n':
-                    raise Exception(
-                        "The file " + f + " does not end with 2 new lines."
-                    )
+                check_valid_section(f.split("/")[-1], section)
                 outfile.write(section)
 
 
@@ -74,7 +113,7 @@ def cleanup_env():
 
 def main():
     prepare_env()
-    glob_spec_to("tmp/full_spec.rst")
+    cat_spec_sections_to("tmp/full_spec.rst")
     run_through_template("tmp/full_spec.rst")
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")
     run_through_template("tmp/howto.rst")

From 9ed39665c8f70d9c704738d07a39877267e0a4da Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 11:09:04 +0100
Subject: [PATCH 32/85] PR feedback

---
 specification/00_basis.rst                | 2 +-
 specification/45_typing_notifications.rst | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/specification/00_basis.rst b/specification/00_basis.rst
index 384d583b..a5196e99 100644
--- a/specification/00_basis.rst
+++ b/specification/00_basis.rst
@@ -444,7 +444,7 @@ The ``error`` string will be a human-readable error message, usually a sentence
 explaining what went wrong. The ``errcode`` string will be a unique string
 which can be used to handle an error message e.g. ``M_FORBIDDEN``. These error
 codes should have their namespace first in ALL CAPS, followed by a single _ to
-ease separating the namespace from the error code.. For example, if there was a
+ease separating the namespace from the error code. For example, if there was a
 custom namespace ``com.mydomain.here``, and a
 ``FORBIDDEN`` code, the error code should look like
 ``COM.MYDOMAIN.HERE_FORBIDDEN``. There may be additional keys depending on the
diff --git a/specification/45_typing_notifications.rst b/specification/45_typing_notifications.rst
index 89f06b48..ff00165d 100644
--- a/specification/45_typing_notifications.rst
+++ b/specification/45_typing_notifications.rst
@@ -11,9 +11,9 @@ To set "I am typing for the next N msec"::
   # timeout is in milliseconds; suggested no more than 20 or 30 seconds
 
 This should be re-sent by the client to continue informing the server the user
-is still typing; I suggest a safety margin of 5 seconds before the expected
-timeout runs out. Just keep declaring a new timeout, it will replace the old
-one.
+is still typing; a safety margin of 5 seconds before the expected
+timeout runs out is recommended. Just keep declaring a new timeout, it will
+replace the old one.
 
 To set "I am no longer typing"::
 

From 06e46c1899755d9ae46b66a8b316b4c41b664328 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 11:20:34 +0100
Subject: [PATCH 33/85] Fix on a better numbering format

---
 scripts/gendoc.py | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 6bf7c1a6..1348679a 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -21,9 +21,16 @@ title_style_matchers = {
 }
 TOP_LEVEL = "="
 SECOND_LEVEL = "-"
+FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}_.*\.rst$")
 
 
 def check_valid_section(filename, section):
+    if not re.match(FILE_FORMAT_MATCHER, filename):
+        raise Exception(
+            "The filename of " + filename +" does not match the expected format " +
+            "of '##_##_words-go-here.rst'"
+        )
+
     # we need TWO new lines else the next file's title gets merged
     # the last paragraph *WITHOUT RST PRODUCING A WARNING*
     if not section[-2:] == '\n\n':
@@ -39,21 +46,21 @@ def check_valid_section(filename, section):
             "The file " + filename + " doesn't have a title style line on line 2"
         )
 
-    # anything marked as x0_ is the start of a new top-level section
-    if re.match("^[0-9]+0_", filename):
+    # anything marked as xx_00_ is the start of a new top-level section
+    if re.match("^[0-9]+_00_", filename):
         if not title_style_matchers[TOP_LEVEL].match(title_line):
             raise Exception(
                 "The file " + filename + " is a top-level section because it matches " +
-                "the filename format x0_something.rst but has the wrong title " +
+                "the filename format ##_00_something.rst but has the wrong title " +
                 "style: expected '" + TOP_LEVEL + "' but got '" +
                 title_line[0] + "'"
             )
-    # anything marked as xx_ is the start of a sub-section
-    elif re.match("^[0-9]+_", filename):
+    # anything marked as xx_x0_ is the start of a sub-section
+    elif re.match("^[0-9]+_0[0-9]{1}_", filename):
         if not title_style_matchers[SECOND_LEVEL].match(title_line):
             raise Exception(
                 "The file " + filename + " is a 2nd-level section because it matches " +
-                "the filename format xx_something.rst but has the wrong title " +
+                "the filename format ##_#0_something.rst but has the wrong title " +
                 "style: expected '" + SECOND_LEVEL + "' but got '" +
                 title_line[0] + "'"
             )

From 7f81501762cb700615de837b7501c6bde9deda11 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 11:39:44 +0100
Subject: [PATCH 34/85] Allow relative references to schema to work in python
 and node.

Rename "schema/v1/core" to "schema/v1/core-event-schema".
Add self-referential symlinks to schema/v1/core-event-schema

The python json schema libraries expect that relative references are
relative to the file they are in. The node json schema libraries
expect that relateive references are relative to the first file loaded.

To support both kinds we reference the core event schema using
"core-event-schema/event.json". We then symlink the core-event-schema
directory to both the location of the file refering to "event.json" so
that it will work in python and to the location of the top level file
so that it will work in node.
---
 api/client-server/v1/core                              |  1 -
 api/client-server/v1/core-event-schema                 |  1 +
 api/client-server/v1/presence.yaml                     |  2 +-
 api/client-server/v1/sync.yaml                         | 10 +++++-----
 api/client-server/v1/{events => v1-event-schema}       |  0
 .../schema/v1/core-event-schema/core-event-schema      |  1 +
 .../schema/v1/{core => core-event-schema}/event.json   |  0
 .../msgtype_infos/image_info.json                      |  0
 .../v1/{core => core-event-schema}/room_event.json     |  2 +-
 .../v1/{core => core-event-schema}/state_event.json    |  2 +-
 event-schemas/schema/v1/m.call.answer                  |  2 +-
 event-schemas/schema/v1/m.call.candidates              |  2 +-
 event-schemas/schema/v1/m.call.hangup                  |  2 +-
 event-schemas/schema/v1/m.call.invite                  |  2 +-
 event-schemas/schema/v1/m.room.aliases                 |  2 +-
 event-schemas/schema/v1/m.room.canonical_alias         |  2 +-
 event-schemas/schema/v1/m.room.create                  |  2 +-
 event-schemas/schema/v1/m.room.history_visibility      |  2 +-
 event-schemas/schema/v1/m.room.join_rules              |  2 +-
 event-schemas/schema/v1/m.room.member                  |  2 +-
 event-schemas/schema/v1/m.room.message                 |  2 +-
 event-schemas/schema/v1/m.room.message#m.audio         |  2 +-
 event-schemas/schema/v1/m.room.message#m.emote         |  2 +-
 event-schemas/schema/v1/m.room.message#m.file          |  4 ++--
 event-schemas/schema/v1/m.room.message#m.image         |  4 ++--
 event-schemas/schema/v1/m.room.message#m.location      |  4 ++--
 event-schemas/schema/v1/m.room.message#m.notice        |  2 +-
 event-schemas/schema/v1/m.room.message#m.text          |  2 +-
 event-schemas/schema/v1/m.room.message#m.video         |  4 ++--
 event-schemas/schema/v1/m.room.message.feedback        |  2 +-
 event-schemas/schema/v1/m.room.name                    |  2 +-
 event-schemas/schema/v1/m.room.power_levels            |  2 +-
 event-schemas/schema/v1/m.room.redaction               |  2 +-
 event-schemas/schema/v1/m.room.topic                   |  2 +-
 event-schemas/schema/v1/v1-event-schema                |  1 +
 templating/matrix_templates/units.py                   |  2 +-
 36 files changed, 40 insertions(+), 38 deletions(-)
 delete mode 120000 api/client-server/v1/core
 create mode 120000 api/client-server/v1/core-event-schema
 rename api/client-server/v1/{events => v1-event-schema} (100%)
 create mode 120000 event-schemas/schema/v1/core-event-schema/core-event-schema
 rename event-schemas/schema/v1/{core => core-event-schema}/event.json (100%)
 rename event-schemas/schema/v1/{core => core-event-schema}/msgtype_infos/image_info.json (100%)
 rename event-schemas/schema/v1/{core => core-event-schema}/room_event.json (93%)
 rename event-schemas/schema/v1/{core => core-event-schema}/state_event.json (93%)
 create mode 120000 event-schemas/schema/v1/v1-event-schema

diff --git a/api/client-server/v1/core b/api/client-server/v1/core
deleted file mode 120000
index 9fc0cd87..00000000
--- a/api/client-server/v1/core
+++ /dev/null
@@ -1 +0,0 @@
-events/core
\ No newline at end of file
diff --git a/api/client-server/v1/core-event-schema b/api/client-server/v1/core-event-schema
new file mode 120000
index 00000000..045aecb0
--- /dev/null
+++ b/api/client-server/v1/core-event-schema
@@ -0,0 +1 @@
+v1-event-schema/core-event-schema
\ No newline at end of file
diff --git a/api/client-server/v1/presence.yaml b/api/client-server/v1/presence.yaml
index 472a2d7f..717e9c47 100644
--- a/api/client-server/v1/presence.yaml
+++ b/api/client-server/v1/presence.yaml
@@ -205,4 +205,4 @@ paths:
               type: object
               title: PresenceEvent
               allOf:
-                - "$ref": "events/core/event.json"
+                - "$ref": "core-event-schema/event.json"
diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index b9525491..27c7073c 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -82,7 +82,7 @@ paths:
                   type: object
                   title: RoomEvent
                   allOf:
-                   - "$ref": "events/core/room_event.json"
+                   - "$ref": "core-event-schema/room_event.json"
         400:
           description: "Bad pagination ``from`` parameter."
   "/initialSync":
@@ -253,7 +253,7 @@ paths:
                   type: object
                   title: Event
                   allOf:
-                   - "$ref": "events/core/event.json"
+                   - "$ref": "core-event-schema/event.json"
               rooms:
                 type: array
                 items:
@@ -294,7 +294,7 @@ paths:
                             type: object
                             title: RoomEvent
                             allOf:
-                              - "$ref": "events/core/room_event.json"
+                              - "$ref": "core-event-schema/room_event.json"
                       required: ["start", "end", "chunk"]
                     state:
                       type: array
@@ -307,7 +307,7 @@ paths:
                         title: StateEvent
                         type: object
                         allOf:
-                          - "$ref": "events/core/state_event.json"
+                          - "$ref": "core-event-schema/state_event.json"
                     visibility:
                       type: string
                       enum: ["private", "public"]
@@ -350,6 +350,6 @@ paths:
               }
           schema:
             allOf:
-              - "$ref": "events/core/event.json"
+              - "$ref": "core-event-schema/event.json"
         404:
           description: The event was not found or you do not have permission to read this event.
diff --git a/api/client-server/v1/events b/api/client-server/v1/v1-event-schema
similarity index 100%
rename from api/client-server/v1/events
rename to api/client-server/v1/v1-event-schema
diff --git a/event-schemas/schema/v1/core-event-schema/core-event-schema b/event-schemas/schema/v1/core-event-schema/core-event-schema
new file mode 120000
index 00000000..945c9b46
--- /dev/null
+++ b/event-schemas/schema/v1/core-event-schema/core-event-schema
@@ -0,0 +1 @@
+.
\ No newline at end of file
diff --git a/event-schemas/schema/v1/core/event.json b/event-schemas/schema/v1/core-event-schema/event.json
similarity index 100%
rename from event-schemas/schema/v1/core/event.json
rename to event-schemas/schema/v1/core-event-schema/event.json
diff --git a/event-schemas/schema/v1/core/msgtype_infos/image_info.json b/event-schemas/schema/v1/core-event-schema/msgtype_infos/image_info.json
similarity index 100%
rename from event-schemas/schema/v1/core/msgtype_infos/image_info.json
rename to event-schemas/schema/v1/core-event-schema/msgtype_infos/image_info.json
diff --git a/event-schemas/schema/v1/core/room_event.json b/event-schemas/schema/v1/core-event-schema/room_event.json
similarity index 93%
rename from event-schemas/schema/v1/core/room_event.json
rename to event-schemas/schema/v1/core-event-schema/room_event.json
index 7ac66b30..d5413f8a 100644
--- a/event-schemas/schema/v1/core/room_event.json
+++ b/event-schemas/schema/v1/core-event-schema/room_event.json
@@ -3,7 +3,7 @@
   "title": "Room Event",
   "description": "In addition to the Event fields, Room Events MUST have the following additional field.",
   "allOf":[{
-    "$ref": "core/event.json"
+    "$ref": "core-event-schema/event.json"
   }],
   "properties": {
     "room_id": {
diff --git a/event-schemas/schema/v1/core/state_event.json b/event-schemas/schema/v1/core-event-schema/state_event.json
similarity index 93%
rename from event-schemas/schema/v1/core/state_event.json
rename to event-schemas/schema/v1/core-event-schema/state_event.json
index 60de9413..f70118f4 100644
--- a/event-schemas/schema/v1/core/state_event.json
+++ b/event-schemas/schema/v1/core-event-schema/state_event.json
@@ -3,7 +3,7 @@
   "title": "State Event",
   "description": "In addition to the Room Event fields, State Events have the following additional fields.",
   "allOf":[{
-    "$ref": "core/room_event.json"
+    "$ref": "core-event-schema/room_event.json"
   }],
   "properties": {
     "state_key": {
diff --git a/event-schemas/schema/v1/m.call.answer b/event-schemas/schema/v1/m.call.answer
index abdf4a9b..f8980a8f 100644
--- a/event-schemas/schema/v1/m.call.answer
+++ b/event-schemas/schema/v1/m.call.answer
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by the callee when they wish to answer the call.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.candidates b/event-schemas/schema/v1/m.call.candidates
index 052ead0b..ad84fe86 100644
--- a/event-schemas/schema/v1/m.call.candidates
+++ b/event-schemas/schema/v1/m.call.candidates
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by callers after sending an invite and by the callee after answering. Its purpose is to give the other party additional ICE candidates to try using to communicate.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.hangup b/event-schemas/schema/v1/m.call.hangup
index 383952d8..ded16a0d 100644
--- a/event-schemas/schema/v1/m.call.hangup
+++ b/event-schemas/schema/v1/m.call.hangup
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.invite b/event-schemas/schema/v1/m.call.invite
index a852ff43..bfeb3b7d 100644
--- a/event-schemas/schema/v1/m.call.invite
+++ b/event-schemas/schema/v1/m.call.invite
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by the caller when they wish to establish a call.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.aliases b/event-schemas/schema/v1/m.room.aliases
index 79fd259b..d1ad0fc7 100644
--- a/event-schemas/schema/v1/m.room.aliases
+++ b/event-schemas/schema/v1/m.room.aliases
@@ -4,7 +4,7 @@
     "title": "Informs the room about what room aliases it has been given.",
     "description": "This event is sent by a homeserver directly to inform of changes to the list of aliases it knows about for that room. The ``state_key`` for this event is set to the homeserver which owns the room alias. The entire set of known aliases for the room is the union of all the ``m.room.aliases`` events, one for each homeserver. Clients **should** check the validity of any room alias given in this list before presenting it to the user as trusted fact. The lists given by this event should be considered simply as advice on which aliases might exist, for which the client can perform the lookup to confirm whether it receives the correct room ID.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
index 17caa321..6d56e27e 100644
--- a/event-schemas/schema/v1/m.room.canonical_alias
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -4,7 +4,7 @@
     "title": "Informs the room as to which alias is the canonical one.",
     "description": "This event is used to inform the room about which alias should be considered the canonical one. This could be for display purposes or as suggestion to users which alias to use to advertise the room.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.create b/event-schemas/schema/v1/m.room.create
index ecc2ed63..bd9231f4 100644
--- a/event-schemas/schema/v1/m.room.create
+++ b/event-schemas/schema/v1/m.room.create
@@ -4,7 +4,7 @@
     "title": "The first event in the room.",
     "description": "This is the first event in a room and cannot be changed. It acts as the root of all other events.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
index 48b0a5c4..a35fe779 100644
--- a/event-schemas/schema/v1/m.room.history_visibility
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -4,7 +4,7 @@
     "title": "Controls visibility of history.",
     "description": "This event controls whether a member of a room can see the events that happened in a room from before they joined.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.join_rules b/event-schemas/schema/v1/m.room.join_rules
index 567247eb..8c70dcaf 100644
--- a/event-schemas/schema/v1/m.room.join_rules
+++ b/event-schemas/schema/v1/m.room.join_rules
@@ -4,7 +4,7 @@
     "title": "Describes how users are allowed to join the room.",
     "description": "A room may be ``public`` meaning anyone can join the room without any prior action. Alternatively, it can be ``invite`` meaning that a user who wishes to join the room must first receive an invite to the room from someone already inside of the room. Currently, ``knock`` and ``private`` are reserved keywords which are not implemented.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 67e10d51..a8a442bc 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -4,7 +4,7 @@
     "title": "The current membership state of a user in the room.",
     "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message
index 61b6256f..59edb676 100644
--- a/event-schemas/schema/v1/m.room.message
+++ b/event-schemas/schema/v1/m.room.message
@@ -4,7 +4,7 @@
     "title": "Message",
     "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.audio b/event-schemas/schema/v1/m.room.message#m.audio
index b236e5b0..41587c1d 100644
--- a/event-schemas/schema/v1/m.room.message#m.audio
+++ b/event-schemas/schema/v1/m.room.message#m.audio
@@ -4,7 +4,7 @@
     "title": "AudioMessage",
     "description": "This message represents a single audio clip.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.emote b/event-schemas/schema/v1/m.room.message#m.emote
index 9f17180c..e6bf5e68 100644
--- a/event-schemas/schema/v1/m.room.message#m.emote
+++ b/event-schemas/schema/v1/m.room.message#m.emote
@@ -4,7 +4,7 @@
     "title": "EmoteMessage",
     "description": "This message is similar to ``m.text`` except that the sender is 'performing' the action contained in the ``body`` key, similar to ``/me`` in IRC. This message should be prefixed by the name of the sender. This message could also be represented in a different colour to distinguish it from regular ``m.text`` messages.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.file b/event-schemas/schema/v1/m.room.message#m.file
index d8c63b33..ed996694 100644
--- a/event-schemas/schema/v1/m.room.message#m.file
+++ b/event-schemas/schema/v1/m.room.message#m.file
@@ -4,7 +4,7 @@
     "title": "FileMessage",
     "description": "This message represents a generic file.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
@@ -50,7 +50,7 @@
                     "title": "ImageInfo",
                     "description": "Metadata about the image referred to in ``thumbnail_url``.",
                     "allOf": [{
-                        "$ref": "core/msgtype_infos/image_info.json"
+                        "$ref": "core-event-schema/msgtype_infos/image_info.json"
                     }]
                 }
             },
diff --git a/event-schemas/schema/v1/m.room.message#m.image b/event-schemas/schema/v1/m.room.message#m.image
index ca22654f..8baee04e 100644
--- a/event-schemas/schema/v1/m.room.message#m.image
+++ b/event-schemas/schema/v1/m.room.message#m.image
@@ -4,7 +4,7 @@
     "title": "ImageMessage",
     "description": "This message represents a single image and an optional thumbnail.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
@@ -31,7 +31,7 @@
                     "title": "ImageInfo",
                     "description": "Metadata about the image referred to in ``thumbnail_url``.",
                     "allOf": [{
-                        "$ref": "core/msgtype_infos/image_info.json"
+                        "$ref": "core-event-schema/msgtype_infos/image_info.json"
                     }]
                 },
                 "info": {
diff --git a/event-schemas/schema/v1/m.room.message#m.location b/event-schemas/schema/v1/m.room.message#m.location
index 919c3325..9e59a0a9 100644
--- a/event-schemas/schema/v1/m.room.message#m.location
+++ b/event-schemas/schema/v1/m.room.message#m.location
@@ -4,7 +4,7 @@
     "title": "LocationMessage",
     "description": "This message represents a real-world location.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
@@ -30,7 +30,7 @@
                     "type": "object",
                     "title": "ImageInfo",
                     "allOf": [{
-                        "$ref": "core/msgtype_infos/image_info.json"
+                        "$ref": "core-event-schema/msgtype_infos/image_info.json"
                     }]
                 }
             },
diff --git a/event-schemas/schema/v1/m.room.message#m.notice b/event-schemas/schema/v1/m.room.message#m.notice
index e6ce6f89..5fd217f8 100644
--- a/event-schemas/schema/v1/m.room.message#m.notice
+++ b/event-schemas/schema/v1/m.room.message#m.notice
@@ -4,7 +4,7 @@
     "title": "NoticeMessage",
     "description": "A m.notice message should be considered similar to a plain m.text message except that clients should visually distinguish it in some way. It is intended to be used by automated clients, such as bots, bridges, and other entities, rather than humans. Additionally, such automated agents which watch a room for messages and respond to them ought to ignore m.notice messages. This helps to prevent infinite-loop situations where two automated clients continuously exchange messages, as each responds to the other.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.text b/event-schemas/schema/v1/m.room.message#m.text
index ceadbcd0..e4b5ab11 100644
--- a/event-schemas/schema/v1/m.room.message#m.text
+++ b/event-schemas/schema/v1/m.room.message#m.text
@@ -4,7 +4,7 @@
     "title": "TextMessage",
     "description": "This message is the most basic message and is used to represent text.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.video b/event-schemas/schema/v1/m.room.message#m.video
index 667832ff..2de0f26a 100644
--- a/event-schemas/schema/v1/m.room.message#m.video
+++ b/event-schemas/schema/v1/m.room.message#m.video
@@ -4,7 +4,7 @@
     "title": "VideoMessage",
     "description": "This message represents a single video clip.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
@@ -55,7 +55,7 @@
                             "type": "object",
                             "title": "ImageInfo",
                             "allOf": [{
-                                "$ref": "core/msgtype_infos/image_info.json"
+                                "$ref": "core-event-schema/msgtype_infos/image_info.json"
                             }]
                         }
                     }
diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback
index b662e9e6..2dc63b8a 100644
--- a/event-schemas/schema/v1/m.room.message.feedback
+++ b/event-schemas/schema/v1/m.room.message.feedback
@@ -4,7 +4,7 @@
     "title": "MessageFeedback",
     "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.name b/event-schemas/schema/v1/m.room.name
index 5565147c..0e212a23 100644
--- a/event-schemas/schema/v1/m.room.name
+++ b/event-schemas/schema/v1/m.room.name
@@ -4,7 +4,7 @@
     "description": "A room has an opaque room ID which is not human-friendly to read. A room alias is human-friendly, but not all rooms have room aliases. The room name is a human-friendly string designed to be displayed to the end-user. The room name is not unique, as multiple rooms can have the same room name set. The room name can also be set when creating a room using ``/createRoom`` with the ``name`` key.",
     "type": "object",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.power_levels b/event-schemas/schema/v1/m.room.power_levels
index 95850fcb..a682a214 100644
--- a/event-schemas/schema/v1/m.room.power_levels
+++ b/event-schemas/schema/v1/m.room.power_levels
@@ -4,7 +4,7 @@
     "title": "Defines the power levels (privileges) of users in the room.",
     "description": "This event specifies the minimum level a user must have in order to perform a certain action. It also specifies the levels of each user in the room. If a ``user_id`` is in the ``users`` list, then that ``user_id`` has the associated power level. Otherwise they have the default level ``users_default``. If ``users_default`` is not supplied, it is assumed to be 0. The level required to send a certain event is governed by ``events``, ``state_default`` and ``events_default``. If an event type is specified in ``events``, then the user must have at least the level specified in order to send that event. If the event type is not supplied, it defaults to ``events_default`` for Message Events and ``state_default`` for State Events.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.redaction b/event-schemas/schema/v1/m.room.redaction
index b095b17d..5168e133 100644
--- a/event-schemas/schema/v1/m.room.redaction
+++ b/event-schemas/schema/v1/m.room.redaction
@@ -4,7 +4,7 @@
     "title": "Redaction",
     "description": "Events can be redacted by either room or server admins. Redacting an event means that all keys not required by the protocol are stripped off, allowing admins to remove offensive or illegal content that may have been attached to any event. This cannot be undone, allowing server owners to physically delete the offending data.  There is also a concept of a moderator hiding a message event, which can be undone, but cannot be applied to state events. The event that has been redacted is specified in the ``redacts`` event level key.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.topic b/event-schemas/schema/v1/m.room.topic
index 32319c38..f88f56ca 100644
--- a/event-schemas/schema/v1/m.room.topic
+++ b/event-schemas/schema/v1/m.room.topic
@@ -4,7 +4,7 @@
     "title": "Topic",
     "description": "A topic is a short message detailing what is currently being discussed in the room.  It can also be used as a way to display extra information about the room, which may not be suitable for the room name. The room topic can also be set when creating a room using ``/createRoom`` with the ``topic`` key.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/v1-event-schema b/event-schemas/schema/v1/v1-event-schema
new file mode 120000
index 00000000..945c9b46
--- /dev/null
+++ b/event-schemas/schema/v1/v1-event-schema
@@ -0,0 +1 @@
+.
\ No newline at end of file
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 4b8127ae..8e803d3b 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -281,7 +281,7 @@ class MatrixUnits(Units):
         return apis
 
     def load_common_event_fields(self):
-        path = "../event-schemas/schema/v1/core"
+        path = "../event-schemas/schema/v1/core-event-schema"
         event_types = {}
 
         for (root, dirs, files) in os.walk(path):

From 1f6b12b3e828696a588fed9a0028e59991d28b94 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 11:43:53 +0100
Subject: [PATCH 35/85] Modify and enforce the file format/structure used

Convert the file format to be of the form ##_##_something.rst where the
first ## is the top-level section number and the second ## is the
second-level section number, e.g. 07_01_push_cs_api.rst means
Section 7.1 - This is now enforced in gendoc.py along with the title line
style that should be used (= for top-level, - for 2nd level) which will
give helpful suggestions if you trip up. This feels much more intuitive
now looking in /specification
---
 scripts/gendoc.py                                  | 12 +++++++-----
 specification/{00_basis.rst => 00_00_intro.rst}    |  0
 ..._server_api.rst => 01_00_client_server_api.rst} |  0
 ...ications.rst => 01_01_typing_notifications.rst} |  8 ++++----
 .../{46_receipts.rst => 01_02_receipts.rst}        |  8 ++++----
 ...visibility.rst => 01_03_history_visibility.rst} |  2 +-
 specification/{20_events.rst => 02_00_events.rst}  |  0
 .../{22_voip_events.rst => 02_01_voip_events.rst}  |  0
 ...1_event_signing.rst => 02_02_event_signing.rst} | 14 +++++++-------
 ...e_api.rst => 03_00_application_service_api.rst} |  0
 ..._server_api.rst => 04_00_server_server_api.rst} |  0
 ...repository.rst => 05_00_content_repository.rst} |  0
 ...ryption.rst => 06_00_end_to_end_encryption.rst} |  0
 ...2_push_overview.rst => 07_00_push_overview.rst} |  0
 .../{43_push_cs_api.rst => 07_01_push_cs_api.rst}  |  0
 ..._push_gw_api.rst => 07_02_push_push_gw_api.rst} |  0
 ...n_core_apis.rst => 08_00_address_book_repo.rst} |  0
 ...tity_servers.rst => 09_00_identity_servers.rst} |  0
 .../{50_appendices.rst => 10_00_appendices.rst}    |  0
 19 files changed, 23 insertions(+), 21 deletions(-)
 rename specification/{00_basis.rst => 00_00_intro.rst} (100%)
 rename specification/{10_client_server_api.rst => 01_00_client_server_api.rst} (100%)
 rename specification/{45_typing_notifications.rst => 01_01_typing_notifications.rst} (96%)
 rename specification/{46_receipts.rst => 01_02_receipts.rst} (94%)
 rename specification/{47_history_visibility.rst => 01_03_history_visibility.rst} (97%)
 rename specification/{20_events.rst => 02_00_events.rst} (100%)
 rename specification/{22_voip_events.rst => 02_01_voip_events.rst} (100%)
 rename specification/{31_event_signing.rst => 02_02_event_signing.rst} (99%)
 rename specification/{25_application_service_api.rst => 03_00_application_service_api.rst} (100%)
 rename specification/{30_server_server_api.rst => 04_00_server_server_api.rst} (100%)
 rename specification/{40_content_repository.rst => 05_00_content_repository.rst} (100%)
 rename specification/{41_end_to_end_encryption.rst => 06_00_end_to_end_encryption.rst} (100%)
 rename specification/{42_push_overview.rst => 07_00_push_overview.rst} (100%)
 rename specification/{43_push_cs_api.rst => 07_01_push_cs_api.rst} (100%)
 rename specification/{44_push_push_gw_api.rst => 07_02_push_push_gw_api.rst} (100%)
 rename specification/{49_other_non_core_apis.rst => 08_00_address_book_repo.rst} (100%)
 rename specification/{60_identity_servers.rst => 09_00_identity_servers.rst} (100%)
 rename specification/{50_appendices.rst => 10_00_appendices.rst} (100%)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 1348679a..c97995e8 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -21,7 +21,7 @@ title_style_matchers = {
 }
 TOP_LEVEL = "="
 SECOND_LEVEL = "-"
-FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}_.*\.rst$")
+FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}[a-z]*_.*\.rst$")
 
 
 def check_valid_section(filename, section):
@@ -55,14 +55,16 @@ def check_valid_section(filename, section):
                 "style: expected '" + TOP_LEVEL + "' but got '" +
                 title_line[0] + "'"
             )
-    # anything marked as xx_x0_ is the start of a sub-section
-    elif re.match("^[0-9]+_0[0-9]{1}_", filename):
+    # anything marked as xx_xx_ is the start of a sub-section
+    elif re.match("^[0-9]+_[0-9]{2}_", filename):
         if not title_style_matchers[SECOND_LEVEL].match(title_line):
             raise Exception(
                 "The file " + filename + " is a 2nd-level section because it matches " +
-                "the filename format ##_#0_something.rst but has the wrong title " +
+                "the filename format ##_##_something.rst but has the wrong title " +
                 "style: expected '" + SECOND_LEVEL + "' but got '" +
-                title_line[0] + "'"
+                title_line[0] + "' - If this is meant to be a 3rd/4th/5th-level section " +
+                "then use the form '##_##b_something.rst' which will not apply this " +
+                "check."
             )
 
 def cat_spec_sections_to(out_file_name):
diff --git a/specification/00_basis.rst b/specification/00_00_intro.rst
similarity index 100%
rename from specification/00_basis.rst
rename to specification/00_00_intro.rst
diff --git a/specification/10_client_server_api.rst b/specification/01_00_client_server_api.rst
similarity index 100%
rename from specification/10_client_server_api.rst
rename to specification/01_00_client_server_api.rst
diff --git a/specification/45_typing_notifications.rst b/specification/01_01_typing_notifications.rst
similarity index 96%
rename from specification/45_typing_notifications.rst
rename to specification/01_01_typing_notifications.rst
index ff00165d..25b714ab 100644
--- a/specification/45_typing_notifications.rst
+++ b/specification/01_01_typing_notifications.rst
@@ -1,8 +1,8 @@
 Typing Notifications
-====================
+--------------------
 
 Client APIs
------------
+~~~~~~~~~~~
 
 To set "I am typing for the next N msec"::
 
@@ -21,7 +21,7 @@ To set "I am no longer typing"::
   Content: { "typing": false }
 
 Client Events
--------------
+~~~~~~~~~~~~~
 
 All room members will receive an event on the event stream::
 
@@ -39,7 +39,7 @@ users who are not currently typing, as that list gets big quickly. The client
 should mark as not typing, any user ID who is not in that list.
 
 Server APIs
------------
+~~~~~~~~~~~
 
 Servers will emit EDUs in the following form::
 
diff --git a/specification/46_receipts.rst b/specification/01_02_receipts.rst
similarity index 94%
rename from specification/46_receipts.rst
rename to specification/01_02_receipts.rst
index a0a90c16..e2f83eea 100644
--- a/specification/46_receipts.rst
+++ b/specification/01_02_receipts.rst
@@ -1,5 +1,5 @@
 Receipts
-========
+--------
 
 Receipts are used to publish which events in a room the user or their devices
 have interacted with. For example, which events the user has read. For
@@ -7,7 +7,7 @@ efficiency this is done as "up to" markers, i.e. marking a particular event
 as, say, ``read`` indicates the user has read all events *up to* that event.
 
 Client-Server API
------------------
+~~~~~~~~~~~~~~~~~
 
 Clients will receive receipts in the following format::
 
@@ -58,7 +58,7 @@ other users. The server will automatically set the ``ts`` field.
 
 
 Server-Server API
------------------
+~~~~~~~~~~~~~~~~~
 
 Receipts are sent across federation as EDUs with type ``m.receipt``. The
 format of the EDUs are::
@@ -73,5 +73,5 @@ format of the EDUs are::
         ...
     }
 
-These are always sent as deltas to previously sent reciepts.
+These are always sent as deltas to previously sent receipts.
 
diff --git a/specification/47_history_visibility.rst b/specification/01_03_history_visibility.rst
similarity index 97%
rename from specification/47_history_visibility.rst
rename to specification/01_03_history_visibility.rst
index b1630f1e..01c2e419 100644
--- a/specification/47_history_visibility.rst
+++ b/specification/01_03_history_visibility.rst
@@ -1,5 +1,5 @@
 Room History Visibility
-=======================
+-----------------------
 
 Whether a member of a room can see the events that happened in a room from
 before they joined the room is controlled by the ``history_visibility`` key
diff --git a/specification/20_events.rst b/specification/02_00_events.rst
similarity index 100%
rename from specification/20_events.rst
rename to specification/02_00_events.rst
diff --git a/specification/22_voip_events.rst b/specification/02_01_voip_events.rst
similarity index 100%
rename from specification/22_voip_events.rst
rename to specification/02_01_voip_events.rst
diff --git a/specification/31_event_signing.rst b/specification/02_02_event_signing.rst
similarity index 99%
rename from specification/31_event_signing.rst
rename to specification/02_02_event_signing.rst
index f3a22d1d..bde58f0c 100644
--- a/specification/31_event_signing.rst
+++ b/specification/02_02_event_signing.rst
@@ -1,8 +1,8 @@
 Signing Events
-==============
+--------------
 
 Canonical JSON
---------------
+~~~~~~~~~~~~~~
 
 Matrix events are represented using JSON objects. If we want to sign JSON
 events we need to encode the JSON as a binary string. Unfortunately the same
@@ -38,7 +38,7 @@ using this representation.
      ).encode("UTF-8")
 
 Grammar
-~~~~~~~
++++++++
 
 Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing
 insignificant whitespace, fractions, exponents and redundant character escapes
@@ -69,14 +69,14 @@ insignificant whitespace, fractions, exponents and redundant character escapes
            / %x75.30.30.31 (%x30-39 / %x61-66)        ; u001X
 
 Signing JSON
-------------
+~~~~~~~~~~~~
 
 We can now sign a JSON object by encoding it as a sequence of bytes, computing
 the signature for that sequence and then adding the signature to the original
 JSON object.
 
 Signing Details
-~~~~~~~~~~~~~~~
++++++++++++++++
 
 JSON is signed by encoding the JSON object without ``signatures`` or keys grouped
 as ``unsigned``, using the canonical encoding described above. The JSON bytes are then signed using the
@@ -133,7 +133,7 @@ and additional signatures.
       return json_object
 
 Checking for a Signature
-~~~~~~~~~~~~~~~~~~~~~~~~
+++++++++++++++++++++++++
 
 To check if an entity has signed a JSON object a server does the following
 
@@ -151,7 +151,7 @@ To check if an entity has signed a JSON object a server does the following
    the check fails. Otherwise the check succeeds.
 
 Signing Events
---------------
+~~~~~~~~~~~~~~
 
 Signing events is a more complicated process since servers can choose to redact
 non-essential parts of an event. Before signing the event it is encoded as
diff --git a/specification/25_application_service_api.rst b/specification/03_00_application_service_api.rst
similarity index 100%
rename from specification/25_application_service_api.rst
rename to specification/03_00_application_service_api.rst
diff --git a/specification/30_server_server_api.rst b/specification/04_00_server_server_api.rst
similarity index 100%
rename from specification/30_server_server_api.rst
rename to specification/04_00_server_server_api.rst
diff --git a/specification/40_content_repository.rst b/specification/05_00_content_repository.rst
similarity index 100%
rename from specification/40_content_repository.rst
rename to specification/05_00_content_repository.rst
diff --git a/specification/41_end_to_end_encryption.rst b/specification/06_00_end_to_end_encryption.rst
similarity index 100%
rename from specification/41_end_to_end_encryption.rst
rename to specification/06_00_end_to_end_encryption.rst
diff --git a/specification/42_push_overview.rst b/specification/07_00_push_overview.rst
similarity index 100%
rename from specification/42_push_overview.rst
rename to specification/07_00_push_overview.rst
diff --git a/specification/43_push_cs_api.rst b/specification/07_01_push_cs_api.rst
similarity index 100%
rename from specification/43_push_cs_api.rst
rename to specification/07_01_push_cs_api.rst
diff --git a/specification/44_push_push_gw_api.rst b/specification/07_02_push_push_gw_api.rst
similarity index 100%
rename from specification/44_push_push_gw_api.rst
rename to specification/07_02_push_push_gw_api.rst
diff --git a/specification/49_other_non_core_apis.rst b/specification/08_00_address_book_repo.rst
similarity index 100%
rename from specification/49_other_non_core_apis.rst
rename to specification/08_00_address_book_repo.rst
diff --git a/specification/60_identity_servers.rst b/specification/09_00_identity_servers.rst
similarity index 100%
rename from specification/60_identity_servers.rst
rename to specification/09_00_identity_servers.rst
diff --git a/specification/50_appendices.rst b/specification/10_00_appendices.rst
similarity index 100%
rename from specification/50_appendices.rst
rename to specification/10_00_appendices.rst

From 63f08bace61b01d629bcdc3beaaf8870c7bfd993 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 14:40:48 +0100
Subject: [PATCH 36/85] Fix the examples in the swagger API documentation to be
 valid JSON

---
 api/client-server/v1/presence.yaml   | 6 +++---
 api/client-server/v1/sync.yaml       | 2 +-
 templating/matrix_templates/units.py | 2 ++
 3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/api/client-server/v1/presence.yaml b/api/client-server/v1/presence.yaml
index 717e9c47..5684398b 100644
--- a/api/client-server/v1/presence.yaml
+++ b/api/client-server/v1/presence.yaml
@@ -101,7 +101,7 @@ paths:
                   The length of time in milliseconds since an action was performed
                   by this user.
               status_msg:
-                type: string
+                type: [string, "null"]
                 description: The state message for this user if one was set.
         404:
           description: |-
@@ -185,7 +185,7 @@ paths:
                     "last_active_ago": 395,
                     "presence": "offline",
                     "user_id": "@alice:matrix.org"
-                  }
+                  },
                   "type": "m.presence"
                 },
                 {
@@ -195,7 +195,7 @@ paths:
                     "last_active_ago": 16874,
                     "presence": "online",
                     "user_id": "@marisa:matrix.org"
-                  }
+                  },
                   "type": "m.presence"
                 }
               ]
diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 27c7073c..833c425a 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -343,7 +343,7 @@ paths:
                   "body": "Hello world!",
                   "msgtype": "m.text"
                 },
-                "room_id:" "!wfgy43Sg4a:matrix.org",
+                "room_id:": "!wfgy43Sg4a:matrix.org",
                 "user_id": "@bob:matrix.org",
                 "event_id": "$asfDuShaf7Gafaw:matrix.org",
                 "type": "m.room.message"
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 8e803d3b..3083a51e 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -87,6 +87,8 @@ def get_json_schema_object_fields(obj, enforce_title=False):
                     desc += (
                         " Must be '%s'." % props[key_name]["enum"][0]
                     )
+            if isinstance(value_type, list):
+                value_type = " or ".join(value_type)
 
         fields["rows"].append({
             "key": key_name,

From f99a38ce726e5d04a9be182bd42f43daf73a3364 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 14:43:57 +0100
Subject: [PATCH 37/85] Update the hard-coded paths in templating units.py.

Replace the hard code paths with global variables.
---
 templating/matrix_templates/units.py | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 4b8127ae..701b68e2 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -8,6 +8,16 @@ import subprocess
 import urllib
 import yaml
 
+V1_CLIENT_API = "../api/client-server/v1"
+V1_EVENT_EXAMPLES = "../event-schemas/examples/v1"
+V1_EVENT_SCHEMA = "../event-schemas/schema/v1"
+CORE_EVENT_SCHEMA = "../event-schemas/schema/v1/core-event-schema"
+CHANGELOG = "../CHANGELOG.rst"
+
+ROOM_EVENT = "core-event-schema/room_event.json"
+STATE_EVENT = "core-event-schema/state_event.json"
+
+
 def get_json_schema_object_fields(obj, enforce_title=False):
     # Algorithm:
     # f.e. property => add field info (if field is object then recurse)
@@ -266,7 +276,7 @@ class MatrixUnits(Units):
         }
 
     def load_swagger_apis(self):
-        path = "../api/client-server/v1"
+        path = V1_CLIENT_API
         apis = {}
         for filename in os.listdir(path):
             if not filename.endswith(".yaml"):
@@ -281,7 +291,7 @@ class MatrixUnits(Units):
         return apis
 
     def load_common_event_fields(self):
-        path = "../event-schemas/schema/v1/core"
+        path = CORE_EVENT_SCHEMA
         event_types = {}
 
         for (root, dirs, files) in os.walk(path):
@@ -320,7 +330,7 @@ class MatrixUnits(Units):
         return event_types
 
     def load_event_examples(self):
-        path = "../event-schemas/examples/v1"
+        path = V1_EVENT_EXAMPLES
         examples = {}
         for filename in os.listdir(path):
             if not filename.startswith("m."):
@@ -332,7 +342,7 @@ class MatrixUnits(Units):
         return examples
 
     def load_event_schemas(self):
-        path = "../event-schemas/schema/v1"
+        path = V1_EVENT_SCHEMA
         schemata = {}
 
         for filename in os.listdir(path):
@@ -361,8 +371,8 @@ class MatrixUnits(Units):
 
                 # add typeof
                 base_defs = {
-                    "core#/definitions/room_event": "Message Event",
-                    "core#/definitions/state_event": "State Event"
+                    ROOM_EVENT: "Message Event",
+                    STATE_EVENT: "State Event"
                 }
                 if type(json_schema.get("allOf")) == list:
                     schema["typeof"] = base_defs.get(
@@ -413,7 +423,7 @@ class MatrixUnits(Units):
         return schemata
 
     def load_spec_meta(self):
-        path = "../CHANGELOG.rst"
+        path = CHANGELOG
         title_part = None
         version = None
         changelog_lines = []

From 427e4c8b1b23ab875b24acad5605848abf0d8ee9 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 15:46:37 +0100
Subject: [PATCH 38/85] Log port being listened on

---
 scripts/continuserv/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index e7757c06..dc7705a8 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -56,7 +56,7 @@ func main() {
 	go doPopulate(ch, dir)
 
 	go watchFS(ch, w)
-
+    fmt.Printf("Listening on port %d\n", *port)
 	http.HandleFunc("/", serve)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 

From ba4998a62b28fcd13c9085ae58996d1c857b27c5 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 15:49:01 +0100
Subject: [PATCH 39/85] Apparently tabs are A Thing

---
 scripts/continuserv/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index dc7705a8..658ae0fb 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -56,7 +56,7 @@ func main() {
 	go doPopulate(ch, dir)
 
 	go watchFS(ch, w)
-    fmt.Printf("Listening on port %d\n", *port)
+	fmt.Printf("Listening on port %d\n", *port)
 	http.HandleFunc("/", serve)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 

From cf8abdafd43ad67802b20896d53ab977d6d4beee Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 15:42:49 +0100
Subject: [PATCH 40/85] Fix PR comments

---
 scripts/gendoc.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index c97995e8..e2f51419 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -27,13 +27,13 @@ FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}[a-z]*_.*\.rst$")
 def check_valid_section(filename, section):
     if not re.match(FILE_FORMAT_MATCHER, filename):
         raise Exception(
-            "The filename of " + filename +" does not match the expected format " +
+            "The filename of " + filename + " does not match the expected format " +
             "of '##_##_words-go-here.rst'"
         )
 
     # we need TWO new lines else the next file's title gets merged
     # the last paragraph *WITHOUT RST PRODUCING A WARNING*
-    if not section[-2:] == '\n\n':
+    if not section[-2:] == "\n\n":
         raise Exception(
             "The file " + filename + " does not end with 2 new lines."
         )
@@ -72,7 +72,7 @@ def cat_spec_sections_to(out_file_name):
         for f in sorted(glob.glob("../specification/*.rst")):
             with open(f, "rb") as infile:
                 section = infile.read()
-                check_valid_section(f.split("/")[-1], section)
+                check_valid_section(os.path.basename(f), section)
                 outfile.write(section)
 
 

From 299a4356d49e9b694c1fb6340cda2669e8e30377 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 16:10:21 +0100
Subject: [PATCH 41/85] Add script to check that the example responses in the
 swagger matches the examples.

---
 api/check_examples.py | 75 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)
 create mode 100755 api/check_examples.py

diff --git a/api/check_examples.py b/api/check_examples.py
new file mode 100755
index 00000000..cec8ac09
--- /dev/null
+++ b/api/check_examples.py
@@ -0,0 +1,75 @@
+#! /usr/bin/env python
+
+import sys
+
+def import_error(module, package, debian, error):
+    sys.stderr.write((
+        "Error importing %(module)s: %(error)r\n"
+        "To install %(module)s run:\n"
+        "  pip install %(package)s\n"
+        "or on Debian run:\n"
+        "  sudo apt-get install python-%(debian)s\n"
+    ) % locals())
+    if __name__=='__main__':
+        sys.exit(1)
+
+try:
+    import jsonschema
+except ImportError as e:
+    import_error("jsonschema", "jsonschema", "jsonschema", e)
+    raise
+
+try:
+    import yaml
+except ImportError as e:
+    import_error("yaml", "PyYAML", "yaml", e)
+    raise
+
+import json
+import os
+
+def check_response(filepath, request, code, response):
+    try:
+        example = json.loads(
+            response.get('examples', {}).get('application/json', "null")
+        )
+    except Exception as e:
+        raise ValueError("Error parsing JSON example response for %r %r" % (
+            request, code
+        ), e)
+    schema = response.get('schema')
+    fileurl = "file://" + os.path.abspath(filepath)
+    if example and schema:
+        try:
+            print ("Checking schema for: %r %r %r" % (filepath, request, code))
+            # Setting the 'id' tells jsonschema where the file is so that it
+            # can correctly resolve relative $ref references in the schema
+            schema['id'] = fileurl
+            jsonschema.validate(example, schema)
+        except Exception as e:
+            raise ValueError("Error validating JSON schema for %r %r" %(
+                request, code
+            ), e)
+
+
+def check_swagger_file(filepath):
+    with open(filepath) as f:
+        swagger = yaml.load(f)
+
+    for path, path_api in swagger['paths'].items():
+        for method, request_api in path_api.items():
+            request = "%s %s" % (method.upper(), path)
+            try:
+                responses = request_api['responses']
+            except KeyError:
+                raise ValueError("No responses for %r" % (request,))
+            for code, response in responses.items():
+                check_response(filepath, request, code, response)
+
+
+if __name__=='__main__':
+    for path in sys.argv[1:]:
+        try:
+            check_swagger_file(path)
+        except Exception as e:
+            raise ValueError("Error checking file %r" % (path,), e)

From 9896f98e2b63fd6204482bd9c7b8ba699487576d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 16:21:48 +0100
Subject: [PATCH 42/85] Search for yaml swagger files if check_examples.py is
 run without arguments.

---
 api/check_examples.py | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/api/check_examples.py b/api/check_examples.py
index cec8ac09..9b2f1fe9 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -1,6 +1,9 @@
 #! /usr/bin/env python
 
 import sys
+import json
+import os
+
 
 def import_error(module, package, debian, error):
     sys.stderr.write((
@@ -10,7 +13,7 @@ def import_error(module, package, debian, error):
         "or on Debian run:\n"
         "  sudo apt-get install python-%(debian)s\n"
     ) % locals())
-    if __name__=='__main__':
+    if __name__ == '__main__':
         sys.exit(1)
 
 try:
@@ -25,8 +28,6 @@ except ImportError as e:
     import_error("yaml", "PyYAML", "yaml", e)
     raise
 
-import json
-import os
 
 def check_response(filepath, request, code, response):
     try:
@@ -47,7 +48,7 @@ def check_response(filepath, request, code, response):
             schema['id'] = fileurl
             jsonschema.validate(example, schema)
         except Exception as e:
-            raise ValueError("Error validating JSON schema for %r %r" %(
+            raise ValueError("Error validating JSON schema for %r %r" % (
                 request, code
             ), e)
 
@@ -56,7 +57,7 @@ def check_swagger_file(filepath):
     with open(filepath) as f:
         swagger = yaml.load(f)
 
-    for path, path_api in swagger['paths'].items():
+    for path, path_api in swagger.get('paths', {}).items():
         for method, request_api in path_api.items():
             request = "%s %s" % (method.upper(), path)
             try:
@@ -67,8 +68,15 @@ def check_swagger_file(filepath):
                 check_response(filepath, request, code, response)
 
 
-if __name__=='__main__':
-    for path in sys.argv[1:]:
+if __name__ == '__main__':
+    paths = sys.argv[1:]
+    if not paths:
+        paths = []
+        for (root, dirs, files) in os.walk(os.curdir):
+            for filename in files:
+                if filename.endswith(".yaml"):
+                    paths.append(os.path.join(root, filename))
+    for path in paths:
         try:
             check_swagger_file(path)
         except Exception as e:

From f827765ba1999073bc9357695b27bcc3e7e0c30c Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 16:35:27 +0100
Subject: [PATCH 43/85] Make to code to skip checking swagger responses which
 don't have an application/json example clearer.

---
 api/check_examples.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/api/check_examples.py b/api/check_examples.py
index 9b2f1fe9..00e75263 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -30,10 +30,11 @@ except ImportError as e:
 
 
 def check_response(filepath, request, code, response):
+    example = None
     try:
-        example = json.loads(
-            response.get('examples', {}).get('application/json', "null")
-        )
+        example_json = response.get('examples', {}).get('application/json')
+        if example_json:
+            example = json.loads(example_json)
     except Exception as e:
         raise ValueError("Error parsing JSON example response for %r %r" % (
             request, code

From 6b5b8432b3761484f3e36910575f5fd844c09400 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 17:26:10 +0100
Subject: [PATCH 44/85] Turn on code highlighting for HTTP api responses and
 add a code highlighting stylesheet for the specification.

---
 scripts/codehighlight.css                           | 6 ++++++
 scripts/gendoc.py                                   | 2 +-
 templating/matrix_templates/templates/http-api.tmpl | 4 +++-
 3 files changed, 10 insertions(+), 2 deletions(-)
 create mode 100644 scripts/codehighlight.css

diff --git a/scripts/codehighlight.css b/scripts/codehighlight.css
new file mode 100644
index 00000000..5c9b0c36
--- /dev/null
+++ b/scripts/codehighlight.css
@@ -0,0 +1,6 @@
+pre.code .comment, code .comment { color: green }
+pre.code .keyword, code .keyword { color: darkred; font-weight: bold }
+pre.code .name.builtin, code .name.builtin { color: darkred; font-weight: bold }
+pre.code .literal.number, code .literal.number { color: blue }
+pre.code .name.tag, code .name.tag { color: darkgreen }
+pre.code .literal.string, code .literal.string { color: darkblue }
diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 3521efed..a821aea7 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -12,7 +12,7 @@ import sys
 os.chdir(os.path.dirname(os.path.abspath(__file__)))
 
 stylesheets = {
-    "stylesheet_path": ["basic.css", "nature.css"]
+    "stylesheet_path": ["basic.css", "nature.css", "codehighlight.css"]
 }
 
 title_style_matchers = {
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index a0b25924..a03ffa7d 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -62,7 +62,9 @@ Response{{"s" if endpoint.example.responses|length > 1 else "" }}:
 
 {{res["description"]}}
 
-Example::
+Example
+
+.. code:: json
 
   {{res["example"] | indent_block(2)}}
 

From 6a2c4d27fc187d6efb755705be9eca7a07dec26f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 17:58:44 +0100
Subject: [PATCH 45/85] Update the docs for room v1 api

---
 api/client-server/v1/rooms.yaml       | 10 +++++-----
 event-schemas/schema/v1/m.room.member |  2 +-
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index c038dba5..4902560b 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -168,7 +168,7 @@ paths:
               title: StateEvent
               type: object
               allOf:
-                - "$ref": "events/core/state_event.json"
+                - "$ref": "core-event-schema/state_event.json"
         403:
           description: You are not joined to the room.
 
@@ -344,7 +344,7 @@ paths:
                       type: object
                       title: RoomEvent
                       allOf:
-                        - "$ref": "events/core/room_event.json"
+                        - "$ref": "core-event-schema/room_event.json"
                 required: ["start", "end", "chunk"]
               state:
                 type: array
@@ -357,7 +357,7 @@ paths:
                   title: StateEvent
                   type: object
                   allOf:
-                    - "$ref": "events/core/state_event.json"
+                    - "$ref": "core-event-schema/state_event.json"
               visibility:
                 type: string
                 enum: ["private", "public"]
@@ -414,7 +414,7 @@ paths:
                     "state_key": "@bob:example.com",
                     "type": "m.room.member",
                     "user_id": "@bob:example.com"
-                  },
+                  }
                 ]
               }
           schema:
@@ -426,5 +426,5 @@ paths:
                   title: MemberEvent
                   type: object
                   allOf:
-                    - "$ref": "events/m.room.member"
+                    - "$ref": "v1-event-schema/m.room.member"
 
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 48f1b034..49b9f5b8 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -19,7 +19,7 @@
                     "description": "The avatar URL for this user, if any. This is added by the homeserver."
                 },
                 "displayname": {
-                    "type": "string",
+                    "type": ["string", "null"],
                     "description": "The display name for this user, if any. This is added by the homeserver."
                 }
             },

From 52640eb2053e155c7f33468c0e6c74512b062b0b Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 13:02:37 +0100
Subject: [PATCH 46/85] Add a python script for checking that the examples
 match the event schema.

Does the same checks as check.sh, but is a *lot* faster making it
suitable for using as a pre-commit hook.

I don't suggest replacing check.sh since it's good to check that the
schema works with multiple implementations of jsonschema.
---
 event-schemas/check_examples.py | 62 +++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)
 create mode 100755 event-schemas/check_examples.py

diff --git a/event-schemas/check_examples.py b/event-schemas/check_examples.py
new file mode 100755
index 00000000..5fc08f4a
--- /dev/null
+++ b/event-schemas/check_examples.py
@@ -0,0 +1,62 @@
+#! /usr/bin/env python
+
+import sys
+import json
+import os
+
+
+def import_error(module, package, debian, error):
+    sys.stderr.write((
+        "Error importing %(module)s: %(error)r\n"
+        "To install %(module)s run:\n"
+        "  pip install %(package)s\n"
+        "or on Debian run:\n"
+        "  sudo apt-get install python-%(debian)s\n"
+    ) % locals())
+    if __name__ == '__main__':
+        sys.exit(1)
+
+try:
+    import jsonschema
+except ImportError as e:
+    import_error("jsonschema", "jsonschema", "jsonschema", e)
+    raise
+
+try:
+    import yaml
+except ImportError as e:
+    import_error("yaml", "PyYAML", "yaml", e)
+    raise
+
+
+def check_example_file(examplepath, schemapath):
+    with open(examplepath) as f:
+        example = yaml.load(f)
+
+    with open(schemapath) as f:
+        schema = yaml.load(f)
+
+    fileurl = "file://" + os.path.abspath(schemapath)
+
+    print ("Checking schema for: %r %r" % (examplepath, schemapath))
+    # Setting the 'id' tells jsonschema where the file is so that it
+    # can correctly resolve relative $ref references in the schema
+    schema['id'] = fileurl
+    try:
+        jsonschema.validate(example, schema)
+    except:
+        raise ValueError("Error validating JSON schema for %r %r" % (
+            examplepath, schemapath
+        ), e)
+
+
+def check_example_dir(exampledir, schemadir):
+    for root, dirs, files in os.walk(exampledir):
+        for filename in files:
+            examplepath = os.path.join(root, filename)
+            schemapath = examplepath.replace(exampledir, schemadir)
+            check_example_file(examplepath, schemapath)
+
+
+if __name__ == '__main__':
+    check_example_dir("examples", "schema")

From f2c952ec5855bb120814a736bf482a19e1c43fd3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:13:18 +0100
Subject: [PATCH 47/85] Reshuffle spec into sections roughly right as per PR
 #52

---
 .../address-book-repo.rst                     |  0
 ..._repository.rst => 02_03_content_repo.rst} |  0
 ...on.rst => 02_04_end_to_end_encryption.rst} |  0
 ...ons.rst => 02_05_typing_notifications.rst} |  0
 ...{01_02_receipts.rst => 02_06_receipts.rst} |  0
 ...ility.rst => 02_07_history_visibility.rst} |  0
 ...h_overview.rst => 02_08_push_overview.rst} |  0
 ...push_cs_api.rst => 02_08b_push_cs_api.rst} |  0
 ...push_gw_api.rst => 02_08c_push_cs_api.rst} |  0
 .../03_00_application_service_api.rst         | 22 -------------------
 ...servers.rst => 05_00_identity_servers.rst} |  0
 ...00_appendices.rst => 06_00_appendices.rst} |  0
 12 files changed, 22 deletions(-)
 rename specification/08_00_address_book_repo.rst => drafts/address-book-repo.rst (100%)
 rename specification/{05_00_content_repository.rst => 02_03_content_repo.rst} (100%)
 rename specification/{06_00_end_to_end_encryption.rst => 02_04_end_to_end_encryption.rst} (100%)
 rename specification/{01_01_typing_notifications.rst => 02_05_typing_notifications.rst} (100%)
 rename specification/{01_02_receipts.rst => 02_06_receipts.rst} (100%)
 rename specification/{01_03_history_visibility.rst => 02_07_history_visibility.rst} (100%)
 rename specification/{07_00_push_overview.rst => 02_08_push_overview.rst} (100%)
 rename specification/{07_01_push_cs_api.rst => 02_08b_push_cs_api.rst} (100%)
 rename specification/{07_02_push_push_gw_api.rst => 02_08c_push_cs_api.rst} (100%)
 rename specification/{09_00_identity_servers.rst => 05_00_identity_servers.rst} (100%)
 rename specification/{10_00_appendices.rst => 06_00_appendices.rst} (100%)

diff --git a/specification/08_00_address_book_repo.rst b/drafts/address-book-repo.rst
similarity index 100%
rename from specification/08_00_address_book_repo.rst
rename to drafts/address-book-repo.rst
diff --git a/specification/05_00_content_repository.rst b/specification/02_03_content_repo.rst
similarity index 100%
rename from specification/05_00_content_repository.rst
rename to specification/02_03_content_repo.rst
diff --git a/specification/06_00_end_to_end_encryption.rst b/specification/02_04_end_to_end_encryption.rst
similarity index 100%
rename from specification/06_00_end_to_end_encryption.rst
rename to specification/02_04_end_to_end_encryption.rst
diff --git a/specification/01_01_typing_notifications.rst b/specification/02_05_typing_notifications.rst
similarity index 100%
rename from specification/01_01_typing_notifications.rst
rename to specification/02_05_typing_notifications.rst
diff --git a/specification/01_02_receipts.rst b/specification/02_06_receipts.rst
similarity index 100%
rename from specification/01_02_receipts.rst
rename to specification/02_06_receipts.rst
diff --git a/specification/01_03_history_visibility.rst b/specification/02_07_history_visibility.rst
similarity index 100%
rename from specification/01_03_history_visibility.rst
rename to specification/02_07_history_visibility.rst
diff --git a/specification/07_00_push_overview.rst b/specification/02_08_push_overview.rst
similarity index 100%
rename from specification/07_00_push_overview.rst
rename to specification/02_08_push_overview.rst
diff --git a/specification/07_01_push_cs_api.rst b/specification/02_08b_push_cs_api.rst
similarity index 100%
rename from specification/07_01_push_cs_api.rst
rename to specification/02_08b_push_cs_api.rst
diff --git a/specification/07_02_push_push_gw_api.rst b/specification/02_08c_push_cs_api.rst
similarity index 100%
rename from specification/07_02_push_push_gw_api.rst
rename to specification/02_08c_push_cs_api.rst
diff --git a/specification/03_00_application_service_api.rst b/specification/03_00_application_service_api.rst
index 9b58c861..2674ba44 100644
--- a/specification/03_00_application_service_api.rst
+++ b/specification/03_00_application_service_api.rst
@@ -400,25 +400,3 @@ in their content to provide a way for Matrix clients to link into the 'native'
 client from which the event originated.  For instance, this could contain the
 message-ID for emails/nntp posts, or a link to a blog comment when gatewaying
 blog comment traffic in & out of matrix
-
-Active Application Services
-----------------------------
-.. NOTE::
-  This section is a work in progress.
-
-.. TODO-spec
-  API that provides hooks into the server so that you can intercept and
-  manipulate events, and/or insert virtual users & rooms into the server.
-
-Policy Servers
-==============
-.. NOTE::
-  This section is a work in progress.
-
-.. TODO-spec
-  We should mention them in the Architecture section at least: how they fit
-  into the picture.
-
-Enforcing policies
-------------------
-
diff --git a/specification/09_00_identity_servers.rst b/specification/05_00_identity_servers.rst
similarity index 100%
rename from specification/09_00_identity_servers.rst
rename to specification/05_00_identity_servers.rst
diff --git a/specification/10_00_appendices.rst b/specification/06_00_appendices.rst
similarity index 100%
rename from specification/10_00_appendices.rst
rename to specification/06_00_appendices.rst

From c77ef1a2ccca95149c4d9d9d9ee965068190fd23 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:17:31 +0100
Subject: [PATCH 48/85] Add stub files feature_profiles and modules

---
 specification/00_01_feature_profiles.rst                          | 0
 specification/{02_00_events.rst => 00_02a_events.rst}             | 0
 .../{02_02_event_signing.rst => 00_02b_event_signing.rst}         | 0
 specification/02_00_modules.rst                                   | 0
 .../{02_08_push_overview.rst => 02_08a_push_overview.rst}         | 0
 5 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 specification/00_01_feature_profiles.rst
 rename specification/{02_00_events.rst => 00_02a_events.rst} (100%)
 rename specification/{02_02_event_signing.rst => 00_02b_event_signing.rst} (100%)
 create mode 100644 specification/02_00_modules.rst
 rename specification/{02_08_push_overview.rst => 02_08a_push_overview.rst} (100%)

diff --git a/specification/00_01_feature_profiles.rst b/specification/00_01_feature_profiles.rst
new file mode 100644
index 00000000..e69de29b
diff --git a/specification/02_00_events.rst b/specification/00_02a_events.rst
similarity index 100%
rename from specification/02_00_events.rst
rename to specification/00_02a_events.rst
diff --git a/specification/02_02_event_signing.rst b/specification/00_02b_event_signing.rst
similarity index 100%
rename from specification/02_02_event_signing.rst
rename to specification/00_02b_event_signing.rst
diff --git a/specification/02_00_modules.rst b/specification/02_00_modules.rst
new file mode 100644
index 00000000..e69de29b
diff --git a/specification/02_08_push_overview.rst b/specification/02_08a_push_overview.rst
similarity index 100%
rename from specification/02_08_push_overview.rst
rename to specification/02_08a_push_overview.rst

From f520ac9d828890193772764e902c3ae5b4b77c1d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:25:52 +0100
Subject: [PATCH 49/85] Create a modules folder to group together similar parts
 of the spec

---
 .../{02_01_voip_events.rst => modules/01_00_voip_events.rst}      | 0
 .../02_00_typing_notifications.rst}                               | 0
 specification/{02_06_receipts.rst => modules/03_00_receipts.rst}  | 0
 .../{02_03_content_repo.rst => modules/04_00_content_repo.rst}    | 0
 .../05_00_end_to_end_encryption.rst}                              | 0
 .../06_00_history_visibility.rst}                                 | 0
 .../{02_08a_push_overview.rst => modules/07_00_push_overview.rst} | 0
 .../{02_08b_push_cs_api.rst => modules/07_01_push_cs_api.rst}     | 0
 .../07_02_push_push_gw_api.rst}                                   | 0
 9 files changed, 0 insertions(+), 0 deletions(-)
 rename specification/{02_01_voip_events.rst => modules/01_00_voip_events.rst} (100%)
 rename specification/{02_05_typing_notifications.rst => modules/02_00_typing_notifications.rst} (100%)
 rename specification/{02_06_receipts.rst => modules/03_00_receipts.rst} (100%)
 rename specification/{02_03_content_repo.rst => modules/04_00_content_repo.rst} (100%)
 rename specification/{02_04_end_to_end_encryption.rst => modules/05_00_end_to_end_encryption.rst} (100%)
 rename specification/{02_07_history_visibility.rst => modules/06_00_history_visibility.rst} (100%)
 rename specification/{02_08a_push_overview.rst => modules/07_00_push_overview.rst} (100%)
 rename specification/{02_08b_push_cs_api.rst => modules/07_01_push_cs_api.rst} (100%)
 rename specification/{02_08c_push_cs_api.rst => modules/07_02_push_push_gw_api.rst} (100%)

diff --git a/specification/02_01_voip_events.rst b/specification/modules/01_00_voip_events.rst
similarity index 100%
rename from specification/02_01_voip_events.rst
rename to specification/modules/01_00_voip_events.rst
diff --git a/specification/02_05_typing_notifications.rst b/specification/modules/02_00_typing_notifications.rst
similarity index 100%
rename from specification/02_05_typing_notifications.rst
rename to specification/modules/02_00_typing_notifications.rst
diff --git a/specification/02_06_receipts.rst b/specification/modules/03_00_receipts.rst
similarity index 100%
rename from specification/02_06_receipts.rst
rename to specification/modules/03_00_receipts.rst
diff --git a/specification/02_03_content_repo.rst b/specification/modules/04_00_content_repo.rst
similarity index 100%
rename from specification/02_03_content_repo.rst
rename to specification/modules/04_00_content_repo.rst
diff --git a/specification/02_04_end_to_end_encryption.rst b/specification/modules/05_00_end_to_end_encryption.rst
similarity index 100%
rename from specification/02_04_end_to_end_encryption.rst
rename to specification/modules/05_00_end_to_end_encryption.rst
diff --git a/specification/02_07_history_visibility.rst b/specification/modules/06_00_history_visibility.rst
similarity index 100%
rename from specification/02_07_history_visibility.rst
rename to specification/modules/06_00_history_visibility.rst
diff --git a/specification/02_08a_push_overview.rst b/specification/modules/07_00_push_overview.rst
similarity index 100%
rename from specification/02_08a_push_overview.rst
rename to specification/modules/07_00_push_overview.rst
diff --git a/specification/02_08b_push_cs_api.rst b/specification/modules/07_01_push_cs_api.rst
similarity index 100%
rename from specification/02_08b_push_cs_api.rst
rename to specification/modules/07_01_push_cs_api.rst
diff --git a/specification/02_08c_push_cs_api.rst b/specification/modules/07_02_push_push_gw_api.rst
similarity index 100%
rename from specification/02_08c_push_cs_api.rst
rename to specification/modules/07_02_push_push_gw_api.rst

From 703f913a47afc3b035a1730e9014e89807370b3e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:31:57 +0100
Subject: [PATCH 50/85] Add a 'targets.yaml' files in /specification

We're well beyond the point now where a simple `cat` of .rst files to "build"
the spec is practical. We may want to slice and dice the spec in different
ways to address various cross-cutting concerns. To this end, there is now a
'targets' file which contains the "build targets" for the spec, which contains
the sorting order for the .rst files. For now, we just have a single
target: 'main'.
---
 specification/targets.yaml | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 specification/targets.yaml

diff --git a/specification/targets.yaml b/specification/targets.yaml
new file mode 100644
index 00000000..845da99d
--- /dev/null
+++ b/specification/targets.yaml
@@ -0,0 +1,25 @@
+targets:
+  main:
+    files:
+      - 00_00_intro.rst
+      - 00_01_feature_profiles.rst
+      - 00_02a_events.rst
+      - 00_02b_event_signing.rst
+      - 01_00_client_server_api.rst
+      - 02_00_modules.rst
+      - "group:module"
+      - 03_00_application_service_api.rst
+      - 04_00_server_server_api.rst
+      - 05_00_identity_servers.rst
+      - 06_00_appendices.rst
+groups:
+  modules:
+    - modules/01_00_voip_events.rst
+    - modules/02_00_typing_notifications.rst
+    - modules/03_00_receipts.rst
+    - modules/04_00_content_repo.rst
+    - modules/05_00_end_to_end_encryption.rst
+    - modules/06_00_history_visibility.rst
+    - modules/07_00_push_overview.rst
+    - modules/07_01_push_cs_api.rst
+    - modules/07_02_push_push_gw_api.rst

From 568982ee2e5a3dbf2f76556475dfdd8fae96998d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:49:54 +0100
Subject: [PATCH 51/85] Represent nested deps in targets.yaml along with title
 styles

This will allow us to programatically position .rst snippets *anywhere*
which will for once and for all remove the horrid title level mismatch bugs.

We require this in order to allow people to re-shuffle the spec without
having to adjust the spec itself (e.g. 2 targets with different levels of
nesting).
---
 specification/targets.yaml | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/specification/targets.yaml b/specification/targets.yaml
index 845da99d..b61e2e20 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -20,6 +20,10 @@ groups:
     - modules/04_00_content_repo.rst
     - modules/05_00_end_to_end_encryption.rst
     - modules/06_00_history_visibility.rst
-    - modules/07_00_push_overview.rst
-    - modules/07_01_push_cs_api.rst
-    - modules/07_02_push_push_gw_api.rst
+    - 1: modules/07_00_push_overview.rst
+      2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
+title_styles:
+ - "="
+ - "-"
+ - "~"
+ - "+"

From 6ba9b29b3b4ffe9d19c30302e36813b1835eb117 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 15:04:03 +0100
Subject: [PATCH 52/85] Report all the errors in schemas/check_examples, not
 just the first error.

---
 event-schemas/check_examples.py | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/event-schemas/check_examples.py b/event-schemas/check_examples.py
index 5fc08f4a..8675396e 100755
--- a/event-schemas/check_examples.py
+++ b/event-schemas/check_examples.py
@@ -3,6 +3,7 @@
 import sys
 import json
 import os
+import traceback
 
 
 def import_error(module, package, debian, error):
@@ -44,19 +45,29 @@ def check_example_file(examplepath, schemapath):
     schema['id'] = fileurl
     try:
         jsonschema.validate(example, schema)
-    except:
+    except Exception as e:
         raise ValueError("Error validating JSON schema for %r %r" % (
             examplepath, schemapath
         ), e)
 
 
 def check_example_dir(exampledir, schemadir):
+    errors = []
     for root, dirs, files in os.walk(exampledir):
         for filename in files:
             examplepath = os.path.join(root, filename)
             schemapath = examplepath.replace(exampledir, schemadir)
-            check_example_file(examplepath, schemapath)
-
+            try:
+                check_example_file(examplepath, schemapath)
+            except Exception as e:
+                errors.append(sys.exc_info())
+    for (exc_type, exc_value, exc_trace) in errors:
+        traceback.print_exception(exc_type, exc_value, exc_trace)
+    if errors:
+        raise ValueError("Error validating examples")
 
 if __name__ == '__main__':
-    check_example_dir("examples", "schema")
+    try:
+        check_example_dir("examples", "schema")
+    except:
+        sys.exit(1)

From 8974b2b67bbc2acf0c482c646be6b07b230a1a15 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 15:05:10 +0100
Subject: [PATCH 53/85] Skip files that start with ".", e.g. vim swp files.

---
 event-schemas/check_examples.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/event-schemas/check_examples.py b/event-schemas/check_examples.py
index 8675396e..e54d3a1c 100755
--- a/event-schemas/check_examples.py
+++ b/event-schemas/check_examples.py
@@ -55,6 +55,9 @@ def check_example_dir(exampledir, schemadir):
     errors = []
     for root, dirs, files in os.walk(exampledir):
         for filename in files:
+            if filename.startswith("."):
+                # Skip over any vim .swp files.
+                continue
             examplepath = os.path.join(root, filename)
             schemapath = examplepath.replace(exampledir, schemadir)
             try:

From 2c31731262b3258307dd7e5eb8364d71817c4749 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 15:16:02 +0100
Subject: [PATCH 54/85] Add the jenkins command to source control so that we
 can update it without having to fiddle with the jenkins UI.

It also allow us to move files without breaking the CI since we
won't be hard coding the locations of scripts in the jenkins UI.
---
 jenkins.sh | 9 +++++++++
 1 file changed, 9 insertions(+)
 create mode 100755 jenkins.sh

diff --git a/jenkins.sh b/jenkins.sh
new file mode 100755
index 00000000..e1043644
--- /dev/null
+++ b/jenkins.sh
@@ -0,0 +1,9 @@
+#! /bin/bash
+
+set -ex
+
+(cd event-schemas/ && ./check_examples.py)
+(cd api && ./check_examples.py)
+(cd scripts && ./gendoc.py)
+(cd api && npm install && node validator.js -s "client-server/v1")
+(cd event-schemas/ && ./check.sh)

From 8590cc84b51ba81b14b0f73988879368c26e649f Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 15:33:28 +0100
Subject: [PATCH 55/85] Process and extract targets in gendoc.py

---
 scripts/gendoc.py          | 74 +++++++++++++++++++++++++++++++-------
 specification/targets.yaml | 10 +++---
 2 files changed, 68 insertions(+), 16 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 3521efed..69c39dcc 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -8,6 +8,7 @@ import re
 import shutil
 import subprocess
 import sys
+import yaml
 
 os.chdir(os.path.dirname(os.path.abspath(__file__)))
 
@@ -24,7 +25,7 @@ SECOND_LEVEL = "-"
 FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}[a-z]*_.*\.rst$")
 
 
-def check_valid_section(filename, section):
+def check_valid_section_old(filename, section):
     if not re.match(FILE_FORMAT_MATCHER, filename):
         raise Exception(
             "The filename of " + filename + " does not match the expected format " +
@@ -67,13 +68,19 @@ def check_valid_section(filename, section):
                 "check."
             )
 
-def cat_spec_sections_to(out_file_name):
-    with open(out_file_name, "wb") as outfile:
-        for f in sorted(glob.glob("../specification/*.rst")):
-            with open(f, "rb") as infile:
-                section = infile.read()
-                check_valid_section(os.path.basename(f), section)
-                outfile.write(section)
+def check_valid_section(section):
+    pass
+
+
+def get_rst(file_info, target):
+    pass
+
+def build_spec(target, out_filename):
+    with open(out_filename, "wb") as outfile:
+        for file_info in target["files"]:
+            section = get_rst(file_info, target)
+            check_valid_section(section)
+            outfile.write(section)
 
 
 def rst2html(i, o):
@@ -88,6 +95,7 @@ def rst2html(i, o):
                 settings_overrides=stylesheets
             )
 
+
 def run_through_template(input):
     tmpfile = './tmp/output'
     try:
@@ -107,6 +115,43 @@ def run_through_template(input):
             sys.stderr.write(f.read() + "\n")
         raise
 
+
+def get_build_target(targets_listing, target_name):
+    build_target = {
+        "title_styles": [],
+        "files": []
+    }
+    with open(targets_listing, "r") as targ_file:
+        all_targets = yaml.load(targ_file.read())
+        build_target["title_styles"] = all_targets["title_styles"]
+        target = all_targets["targets"].get(target_name)
+        if not target:
+            raise Exception(
+                "No target by the name '" + target_name + "' exists in '" +
+                targets_listing + "'."
+            )
+        if not isinstance(target.get("files"), list):
+            raise Exception(
+                "Found target but 'files' key is not a list."
+            )
+        resolved_files = []
+        for f in target["files"]:
+            if isinstance(f, basestring) and f.startswith("group:"):
+                # copy across the group of files specified
+                group_name = f[len("group:"):]
+                group = all_targets.get("groups", {}).get(group_name)
+                if not isinstance(group, list):
+                    raise Exception(
+                        "Tried to find group '" + group_name + "' but either " +
+                        "it doesn't exist or it isn't a list of files."
+                    )
+                resolved_files.extend(group)
+            else:
+                resolved_files.append(f)
+        build_target["files"] = resolved_files
+    return build_target
+
+
 def prepare_env():
     try:
         os.makedirs("./gen")
@@ -116,13 +161,17 @@ def prepare_env():
         os.makedirs("./tmp")
     except OSError:
         pass
-    
+
+
 def cleanup_env():
     shutil.rmtree("./tmp")
 
-def main():
+
+def main(target_name):
     prepare_env()
-    cat_spec_sections_to("tmp/full_spec.rst")
+    target = get_build_target("../specification/targets.yaml", target_name)
+    print target
+    build_spec(target=target, out_filename="tmp/full_spec.rst")
     run_through_template("tmp/full_spec.rst")
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")
     run_through_template("tmp/howto.rst")
@@ -131,6 +180,7 @@ def main():
     if "--nodelete" not in sys.argv:
         cleanup_env()
 
+
 if __name__ == '__main__':
     if len(sys.argv) > 1 and sys.argv[1:] != ["--nodelete"]:
         # we accept almost no args, so they don't know what they're doing!
@@ -145,4 +195,4 @@ if __name__ == '__main__':
         print "Requirements:"
         print " - This script requires Jinja2 and rst2html (docutils)."
         sys.exit(0)
-    main()
+    main("main")
diff --git a/specification/targets.yaml b/specification/targets.yaml
index b61e2e20..c4828ef1 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -1,18 +1,18 @@
 targets:
-  main:
-    files:
+  main:  # arbitrary name to identify this build target
+    files:  # the sort order of files to cat
       - 00_00_intro.rst
       - 00_01_feature_profiles.rst
       - 00_02a_events.rst
       - 00_02b_event_signing.rst
       - 01_00_client_server_api.rst
       - 02_00_modules.rst
-      - "group:module"
+      - "group:modules"  # reference a group of files
       - 03_00_application_service_api.rst
       - 04_00_server_server_api.rst
       - 05_00_identity_servers.rst
       - 06_00_appendices.rst
-groups:
+groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
     - modules/01_00_voip_events.rst
     - modules/02_00_typing_notifications.rst
@@ -20,6 +20,7 @@ groups:
     - modules/04_00_content_repo.rst
     - modules/05_00_end_to_end_encryption.rst
     - modules/06_00_history_visibility.rst
+    # Mark a nested file dependency
     - 1: modules/07_00_push_overview.rst
       2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
 title_styles:
@@ -27,3 +28,4 @@ title_styles:
  - "-"
  - "~"
  - "+"
+

From 65ce95249f99fb67e61377e5b35734522a6d7ba3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 16:54:30 +0100
Subject: [PATCH 56/85] cat the spec according to the build target. Remove old
 checks as they are now obsolete since we don't care about the filename

---
 scripts/gendoc.py | 121 ++++++++++++++++++++++++----------------------
 1 file changed, 62 insertions(+), 59 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 69c39dcc..22aaad11 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -16,70 +16,74 @@ stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css"]
 }
 
-title_style_matchers = {
-    "=": re.compile("^=+$"),
-    "-": re.compile("^-+$")
-}
-TOP_LEVEL = "="
-SECOND_LEVEL = "-"
-FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}[a-z]*_.*\.rst$")
-
-
-def check_valid_section_old(filename, section):
-    if not re.match(FILE_FORMAT_MATCHER, filename):
-        raise Exception(
-            "The filename of " + filename + " does not match the expected format " +
-            "of '##_##_words-go-here.rst'"
-        )
-
-    # we need TWO new lines else the next file's title gets merged
-    # the last paragraph *WITHOUT RST PRODUCING A WARNING*
-    if not section[-2:] == "\n\n":
-        raise Exception(
-            "The file " + filename + " does not end with 2 new lines."
-        )
-
-    # Enforce some rules to reduce the risk of having mismatched title
-    # styles.
-    title_line = section.split("\n")[1]
-    if title_line != (len(title_line) * title_line[0]):
-        raise Exception(
-            "The file " + filename + " doesn't have a title style line on line 2"
-        )
-
-    # anything marked as xx_00_ is the start of a new top-level section
-    if re.match("^[0-9]+_00_", filename):
-        if not title_style_matchers[TOP_LEVEL].match(title_line):
-            raise Exception(
-                "The file " + filename + " is a top-level section because it matches " +
-                "the filename format ##_00_something.rst but has the wrong title " +
-                "style: expected '" + TOP_LEVEL + "' but got '" +
-                title_line[0] + "'"
-            )
-    # anything marked as xx_xx_ is the start of a sub-section
-    elif re.match("^[0-9]+_[0-9]{2}_", filename):
-        if not title_style_matchers[SECOND_LEVEL].match(title_line):
-            raise Exception(
-                "The file " + filename + " is a 2nd-level section because it matches " +
-                "the filename format ##_##_something.rst but has the wrong title " +
-                "style: expected '" + SECOND_LEVEL + "' but got '" +
-                title_line[0] + "' - If this is meant to be a 3rd/4th/5th-level section " +
-                "then use the form '##_##b_something.rst' which will not apply this " +
-                "check."
-            )
-
-def check_valid_section(section):
-    pass
 
+def _list_get(l, index, default=None):
+    try:
+        return l[index]
+    except IndexError:
+        return default
+
+
+def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
+    rst_lines = []
+    title_chars = "".join(title_styles)
+    title_regex = re.compile("^[" + re.escape(title_chars) + "]+$")
+
+    curr_title_level = title_level
+    for i, line in enumerate(file_stream, 1):
+        if title_regex.match(line):
+            line_title_level = title_styles.index(line[0])
+            # Allowed to go 1 deeper or any number shallower
+            if curr_title_level - line_title_level < -1:
+                raise Exception(
+                    ("File '%s' line '%s' has a title " +
+                    "style '%s' which doesn't match one of the " +
+                    "allowed title styles of %s because the " +
+                    "title level before this line was '%s'") %
+                    (filename, (i + 1), line[0], title_styles,
+                    title_styles[curr_title_level])
+                )
+            curr_title_level = line_title_level
+            rst_lines.append(line)
+        else:
+            rst_lines.append(line)
+    return "".join(rst_lines)
+
+
+
+def get_rst(file_info, title_level, title_styles, spec_dir):
+    # string are file paths to RST blobs
+    if isinstance(file_info, basestring):
+        with open(spec_dir + file_info, "r") as f:
+            return load_with_adjusted_titles(file_info, f, title_level, title_styles)
+    # dicts look like {1: filepath, 2: filepath} where the key is the title level
+    elif isinstance(file_info, dict):
+        levels = sorted(file_info.keys())
+        rst = []
+        for l in levels:
+            rst.append(get_rst(file_info[l], l, title_styles, spec_dir))
+        return "".join(rst)
+    # lists are multiple file paths e.g. [filepath, filepath]
+    elif isinstance(file_info, list):
+        rst = []
+        for f in file_info:
+            rst.append(get_rst(f, title_level, title_styles, spec_dir))
+        return "".join(rst)
+    raise Exception(
+        "The following 'file' entry in this target isn't a string, list or dict. " +
+        "It really really should be. Entry: %s" % (file_info,)
+    )
 
-def get_rst(file_info, target):
-    pass
 
 def build_spec(target, out_filename):
     with open(out_filename, "wb") as outfile:
         for file_info in target["files"]:
-            section = get_rst(file_info, target)
-            check_valid_section(section)
+            section = get_rst(
+                file_info=file_info,
+                title_level=0,
+                title_styles=target["title_styles"],
+                spec_dir="../specification/"
+            )
             outfile.write(section)
 
 
@@ -170,7 +174,6 @@ def cleanup_env():
 def main(target_name):
     prepare_env()
     target = get_build_target("../specification/targets.yaml", target_name)
-    print target
     build_spec(target=target, out_filename="tmp/full_spec.rst")
     run_through_template("tmp/full_spec.rst")
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")

From cb41adee707a04591c93726f5002855febf2543f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:10:23 +0100
Subject: [PATCH 57/85] Fix the swagger host to be "localhost:8008" so that it
 can be used in a "Try it now" setup against localhost

---
 api/client-server/v1/rooms.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 4902560b..4494d357 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -2,7 +2,7 @@ swagger: '2.0'
 info:
   title: "Matrix Client-Server v1 Rooms API"
   version: "1.0.0"
-host: example.com:8008
+host: localhost:8008
 schemes:
   - https
   - http

From 615a9575cbad5401997e89c292cc741e0bf5ff23 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:12:29 +0100
Subject: [PATCH 58/85] SPEC-216: Clarify when the room getters will return 403

---
 api/client-server/v1/rooms.yaml | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 4494d357..3bd23c4a 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -57,7 +57,9 @@ paths:
         404:
           description: The room has no state with the given type or key.
         403:
-          description: You are not joined to the room.
+          description: >
+            You aren't a member of the room and weren't previously a
+            member of the room.
 
   "/rooms/{roomId}/state":
     get:
@@ -170,7 +172,9 @@ paths:
               allOf:
                 - "$ref": "core-event-schema/state_event.json"
         403:
-          description: You are not joined to the room.
+          description: >
+            You aren't a member of the room and weren't previously a
+            member of the room.
 
   "/rooms/{roomId}/initialSync":
     get:
@@ -365,6 +369,10 @@ paths:
                   Whether this room is visible to the ``/publicRooms`` API
                   or not."
             required: ["room_id", "membership"]
+        403:
+          description: >
+            You aren't a member of the room and weren't previously a
+            member of the room.
 
   "/rooms/{roomId}/members":
     get:
@@ -427,4 +435,8 @@ paths:
                   type: object
                   allOf:
                     - "$ref": "v1-event-schema/m.room.member"
+        403:
+          description: >
+            You aren't a member of the room and weren't previously a
+            member of the room.
 

From ba6c7d267ca4b5265d2eb1ba19933fe6f790090a Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:17:40 +0100
Subject: [PATCH 59/85] SPEC-216: Document the behaviour of the room getters
 when the user has left the room

---
 api/client-server/v1/rooms.yaml | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 3bd23c4a..7550c5fe 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -24,7 +24,8 @@ paths:
       description: |-
         Looks up the contents of a state event in a room. If the user is
         joined to the room then the state is taken from the current
-        state of the room.
+        state of the room. If the user has left the room then the state is
+        taken from the state of the room when they left.
       security:
         - accessToken: []
       parameters:
@@ -165,7 +166,9 @@ paths:
             title: RoomState
             description: |-
               If the user is a member of the room this will be the
-              current state of the room as a list of events.
+              current state of the room as a list of events. If the user
+              has left the room then this will be the state of the room
+              when they left as a list of events.
             items:
               title: StateEvent
               type: object
@@ -378,7 +381,7 @@ paths:
     get:
       summary: Get the m.room.member events for the room.
       description:
-        Get the list of members of the room.
+        Get the list of members for this room.
       parameters:
         - in: path
           type: string

From 067363c629bcf140e925980d20bcf8b264407cac Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 17:21:08 +0100
Subject: [PATCH 60/85] Get the desired title levels right; print out the
 actual level used to stdout

---
 scripts/gendoc.py                          | 16 ++++++++-----
 specification/modules/00_modules_intro.rst |  0
 specification/targets.yaml                 | 26 +++++++++-------------
 3 files changed, 21 insertions(+), 21 deletions(-)
 create mode 100644 specification/modules/00_modules_intro.rst

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 2e303ea2..f7bde162 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -50,13 +50,13 @@ def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
     return "".join(rst_lines)
 
 
-
 def get_rst(file_info, title_level, title_styles, spec_dir):
     # string are file paths to RST blobs
     if isinstance(file_info, basestring):
+        print "%s %s" % (">" * (1 + title_level), file_info)
         with open(spec_dir + file_info, "r") as f:
             return load_with_adjusted_titles(file_info, f, title_level, title_styles)
-    # dicts look like {1: filepath, 2: filepath} where the key is the title level
+    # dicts look like {0: filepath, 1: filepath} where the key is the title level
     elif isinstance(file_info, dict):
         levels = sorted(file_info.keys())
         rst = []
@@ -144,12 +144,15 @@ def get_build_target(targets_listing, target_name):
                 # copy across the group of files specified
                 group_name = f[len("group:"):]
                 group = all_targets.get("groups", {}).get(group_name)
-                if not isinstance(group, list):
+                if not group:
                     raise Exception(
-                        "Tried to find group '" + group_name + "' but either " +
-                        "it doesn't exist or it isn't a list of files."
+                        "Tried to find group '" + group_name + "' but it " +
+                        "doesn't exist."
                     )
-                resolved_files.extend(group)
+                if isinstance(group, list):
+                    resolved_files.extend(group)
+                else:
+                    resolved_files.append(group)
             else:
                 resolved_files.append(f)
         build_target["files"] = resolved_files
@@ -173,6 +176,7 @@ def cleanup_env():
 
 def main(target_name):
     prepare_env()
+    print "Building spec [target=%s]" % target_name
     target = get_build_target("../specification/targets.yaml", target_name)
     build_spec(target=target, out_filename="tmp/full_spec.rst")
     run_through_template("tmp/full_spec.rst")
diff --git a/specification/modules/00_modules_intro.rst b/specification/modules/00_modules_intro.rst
new file mode 100644
index 00000000..e69de29b
diff --git a/specification/targets.yaml b/specification/targets.yaml
index c4828ef1..e492ad3c 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -14,18 +14,14 @@ targets:
       - 06_00_appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
-    - modules/01_00_voip_events.rst
-    - modules/02_00_typing_notifications.rst
-    - modules/03_00_receipts.rst
-    - modules/04_00_content_repo.rst
-    - modules/05_00_end_to_end_encryption.rst
-    - modules/06_00_history_visibility.rst
-    # Mark a nested file dependency
-    - 1: modules/07_00_push_overview.rst
-      2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
-title_styles:
- - "="
- - "-"
- - "~"
- - "+"
-
+      0: modules/00_modules_intro.rst
+      1:
+         - modules/01_00_voip_events.rst
+         - modules/02_00_typing_notifications.rst
+         - modules/03_00_receipts.rst
+         - modules/04_00_content_repo.rst
+         - modules/05_00_end_to_end_encryption.rst
+         - modules/06_00_history_visibility.rst
+         - 1: modules/07_00_push_overview.rst # Mark a nested file dependency
+           2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
+title_styles: ["=", "-", "~", "+"]

From f60190086a41579d35a7d42caaaee7e69fe27b00 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:30:10 +0100
Subject: [PATCH 61/85] Describe the behaviour of /rooms/{roomId}/member when
 the user has left the room

---
 api/client-server/v1/rooms.yaml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 7550c5fe..e02c75c6 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -391,7 +391,10 @@ paths:
           x-example: "!room:example.com"
       responses:
         200:
-          description: The members of the room.
+          description: |-
+            A list of members of the room. If you are joined to the room then
+            this will be the current members of the room. If you have left te
+            room then this will be the members of the room when you left.
           examples:
             application/json: |-
               {

From 98d91d0c2b3b098618bc91f70079a1a9c64bb546 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:31:35 +0100
Subject: [PATCH 62/85] Make the example room id more "random" so that people
 are less likely to think that it is supposed to be human readable

---
 api/client-server/v1/rooms.yaml | 38 ++++++++++++++++-----------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index e02c75c6..9300d7d1 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -34,7 +34,7 @@ paths:
           name: roomId
           description: The room to look up the state in.
           required: true
-          x-example: "!room:example.com"
+          x-example: "!636q39766251:example.com"
         - in: path
           type: string
           name: eventType
@@ -75,7 +75,7 @@ paths:
           name: roomId
           description: The room to look up the state for.
           required: true
-          x-example: "!room:example.com"
+          x-example: "!636q39766251:example.com"
       responses:
         200:
           description: The current state of the room
@@ -89,7 +89,7 @@ paths:
                     },
                     "event_id": "$14259997323TLwtb:example.com",
                     "origin_server_ts": 1425999732392,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.join_rules",
                     "user_id": "@alice:example.com"
@@ -104,7 +104,7 @@ paths:
                     "event_id": "$1426600438280zExKY:example.com",
                     "membership": "join",
                     "origin_server_ts": 1426600438277,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@alice:example.com",
                     "type": "m.room.member",
                     "user_id": "@alice:example.com"
@@ -116,7 +116,7 @@ paths:
                     },
                     "event_id": "$14259997320KhbwJ:example.com",
                     "origin_server_ts": 1425999732089,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.create",
                     "user_id": "@alice:example.com"
@@ -131,7 +131,7 @@ paths:
                     "event_id": "$1431525430134MxlLX:example.com",
                     "origin_server_ts": 1431525430569,
                     "replaces_state": "$142652023736BSXcM:example.com",
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@bob:example.com",
                     "type": "m.room.member",
                     "user_id": "@bob:example.com"
@@ -155,7 +155,7 @@ paths:
                     },
                     "event_id": "$14259997322mqfaq:example.com",
                     "origin_server_ts": 1425999732285,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.power_levels",
                     "user_id": "@alice:example.com"
@@ -192,7 +192,7 @@ paths:
           name: roomId
           description: The room to get the data.
           required: true
-          x-example: "!room:example.com"
+          x-example: "!636q39766251:example.com"
       responses:
         200:
           description: The current state of the room
@@ -210,7 +210,7 @@ paths:
                       },
                       "event_id": "$14328044851tzTJS:example.com",
                       "origin_server_ts": 1432804485886,
-                      "room_id": "!room:example.com",
+                      "room_id": "!636q39766251:example.com",
                       "type": "m.room.message",
                       "user_id": "@alice:example.com"
                     },
@@ -222,7 +222,7 @@ paths:
                       },
                       "event_id": "$14328044872spjFg:example.com",
                       "origin_server_ts": 1432804487480,
-                      "room_id": "!room:example.com",
+                      "room_id": "!636q39766251:example.com",
                       "type": "m.room.message",
                       "user_id": "@bob:example.com"
                     }
@@ -230,7 +230,7 @@ paths:
                   "end": "s3456_9_0",
                   "start": "t44-3453_9_0"
                 },
-                "room_id": "!room:example.com",
+                "room_id": "!636q39766251:example.com",
                 "state": [
                   {
                     "age": 7148266897,
@@ -239,7 +239,7 @@ paths:
                     },
                     "event_id": "$14259997323TLwtb:example.com",
                     "origin_server_ts": 1425999732392,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.join_rules",
                     "user_id": "@alice:example.com"
@@ -254,7 +254,7 @@ paths:
                     "event_id": "$1426600438280zExKY:example.com",
                     "membership": "join",
                     "origin_server_ts": 1426600438277,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@alice:example.com",
                     "type": "m.room.member",
                     "user_id": "@alice:example.com"
@@ -266,7 +266,7 @@ paths:
                     },
                     "event_id": "$14259997320KhbwJ:example.com",
                     "origin_server_ts": 1425999732089,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.create",
                     "user_id": "@alice:example.com"
@@ -281,7 +281,7 @@ paths:
                     "event_id": "$1431525430134MxlLX:example.com",
                     "origin_server_ts": 1431525430569,
                     "replaces_state": "$142652023736BSXcM:example.com",
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@bob:example.com",
                     "type": "m.room.member",
                     "user_id": "@bob:example.com"
@@ -305,7 +305,7 @@ paths:
                     },
                     "event_id": "$14259997322mqfaq:example.com",
                     "origin_server_ts": 1425999732285,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.power_levels",
                     "user_id": "@alice:example.com"
@@ -388,7 +388,7 @@ paths:
           name: roomId
           description: The room to get the member events for.
           required: true
-          x-example: "!room:example.com"
+          x-example: "!636q39766251:example.com"
       responses:
         200:
           description: |-
@@ -409,7 +409,7 @@ paths:
                     "event_id": "$1426600438280zExKY:example.com",
                     "membership": "join",
                     "origin_server_ts": 1426600438277,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@alice:example.com",
                     "type": "m.room.member",
                     "user_id": "@alice:example.com"
@@ -424,7 +424,7 @@ paths:
                     "event_id": "$1431525430134MxlLX:example.com",
                     "origin_server_ts": 1431525430569,
                     "replaces_state": "$142652023736BSXcM:example.com",
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@bob:example.com",
                     "type": "m.room.member",
                     "user_id": "@bob:example.com"

From f71763b0d3b6d95ed7839f5c9735d21aa59d5d71 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 13:08:15 +0100
Subject: [PATCH 63/85] Implement relative title styles

Templates don't know at what level they will be inserted. Previously, we
hard-coded the title style which is not compatible with the build target
system. Define a set of styles which will be replaced by the gendoc script
when it encounters them:
 '<' : Make this title a sub-heading
 '/' : Make this title a heading at the same level
 '>' : Make this title a super-heading

The build target system is now basically complete and functioning.
---
 scripts/gendoc.py                             | 234 +++++++++++++++---
 specification/00_01_feature_profiles.rst      |   5 +
 specification/02_00_modules.rst               |   5 +
 .../03_00_application_service_api.rst         |   1 +
 specification/04_00_server_server_api.rst     |   4 +-
 specification/modules/00_modules_intro.rst    |   5 +
 specification/targets.yaml                    |  41 +--
 templating/matrix_templates/sections.py       |  35 +--
 .../matrix_templates/templates/msgtypes.tmpl  |   2 +-
 templating/matrix_templates/units.py          |   7 +
 10 files changed, 268 insertions(+), 71 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index f7bde162..308d148f 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -17,57 +17,133 @@ stylesheets = {
 }
 
 
-def _list_get(l, index, default=None):
-    try:
-        return l[index]
-    except IndexError:
-        return default
-
-
+"""
+Read a RST file and replace titles with a different title level if required.
+Args:
+    filename: The name of the file being read (for debugging)
+    file_stream: The open file stream to read from.
+    title_level: The integer which determines the offset to *start* from.
+    title_styles: An array of characters detailing the right title styles to use
+                  e.g. ["=", "-", "~", "+"]
+Returns:
+    string: The file contents with titles adjusted.
+Example:
+    Assume title_styles = ["=", "-", "~", "+"], title_level = 1, and the file
+    when read line-by-line encounters the titles "===", "---", "---", "===", "---".
+    This function will bump every title encountered down a sub-heading e.g.
+    "=" to "-" and "-" to "~" because title_level = 1, so the output would be
+    "---", "~~~", "~~~", "---", "~~~". There is no bumping "up" a title level.
+"""
 def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
     rst_lines = []
     title_chars = "".join(title_styles)
-    title_regex = re.compile("^[" + re.escape(title_chars) + "]+$")
+    title_regex = re.compile("^[" + re.escape(title_chars) + "]{3,}$")
 
-    curr_title_level = title_level
+    prev_line_title_level = 0 # We expect the file to start with '=' titles
+    file_offset = None
+    prev_non_title_line = None
     for i, line in enumerate(file_stream, 1):
-        if title_regex.match(line):
-            line_title_level = title_styles.index(line[0])
-            # Allowed to go 1 deeper or any number shallower
-            if curr_title_level - line_title_level < -1:
-                raise Exception(
-                    ("File '%s' line '%s' has a title " +
-                    "style '%s' which doesn't match one of the " +
-                    "allowed title styles of %s because the " +
-                    "title level before this line was '%s'") %
-                    (filename, (i + 1), line[0], title_styles,
-                    title_styles[curr_title_level])
-                )
-            curr_title_level = line_title_level
+        # ignore anything which isn't a title (e.g. '===============')
+        if not title_regex.match(line):
             rst_lines.append(line)
-        else:
+            prev_non_title_line = line
+            continue
+        # The title underline must match at a minimum the length of the title
+        if len(prev_non_title_line) > len(line):
             rst_lines.append(line)
+            prev_non_title_line = line
+            continue
+
+        line_title_style = line[0]
+        line_title_level = title_styles.index(line_title_style)
+
+        # Not all files will start with "===" and we should be flexible enough
+        # to allow that. The first title we encounter sets the "file offset"
+        # which is added to the title_level desired.
+        if file_offset is None:
+            file_offset = line_title_level
+            if file_offset != 0:
+                print ("     WARNING: %s starts with a title style of '%s' but '%s' " +
+                    "is preferable.") % (filename, line_title_style, title_styles[0])
+
+        # Sanity checks: Make sure that this file is obeying the title levels
+        # specified and bail if it isn't.
+        # The file is allowed to go 1 deeper or any number shallower
+        if prev_line_title_level - line_title_level < -1:
+            raise Exception(
+                ("File '%s' line '%s' has a title " +
+                "style '%s' which doesn't match one of the " +
+                "allowed title styles of %s because the " +
+                "title level before this line was '%s'") %
+                (filename, (i + 1), line_title_style, title_styles,
+                title_styles[prev_line_title_level])
+            )
+        prev_line_title_level = line_title_level
+
+        adjusted_level = (
+            title_level + line_title_level - file_offset
+        )
+
+        # Sanity check: Make sure we can bump down the title and we aren't at the
+        # lowest level already
+        if adjusted_level >= len(title_styles):
+            raise Exception(
+                ("Files '%s' line '%s' has a sub-title level too low and it " +
+                "cannot be adjusted to fit. You can add another level to the " +
+                "'title_styles' key in targets.yaml to fix this.") %
+                (filename, (i + 1))
+            )
+
+        if adjusted_level == line_title_level:
+            # no changes required
+            rst_lines.append(line)
+            continue
+
+        # Adjusting line levels
+        # print (
+        #     "File: %s Adjusting %s to %s because file_offset=%s title_offset=%s" %
+        #     (filename, line_title_style,
+        #         title_styles[adjusted_level],
+        #         file_offset, title_level)
+        # )
+        rst_lines.append(line.replace(
+            line_title_style,
+            title_styles[adjusted_level]
+        ))
+            
     return "".join(rst_lines)
 
 
-def get_rst(file_info, title_level, title_styles, spec_dir):
+def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles):
     # string are file paths to RST blobs
     if isinstance(file_info, basestring):
         print "%s %s" % (">" * (1 + title_level), file_info)
         with open(spec_dir + file_info, "r") as f:
-            return load_with_adjusted_titles(file_info, f, title_level, title_styles)
+            rst = None
+            if adjust_titles:
+                rst = load_with_adjusted_titles(
+                    file_info, f, title_level, title_styles
+                )
+            else:
+                rst = f.read()
+            if rst[-2:] != "\n\n":
+                raise Exception(
+                    ("File %s should end with TWO new-line characters to ensure " +
+                    "file concatenation works correctly.") % (file_info,)
+                )
+            return rst
     # dicts look like {0: filepath, 1: filepath} where the key is the title level
     elif isinstance(file_info, dict):
         levels = sorted(file_info.keys())
         rst = []
         for l in levels:
-            rst.append(get_rst(file_info[l], l, title_styles, spec_dir))
+            rst.append(get_rst(file_info[l], l, title_styles, spec_dir, adjust_titles))
         return "".join(rst)
     # lists are multiple file paths e.g. [filepath, filepath]
     elif isinstance(file_info, list):
         rst = []
         for f in file_info:
-            rst.append(get_rst(f, title_level, title_styles, spec_dir))
+            rst.append(get_rst(f, title_level, title_styles, spec_dir, adjust_titles))
         return "".join(rst)
     raise Exception(
         "The following 'file' entry in this target isn't a string, list or dict. " +
@@ -82,11 +158,74 @@ def build_spec(target, out_filename):
                 file_info=file_info,
                 title_level=0,
                 title_styles=target["title_styles"],
-                spec_dir="../specification/"
+                spec_dir="../specification/",
+                adjust_titles=True
             )
             outfile.write(section)
 
 
+"""
+Replaces relative title styles with actual title styles.
+
+The templating system has no idea what the right title style is when it produces
+RST because it depends on the build target. As a result, it uses relative title
+styles defined in targets.yaml to say "down a level, up a level, same level".
+
+This function replaces these relative titles with actual title styles from the
+array in targets.yaml.
+"""
+def fix_relative_titles(target, filename, out_filename):
+    title_styles = target["title_styles"] # ["=", "-", "~", "+"]
+    relative_title_chars = [ # ["<", "/", ">"]
+        target["relative_title_styles"]["subtitle"],
+        target["relative_title_styles"]["sametitle"],
+        target["relative_title_styles"]["supertitle"]
+    ]
+    relative_title_matcher = re.compile(
+        "^[" + re.escape("".join(relative_title_chars)) + "]{3,}$"
+    )
+    title_matcher = re.compile(
+        "^[" + re.escape("".join(title_styles)) + "]{3,}$"
+    )
+    current_title_style = None
+    with open(filename, "r") as infile:
+        with open(out_filename, "w") as outfile:
+            for line in infile.readlines():
+                if not relative_title_matcher.match(line):
+                    if title_matcher.match(line):
+                        current_title_style = line[0]
+                    outfile.write(line)
+                    continue
+                line_char = line[0]
+                replacement_char = None
+                current_title_level = title_styles.index(current_title_style)
+                if line_char == target["relative_title_styles"]["subtitle"]:
+                    if (current_title_level + 1) == len(title_styles):
+                        raise Exception(
+                            "Encountered sub-title line style but we can't go " +
+                            "any lower."
+                        )
+                    replacement_char = title_styles[current_title_level + 1]
+                elif line_char == target["relative_title_styles"]["sametitle"]:
+                    replacement_char = title_styles[current_title_level]
+                elif line_char == target["relative_title_styles"]["supertitle"]:
+                    if (current_title_level - 1) < 0:
+                        raise Exception(
+                            "Encountered super-title line style but we can't go " +
+                            "any higher."
+                        )
+                    replacement_char = title_styles[current_title_level - 1]
+                else:
+                    raise Exception(
+                        "Unknown relative line char %s" % (line_char,)
+                    )
+
+                outfile.write(
+                    line.replace(line_char, replacement_char)
+                )
+
+
+
 def rst2html(i, o):
     with open(i, "r") as in_file:
         with open(o, "w") as out_file:
@@ -123,11 +262,13 @@ def run_through_template(input):
 def get_build_target(targets_listing, target_name):
     build_target = {
         "title_styles": [],
+        "relative_title_styles": {},
         "files": []
     }
     with open(targets_listing, "r") as targ_file:
         all_targets = yaml.load(targ_file.read())
         build_target["title_styles"] = all_targets["title_styles"]
+        build_target["relative_title_styles"] = all_targets["relative_title_styles"]
         target = all_targets["targets"].get(target_name)
         if not target:
             raise Exception(
@@ -138,17 +279,30 @@ def get_build_target(targets_listing, target_name):
             raise Exception(
                 "Found target but 'files' key is not a list."
             )
+
+        def get_group(group_id):
+            group_name = group_id[len("group:"):]
+            group = all_targets.get("groups", {}).get(group_name)
+            if not group:
+                raise Exception(
+                    "Tried to find group '" + group_name + "' but it " +
+                    "doesn't exist."
+                )
+            return group
+
         resolved_files = []
         for f in target["files"]:
+            group = None
             if isinstance(f, basestring) and f.startswith("group:"):
-                # copy across the group of files specified
-                group_name = f[len("group:"):]
-                group = all_targets.get("groups", {}).get(group_name)
-                if not group:
-                    raise Exception(
-                        "Tried to find group '" + group_name + "' but it " +
-                        "doesn't exist."
-                    )
+                group = get_group(f)
+            elif isinstance(f, dict):
+                for (k, v) in f.iteritems():
+                    if isinstance(v, basestring) and v.startswith("group:"):
+                        f[k] = get_group(v)
+                resolved_files.append(f)
+                continue
+
+            if group:
                 if isinstance(group, list):
                     resolved_files.extend(group)
                 else:
@@ -178,8 +332,12 @@ def main(target_name):
     prepare_env()
     print "Building spec [target=%s]" % target_name
     target = get_build_target("../specification/targets.yaml", target_name)
-    build_spec(target=target, out_filename="tmp/full_spec.rst")
-    run_through_template("tmp/full_spec.rst")
+    build_spec(target=target, out_filename="tmp/templated_spec.rst")
+    run_through_template("tmp/templated_spec.rst")
+    fix_relative_titles(
+        target=target, filename="tmp/templated_spec.rst",
+        out_filename="tmp/full_spec.rst"
+    )
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")
     run_through_template("tmp/howto.rst")
     rst2html("tmp/full_spec.rst", "gen/specification.html")
diff --git a/specification/00_01_feature_profiles.rst b/specification/00_01_feature_profiles.rst
index e69de29b..155e51c5 100644
--- a/specification/00_01_feature_profiles.rst
+++ b/specification/00_01_feature_profiles.rst
@@ -0,0 +1,5 @@
+Feature Profiles
+================
+
+Feature profiles blurb goes here.
+
diff --git a/specification/02_00_modules.rst b/specification/02_00_modules.rst
index e69de29b..ab35fe9f 100644
--- a/specification/02_00_modules.rst
+++ b/specification/02_00_modules.rst
@@ -0,0 +1,5 @@
+Modules
+=======
+
+Modules intro here.
+
diff --git a/specification/03_00_application_service_api.rst b/specification/03_00_application_service_api.rst
index 2674ba44..e982390b 100644
--- a/specification/03_00_application_service_api.rst
+++ b/specification/03_00_application_service_api.rst
@@ -400,3 +400,4 @@ in their content to provide a way for Matrix clients to link into the 'native'
 client from which the event originated.  For instance, this could contain the
 message-ID for emails/nntp posts, or a link to a blog comment when gatewaying
 blog comment traffic in & out of matrix
+
diff --git a/specification/04_00_server_server_api.rst b/specification/04_00_server_server_api.rst
index f5cadabf..8d1f8898 100644
--- a/specification/04_00_server_server_api.rst
+++ b/specification/04_00_server_server_api.rst
@@ -92,7 +92,7 @@ server by querying other servers.
 .. _Perspectives Project: http://perspectives-project.org/
 
 Publishing Keys
-_______________
+^^^^^^^^^^^^^^^
 
 Home servers publish the allowed TLS fingerprints and signing keys in a JSON
 object at ``/_matrix/key/v2/server/{key_id}``. The response contains a list of
@@ -178,7 +178,7 @@ events sent by that server can still be checked.
     }
 
 Querying Keys Through Another Server
-____________________________________
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Servers may offer a query API ``_matrix/key/v2/query/`` for getting the keys
 for another server. This API can be used to GET at list of JSON objects for a
diff --git a/specification/modules/00_modules_intro.rst b/specification/modules/00_modules_intro.rst
index e69de29b..fdfcbbdc 100644
--- a/specification/modules/00_modules_intro.rst
+++ b/specification/modules/00_modules_intro.rst
@@ -0,0 +1,5 @@
+Modules
+=======
+
+Modules blurb goes here.
+
diff --git a/specification/targets.yaml b/specification/targets.yaml
index e492ad3c..7bd3d882 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -2,26 +2,37 @@ targets:
   main:  # arbitrary name to identify this build target
     files:  # the sort order of files to cat
       - 00_00_intro.rst
-      - 00_01_feature_profiles.rst
-      - 00_02a_events.rst
-      - 00_02b_event_signing.rst
+      - { 1: 00_01_feature_profiles.rst }
+      - { 1: 00_02a_events.rst }
+      - { 1: 00_02b_event_signing.rst }
       - 01_00_client_server_api.rst
       - 02_00_modules.rst
-      - "group:modules"  # reference a group of files
+      - { 1: "group:modules" }  # reference a group of files
       - 03_00_application_service_api.rst
       - 04_00_server_server_api.rst
       - 05_00_identity_servers.rst
       - 06_00_appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
-      0: modules/00_modules_intro.rst
-      1:
-         - modules/01_00_voip_events.rst
-         - modules/02_00_typing_notifications.rst
-         - modules/03_00_receipts.rst
-         - modules/04_00_content_repo.rst
-         - modules/05_00_end_to_end_encryption.rst
-         - modules/06_00_history_visibility.rst
-         - 1: modules/07_00_push_overview.rst # Mark a nested file dependency
-           2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
-title_styles: ["=", "-", "~", "+"]
+    - modules/00_modules_intro.rst
+    - modules/01_00_voip_events.rst
+    - modules/02_00_typing_notifications.rst
+    - modules/03_00_receipts.rst
+    - modules/04_00_content_repo.rst
+    - modules/05_00_end_to_end_encryption.rst
+    - modules/06_00_history_visibility.rst
+    - modules/07_00_push_overview.rst
+    - { 2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst] }
+
+title_styles: ["=", "-", "~", "+", "^"]
+
+# The templating system doesn't know the right title style to use when generating
+# RST. These symbols are 'relative' to say "make a sub-title" (-1), "make a title
+# at the same level (0)", or "make a title one above (+1)". The gendoc script
+# will inspect this file and replace these relative styles with actual title
+# styles. The templating system will also inspect this file to know which symbols
+# to inject.
+relative_title_styles:
+  subtitle: "<"
+  sametitle: "/"
+  supertitle: ">"
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 729157bb..6072222a 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -23,10 +23,13 @@ class MatrixSections(Sections):
         spec_meta = self.units.get("spec_meta")
         return spec_meta["changelog"]
 
-    def _render_events(self, filterFn, sortFn, title_kind="~"):
+    def _render_events(self, filterFn, sortFn):
         template = self.env.get_template("events.tmpl")
         examples = self.units.get("event_examples")
         schemas = self.units.get("event_schemas")
+        subtitle_title_char = self.units.get("spec_targets")[
+            "relative_title_styles"
+        ]["subtitle"]
         sections = []
         for event_name in sortFn(schemas):
             if not filterFn(event_name):
@@ -34,14 +37,16 @@ class MatrixSections(Sections):
             sections.append(template.render(
                 example=examples[event_name], 
                 event=schemas[event_name],
-                title_kind=title_kind
+                title_kind=subtitle_title_char
             ))
         return "\n\n".join(sections)
 
-    def _render_http_api_group(self, group, sortFnOrPathList=None, 
-                               title_kind="-"):
+    def _render_http_api_group(self, group, sortFnOrPathList=None):
         template = self.env.get_template("http-api.tmpl")
         http_api = self.units.get("swagger_apis")[group]["__meta"]
+        subtitle_title_char = self.units.get("spec_targets")[
+            "relative_title_styles"
+        ]["subtitle"]
         sections = []
         endpoints = []
         if sortFnOrPathList:
@@ -67,15 +72,14 @@ class MatrixSections(Sections):
         for endpoint in endpoints:
             sections.append(template.render(
                 endpoint=endpoint,
-                title_kind=title_kind
+                title_kind=subtitle_title_char
             ))
         return "\n\n".join(sections)
 
     def render_profile_http_api(self):
         return self._render_http_api_group(
             "profile", 
-            sortFnOrPathList=["displayname", "avatar_url"],
-            title_kind="~"
+            sortFnOrPathList=["displayname", "avatar_url"]
         )
 
     def render_sync_http_api(self):
@@ -86,20 +90,17 @@ class MatrixSections(Sections):
     def render_presence_http_api(self):
         return self._render_http_api_group(
             "presence",
-            sortFnOrPathList=["status"],
-            title_kind="~"
+            sortFnOrPathList=["status"]
         )
 
     def render_membership_http_api(self):
         return self._render_http_api_group(
-            "membership",
-            title_kind="~"
+            "membership"
         )
 
     def render_login_http_api(self):
         return self._render_http_api_group(
-            "login",
-            title_kind="~"
+            "login"
         )
 
     def render_room_events(self):
@@ -114,6 +115,9 @@ class MatrixSections(Sections):
         template = self.env.get_template("msgtypes.tmpl")
         examples = self.units.get("event_examples")
         schemas = self.units.get("event_schemas")
+        subtitle_title_char = self.units.get("spec_targets")[
+            "relative_title_styles"
+        ]["subtitle"]
         sections = []
         msgtype_order = [
             "m.room.message#m.text", "m.room.message#m.emote",
@@ -129,7 +133,8 @@ class MatrixSections(Sections):
                 continue
             sections.append(template.render(
                 example=examples[event_name], 
-                event=schemas[event_name]
+                event=schemas[event_name],
+                title_kind=subtitle_title_char
             ))
         return "\n\n".join(sections)
 
@@ -150,7 +155,7 @@ class MatrixSections(Sections):
     def render_presence_events(self):
         def filterFn(eventType):
             return eventType.startswith("m.presence")
-        return self._render_events(filterFn, sorted, title_kind="+")
+        return self._render_events(filterFn, sorted)
 
     def _render_ce_type(self, type):
         template = self.env.get_template("common-event-fields.tmpl")
diff --git a/templating/matrix_templates/templates/msgtypes.tmpl b/templating/matrix_templates/templates/msgtypes.tmpl
index 29e86160..f7862451 100644
--- a/templating/matrix_templates/templates/msgtypes.tmpl
+++ b/templating/matrix_templates/templates/msgtypes.tmpl
@@ -1,5 +1,5 @@
 ``{{event.msgtype}}``
-{{(4 + event.msgtype | length) * '+'}}
+{{(4 + event.msgtype | length) * title_kind}}
 {{event.desc | wrap(80)}}
 {% for table in event.content_fields -%}
 {{"``"+table.title+"``" if table.title else "" }}
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 1a9d981a..0096bbfa 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -13,6 +13,7 @@ V1_EVENT_EXAMPLES = "../event-schemas/examples/v1"
 V1_EVENT_SCHEMA = "../event-schemas/schema/v1"
 CORE_EVENT_SCHEMA = "../event-schemas/schema/v1/core-event-schema"
 CHANGELOG = "../CHANGELOG.rst"
+TARGETS = "../specification/targets.yaml"
 
 ROOM_EVENT = "core-event-schema/room_event.json"
 STATE_EVENT = "core-event-schema/state_event.json"
@@ -466,6 +467,12 @@ class MatrixUnits(Units):
             "changelog": "".join(changelog_lines)
         }
 
+
+    def load_spec_targets(self):
+        with open(TARGETS, "r") as f:
+            return yaml.load(f.read())
+
+
     def load_git_version(self):
         null = open(os.devnull, 'w')
         cwd = os.path.dirname(os.path.abspath(__file__))

From e8cdfcbba28223055a182e06c1ee1a4f04cc3738 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 14:01:55 +0100
Subject: [PATCH 64/85] Rename spec files to not have numbers everywhere;
 update targets.yaml

---
 ..._event_signing.rst => 0-event_signing.rst} |  0
 .../{00_02a_events.rst => 0-events.rst}       |  0
 ...re_profiles.rst => 0-feature_profiles.rst} |  0
 .../{00_00_intro.rst => 0-intro.rst}          |  0
 ...server_api.rst => 1-client_server_api.rst} |  0
 .../{02_00_modules.rst => 2-modules.rst}      |  0
 ..._api.rst => 3-application_service_api.rst} |  0
 ...server_api.rst => 4-server_server_api.rst} |  0
 ...ity_servers.rst => 5-identity_servers.rst} |  0
 ...{06_00_appendices.rst => 6-appendices.rst} |  0
 ...4_00_content_repo.rst => content_repo.rst} |  0
 ...cryption.rst => end_to_end_encryption.rst} |  0
 ..._visibility.rst => history_visibility.rst} |  0
 .../{00_modules_intro.rst => intro.rst}       |  0
 ...{07_01_push_cs_api.rst => push_cs_api.rst} |  0
 ...00_push_overview.rst => push_overview.rst} |  0
 ...h_push_gw_api.rst => push_push_gw_api.rst} |  0
 .../{03_00_receipts.rst => receipts.rst}      |  0
 ...fications.rst => typing_notifications.rst} |  0
 ...{01_00_voip_events.rst => voip_events.rst} |  0
 specification/targets.yaml                    | 38 +++++++++----------
 21 files changed, 19 insertions(+), 19 deletions(-)
 rename specification/{00_02b_event_signing.rst => 0-event_signing.rst} (100%)
 rename specification/{00_02a_events.rst => 0-events.rst} (100%)
 rename specification/{00_01_feature_profiles.rst => 0-feature_profiles.rst} (100%)
 rename specification/{00_00_intro.rst => 0-intro.rst} (100%)
 rename specification/{01_00_client_server_api.rst => 1-client_server_api.rst} (100%)
 rename specification/{02_00_modules.rst => 2-modules.rst} (100%)
 rename specification/{03_00_application_service_api.rst => 3-application_service_api.rst} (100%)
 rename specification/{04_00_server_server_api.rst => 4-server_server_api.rst} (100%)
 rename specification/{05_00_identity_servers.rst => 5-identity_servers.rst} (100%)
 rename specification/{06_00_appendices.rst => 6-appendices.rst} (100%)
 rename specification/modules/{04_00_content_repo.rst => content_repo.rst} (100%)
 rename specification/modules/{05_00_end_to_end_encryption.rst => end_to_end_encryption.rst} (100%)
 rename specification/modules/{06_00_history_visibility.rst => history_visibility.rst} (100%)
 rename specification/modules/{00_modules_intro.rst => intro.rst} (100%)
 rename specification/modules/{07_01_push_cs_api.rst => push_cs_api.rst} (100%)
 rename specification/modules/{07_00_push_overview.rst => push_overview.rst} (100%)
 rename specification/modules/{07_02_push_push_gw_api.rst => push_push_gw_api.rst} (100%)
 rename specification/modules/{03_00_receipts.rst => receipts.rst} (100%)
 rename specification/modules/{02_00_typing_notifications.rst => typing_notifications.rst} (100%)
 rename specification/modules/{01_00_voip_events.rst => voip_events.rst} (100%)

diff --git a/specification/00_02b_event_signing.rst b/specification/0-event_signing.rst
similarity index 100%
rename from specification/00_02b_event_signing.rst
rename to specification/0-event_signing.rst
diff --git a/specification/00_02a_events.rst b/specification/0-events.rst
similarity index 100%
rename from specification/00_02a_events.rst
rename to specification/0-events.rst
diff --git a/specification/00_01_feature_profiles.rst b/specification/0-feature_profiles.rst
similarity index 100%
rename from specification/00_01_feature_profiles.rst
rename to specification/0-feature_profiles.rst
diff --git a/specification/00_00_intro.rst b/specification/0-intro.rst
similarity index 100%
rename from specification/00_00_intro.rst
rename to specification/0-intro.rst
diff --git a/specification/01_00_client_server_api.rst b/specification/1-client_server_api.rst
similarity index 100%
rename from specification/01_00_client_server_api.rst
rename to specification/1-client_server_api.rst
diff --git a/specification/02_00_modules.rst b/specification/2-modules.rst
similarity index 100%
rename from specification/02_00_modules.rst
rename to specification/2-modules.rst
diff --git a/specification/03_00_application_service_api.rst b/specification/3-application_service_api.rst
similarity index 100%
rename from specification/03_00_application_service_api.rst
rename to specification/3-application_service_api.rst
diff --git a/specification/04_00_server_server_api.rst b/specification/4-server_server_api.rst
similarity index 100%
rename from specification/04_00_server_server_api.rst
rename to specification/4-server_server_api.rst
diff --git a/specification/05_00_identity_servers.rst b/specification/5-identity_servers.rst
similarity index 100%
rename from specification/05_00_identity_servers.rst
rename to specification/5-identity_servers.rst
diff --git a/specification/06_00_appendices.rst b/specification/6-appendices.rst
similarity index 100%
rename from specification/06_00_appendices.rst
rename to specification/6-appendices.rst
diff --git a/specification/modules/04_00_content_repo.rst b/specification/modules/content_repo.rst
similarity index 100%
rename from specification/modules/04_00_content_repo.rst
rename to specification/modules/content_repo.rst
diff --git a/specification/modules/05_00_end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst
similarity index 100%
rename from specification/modules/05_00_end_to_end_encryption.rst
rename to specification/modules/end_to_end_encryption.rst
diff --git a/specification/modules/06_00_history_visibility.rst b/specification/modules/history_visibility.rst
similarity index 100%
rename from specification/modules/06_00_history_visibility.rst
rename to specification/modules/history_visibility.rst
diff --git a/specification/modules/00_modules_intro.rst b/specification/modules/intro.rst
similarity index 100%
rename from specification/modules/00_modules_intro.rst
rename to specification/modules/intro.rst
diff --git a/specification/modules/07_01_push_cs_api.rst b/specification/modules/push_cs_api.rst
similarity index 100%
rename from specification/modules/07_01_push_cs_api.rst
rename to specification/modules/push_cs_api.rst
diff --git a/specification/modules/07_00_push_overview.rst b/specification/modules/push_overview.rst
similarity index 100%
rename from specification/modules/07_00_push_overview.rst
rename to specification/modules/push_overview.rst
diff --git a/specification/modules/07_02_push_push_gw_api.rst b/specification/modules/push_push_gw_api.rst
similarity index 100%
rename from specification/modules/07_02_push_push_gw_api.rst
rename to specification/modules/push_push_gw_api.rst
diff --git a/specification/modules/03_00_receipts.rst b/specification/modules/receipts.rst
similarity index 100%
rename from specification/modules/03_00_receipts.rst
rename to specification/modules/receipts.rst
diff --git a/specification/modules/02_00_typing_notifications.rst b/specification/modules/typing_notifications.rst
similarity index 100%
rename from specification/modules/02_00_typing_notifications.rst
rename to specification/modules/typing_notifications.rst
diff --git a/specification/modules/01_00_voip_events.rst b/specification/modules/voip_events.rst
similarity index 100%
rename from specification/modules/01_00_voip_events.rst
rename to specification/modules/voip_events.rst
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 7bd3d882..df61e7a1 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -1,28 +1,28 @@
 targets:
   main:  # arbitrary name to identify this build target
     files:  # the sort order of files to cat
-      - 00_00_intro.rst
-      - { 1: 00_01_feature_profiles.rst }
-      - { 1: 00_02a_events.rst }
-      - { 1: 00_02b_event_signing.rst }
-      - 01_00_client_server_api.rst
-      - 02_00_modules.rst
+      - 0-intro.rst
+      - { 1: 0-feature_profiles.rst }
+      - { 1: 0-events.rst }
+      - { 1: 0-event_signing.rst }
+      - 1-client_server_api.rst
+      - 2-modules.rst
       - { 1: "group:modules" }  # reference a group of files
-      - 03_00_application_service_api.rst
-      - 04_00_server_server_api.rst
-      - 05_00_identity_servers.rst
-      - 06_00_appendices.rst
+      - 3-application_service_api.rst
+      - 4-server_server_api.rst
+      - 5-identity_servers.rst
+      - 6-appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
-    - modules/00_modules_intro.rst
-    - modules/01_00_voip_events.rst
-    - modules/02_00_typing_notifications.rst
-    - modules/03_00_receipts.rst
-    - modules/04_00_content_repo.rst
-    - modules/05_00_end_to_end_encryption.rst
-    - modules/06_00_history_visibility.rst
-    - modules/07_00_push_overview.rst
-    - { 2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst] }
+    - modules/intro.rst
+    - modules/voip_events.rst
+    - modules/typing_notifications.rst
+    - modules/receipts.rst
+    - modules/content_repo.rst
+    - modules/end_to_end_encryption.rst
+    - modules/history_visibility.rst
+    - modules/push_overview.rst
+    - { 2: [modules/push_cs_api.rst , modules/push_push_gw_api.rst] }
 
 title_styles: ["=", "-", "~", "+", "^"]
 

From 0c0ac3e814c763e271d7a0c9b5c9b5e824b2f2f3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 14:07:23 +0100
Subject: [PATCH 65/85] Fix common event fields template to use subtitle char

---
 specification/modules/intro.rst                            | 5 -----
 specification/targets.yaml                                 | 1 -
 templating/matrix_templates/sections.py                    | 7 ++++++-
 .../matrix_templates/templates/common-event-fields.tmpl    | 2 +-
 4 files changed, 7 insertions(+), 8 deletions(-)
 delete mode 100644 specification/modules/intro.rst

diff --git a/specification/modules/intro.rst b/specification/modules/intro.rst
deleted file mode 100644
index fdfcbbdc..00000000
--- a/specification/modules/intro.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Modules
-=======
-
-Modules blurb goes here.
-
diff --git a/specification/targets.yaml b/specification/targets.yaml
index df61e7a1..96527dae 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -14,7 +14,6 @@ targets:
       - 6-appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
-    - modules/intro.rst
     - modules/voip_events.rst
     - modules/typing_notifications.rst
     - modules/receipts.rst
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 6072222a..78f011b9 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -160,7 +160,12 @@ class MatrixSections(Sections):
     def _render_ce_type(self, type):
         template = self.env.get_template("common-event-fields.tmpl")
         ce_types = self.units.get("common_event_fields")
-        return template.render(common_event=ce_types[type])
+        subtitle_title_char = self.units.get("spec_targets")[
+            "relative_title_styles"
+        ]["subtitle"]
+        return template.render(
+            common_event=ce_types[type], title_kind=subtitle_title_char
+        )
 
     def render_common_event_fields(self):
         return self._render_ce_type("event")
diff --git a/templating/matrix_templates/templates/common-event-fields.tmpl b/templating/matrix_templates/templates/common-event-fields.tmpl
index 2a3f7ec5..3f16be3d 100644
--- a/templating/matrix_templates/templates/common-event-fields.tmpl
+++ b/templating/matrix_templates/templates/common-event-fields.tmpl
@@ -1,5 +1,5 @@
 {{common_event.title}} Fields
-{{(7 + common_event.title | length) * '-'}}
+{{(7 + common_event.title | length) * title_kind}}
 
 {{common_event.desc | wrap(80)}}
 

From 056b5eba22e94b613c50b6255b23152553bced2e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 16:01:22 +0100
Subject: [PATCH 66/85] Partially handle representing top-level array responses

If an HTTP API returned a top-level array response, the templating system
would fail to create a table for it. This is now partially fixed by pulling
out the type of the elements (no recursion is done to populate nested tables)
---
 templating/matrix_templates/units.py | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 84c96ce3..975f7c80 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -247,7 +247,8 @@ class MatrixUnits(Units):
                             "rows": [{
                                 "key": good_response["schema"].get("name", ""),
                                 "type": res_type,
-                                "desc": res.get("description", "")
+                                "desc": res.get("description", ""),
+                                "req_str": ""
                             }]
                         })
                     elif res_type and Units.prop(good_response, "schema/properties"):
@@ -257,6 +258,24 @@ class MatrixUnits(Units):
                         for table in res_tables:
                             if "no-table" not in table:
                                 endpoint["res_tables"].append(table)
+                    elif res_type and Units.prop(good_response, "schema/items"):
+                        # response is an array:
+                        # FIXME: Doesn't recurse at all.
+                        schema = good_response["schema"]
+                        array_type = Units.prop(schema, "items/type")
+                        if Units.prop(schema, "items/allOf"):
+                            array_type = (
+                                Units.prop(schema, "items/title")
+                            )
+                        endpoint["res_tables"].append({
+                            "title": schema.get("title", ""),
+                            "rows": [{
+                                "key": "N/A",
+                                "type": ("[%s]" % array_type),
+                                "desc": schema.get("description", ""),
+                                "req_str": ""
+                            }]
+                        })
 
                 endpoints.append(endpoint)
 

From 388aeefac000d89e9b775e1fb7905b8e8f178ae2 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 16:07:19 +0100
Subject: [PATCH 67/85] Remove obsolete key

---
 templating/matrix_templates/sections.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 15dca9b1..cacaa06a 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -105,8 +105,7 @@ class MatrixSections(Sections):
 
     def render_rooms_http_api(self):
         return self._render_http_api_group(
-            "rooms",
-            title_kind="+"
+            "rooms"
         )
 
     def render_room_events(self):

From b21859836d1b6ca0df6f4072c393211706eae9a8 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 16:11:16 +0100
Subject: [PATCH 68/85] Print stdout of build.py in verbose mode

---
 scripts/gendoc.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 308d148f..6c95e33c 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -243,9 +243,9 @@ def run_through_template(input):
     tmpfile = './tmp/output'
     try:
         with open(tmpfile, 'w') as out:
-            subprocess.check_output(
+            print subprocess.check_output(
                 [
-                    'python', 'build.py',
+                    'python', 'build.py', "-v",
                     "-i", "matrix_templates",
                     "-o", "../scripts/tmp",
                     "../scripts/"+input

From 16693a644aead93c67ad1f10d3afb1d7fc0169a5 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 16:59:49 +0100
Subject: [PATCH 69/85] Various review fixes

---
 scripts/gendoc.py                    | 105 ++++++++++++++++-----------
 specification/0-feature_profiles.rst |   2 -
 specification/2-modules.rst          |   2 -
 3 files changed, 62 insertions(+), 47 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 6c95e33c..26ee5a48 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -118,7 +118,7 @@ def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles):
     # string are file paths to RST blobs
     if isinstance(file_info, basestring):
         print "%s %s" % (">" * (1 + title_level), file_info)
-        with open(spec_dir + file_info, "r") as f:
+        with open(os.path.join(spec_dir, file_info), "r") as f:
             rst = None
             if adjust_titles:
                 rst = load_with_adjusted_titles(
@@ -175,8 +175,8 @@ This function replaces these relative titles with actual title styles from the
 array in targets.yaml.
 """
 def fix_relative_titles(target, filename, out_filename):
-    title_styles = target["title_styles"] # ["=", "-", "~", "+"]
-    relative_title_chars = [ # ["<", "/", ">"]
+    title_styles = target["title_styles"]
+    relative_title_chars = [
         target["relative_title_styles"]["subtitle"],
         target["relative_title_styles"]["sametitle"],
         target["relative_title_styles"]["supertitle"]
@@ -259,6 +259,17 @@ def run_through_template(input):
         raise
 
 
+"""
+Extract and resolve groups for the given target in the given targets listing.
+Args:
+    targets_listing (str): The path to a YAML file containing a list of targets
+    target_name (str): The name of the target to extract from the listings.
+Returns:
+    dict: Containing "filees" (a list of file paths), "relative_title_styles"
+          (a dict of relative style keyword to title character) and "title_styles"
+          (a list of characters which represent the global title style to follow,
+           with the top section title first, the second section second, and so on.)
+"""
 def get_build_target(targets_listing, target_name):
     build_target = {
         "title_styles": [],
@@ -267,49 +278,57 @@ def get_build_target(targets_listing, target_name):
     }
     with open(targets_listing, "r") as targ_file:
         all_targets = yaml.load(targ_file.read())
-        build_target["title_styles"] = all_targets["title_styles"]
-        build_target["relative_title_styles"] = all_targets["relative_title_styles"]
-        target = all_targets["targets"].get(target_name)
-        if not target:
-            raise Exception(
-                "No target by the name '" + target_name + "' exists in '" +
-                targets_listing + "'."
-            )
-        if not isinstance(target.get("files"), list):
+
+    build_target["title_styles"] = all_targets["title_styles"]
+    build_target["relative_title_styles"] = all_targets["relative_title_styles"]
+    target = all_targets["targets"].get(target_name)
+    if not target:
+        raise Exception(
+            "No target by the name '" + target_name + "' exists in '" +
+            targets_listing + "'."
+        )
+    if not isinstance(target.get("files"), list):
+        raise Exception(
+            "Found target but 'files' key is not a list."
+        )
+
+    def get_group(group_id):
+        group_name = group_id[len("group:"):]
+        group = all_targets.get("groups", {}).get(group_name)
+        if not group:
             raise Exception(
-                "Found target but 'files' key is not a list."
+                "Tried to find group '" + group_name + "' but it " +
+                "doesn't exist."
             )
-
-        def get_group(group_id):
-            group_name = group_id[len("group:"):]
-            group = all_targets.get("groups", {}).get(group_name)
-            if not group:
-                raise Exception(
-                    "Tried to find group '" + group_name + "' but it " +
-                    "doesn't exist."
-                )
-            return group
-
-        resolved_files = []
-        for f in target["files"]:
-            group = None
-            if isinstance(f, basestring) and f.startswith("group:"):
-                group = get_group(f)
-            elif isinstance(f, dict):
-                for (k, v) in f.iteritems():
-                    if isinstance(v, basestring) and v.startswith("group:"):
-                        f[k] = get_group(v)
-                resolved_files.append(f)
-                continue
-
-            if group:
-                if isinstance(group, list):
-                    resolved_files.extend(group)
-                else:
-                    resolved_files.append(group)
+        return group
+
+    resolved_files = []
+    for file_entry in target["files"]:
+        # file_entry is a group id
+        if isinstance(file_entry, basestring) and file_entry.startswith("group:"):
+            group = get_group(file_entry)
+            # The group may be resolved to a list of file entries, in which case
+            # we want to extend the array to insert each of them rather than
+            # insert the entire list as a single element (which is what append does)
+            if isinstance(group, list):
+                resolved_files.extend(group)
             else:
-                resolved_files.append(f)
-        build_target["files"] = resolved_files
+                resolved_files.append(group)
+        # file_entry is a dict which has more file entries as values
+        elif isinstance(file_entry, dict):
+            resolved_entry = {}
+            for (k, v) in file_entry.iteritems():
+                if isinstance(v, basestring) and v.startswith("group:"):
+                    resolved_entry[k] = get_group(v)
+                else:
+                    # map across without editing (e.g. normal file path)
+                    resolved_entry[k] = v
+            resolved_files.append(resolved_entry)
+            continue
+        # file_entry is just a plain ol' file path
+        else:
+            resolved_files.append(file_entry)
+    build_target["files"] = resolved_files
     return build_target
 
 
diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 155e51c5..234e14db 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -1,5 +1,3 @@
 Feature Profiles
 ================
 
-Feature profiles blurb goes here.
-
diff --git a/specification/2-modules.rst b/specification/2-modules.rst
index ab35fe9f..0aad77e1 100644
--- a/specification/2-modules.rst
+++ b/specification/2-modules.rst
@@ -1,5 +1,3 @@
 Modules
 =======
 
-Modules intro here.
-

From b49cb57fe596eb942759f67a054906386854867a Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 17:32:31 +0100
Subject: [PATCH 70/85] Move events sections to CS API

---
 specification/targets.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/targets.yaml b/specification/targets.yaml
index 96527dae..92ad9bb4 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -3,9 +3,9 @@ targets:
     files:  # the sort order of files to cat
       - 0-intro.rst
       - { 1: 0-feature_profiles.rst }
+      - 1-client_server_api.rst
       - { 1: 0-events.rst }
       - { 1: 0-event_signing.rst }
-      - 1-client_server_api.rst
       - 2-modules.rst
       - { 1: "group:modules" }  # reference a group of files
       - 3-application_service_api.rst

From 5b134119bd53200f00da3b52557ff88b739f8ad5 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 09:59:54 +0100
Subject: [PATCH 71/85] Add presence module; fix relative title bug

If a relative title appeared after an HTTP API table, it
would insert the wrong level because it thought that part
of the table was a title.
---
 specification/1-client_server_api.rst         |  6 --
 specification/modules/presence.rst            | 60 +++++++++++++++++++
 specification/targets.yaml                    |  1 +
 .../matrix_templates/templates/http-api.tmpl  |  2 +-
 4 files changed, 62 insertions(+), 7 deletions(-)
 create mode 100644 specification/modules/presence.rst

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index ad39dafc..93e3cb90 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -1054,12 +1054,6 @@ medium
 address
   The textual address of the 3pid, eg. the email address
 
-Presence
---------
-.. TODO-spec
-  - Define how users receive presence invites, and how they accept/decline them
-
-{{presence_http_api}}
 
 Profiles
 --------
diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
new file mode 100644
index 00000000..6f644879
--- /dev/null
+++ b/specification/modules/presence.rst
@@ -0,0 +1,60 @@
+Presence Events
+===============
+
+{{presence_events}}
+ 
+Each user has the concept of presence information. This encodes the
+"availability" of that user, suitable for display on other user's clients.
+This is transmitted as an ``m.presence`` event and is one of the few events
+which are sent *outside the context of a room*. The basic piece of presence
+information is represented by the ``presence`` key, which is an enum of one
+of the following:
+
+      - ``online`` : The default state when the user is connected to an event
+        stream.
+      - ``unavailable`` : The user is not reachable at this time.
+      - ``offline`` : The user is not connected to an event stream.
+      - ``free_for_chat`` : The user is generally willing to receive messages
+        moreso than default.
+      - ``hidden`` : Behaves as offline, but allows the user to see the client
+        state anyway and generally interact with client features. (Not yet
+        implemented in synapse).
+
+In addition, the server maintains a timestamp of the last time it saw a
+pro-active event from the user; either sending a message to a room, or
+changing presence state from a lower to a higher level of availability
+(thus: changing state from ``unavailable`` to ``online`` counts as a
+proactive event, whereas in the other direction it will not). This timestamp
+is presented via a key called ``last_active_ago``, which gives the relative
+number of milliseconds since the message is generated/emitted that the user
+was last seen active.
+
+Presence HTTP API
+-----------------
+.. TODO-spec
+  - Define how users receive presence invites, and how they accept/decline them
+
+{{presence_http_api}}
+    
+
+Events on Change of Profile Information
+---------------------------------------
+Because the profile displayname and avatar information are likely to be used in
+many places of a client's display, changes to these fields cause an automatic
+propagation event to occur, informing likely-interested parties of the new
+values. This change is conveyed using two separate mechanisms:
+
+ - a ``m.room.member`` event is sent to every room the user is a member of,
+   to update the ``displayname`` and ``avatar_url``.
+ - a ``m.presence`` presence status update is sent, again containing the new values of the
+   ``displayname`` and ``avatar_url`` keys, in addition to the required
+   ``presence`` key containing the current presence state of the user.
+
+Both of these should be done automatically by the home server when a user
+successfully changes their displayname or avatar URL fields.
+
+Additionally, when home servers emit room membership events for their own
+users, they should include the displayname and avatar URL fields in these
+events so that clients already have these details to hand, and do not have to
+perform extra roundtrips to query it.
+
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 92ad9bb4..454f23b9 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -17,6 +17,7 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/voip_events.rst
     - modules/typing_notifications.rst
     - modules/receipts.rst
+    - modules/presence.rst
     - modules/content_repo.rst
     - modules/end_to_end_encryption.rst
     - modules/history_visibility.rst
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index a03ffa7d..eb3f3e64 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -18,7 +18,7 @@ Request format:
 ================== ================= ===========================================
 {% for loc in endpoint.req_param_by_loc -%}
 *{{loc}} parameters*
---------------------------------------------------------------------------------
+-------------------------------------------------------------------------------- 
 {% for param in endpoint.req_param_by_loc[loc] -%}
 {{param.key}}{{param.type|indent(19-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(37)}}
 {% endfor -%}

From 51153462971d3c6ffd4fae4672504e857532c5c2 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 10:48:49 +0100
Subject: [PATCH 72/85] Add instant_messaging module; modify batesian section
 rules

Previously, all `m.room.*` events were wodged into `{{room_events}}` which
isn't great when you want to pull specific ones out. Batesian had a 1:1
mapping of `render_foo()` to a section `{{foo}}`, and having to constantly
add functions for new types is a PITA. Batesian now supports returning a
`dict` instead of a section `string` where the keys are the `{{foo}}` and
the value is what will be inserted. Also add conflicting section key checks
to avoid multiple definitions of the same `{{foo}}`. Define dicts for
event schemata and swagger HTTP APIs.

Using this new feature, split out the instant messaging stuff from the events
section, and replace `{{room_events}}` with a list of specific events e.g.
`{{m_room_member_event}}`.
---
 specification/0-events.rst                  | 82 ++++-----------------
 specification/modules/instant_messaging.rst | 27 +++++++
 specification/modules/presence.rst          | 11 ++-
 specification/targets.yaml                  |  1 +
 templating/batesian/sections.py             | 38 ++++++++--
 templating/matrix_templates/sections.py     | 58 +++++++--------
 6 files changed, 111 insertions(+), 106 deletions(-)
 create mode 100644 specification/modules/instant_messaging.rst

diff --git a/specification/0-events.rst b/specification/0-events.rst
index ce36b040..a50d01fa 100644
--- a/specification/0-events.rst
+++ b/specification/0-events.rst
@@ -20,71 +20,19 @@ Room Events
 This specification outlines several standard event types, all of which are
 prefixed with ``m.``
 
-{{room_events}}
-
-m.room.message msgtypes
-~~~~~~~~~~~~~~~~~~~~~~~
-
-.. TODO-spec
-   How a client should handle unknown message types.
-
-
-Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type
-of message being sent. Each type has their own required and optional keys, as
-outlined below.
-
-{{msgtype_events}}
-
-Presence Events
-~~~~~~~~~~~~~~~
-
-{{presence_events}}
- 
-Each user has the concept of presence information. This encodes the
-"availability" of that user, suitable for display on other user's clients.
-This is transmitted as an ``m.presence`` event and is one of the few events
-which are sent *outside the context of a room*. The basic piece of presence
-information is represented by the ``presence`` key, which is an enum of one
-of the following:
-
-      - ``online`` : The default state when the user is connected to an event
-        stream.
-      - ``unavailable`` : The user is not reachable at this time.
-      - ``offline`` : The user is not connected to an event stream.
-      - ``free_for_chat`` : The user is generally willing to receive messages
-        moreso than default.
-      - ``hidden`` : Behaves as offline, but allows the user to see the client
-        state anyway and generally interact with client features. (Not yet
-        implemented in synapse).
-
-In addition, the server maintains a timestamp of the last time it saw a
-pro-active event from the user; either sending a message to a room, or
-changing presence state from a lower to a higher level of availability
-(thus: changing state from ``unavailable`` to ``online`` counts as a
-proactive event, whereas in the other direction it will not). This timestamp
-is presented via a key called ``last_active_ago``, which gives the relative
-number of milliseconds since the message is generated/emitted that the user
-was last seen active.
-    
-
-Events on Change of Profile Information
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Because the profile displayname and avatar information are likely to be used in
-many places of a client's display, changes to these fields cause an automatic
-propagation event to occur, informing likely-interested parties of the new
-values. This change is conveyed using two separate mechanisms:
-
- - a ``m.room.member`` event is sent to every room the user is a member of,
-   to update the ``displayname`` and ``avatar_url``.
- - a ``m.presence`` presence status update is sent, again containing the new values of the
-   ``displayname`` and ``avatar_url`` keys, in addition to the required
-   ``presence`` key containing the current presence state of the user.
-
-Both of these should be done automatically by the home server when a user
-successfully changes their displayname or avatar URL fields.
-
-Additionally, when home servers emit room membership events for their own
-users, they should include the displayname and avatar URL fields in these
-events so that clients already have these details to hand, and do not have to
-perform extra roundtrips to query it.
+{{m_room_aliases_event}}
+
+{{m_room_canonical_aliases_event}}
+
+{{m_room_create_event}}
+
+{{m_room_history_visibility_event}}
+
+{{m_room_join_rules_event}}
+
+{{m_room_member_event}}
+
+{{m_room_power_levels_event}}
+
+{{m_room_redaction_event}}
 
diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
new file mode 100644
index 00000000..7f582ca4
--- /dev/null
+++ b/specification/modules/instant_messaging.rst
@@ -0,0 +1,27 @@
+Instant Messaging
+=================
+
+Events
+------
+
+{{m_room_message_event}}
+
+{{m_room_message_feedback_event}}
+
+{{m_room_name_event}}
+
+{{m_room_topic_event}}
+
+m.room.message msgtypes
+-----------------------
+
+.. TODO-spec
+   How a client should handle unknown message types.
+
+
+Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type
+of message being sent. Each type has their own required and optional keys, as
+outlined below.
+
+{{msgtype_events}}
+
diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
index 6f644879..ddd2adff 100644
--- a/specification/modules/presence.rst
+++ b/specification/modules/presence.rst
@@ -1,7 +1,5 @@
-Presence Events
-===============
-
-{{presence_events}}
+Presence
+========
  
 Each user has the concept of presence information. This encodes the
 "availability" of that user, suitable for display on other user's clients.
@@ -29,6 +27,11 @@ is presented via a key called ``last_active_ago``, which gives the relative
 number of milliseconds since the message is generated/emitted that the user
 was last seen active.
 
+Events
+------
+
+{{presence_events}}
+
 Presence HTTP API
 -----------------
 .. TODO-spec
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 454f23b9..e33bc785 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -14,6 +14,7 @@ targets:
       - 6-appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
+    - modules/instant_messaging.rst
     - modules/voip_events.rst
     - modules/typing_notifications.rst
     - modules/receipts.rst
diff --git a/templating/batesian/sections.py b/templating/batesian/sections.py
index 11c34fb3..31849389 100644
--- a/templating/batesian/sections.py
+++ b/templating/batesian/sections.py
@@ -27,12 +27,38 @@ class Sections(object):
             section_key = func_name[len("render_"):]
             self.log("Generating section '%s'" % section_key)
             section = func()
-            if not isinstance(section, basestring):
+            if isinstance(section, basestring):
+                if section_key in section_dict:
+                    raise Exception(
+                        ("%s : Section %s already exists. It must have been " +
+                        "generated dynamically. Check which render_ methods " +
+                        "return a dict.") %
+                        (func_name, section_key)
+                    )
+                section_dict[section_key] = section
+                self.log(
+                    "  Generated. Snippet => %s" % section[:60].replace("\n","")
+                )
+            elif isinstance(section, dict):
+                self.log("  Generated multiple sections:")
+                for (k, v) in section.iteritems():
+                    if not isinstance(k, basestring) or not isinstance(v, basestring):
+                        raise Exception(
+                            ("Method %s returned multiple sections as a dict but " +
+                            "expected the dict elements to be strings but they aren't.") %
+                            (func_name, )
+                        )
+                    if k in section_dict:
+                        raise Exception(
+                            "%s tried to produce section %s which already exists." %
+                            (func_name, k)
+                        )
+                    section_dict[k] = v
+                    self.log(
+                        "  %s  =>  %s" % (k, v[:60].replace("\n",""))
+                    )
+            else:
                 raise Exception(
-                    "Section function '%s' didn't return a string!" % func_name
+                    "Section function '%s' didn't return a string/dict!" % func_name
                 )
-            section_dict[section_key] = section
-            self.log(
-                "  Generated. Snippet => %s" % section[:60].replace("\n","")
-            )
         return section_dict
\ No newline at end of file
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index cacaa06a..e75a75af 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -76,37 +76,36 @@ class MatrixSections(Sections):
             ))
         return "\n\n".join(sections)
 
-    def render_profile_http_api(self):
-        return self._render_http_api_group(
-            "profile", 
-            sortFnOrPathList=["displayname", "avatar_url"]
-        )
-
-    def render_sync_http_api(self):
-        return self._render_http_api_group(
-            "sync"
-        )
-
-    def render_presence_http_api(self):
-        return self._render_http_api_group(
-            "presence",
-            sortFnOrPathList=["status"]
-        )
-
-    def render_membership_http_api(self):
-        return self._render_http_api_group(
-            "membership"
-        )
 
-    def render_login_http_api(self):
-        return self._render_http_api_group(
-            "login"
-        )
+    # Special function: Returning a dict will specify multiple sections where
+    # the key is the section name and the value is the value of the section
+    def render_group_http_apis(self):
+        # map all swagger_apis to the form $GROUP_http_api
+        swagger_groups = self.units.get("swagger_apis").keys()
+        renders = {}
+        for group in swagger_groups:
+            sortFnOrPathList = None
+            if group == "presence":
+                sortFnOrPathList = ["status"]
+            elif group == "profile":
+                sortFnOrPathList=["displayname", "avatar_url"]
+            renders[group + "_http_api"] = self._render_http_api_group(
+                group, sortFnOrPathList
+            )
+        return renders
 
-    def render_rooms_http_api(self):
-        return self._render_http_api_group(
-            "rooms"
-        )
+    # Special function: Returning a dict will specify multiple sections where
+    # the key is the section name and the value is the value of the section
+    def render_group_events(self):
+        # map all event schemata to the form $EVENTTYPE_event with s/./_/g
+        # e.g. m_room_topic_event
+        schemas = self.units.get("event_schemas")
+        renders = {}
+        for event_type in schemas:
+            renders[event_type.replace(".", "_") + "_event"] = self._render_events(
+                lambda x: x == event_type, sorted
+            )
+        return renders
 
     def render_room_events(self):
         def filterFn(eventType):
@@ -180,3 +179,4 @@ class MatrixSections(Sections):
 
     def render_common_state_event_fields(self):
         return self._render_ce_type("state_event")
+

From 29bae1579017c05d8ff5f5cf415459cebf0bac12 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 11:30:07 +0100
Subject: [PATCH 73/85] Fix typo

---
 specification/0-events.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/0-events.rst b/specification/0-events.rst
index a50d01fa..16948462 100644
--- a/specification/0-events.rst
+++ b/specification/0-events.rst
@@ -22,7 +22,7 @@ prefixed with ``m.``
 
 {{m_room_aliases_event}}
 
-{{m_room_canonical_aliases_event}}
+{{m_room_canonical_alias_event}}
 
 {{m_room_create_event}}
 

From 1da64db302f07019e410e607775c9efa0702d100 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 14:29:37 +0100
Subject: [PATCH 74/85] Use relative depths for groups instead of absolute ones

This means the group can be agnostic to how deeply nested it is,
improving reusability of groups.
---
 scripts/gendoc.py          | 32 ++++++++++++++++++++++++--------
 specification/targets.yaml |  3 ++-
 2 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 26ee5a48..364b8e65 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -1,6 +1,7 @@
 #! /usr/bin/env python
 
 from docutils.core import publish_file
+import copy
 import fileinput
 import glob
 import os
@@ -292,21 +293,32 @@ def get_build_target(targets_listing, target_name):
             "Found target but 'files' key is not a list."
         )
 
-    def get_group(group_id):
+    def get_group(group_id, depth):
         group_name = group_id[len("group:"):]
         group = all_targets.get("groups", {}).get(group_name)
         if not group:
             raise Exception(
-                "Tried to find group '" + group_name + "' but it " +
-                "doesn't exist."
+                "Tried to find group '%s' but it doesn't exist." % group_name
             )
+        if not isinstance(group, list):
+            raise Exception(
+                "Expected group '%s' to be a list but it isn't." % group_name
+            )
+        # deep copy so changes to depths don't contaminate multiple uses of this group
+        group = copy.deepcopy(group)
+        # swap relative depths for absolute ones
+        for i, entry in enumerate(group):
+            if isinstance(entry, dict):
+                group[i] = {
+                    (rel_depth + depth): v for (rel_depth, v) in entry.items()
+                }
         return group
 
     resolved_files = []
     for file_entry in target["files"]:
         # file_entry is a group id
         if isinstance(file_entry, basestring) and file_entry.startswith("group:"):
-            group = get_group(file_entry)
+            group = get_group(file_entry, 0)
             # The group may be resolved to a list of file entries, in which case
             # we want to extend the array to insert each of them rather than
             # insert the entire list as a single element (which is what append does)
@@ -317,12 +329,16 @@ def get_build_target(targets_listing, target_name):
         # file_entry is a dict which has more file entries as values
         elif isinstance(file_entry, dict):
             resolved_entry = {}
-            for (k, v) in file_entry.iteritems():
-                if isinstance(v, basestring) and v.startswith("group:"):
-                    resolved_entry[k] = get_group(v)
+            for (depth, entry) in file_entry.iteritems():
+                if not isinstance(entry, basestring):
+                    raise Exception(
+                        "Double-nested depths are not supported. Entry: %s" % (file_entry,)
+                    )
+                if entry.startswith("group:"):
+                    resolved_entry[depth] = get_group(entry, depth)
                 else:
                     # map across without editing (e.g. normal file path)
-                    resolved_entry[k] = v
+                    resolved_entry[depth] = entry
             resolved_files.append(resolved_entry)
             continue
         # file_entry is just a plain ol' file path
diff --git a/specification/targets.yaml b/specification/targets.yaml
index e33bc785..62585c69 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -23,7 +23,8 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/end_to_end_encryption.rst
     - modules/history_visibility.rst
     - modules/push_overview.rst
-    - { 2: [modules/push_cs_api.rst , modules/push_push_gw_api.rst] }
+      # relative depth
+    - { 1: [modules/push_cs_api.rst , modules/push_push_gw_api.rst] }
 
 title_styles: ["=", "-", "~", "+", "^"]
 

From 064a2c91721a285cf03dcd0505e3c01a8af39256 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 14:59:57 +0100
Subject: [PATCH 75/85] Use argparse and log functions for gendoc.py

gendoc.py has become more complex such that we actually want to pass things
to it like `--verbose`, `--nodelete`, `--target`, so use `argparse` to do this
like we have `build.py`. Pass through `-v` flags to `build.py`.
---
 scripts/gendoc.py | 91 +++++++++++++++++++++++++++++------------------
 1 file changed, 56 insertions(+), 35 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 364b8e65..d2b0d75e 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -1,5 +1,6 @@
 #! /usr/bin/env python
 
+from argparse import ArgumentParser
 from docutils.core import publish_file
 import copy
 import fileinput
@@ -17,6 +18,7 @@ stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css", "codehighlight.css"]
 }
 
+VERBOSE = False
 
 """
 Read a RST file and replace titles with a different title level if required.
@@ -64,8 +66,8 @@ def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
         if file_offset is None:
             file_offset = line_title_level
             if file_offset != 0:
-                print ("     WARNING: %s starts with a title style of '%s' but '%s' " +
-                    "is preferable.") % (filename, line_title_style, title_styles[0])
+                logv(("     WARNING: %s starts with a title style of '%s' but '%s' " +
+                    "is preferable.") % (filename, line_title_style, title_styles[0]))
 
         # Sanity checks: Make sure that this file is obeying the title levels
         # specified and bail if it isn't.
@@ -101,12 +103,11 @@ def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
             continue
 
         # Adjusting line levels
-        # print (
-        #     "File: %s Adjusting %s to %s because file_offset=%s title_offset=%s" %
-        #     (filename, line_title_style,
-        #         title_styles[adjusted_level],
-        #         file_offset, title_level)
-        # )
+        logv(
+            "File: %s Adjusting %s to %s because file_offset=%s title_offset=%s" %
+            (filename, line_title_style, title_styles[adjusted_level],
+                file_offset, title_level)
+        )
         rst_lines.append(line.replace(
             line_title_style,
             title_styles[adjusted_level]
@@ -118,7 +119,7 @@ def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
 def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles):
     # string are file paths to RST blobs
     if isinstance(file_info, basestring):
-        print "%s %s" % (">" * (1 + title_level), file_info)
+        log("%s %s" % (">" * (1 + title_level), file_info))
         with open(os.path.join(spec_dir, file_info), "r") as f:
             rst = None
             if adjust_titles:
@@ -244,15 +245,21 @@ def run_through_template(input):
     tmpfile = './tmp/output'
     try:
         with open(tmpfile, 'w') as out:
-            print subprocess.check_output(
-                [
-                    'python', 'build.py', "-v",
-                    "-i", "matrix_templates",
-                    "-o", "../scripts/tmp",
-                    "../scripts/"+input
-                ],
-                stderr=out,
-                cwd="../templating",
+            args = [
+                'python', 'build.py',
+                "-i", "matrix_templates",
+                "-o", "../scripts/tmp",
+                "../scripts/"+input
+            ]
+            if VERBOSE:
+                args.insert(2, "-v")
+            log(" ==== build.py output ==== ")
+            log(
+                subprocess.check_output(
+                    args,
+                    stderr=out,
+                    cwd="../templating"
+                )
             )
     except subprocess.CalledProcessError as e:
         with open(tmpfile, 'r') as f:
@@ -347,6 +354,13 @@ def get_build_target(targets_listing, target_name):
     build_target["files"] = resolved_files
     return build_target
 
+def log(line):
+    print "gendoc: %s" % line
+
+def logv(line):
+    if VERBOSE:
+        print "gendoc:V: %s" % line
+
 
 def prepare_env():
     try:
@@ -363,9 +377,9 @@ def cleanup_env():
     shutil.rmtree("./tmp")
 
 
-def main(target_name):
+def main(target_name, keep_intermediates):
     prepare_env()
-    print "Building spec [target=%s]" % target_name
+    log("Building spec [target=%s]" % target_name)
     target = get_build_target("../specification/targets.yaml", target_name)
     build_spec(target=target, out_filename="tmp/templated_spec.rst")
     run_through_template("tmp/templated_spec.rst")
@@ -377,22 +391,29 @@ def main(target_name):
     run_through_template("tmp/howto.rst")
     rst2html("tmp/full_spec.rst", "gen/specification.html")
     rst2html("tmp/howto.rst", "gen/howtos.html")
-    if "--nodelete" not in sys.argv:
+    if not keep_intermediates:
         cleanup_env()
 
 
 if __name__ == '__main__':
-    if len(sys.argv) > 1 and sys.argv[1:] != ["--nodelete"]:
-        # we accept almost no args, so they don't know what they're doing!
-        print "gendoc.py - Generate the Matrix specification as HTML."
-        print "Usage:"
-        print "  python gendoc.py [--nodelete]"
-        print ""
-        print "The specification can then be found in the gen/ folder."
-        print ("If --nodelete was specified, intermediate files will be "
-               "present in the tmp/ folder.")
-        print ""
-        print "Requirements:"
-        print " - This script requires Jinja2 and rst2html (docutils)."
-        sys.exit(0)
-    main("main")
+    parser = ArgumentParser(
+        "gendoc.py - Generate the Matrix specification as HTML to the gen/ folder."
+    )
+    parser.add_argument(
+        "--nodelete", "-n", action="store_true",
+        help="Do not delete intermediate files. They will be found in tmp/"
+    )
+    parser.add_argument(
+        "--target", "-t", default="main",
+        help="Specify the build target to build from specification/targets.yaml"
+    )
+    parser.add_argument(
+        "--verbose", "-v", action="store_true",
+        help="Turn on verbose mode."
+    )
+    args = parser.parse_args()
+    if not args.target:
+        parser.print_help()
+        sys.exit(1)
+    VERBOSE = args.verbose
+    main(args.target, args.nodelete)

From f1adad5fb3b2c6f03d1b635880726a661ba7bb03 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 15:10:55 +0100
Subject: [PATCH 76/85] Add more logging with file prefixes

This makes the handoff between gendoc and batesian clearer in the logs.
---
 scripts/gendoc.py               | 11 +++++------
 templating/batesian/sections.py |  2 +-
 templating/batesian/units.py    |  2 +-
 templating/build.py             | 14 ++++++++------
 4 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index d2b0d75e..611d0d10 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -253,13 +253,12 @@ def run_through_template(input):
             ]
             if VERBOSE:
                 args.insert(2, "-v")
+            log("EXEC: %s" % " ".join(args))
             log(" ==== build.py output ==== ")
-            log(
-                subprocess.check_output(
-                    args,
-                    stderr=out,
-                    cwd="../templating"
-                )
+            print subprocess.check_output(
+                args,
+                stderr=out,
+                cwd="../templating"
             )
     except subprocess.CalledProcessError as e:
         with open(tmpfile, 'r') as f:
diff --git a/templating/batesian/sections.py b/templating/batesian/sections.py
index 31849389..c18c9a5f 100644
--- a/templating/batesian/sections.py
+++ b/templating/batesian/sections.py
@@ -16,7 +16,7 @@ class Sections(object):
 
     def log(self, text):
         if self.debug:
-            print text
+            print "batesian:sections: %s" % text
 
     def get_sections(self):
         render_list = inspect.getmembers(self, predicate=inspect.ismethod)
diff --git a/templating/batesian/units.py b/templating/batesian/units.py
index c20a2d1f..993f253d 100644
--- a/templating/batesian/units.py
+++ b/templating/batesian/units.py
@@ -22,7 +22,7 @@ class Units(object):
 
     def log(self, text):
         if self.debug:
-            print text
+            print "batesian:units: %s" % text
 
     def get_units(self, debug=False):
         unit_list = inspect.getmembers(self, predicate=inspect.ismethod)
diff --git a/templating/build.py b/templating/build.py
index ac3d2491..013248f4 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -52,8 +52,8 @@ def create_from_template(template, sections):
 def check_unaccessed(name, store):
     unaccessed_keys = store.get_unaccessed_set()
     if len(unaccessed_keys) > 0:
-        print "Found %s unused %s keys." % (len(unaccessed_keys), name)
-        print unaccessed_keys
+        log("Found %s unused %s keys." % (len(unaccessed_keys), name))
+        log(unaccessed_keys)
 
 def main(input_module, file_stream=None, out_dir=None, verbose=False):
     if out_dir and not os.path.exists(out_dir):
@@ -121,17 +121,19 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
         return
 
     # check the input files and substitute in sections where required
-    print "Parsing input template: %s" % file_stream.name
+    log("Parsing input template: %s" % file_stream.name)
     temp = Template(file_stream.read())
-    print "Creating output for: %s" % file_stream.name
+    log("Creating output for: %s" % file_stream.name)
     output = create_from_template(temp, sections)
     with open(
             os.path.join(out_dir, os.path.basename(file_stream.name)), "w"
             ) as f:
         f.write(output)
-    print "Output file for: %s" % file_stream.name
+    log("Output file for: %s" % file_stream.name)
     check_unaccessed("units", units)
 
+def log(line):
+    print "batesian: %s" % line
 
 if __name__ == '__main__':
     parser = ArgumentParser(
@@ -175,7 +177,7 @@ if __name__ == '__main__':
         sys.exit(0)
 
     if not args.file:
-        print "No file supplied."
+        log("No file supplied.")
         parser.print_help()
         sys.exit(1)
 

From 6afdfc0771265ded3b7e0752c7ee438b09f076cb Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 15:36:13 +0100
Subject: [PATCH 77/85] Add more logging and make logging context clearer

This is now actually useful if you want to debug why your swagger YAML
isn't producing a table you think it should be.
---
 scripts/gendoc.py                    |  8 ++++----
 templating/batesian/units.py         |  6 +++++-
 templating/matrix_templates/units.py | 28 +++++++++++++++++++++++++---
 3 files changed, 34 insertions(+), 8 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 611d0d10..1655d6f0 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -241,7 +241,7 @@ def rst2html(i, o):
             )
 
 
-def run_through_template(input):
+def run_through_template(input, set_verbose):
     tmpfile = './tmp/output'
     try:
         with open(tmpfile, 'w') as out:
@@ -251,7 +251,7 @@ def run_through_template(input):
                 "-o", "../scripts/tmp",
                 "../scripts/"+input
             ]
-            if VERBOSE:
+            if set_verbose:
                 args.insert(2, "-v")
             log("EXEC: %s" % " ".join(args))
             log(" ==== build.py output ==== ")
@@ -381,13 +381,13 @@ def main(target_name, keep_intermediates):
     log("Building spec [target=%s]" % target_name)
     target = get_build_target("../specification/targets.yaml", target_name)
     build_spec(target=target, out_filename="tmp/templated_spec.rst")
-    run_through_template("tmp/templated_spec.rst")
+    run_through_template("tmp/templated_spec.rst", VERBOSE)
     fix_relative_titles(
         target=target, filename="tmp/templated_spec.rst",
         out_filename="tmp/full_spec.rst"
     )
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")
-    run_through_template("tmp/howto.rst")
+    run_through_template("tmp/howto.rst", False)  # too spammy to mark -v on this
     rst2html("tmp/full_spec.rst", "gen/specification.html")
     rst2html("tmp/howto.rst", "gen/howtos.html")
     if not keep_intermediates:
diff --git a/templating/batesian/units.py b/templating/batesian/units.py
index 993f253d..144cf245 100644
--- a/templating/batesian/units.py
+++ b/templating/batesian/units.py
@@ -22,7 +22,11 @@ class Units(object):
 
     def log(self, text):
         if self.debug:
-            print "batesian:units: %s" % text
+            func_name = ""
+            trace = inspect.stack()
+            if len(trace) > 1 and len(trace[1]) > 2:
+                func_name = trace[1][3] + ":"
+            print "batesian:units:%s %s" % (func_name, text)
 
     def get_units(self, debug=False):
         unit_list = inspect.getmembers(self, predicate=inspect.ismethod)
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index a5c6a815..50fa784e 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -1,4 +1,12 @@
-"""Contains all the units for the spec."""
+"""
+Contains all the units for the spec.
+
+This file loads swagger and JSON schema files and parses out the useful bits
+and returns them as Units for use in Batesian.
+
+For the actual conversion of data -> RST (including templates), see the sections
+file instead.
+"""
 from batesian.units import Units
 import inspect
 import json
@@ -134,7 +142,7 @@ class MatrixUnits(Units):
                         "good_response": ""
                     }
                 }
-                self.log(".o.O.o. Endpoint: %s %s" % (method, path))
+                self.log(" ------- Endpoint: %s %s ------- " % (method, path))
                 for param in single_api.get("parameters", []):
                     # description
                     desc = param.get("description", "")
@@ -183,6 +191,9 @@ class MatrixUnits(Units):
                             "desc": json_body[key]["description"]
                         })
                 # endfor[param]
+                for row in endpoint["req_params"]:
+                    self.log("Request parameter: %s" % row)
+
                 # group params by location to ease templating
                 endpoint["req_param_by_loc"] = {
                     #   path: [...], query: [...], body: [...]
@@ -240,6 +251,7 @@ class MatrixUnits(Units):
 
                 # add response params if this API has any.
                 if good_response:
+                    self.log("Found a 200 response for this API")
                     res_type = Units.prop(good_response, "schema/type")
                     if res_type and res_type not in ["object", "array"]:
                         # response is a raw string or something like that
@@ -278,6 +290,16 @@ class MatrixUnits(Units):
                             }]
                         })
 
+                for response_table in endpoint["res_tables"]:
+                    self.log("Response: %s" % response_table["title"])
+                    for r in response_table["rows"]:
+                        self.log("Row: %s" % r)
+                if len(endpoint["res_tables"]) == 0:
+                    self.log(
+                        "This API appears to have no response table. Are you " +
+                        "sure this API returns no parameters?"
+                    )
+
                 endpoints.append(endpoint)
 
                 aliases = single_api.get("x-alias", None)
@@ -475,7 +497,7 @@ class MatrixUnits(Units):
             if re.match("^v[0-9\.]+$", word):
                 version = word[1:]  # strip the 'v'
 
-        self.log("Version: %s Title part: %s Changelog lines: %s" % (
+        self.log("Version: %s Title part: %s Changelog line count: %s" % (
             version, title_part, len(changelog_lines)
         ))
         if not version or len(changelog_lines) == 0:

From 385b6c4759515b8b79151984e28932968ba9dc0c Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 24 Sep 2015 13:37:32 +0100
Subject: [PATCH 78/85] Only validate a file if it ends with ".yaml".

Otherwise we try to validate vim .swp files.
---
 api/validator.js | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/api/validator.js b/api/validator.js
index 6eef652f..3b89a5a3 100644
--- a/api/validator.js
+++ b/api/validator.js
@@ -44,7 +44,8 @@ if (isDir) {
             process.exit(1);
         }
         files.forEach(function(f) {
-            if (f.indexOf(".yaml") > 0) {
+            var suffix = ".yaml";
+            if (f.indexOf(suffix, f.length - suffix.length) > 0) {
                 parser.parse(path.join(opts.schema, f), function(err, api, metadata) {
                     if (!err) {
                         console.log("%s is valid.", f);

From f34722485f5984d5bbf3bc26f50bd54949730295 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 24 Sep 2015 16:50:27 +0100
Subject: [PATCH 79/85] Check the request example JSON matches the schema

---
 api/check_examples.py | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/api/check_examples.py b/api/check_examples.py
index 00e75263..a0cd0658 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -29,6 +29,33 @@ except ImportError as e:
     raise
 
 
+def check_parameter(filepath, request, parameter):
+    schema = parameter.get("schema")
+    example = None
+    try:
+        example_json = schema.get('example')
+        if example_json:
+            example = json.loads(example_json)
+    except Exception as e:
+        raise ValueError("Error parsing JSON example request for %r" % (
+            request
+        ), e)
+    fileurl = "file://" + os.path.abspath(filepath)
+    if example and schema:
+        try:
+            print ("Checking request schema for: %r %r" % (
+                filepath, request
+            ))
+            # Setting the 'id' tells jsonschema where the file is so that it
+            # can correctly resolve relative $ref references in the schema
+            schema['id'] = fileurl
+            jsonschema.validate(example, schema)
+        except Exception as e:
+            raise ValueError("Error validating JSON schema for %r %r" % (
+                request, code
+            ), e)
+
+
 def check_response(filepath, request, code, response):
     example = None
     try:
@@ -43,7 +70,9 @@ def check_response(filepath, request, code, response):
     fileurl = "file://" + os.path.abspath(filepath)
     if example and schema:
         try:
-            print ("Checking schema for: %r %r %r" % (filepath, request, code))
+            print ("Checking response schema for: %r %r %r" % (
+                filepath, request, code
+            ))
             # Setting the 'id' tells jsonschema where the file is so that it
             # can correctly resolve relative $ref references in the schema
             schema['id'] = fileurl
@@ -59,8 +88,13 @@ def check_swagger_file(filepath):
         swagger = yaml.load(f)
 
     for path, path_api in swagger.get('paths', {}).items():
+
         for method, request_api in path_api.items():
             request = "%s %s" % (method.upper(), path)
+            for parameter in request_api.get('parameters', ()):
+                if parameter['in'] == 'body':
+                    check_parameter(filepath, request, parameter)
+
             try:
                 responses = request_api['responses']
             except KeyError:

From 6c89e6ea6717d9cd07afed7906add62ab82c0431 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:03:46 +0100
Subject: [PATCH 80/85] Wrap refresh_token in `s

---
 api/client-server/v1/login.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index 0852db6b..a4fd8b95 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -66,7 +66,7 @@ paths:
                 description: |-
                   An access token for the account.
                   This access token can then be used to authorize other requests.
-                  The access token may expire at some point, and if so, it SHOULD come with a refresh_token.
+                  The access token may expire at some point, and if so, it SHOULD come with a ``refresh_token``.
                   There is no specific error message to indicate that a request has failed because
                   an access token has expired; instead, if a client has reason to believe its
                   access token is valid, and it receives an auth error, they should attempt to
@@ -132,7 +132,7 @@ paths:
                 description: |-
                   An access token for the account.
                   This access token can then be used to authorize other requests.
-                  The access token may expire at some point, and if so, it SHOULD come with a refresh_token.
+                  The access token may expire at some point, and if so, it SHOULD come with a ``refresh_token``.
               refresh_token:
                 type: string
                 description: (optional) A ``refresh_token`` may be exchanged for a new ``access_token`` using the TODO Linkify /tokenrefresh API endpoint.

From fc87f4cdb03cca548ee7ec499949c1d52d5104a3 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:10:15 +0100
Subject: [PATCH 81/85] Remove unused keys

---
 api/client-server/v1/login.yaml | 2 --
 1 file changed, 2 deletions(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index a4fd8b95..4cb564b7 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -29,7 +29,6 @@ paths:
       parameters:
         - in: body
           name: body
-          required: true
           schema:
             type: object
             example: |-
@@ -100,7 +99,6 @@ paths:
       parameters:
         - in: body
           name: body
-          required: true
           schema:
             type: object
             example: |-

From 5c4398c181bbf9da4aada6d4f8a3bf5b0d49c877 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:10:49 +0100
Subject: [PATCH 82/85] Remove superfluous comma

---
 api/client-server/v1/login.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index 4cb564b7..e415e798 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -114,7 +114,7 @@ paths:
         200:
           description: |-
             The refresh token was accepted, and a new access token has been issued.
-            The passed refresh token is no longer valid, and cannot be used.
+            The passed refresh token is no longer valid and cannot be used.
             A new refresh token may have been returned.
           examples:
             application/json: |-

From 6c1491b3bafb3a806d8fdf4ed59eacdf4df79d06 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:17:11 +0100
Subject: [PATCH 83/85] Respond to some review comments

---
 api/client-server/v1/login.yaml | 3 ++-
 drafts/macaroons_caveats.rst    | 4 +++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index e415e798..3d415c29 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -115,7 +115,8 @@ paths:
           description: |-
             The refresh token was accepted, and a new access token has been issued.
             The passed refresh token is no longer valid and cannot be used.
-            A new refresh token may have been returned.
+            A new refresh token will have been returned unless some policy does
+            not allow the user to continue to renew their session.
           examples:
             application/json: |-
               {
diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index 791d217a..c4b6b6a4 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -1,7 +1,9 @@
 Macaroon Caveats
 ================
 
-Macaroons (http://theory.stanford.edu/~ataly/Papers/macaroons.pdf) are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
+`Macaroons`_ are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
+
+.. _Macaroons: http://theory.stanford.edu/~ataly/Papers/macaroons.pdf)
 
 Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
 

From f5d436bd808ac9a1f20d816dccceac63a9202546 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:18:09 +0100
Subject: [PATCH 84/85] Remove extraneous )

---
 drafts/macaroons_caveats.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index c4b6b6a4..93622c3d 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -3,7 +3,7 @@ Macaroon Caveats
 
 `Macaroons`_ are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
 
-.. _Macaroons: http://theory.stanford.edu/~ataly/Papers/macaroons.pdf)
+.. _Macaroons: http://theory.stanford.edu/~ataly/Papers/macaroons.pdf
 
 Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
 

From cd6f15f62740075c2e8d1914fe77f3596046b166 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:34:24 +0100
Subject: [PATCH 85/85] Remove obsolete comment

---
 api/client-server/v1/membership.yaml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index 04922656..41f2febc 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -69,7 +69,6 @@ paths:
   "/rooms/{roomId}/invite":
     post:
       summary: Invite a user to participate in a particular room.
-      # It's a crying shame that I don't know how to force line breaks.
       description: |-
         This API invites a user to participate in a particular room.
         They do not start participating in the room until they actually join the